Dual Dynamic Consistency Regularization for Semi-Supervised Domain Adaptation

Abstract

The Vision Transformer (ViT) model serves as a powerful model to capture and comprehend global information, particularly when trained on extensive datasets. Conversely, the Convolutional Neural Network (CNN) model is beneficial to training with small datasets for retaining essential local information. Inspired by these properties of ViT and CNN models, we introduce a hybrid framework that smoothly increases the cross-domain generalization in Semi-supervised Domain Adaptation (SSDA). To achieve this goal, we first train the ViT model on abundant labeled source data, while the CNN model is trained on a few labeled target samples. Then, these models dynamically exchange their knowledge for potential generalization to unlabeled target data via the proposed method, named Dual Dynamic Consistency Regularization (D <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> CR). Specifically, the ViT model provides its pseudo labels to update the global perspective for the CNN model. Similarly, the CNN model offers pseudo labels to complement the local perspective for the ViT model. The previous methods use a fixed threshold algorithm for the pseudo-labeling process. However, we utilize the dynamic threshold strategy to create pseudo labels for the bi-directional consistency regularization learning between the ViT and CNN models. We verify our approach across several SSDA benchmark datasets. The outstanding experimental results provide strong evidence of the effectiveness and superiority of our approach over previous state-of-the-art SSDA methods.