2024 · 15 citations · 29 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningNatural Language ProcessingDeep LearningRepresentation LearningSelf-supervised LearningLabelsPretrainingPseudo-labelingSelf-supervised Fine-tuningSelf-supervised PretrainingSelf-trainingTask-agnostic PretrainingTuning
Pre-trained code models have recently achieved substantial improvements in many code intelligence tasks. These models are first pre-trained on large-scale unlabeled datasets in a task-agnostic manner using self-supervised learning, and then fine-tuned on labeled datasets in downstream tasks. However, the labeled datasets are usually limited in size (i.e., human intensive efforts), which may hinder the performance of pre-trained code models in specific tasks. To mitigate this, one possible solution is to leverage the large-scale unlabeled data in the tuning stage by pseudo-labeling, i.e., generating pseudo labels for unlabeled data and further training the pre-trained code models with the pseudo-labeled data. However, directly employing the pseudo-labeled data can bring a large amount of noise, i.e., incorrect labels, leading to suboptimal performance. How to effectively leverage the noisy pseudo-labeled data is a challenging yet under-explored problem.
29
Information theory and statistics
Journal of the Franklin Institute · 1959 · 7.2K citations
Practical Nonparametric Statistics
Vasant Bhapkar, W. J. Conover · Technometrics · 1972 · 1.4K citations