2024 · 52 citations · 27 references
ComputingComputer ScienceNatural Language ProcessingProgramming LanguagesSoftware EngineeringLanguage ModelingSoftware MetricsBenchmark DatasetsLanguage Processing ModelsMaskingMetricsOnline Evaluation
Transformer-based language models for automatic code completion have shown great promise so far, yet the evaluation of these models rarely uses real data. This study provides both quantitative and qualitative assessments of three public code language models when completing real-world code. We first developed an open-source IDE extension, Code4Me, for the online evaluation of the models. We collected real auto-completion usage data for over a year from more than 1200 users, resulting in over 600K valid completions. These models were then evaluated using six standard metrics across twelve programming languages. Next, we conducted a qualitative study of 1690 real-world completion requests to identify the reasons behind the poor model performance. A comparative analysis of the models' performance in online and offline settings was also performed, using benchmark synthetic datasets and two masking strategies.
27
Advances in neural information processing systems 7
Computers & Mathematics with Applications · 1996 · 14.4K citations
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Zhangyin Feng, Daya Guo, Duyu Tang et al. · 2020 · 2.4K citations · Full text
Yue Wang, Weishi Wang, Shafiq Joty et al. · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021 · 1.3K citations · Full text