Proceedings of the 44th International Conference on Software Engineering · 2022 · 42 citations · 18 references
ComputingComputer ScienceNatural Language ProcessingProgramming LanguagesSoftware EngineeringDomain-specific LanguagesLanguage ModelingMeta-learningQuery ProcessingSemantic SearchTransfer LearningDomain AdaptationFew-shot Domain AdaptationFew-shot LearningLanguage Model Fine-tuningProgram RepresentationProgramming Models
Recently, pre-trained programming language models such as CodeBERT have demonstrated substantial gains in code search. Despite their success, they rely on the availability of large amounts of parallel data to fine-tune the semantic mappings between queries and code. This restricts their practicality in domain-specific languages with relatively scarce and expensive data. In this paper, we propose CDCS, a novel approach for domain-specific code search. CDCS employs a transfer learning framework where an initial program representation model is pre-trained on a large corpus of common programming languages (such as Java and Python), and is further adapted to domain-specific languages such as Solidity and SQL. Unlike cross-language CodeBERT, which is directly fine-tuned in the target language, CDCS adapts a few-shot meta-learning algorithm called MAML to learn the good initialization of model parameters, which can be best reused in a domain-specific language. We evaluate the proposed approach on two domain-specific languages, namely Solidity and SQL, with model transferred from two widely used languages (Python and Java). Experimental results show that CDCS significantly outperforms conventional pre-trained code models that are directly fine-tuned in domain-specific languages, and it is particularly effective for scarce data.
18
CodeBERT: A Pre-Trained Model for Programming and Natural Languages
Zhangyin Feng, Daya Guo, Duyu Tang et al. · 2020 · 2.4K citations · Full text
Meta-Transfer Learning for Few-Shot Learning
Qianru Sun, Yaoyao Liu, Tat‐Seng Chua et al. · 2019 · 1.3K citations
Yue Wang, Weishi Wang, Shafiq Joty et al. · Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing · 2021 · 1.3K citations · Full text