EL-CodeBert: Better Exploiting CodeBert to Support Source Code-Related Classification Tasks

Abstract

With the development of deep learning and natural language processing techniques, the performance of many source code-related tasks can be improved by using pre-trained models. Of these pre-trained models, CodeBert is a bi-modal pre-trained model for programming languages and natural languages, which has been successfully used in current source code-related tasks. These previous studies mainly use the output vector of CodeBert’s last layer as the code semantic representation for fine-tuning downstream source code-related tasks. However, this setting may miss the valuable representational information, which may be captured by other layers of CodeBert.