On the Effectiveness of Large Language Models in Statement-level Code Summarization

Abstract

Code comments are crucial for program comprehension, and the automated generation of comments greatly enhances the efficiency of code commenting. Statement-level code summarization represents the finest granularity of code summarization, typically encompassing explanations of the purpose and functionality of code statements. Large Language Models (LLMs) are deep learning models trained on massive text data. They possess not only the capability to generate natural language text but also to deeply understand the meaning of text, applicable to various natural language processing tasks such as text summarization, question answering, and translation. Currently, LLMs have demonstrated the ability to generate summaries for code. In this paper, we systematically investigate the capability of LLMs to generate statement-level code summaries. we construct a dataset for statement-level code summarization and evaluate the ability of large language models on statement-level code summarization. For further research, we investigate the impact of different prompting techniques. We also study the effect of the temperature parameter on the quality of generated summaries. Additionally, we compare large language models with the state-of-the-art pretrained model CodeT5 and find out that large language models have great potential to replace pre-trained models and become the new state-of-the-art models on statement-level code summarization. To ensure the reliability of automatic evaluation, we also conduct human evaluation. Our findings emphasize the transformative potential of LLMs on statement-level code summarization and the challenges yet to be overcome.