Empirical studies on the NLP techniques for source code data preprocessing

Abstract

Program comprehension usually focuses on the significance of textual information to capture the programmers’ intent and knowledge in the software, in particular the source code. In the source code, most of the data is unstructured data, such as the natural language text in comments and identifier names. Researchers in software engineering community have developed many techniques for handling such unstructured data, such as natural language processing (NLP) and information retrieval (IR). Before using the IR technique on the unstructured source code, we must preprocess the text identifies and comments since these data is different from that used in our daily life. During this process, several operations, i.e, tokenization, splitting, stemming, etc. are usually used for preprocessing the unstructured source code. These preprocessing operations will affect the quality of the data used in the IR process. But how these preprocessing operations affect the results of IR is still an open problem. To the best of our knowledge, there are still no studies focusin on this problem. This paper attempts to fill this gap, and conducts some empirical studies to show what are the differences before and after these preprocessing operations. The results show some interesting phenomena based on using or not using these preprocessing operations.