Winning some of the document preprocessing challenges in a text mining process

Abstract

Considering the huge growth of the number of documents in the digital universe and the possibility of obtaining some competitive advantage in processing them, this paper describes some of the difficulties of working with text collections. More specifically, it shows some of the challenges on the step considered one of the most important of the Text Mining process - the data preprocessing - focusing on two of its main tasks: attribute generation and selection, considering not only single terms but composed terms too. In order to overcome the challenges imposed by these problems, this paper presents efficient unsupervised solutions. The application of these solutions in three real data sets is presented in order to evaluate them and to show a way to treat the data step by step. Good results were obtained at the end of the whole process.