Designing preprocessing framework (ERT) for text mining application

Abstract

Text mining is the progression of originating high superiority information from text. As the majority information is presently accumulated as text, text mining is alleged to enclose a high profitable potential significance. The preprocessing steps are performed individually based on their need and there is no common framework for preprocessing. This paper focuses on designing preprocessing framework for text mining application by gathering the frequently used preprocessing steps. This framework focuses on three major preprocessing tasks Expansion, Removal and Tokenization (ERT). The ERT takes corpus as input and generate the list of tokens; the tokens has been equipped to perform all learning algorithms. ERT framework helps to perform all the preprocessing steps quickly and correctly.