A new approach to feature selection in text classification

Abstract

Text classification is the process of automatically assigning predefined categories to free text, which is very important to information retrieval and many other applications. Of it, the first important thing is to effectively represent a text to characterize it as belonging to a specified category based on its content and thus make the following phase of classifier training and using more effective and efficient regarding to the final classification performance. In this paper, an effective and efficient new method called variance-mean based feature filtering method of feature selection to do feature reduction in the representation phase for text classification is proposed. It keeps the best features, and thus improves the final performance, e.g. macro-f1 to 0.92 and simultaneously decreases the computing time for representing the incoming text waiting to be classified dramatically, which is important because it occurs on line and is time-critical. The effectiveness and efficiency are especially obvious when applied to Chinese language text.