Exploiting data preparation to enhance mining and knowledge discovery

Abstract

One of the major obstacles to using organizational data for mining and knowledge discovery is that, in most cases, it is not amenable for mining in its natural form. Using a data set from a large tertiary-care hospital, we provide strong empirical evidence that data enhancement by the introduction of new attributes, along with judicious aggregation of existing attributes, results in higher-quality knowledge discovery. Interestingly, we also found that there is a differential impact of data set enhancements on the performance of different data mining algorithms. We define and use several measures, including entropy, rule complexity and resonance, to evaluate the quality and usefulness of the knowledge discovered.