Data discretization: taxonomy and big data challenge

TLDR

Let's collect: Background sentences: - "Discretization of numerical data is one of the most influential data preprocessing tasks in knowledge discovery and data mining." - "The purpose of attribute discretization is to find concise data representations as categories which are adequate for the learning task retaining as much information in the original continuous attribute as possible." - "Despite the great impact of discretization as data preprocessing technique, few elementary approaches have been developed in the literature for Big Data." So background: summarise field context: discretization is key preprocessing, aims to produce concise categories retaining info, but few approaches for Big Data. Purpose sentences: - From [Purpose, Mechanism] line: "In this article, we present an updated overview of discretization techniques in conjunction with a complete taxonomy of the leading discretizers." - From [Purpose, Mechanism] second: "The purpose of this article is twofold: a comprehensive taxonomy of discretization techniques to help the practitioners in the use of the algorithms is presented; the article aims is to demonstrate that standard discretization methods can be parallelized in Big Data platforms such as Apache Spark, boosting both performance and accuracy." - From [Purpose, Findings] line: "We thus propose a distributed implementation of one of the most well‐known discretizers based on Information Theory, obtaining better results than the one produced by: the entropy minimization discretizer proposed by Fayyad and Irani." - From [Purpose] line: "Our scheme goes beyond a simple parallelization and it is intended to be the first to face the Big Data challenge." So purpose: summarise aims: provide taxonomy, show parallelization in Spark, propose distributed implementation of Info Theory discretizer with better results, first to tackle Big Data challenge. Mechanism sentences: - From [Purpose, Mechanism] lines: same as above: "In this article, we present an updated overview of discretization techniques in conjunction with a complete taxonomy of the leading discretizers." and "The purpose of this article is twofold: a comprehensive taxonomy of discretization techniques to help the practitioners in the use of the algorithms is presented; the article aims is to demonstrate that standard discretization methods can be parallelized in Big Data platforms such as Apache Spark, boosting both performance and accuracy." - Also from [Purpose, Findings] line: "We thus propose a distributed implementation of one of the most well‐known discretizers based on Information Theory, obtaining better results than the one produced by: the entropy minimization discretizer proposed by Fayyad and Irani." - From [Purpose] line: "Our scheme goes beyond a simple parallelization and it is intended to be the first to face the Big Data challenge." So mechanism: updated overview, taxonomy, parallelization in Spark, distributed implementation of Info Theory discretizer, improved results. Findings sentences: - From [Purpose, Findings] line: "We thus propose a distributed implementation of one of the most well‐known discretizers based on Information Theory, obtaining better results than the one produced by: the entropy minimization discretizer proposed by Fayyad and Irani." So findings: distributed Info Theory discretizer yields better results than Fayyad & Irani entropy minimization discretizer. Other sentences: - "[Other] WIREs Data Mining Knowl Discov 2016, 6:5–21." - "[Other] doi: 10.1002/widm.1173 This article is categorized under: Technologies > Classification Technologies > Data Preprocessing" So other: provide citation info: WIREs Data Mining Knowledge Discovery 2016, 6:5–21, doi 10.1002/widm.1173, categorized under Technologies > Classification Technologies > Data Preprocessing.

Abstract

Discretization of numerical data is one of the most influential data preprocessing tasks in knowledge discovery and data mining. The purpose of attribute discretization is to find concise data representations as categories which are adequate for the learning task retaining as much information in the original continuous attribute as possible. In this article, we present an updated overview of discretization techniques in conjunction with a complete taxonomy of the leading discretizers. Despite the great impact of discretization as data preprocessing technique, few elementary approaches have been developed in the literature for Big Data. The purpose of this article is twofold: a comprehensive taxonomy of discretization techniques to help the practitioners in the use of the algorithms is presented; the article aims is to demonstrate that standard discretization methods can be parallelized in Big Data platforms such as Apache Spark, boosting both performance and accuracy. We thus propose a distributed implementation of one of the most well‐known discretizers based on Information Theory, obtaining better results than the one produced by: the entropy minimization discretizer proposed by Fayyad and Irani. Our scheme goes beyond a simple parallelization and it is intended to be the first to face the Big Data challenge. WIREs Data Mining Knowl Discov 2016, 6:5–21. doi: 10.1002/widm.1173 This article is categorized under: Technologies > Classification Technologies > Data Preprocessing