2015 · 165 citations · 21 references
ComputingComputer ScienceBig Data AnalyticsMachine LearningReal-time SystemsClassificationMachine Learning For Big DataOnline AlgorithmsReal-time AnalyticsStream ProcessingStreaming AlgorithmsBig Data ClassificationClass ImbalanceClass Imbalance HandlingClass Imbalance LearningOnline EvaluationOnline Machine LearningStreaming Machine LearningTime Scales
The evaluation of classifiers in data streams is fundamental so that poorly-performing models can be identified, and either improved or replaced by better-performing models. This is an increasingly relevant and important task as stream data is generated from more sources, in real-time, in large quantities, and is now considered the largest source of big data. Both researchers and practitioners need to be able to effectively evaluate the performance of the methods they employ. However, there are major challenges for evaluation in a stream. Instances arriving in a data stream are usually time-dependent, and the underlying concept that they represent may evolve over time. Furthermore, the massive quantity of data also tends to exacerbate issues such as class imbalance. Current frameworks for evaluating streaming and online algorithms are able to give predictions in real-time, but as they use a prequential setting, they build only one model, and are thus not able to compute the statistical significance of results in real-time. In this paper we propose a new evaluation methodology for big data streams. This methodology addresses unbalanced data streams, data where change occurs on different time scales, and the question of how to split the data between training and testing, over multiple models.
21
UCI Machine Learning Repository
Arthur Asuncion · Medical Entomology and Zoology · 2007 · 24.4K citations
Classification and Regression Trees.
John Van Ryzin, Leo Breiman, Jerome H. Friedman et al. · Journal of the American Statistical Association · 1986 · 21K citations
Mining time-changing data streams
Geoff Hulten, Laurie Spencer, Pedro Domingos · 2001 · 1.8K citations