Open challenges for data stream mining research

TLDR

Every day, massive streams of sensory, transactional, and web data are generated, making data streams a key component of big data, yet most predictive models are built for controlled settings and ignore real‑world challenges. This article discusses eight open challenges in data stream mining, aiming to expose gaps between current research and practical applications and to propose new, application‑relevant research directions. The challenges span the entire knowledge‑discovery cycle, addressing issues such as data privacy, legacy system integration, incomplete or delayed information, complex data analysis, and algorithm evaluation. Illustrated through practical applications, the analysis offers general recommendations for future research in data stream mining.

Abstract

Every day, huge volumes of sensory, transactional, and web data are continuously generated as streams, which need to be analyzed online as they arrive. Streaming data can be considered as one of the main sources of what is called big data. While predictive modeling for data streams and big data have received a lot of attention over the last decade, many research approaches are typically designed for well-behaved controlled problem settings, overlooking important challenges imposed by real-world applications. This article presents a discussion on eight open challenges for data stream mining. Our goal is to identify gaps between current research and meaningful applications, highlight open problems, and define new application-relevant research directions for data stream mining. The identified challenges cover the full cycle of knowledge discovery and involve such problems as: protecting data privacy, dealing with legacy systems, handling incomplete and delayed information, analysis of complex data, and evaluation of stream mining algorithms. The resulting analysis is illustrated by practical applications and provides general suggestions concerning lines of future research in data stream mining.