Unlocking the power of machine learning in big data: a scoping survey

Abstract

Machine learning (ML) plays a crucial role in big data (BD) by serving as the cornerstone of efficient data processing and analysis. In particular, ML provides BD with the ability to extract valuable insights from the large data sets. Therefore, this study conducted a scoping survey to define the role of ML in BD by exploring its history and evolution. Subsequently, a framework of ML in BD is proposed, emphasizing its practical applications in addressing the challenges presented by the volume, velocity, variety, and veracity of data. Moreover, BD analytics is described, showcasing how ML paradigms contribute to decision-making. This is followed by an illustration of real-world applications across diverse industries regarding the transformative impact of ML on BD. The survey findings highlight the integration of ML in BD considering volume, variety, velocity, and reliability. Scalable storage solutions, advanced computational architectures, and distributed ML are the avenues for shaping the ML landscape. Similarly, embedding intelligence in preprocessing and interoperable models is an avenue considering veracity. Real-time processing frameworks, temporally aware ML, and edge--computing integration are avenues considering velocity. Automated data quality assurance, explainable artificial intelligence and transparency, and blockchain technology for data provenance are avenues in veracity.