ATD Learning: A secure, smart, and decentralised learning method for big data environments

Abstract

Big data and its distributed approach to data management have evolved significantly in recent years, giving rise to a huge volume of data generated from new services, devices (e.g. IoT), and applications. Recently, federated learning (FL) has been proposed for training deep learning models on distributed data in order to address significant challenges previously described in the literature, e.g., those concerns considering privacy, security, computational overhead, and legal restrictions. Nevertheless, while FL adequately addresses the above-mentioned limitations, there are still lingering drawbacks regarding privacy, security, scalability, single point of failure, and conflicting security policies. Specifically, the handling of sensitive data complicates the sharing and utilisation of data without breaching confidentiality. In this work, we propose a novel learning approach, named the AI-To-Data (ATD) learning method, to deal with the previous drawbacks. In particular, ATD proposes a more robust decentralised approach in which AI models are transferred to the data for training rather than centrally aggregating the model’s parameters. Each ATD node operates as both a local and a global entity, i.e. training on local data while synthesising models from other nodes. This new approach preserves data locality while enabling collaborative model training, thus fostering a more secure and integrated learning environment. Additionally, Eye on ATD, a blockchain-based security mechanism, is proposed to be incorporated into ATD to address potential security and privacy vulnerabilities, e.g. malicious participants or tampered updates. Our approach based on the combination of ATD and Eye on ATD has been extensively evaluated using three distinct datasets across multiple nodes considering multi-scenarios of abnormal behaviour detection tasks, including violence. The empirical results demonstrated that our proposal outperforms the state-of-the-art by obtaining an average accuracy of 92.1%. It has been carried out an independent test in order to validate the generalisation of ATD, achieving an accuracy of 89.2%. In addition, the scalability of the ATD has been tested by adding a fourth node with different behaviours, including shoplifting. In the last scenario, ATD achieved an accuracy of 93.3% when considering the four nodes without any negative impact on the performance of the entire system. Finally, it is worth highlighting how ATD ensures compliance with various regulatory frameworks due to ATD facilitates seamless node scalability and supports customisable data governance policies. The code of the proposed framework, both ATD and Eye on ATD, is available at https://github.com/LaithAlzubaidi/ATD/tree/main . • A novel decentralised learning technique, ATD, is proposed. • Fully decentralised; no single point of failure. • Raw data remains local, which maintains privacy. • Achieves superior performance, with an average accuracy of 92.1%. • Demonstrates scalability with 93.32% accuracy across four nodes. • Supports seamless node integration.