Machine-Learning-Based Online Distributed Denial-of-Service Attack Detection Using Spark Streaming

Abstract

In order to cope with the increasing number of cyber attacks, network operators must monitor the whole network situations in real time. Traditional network monitoring method that usually works on a single machine, however, is no longer suitable for the huge traffic data nowadays due to its poor processing ability. In this paper, we propose a machine-learning based online Internet traffic monitoring system using Spark Streaming, a stream- processing-based big data framework, to detect DDoS attacks in real time. The system consists of three parts, collector, messaging system and stream processor. We use a correlation-based feature selection method and choose 4 most necessary network features in our machine- learning-based DDoS detection algorithm. We verify the result of feature selection method by a comparative experiment and compare the detection accuracy of 3 machine learning methods - Naïve Bayes, Logistic Regression and Decision Tree. Finally, we conduct experiments in a cluster with the standalone mode, showing that our system can detect 3 typical DDoS attacks - TCP flooding, UDP flooding and ICMP flooding at the accuracy of more than 99.3%. It also shows the system performs well even for large Internet traffic.