Unsupervised Learning for Network Flow Based Anomaly Detection in the Era of Deep Learning

Abstract

In this research, we investigate and evaluate four unsupervised learning algorithms: K-Means and Self Organizing Maps (SOM), deep autoencoding Gaussian mixture model (DAGMM), and adversarially learned anomaly detection (ALAD) on two benchmark data sets towards network flow-based anomaly detection. We explore different parameters and neural network settings of the learning algorithms, respectively. The returned results show that DAGMM gains the lowest false-positive rate and a relatively high detection rate on one of the data set, whereas SOM achieves the best results on the other data set. By comparing the detection rates of all algorithms on attacks that have no instance in the training set, and we find that K-Means works the better than the other three. Whereas the comparison on the attacks that have few instances in the training set shows that the ALAD works much better than the others since it uses the adversarial sample generation mechanism. We conclude that to achieve the best performance, integration of the traditional and deep-learning algorithms for network flow-based intrusion detection is critical.