A scalable, efficient and informative approach for anomaly‐based intrusion detection systems: theory and practice

Abstract

Abstract In this paper, we present the design and implementation of a new approach for anomaly detection and classification over high speed networks. The proposed approach is based first of all on a data reduction phase through flow sampling by focusing mainly on short lived flows. The second step is then a random aggregation of some descriptors such as a number of SYN packets per flow in two different data structures called Count Min Sketch and Multi‐Layer Reversible Sketch. A sequential change point detection algorithm continuously monitors the sketch cell values. An alarm is raised if a significant change is identified in cell values. With an appropriate definition of the combination of IP header fields that should be used to identify one flow, we are able not only to detect the anomaly but also to classify the anomaly as DoS, DDoS or flash crowd, network scanning and port scanning. We validate our framework for anomaly detection on various real world traffic traces and demonstrate the accuracy of our approach on these real‐life case studies. Our analysis results from online implementation of our algorithm over measurements gathered by a DAG sniffing card are very attractive in terms of accuracy and response time. The proposed approach is very effective in detecting and classifying anomalies, and in providing information by extracting the culprit flows with a high level of accuracy. Copyright © 2010 John Wiley & Sons, Ltd.