Agile and scalable analysis of network events

Abstract

The state of the art in general purpose software systems for large-scale traffic measurement has not progressed much past the venerable libpcap. In this paper we describe a new data analysis system that provides a scalable, flexible system for composing ad-hoc analyses of high-speed, streming data. This agility allows researchers, network security analysts, or network operators to easily compose new analysis functions. A growing tool box of filtering, measurement, and statistical tools allows new approaches to be tested with a minimum of software development. Further, a dynamic type system allows polymorphic analysis modules to operate on arbitrary forms of structured data, thus allowing easy integration of multiple data sources such as packet traces, netflow records, or security logs. In this paper we present this system and demonstrate its capabilities while performing several measurements, such as computing probability density functions, detecting port-scans, and probabilistic counting of traffic traces.