Complex event processing for reactive security monitoring in virtualized computer systems

Abstract

The number of security incidents in computer systems is steadily increasing, despite intrusion detection and prevention mechanisms deployed as countermeasures. Many existing intrusion detection and prevention systems struggle to keep up with new threats posed by zero-day attacks and/or have serious performance impacts through extensive monitoring, questioning their effectiveness in most real-life scenarios. In this paper, we present a new approach for reactive security monitoring in a virtualized computer environment based on minimally-intrusive dynamic sensors deployed vertically across virtualization layers and horizontally within a virtual machine instance. The sensor streams are analyzed using a novel federation of complex event processing engines and an optimized query index to maximize the performance of continuous queries, and the results of the analysis are used to trigger appropriate actions on different virtualization layers in response to detected security anomalies. Furthermore, a novel event store that supports fast event logging is utilized for offline analysis of collected historical data. Experiments show that the proposed system can execute tens of thousands of complex, stateful detection rules simultaneously and trigger actions efficiently and with low latency.