Scalable network-based buffer overflow attack detection

Abstract

Buffer overflow attack is the main attack method that most if not all existing malicious worms use to propagate themselves from machine to machine. Although a great deal of research has been invested in defense mechanisms against buffer overflow attack, most of them require modifications to the network applications and/or the platforms that host them. Being an extension work of CTCP, this paper presents a network-based low performance overhead buffer overflow attack detection system called Nebula 1 NEtwork-based BUffer overfLow Attack detection, which can detect both known and zero-day buffer overflow attacks based solely on the packets observed without requiring any modifications to the end hosts. Moreover, instead of deriving a specific signature for each individual buffer overflow attack instance, Nebula uses a generalized signature that can capture all known variants of buffer overflow attacks while reducing the number of false positives to a negligible level. In addition, Nebula is built on a centralized TCP/IP architecture that effectively defeats all existing NIDS evasion techniques. Finally, Nebula incorporates a payload type identification mechanism that reduces further the false positive rate and scales the proposed buffer overflow attack detection scheme to gigabit network links.