Accurate and Automated System Call Policy-Based Intrusion Prevention

Abstract

One way to prevent control hijacking attack is to compare a network application’s run-time system calls with a pre-defined normal system call behavior model, and raise an alert upon detecting a mismatch. This paper describes a system called PAID, which can automatically derive an accurate system call pattern from the source code of an application, and use it to detect any anomalous behavior at run time with minimal overhead. Because each application’s system call pattern is directly derived from its source code, PAID never raises false positive alarms. Moreover, its false negative rate is very close to zero because PAID uses the sequence of return addresses on the user/kernel stack to uniquely identify each system call instance. Experiments on a fully operational PAID prototype show that PAID can indeed stop all known control hijacking attacks. The run-time latency and throughput penalty of PAID are under 13.02% and 11.52%, respectively, when it is tested against a set of production-mode network applications.