ComputingComputer EngineeringComputer ScienceApplication SecurityComputer SecurityOperating SystemsAnomaly DetectionSystem CallsBehavioral Anomaly DetectionExecution TracingRandomizationSystem Call AnalysisSystem Call Tracing
Many host-based anomaly detection systems monitor a process ostensibly running a known program by observing the system calls the process makes. Numerous improvements to the precision of this approach have been proposed, such as tracking system call sequences, and various "gray-box" extensions such as examining the program counter or return addresses on the stack when system calls are made. In this paper, we perform the first systematic study of a wide spectrum of such methods. We show that prior approaches can be organized along three axes, revealing new possibilities for system-call-based program tracking. Through an empirical analysis of this design space, we shed light on the benefits and costs of various points in the space and identify new regions that appear to outperform prior approaches. In separate contributions, we demonstrate novel mimicry attacks on a recent proposal using return addresses for system-call-based program tracking, and then suggest randomization techniques to make such attacks more difficult.
23
Efficient software-based fault isolation
Robert Wahbe, Steven Lucco, Thomas E. Anderson et al. · 1993 · 1.2K citations · Full text
A Sense of Self for Unix Processes
1996 · 1.1K citations
Mimicry attacks on host-based intrusion detection systems
David Wagner, Paolo Soto · 2002 · 707 citations
Intrusion detection via static analysis
David Wagner, Ralph A. Dean · 2002 · 650 citations