Detecting and Preventing Kernel Rootkit Attacks with Bus Snooping

Abstract

To protect the integrity of operating system kernels, we present <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Vigilare system</i> , a kernel integrity monitor that is architected to snoop the bus traffic of the host system from a separate independent hardware. This <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">snoop-based monitoring</i> enabled by the Vigilare system, overcomes the limitations of the <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">snapshot-based monitoring</i> employed in previous kernel integrity monitoring solutions. Being based on inspecting snapshots collected over a certain interval, the previous hardware-based monitoring solutions cannot detect <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">transient attacks</i> that can occur in between snapshots, and cannot protect the kernel against permanent damage. We implemented three prototypes of the Vigilare system by adding <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Snooper</i> hardware connections module to the host system for bus snooping, and a snapshot-based monitor to be comared with, in order to evaluate the benefit of snoop-based monitoring. The prototypes of Vigilare system detected all the transient attacks and the second one protected the kernel with negligible performance degradation while the snapshot-based monitor could not detect all the attacks and induced considerable performance degradation as much as 10 percent in our tuned STREAM benchmark test.