Phase Space Detection of Virtual Machine Cyber Events Through Hypervisor-Level System Call Analysis

Abstract

The growth of the cloud computing ecosystem has afforded many new opportunities to businesses and consumers alike; however, with this new computing context comes new risks, and much attention has been given to the security dangers inherent in the architecture of cloud-based systems. Researchers, however, have done little to address the risk of advanced persistent threat intrusions, specifically in regard to the use of rootkits, which are powerful, stealthy pieces of malware that have grown in popularity with cybercriminals and nation state actors. These programs threaten a system by acquiring root privilege and then, using a variety of stealth tactics, evading detection and removal by modern anti-malware tools. In this research, we validate that the approach of Oak Ridge National Laboratory's Beholder project is applicable to the context of rootkit detection within a running virtual machine. We do this by collecting and analyzing system calls collected on the hypervisor level. The analysis employs a novel nonlinear, phase-space algorithm to derive time-serial cyber dynamics, and then uses these dynamics to characterize potentially anomalous system behavior through the comparison of nominal and test behavior profiles. Our results demonstrate that this technique is effective in flagging variance between the timing traces of an infected and an uninfected machine, thus indicating the presence of a running rootkit.