Runtime execution monitoring (REM) to detect and prevent malicious code execution

Abstract

Many computer security threats involve execution of unauthorized foreign code on the victim computer. Viruses, network and email worms, Trojan horses, backdoor programs used in denial of service attacks are a few examples. In this paper, we present an architectural technique, which we call runtime execution monitoring (REM), to detect program flow anomalies associated with such malicious code. The key idea in REM is the verification of program code at the hash block (similar to a basic block) level. This is achieved by pre-computing keyed hashes (HMACs) for each hash block during program installation, and then verifying these values during program execution. By verifying program code integrity at the hash block level, REM can monitor instructions whose behavior is typically exploited by malicious code, such as branch, call, return instructions. Performance degradation with REM averages 6.4% on our benchmark programs, which can be reduced to under 5% by increasing the size of the L1 instruction cache.