Survey on malware evasion techniques: State of the art and challenges

Abstract

Nowadays targeted malware attacks against organizations are increasingly becoming more sophisticated, damaging, and difficult to detect. Current intrusion detection technologies are incapable of addressing many of the newer malware evasion techniques such as return-oriented programming and remote library injection. This paper presents a survey on the various techniques employed in malware to evade detection by security systems such as intrusion detection and anti-virus software. The evasion techniques we cover include obfuscation, fragmentation and session splicing, application specific violations, protocol violations, inserting traffic at IDS, denial of service, and code reuse attacks. We also discuss mitigations such as sandboxing, session reassembly, data execution prevention, address space layout randomization, control flow integrity, and Windows 8 ROP mitigation. We also compare evasion techniques with an analysis on the sophistication of the attack, challenges or difficulty to detect, and degree of impact.