Analyzing Malware by Abstracting the Frequent Itemsets in API Call Sequences

Abstract

Analyzing the usage of Windows Application Program Interface (API) is a common way to understand behaviors of Malicious Software (malware) in either static analysis or dynamic analysis methods. In this work, we focus on the usage of frequent messages in API call sequences, and we hypothesize that frequent itemsets composed of API names and/or API arguments could be valuable in the identification of the behavior of malware. For verification, we introduced clustering processes of malware binaries based on their frequent itemsets of API call sequences, and we evaluated the performance of malware clustering. Specific implementation processes for malware clustering, including API calls abstraction, frequent itemsets mining and similarity calculation, are illustrated. The experiment upon a big malware dataset demonstrated that merely using the frequent messages of API call sequences can achieve a high precision for malware clustering while significantly reducing the computation time. This also proves the importance of frequent itemsets in API call sequences for identifying the behavior of malware.