2006 · 572 citations · 28 references
ComputingComputer ScienceCybersecurityComputer SecurityData MiningMachine LearningDecision TreesIntrusion DetectionMalware ClassificationThreat DetectionCharacter N-gramsMachine Learning For Malware ClassificationMachine Learning-based Malware DetectionMalware DetectionNaive BayesNaive Bayes ClassificationWord N-grams
We describe the use of machine learning and data mining to detect and classify malicious executables as they appear in the wild. We gathered 1,971 benign and 1,651 malicious executables and encoded each as a training example using n-grams of byte codes as features. Such processing resulted in more than 255 million distinct n-grams. After selecting the most relevant n-grams for prediction, we evaluated a variety of inductive methods, including naive Bayes, decision trees, support vector machines, and boosting. Ultimately, boosted decision trees outperformed other methods with an area under the ROC curve of 0.996. Results suggest that our methodology will scale to larger collections of executables. We also evaluated how well the methods classified executables based on the function of their payload, such as opening a backdoor and mass-mailing. Areas under the ROC curve for detecting payload function were in the neighborhood of 0.9, which were smaller than those for the detection task. However, we attribute this drop in performance to fewer training examples and to the challenge of obtaining properly labeled examples, rather than to a failing of the methodology or to some inherent difficulty of the classification task. Finally, we applied detectors to 291 malicious executables discovered after we gathered our original collection, and boosted decision trees achieved a true-positive rate of 0.98 for a desired false-positive rate of 0.05. This result is particularly important, for it suggests that our methodology could be used as the basis for an operational system for detecting previously undiscovered malicious executables.
28
THE USE OF MULTIPLE MEASUREMENTS IN TAXONOMIC PROBLEMS
Ronald Aylmer Fisher · Annals of Eugenics · 1936 · 14.7K citations · Full text
A training algorithm for optimal margin classifiers
Bernhard E. Boser, Isabelle Guyon, Vladimir Vapnik · 1992 · 11.6K citations
Experiments with a new boosting algorithm
Yoav Freund, Robert E. Schapire · 1996 · 7.6K citations
A Comparative Study on Feature Selection in Text Categorization
Yiming Yang, Jan Pedersen · 1997 · 4.8K citations
Instance-Based Learning Algorithms
David W. Aha, Dennis Kibler, Marc K. Albert · Machine Learning · 1991 · 4.1K citations · Full text