Enhancing Machine Learning Based Malware Detection Model by Reinforcement Learning

Abstract

Malware detection is getting more and more attention due to the rapid growth of new malware. As a result, machine learning (ML) has become a popular way to detect malware variants. However, machine learning models can also be cheated. Through reinforcement learning (RL), we can generate new malware samples which can bypass the detection of machine learning. In this paper, a RL model on malware generation named gym-plus is designed. Gym-plus is built based on gym-malware with some improvements. As a result, the probability of evading machine learning based static PE malware detection models is increased by 30%. Based on these newly generated samples, we retrain our detecting model to detect unknown threats. In our test, the detection accuracy of malware increased from 15.75% to 93.5%.