VYPER: Vulnerability detection in binary code

Abstract

Abstract This paper presents a method for exploitable vulnerabilities detection in binary code with almost no false positives. It is based on the concolic (a mix of concrete and symbolic) execution of software binary code and the annotation of sensitive memory zones of the corresponding program traces (represented in a formal manner). Three big families of vulnerabilities are considered (taint related, stack overflow, and heap overflow). Based on the angr framework as a supporting software VulnerabilitY detection based on dynamic behavioral PattErn Recognition was developed to demonstrate the viability of the method. Several test cases using custom code, Juliet test base and widely used public libraries were performed showing a high detection potential for exploitable vulnerabilities with a very low rate of false positives.