Unleashing Mayhem on Binary Code

TLDR

Mayhem is a system designed to automatically discover exploitable bugs in binary programs by enabling exploit generation at the binary level. It operates on raw binaries without debugging information, using hybrid symbolic execution and index‑based memory modeling to manage execution paths and reason about symbolic memory indices. Mayhem identified 29 exploitable vulnerabilities in Linux and Windows binaries, including two previously undocumented ones, and each bug is accompanied by a working shell‑spawning exploit.

Abstract

In this paper we present Mayhem, a new system for automatically finding exploitable bugs in binary (i.e., executable) programs. Every bug reported by Mayhem is accompanied by a working shell-spawning exploit. The working exploits ensure soundness and that each bug report is security-critical and actionable. Mayhem works on raw binary code without debugging information. To make exploit generation possible at the binary-level, Mayhem addresses two major technical challenges: actively managing execution paths without exhausting memory, and reasoning about symbolic memory indices, where a load or a store address depends on user input. To this end, we propose two novel techniques: 1) hybrid symbolic execution for combining online and offline (concolic) execution to maximize the benefits of both techniques, and 2) index-based memory modeling, a technique that allows Mayhem to efficiently reason about symbolic memory at the binary level. We used Mayhem to find and demonstrate 29 exploitable vulnerabilities in both Linux and Windows programs, 2 of which were previously undocumented.