Continuous Fuzzing: A Study of the Effectiveness and Scalability of Fuzzing in CI/CD Pipelines

Abstract

While fuzzing can be very costly, it has proven to be a fundamental technique in uncovering bugs (often security related) in many applications. A recent study on bug reports from OSS-Fuzz observed that recent code changes are responsible for 77% of all reported bugs, stressing the importance of continuous testing. With the increased adoption of CI/CD practices in software development, it is only natural to look for effective ways of incorporating fuzzing into continuous security testing. In this paper, we study the effectiveness of fuzz testing in CI/CD pipelines with a focus on security related bugs and seek optimization opportunities to triage commits that do not require fuzzing. Through experimental analysis, we found that the fuzzing effort can be reduced by 63% in three of the nine libraries we analyzed (55% on average). Additionally, we investigate the correlation between fuzzing campaign duration and the effectiveness of fuzzers in vulnerability discovery: a significantly shorter fuzzing campaign facilitates a faster pipeline for developers, while it can still uncover important bugs. Our findings suggest that continuous fuzzing is indeed beneficial for secure software development processes, and that there are many opportunities to improve its effectiveness.