Evaluating how static analysis tools can reduce code review effort

Abstract

Peer code reviews are important for giving and receiving peer feedback, but the code review process is time consuming. Static analysis tools can help reduce reviewer effort by catching common mistakes prior to peer code review. Ideally, contributors would use static analysis tools prior to pull request submission so common mistakes could be addressed first, before invoking the reviewer. To explore the potential efficiency gains for peer reviewers, we explore the overlap between reviewer comments on pull requests and warnings from the PMD static analysis tool. In an empirical study of 274 comments from 92 pull requests on GitHub, we observed that PMD overlapped with nearly 16% of the reviewer comments, indicating a time benefit to the reviewer if static analyzers would have been used prior to pull request submission. Using the non-overlapping set of comments, we identify four additional rules that, if implemented, could further reduce reviewer effort.