Evaluating Complexity, Code Churn, and Developer Activity Metrics as Indicators of Software Vulnerabilities

TLDR

Security inspection requires attacker‑thinking experts who must identify vulnerable code locations, a difficult task because vulnerabilities are rare. The study examines whether source‑code and development‑history metrics can predict vulnerable code locations, enabling experts to prioritize inspection. The authors evaluated complexity, code churn, and developer‑activity metrics across Mozilla Firefox and Red Hat Enterprise Linux to predict vulnerabilities. The models correctly identified over 80 % of vulnerable files with less than 25 % false positives, and would cut inspection effort by more than 71 % of files and 28 % of lines of code compared to random selection.

Abstract

Security inspection and testing require experts in security who think like an attacker. Security experts need to know code locations on which to focus their testing and inspection efforts. Since vulnerabilities are rare occurrences, locating vulnerable code locations can be a challenging task. We investigated whether software metrics obtained from source code and development history are discriminative and predictive of vulnerable code locations. If so, security experts can use this prediction to prioritize security inspection and testing efforts. The metrics we investigated fall into three categories: complexity, code churn, and developer activity metrics. We performed two empirical case studies on large, widely used open-source projects: the Mozilla Firefox web browser and the Red Hat Enterprise Linux kernel. The results indicate that 24 of the 28 metrics collected are discriminative of vulnerabilities for both projects. The models using all three types of metrics together predicted over 80 percent of the known vulnerable files with less than 25 percent false positives for both projects. Compared to a random selection of files for inspection and testing, these models would have reduced the number of files and the number of lines of code to inspect or test by over 71 and 28 percent, respectively, for both projects.