Is finding security holes a good idea?

TLDR

Despite extensive efforts to find and patch security holes, data do not show clear improvement in software quality. The article seeks to quantify the effect of vulnerability finding. The analysis relies on imperfect data and simplifying assumptions, deliberately biasing them toward positive outcomes while explicitly acknowledging counter‑biases. The results suggest that vulnerability finding can be useful, representing the best‑case scenario supported by the data.

Abstract

Despite the large amount of effort that goes toward finding and patching security holes, the available data does not show a clear improvement in software quality as a result. This article aims to measure the effect of vulnerability finding. Any attempt to measure this kind of effect is inherently rough, depending as it does on imperfect data and several simplifying assumptions. Because I'm looking for evidence of usefulness, where possible, I bias such assumptions in favor of a positive result - explicitly calling out those assumptions biased in the opposite direction. Thus, the analysis in this article represents the best-case scenario, consistent with the data and my ability to analyze it, for the vulnerability finding's usefulness