Empirical Analysis of System-Level Vulnerability Metrics through Actual Attacks

TLDR

The Common Vulnerability Scoring System (CVSS) is a widely used standard for scoring vulnerability severity, and although several system‑level aggregation metrics exist, their effectiveness has not been empirically examined. This study statistically evaluates how 18 CVSS‑based security estimation metrics correlate with time‑to‑compromise in 34 real attacks. The analysis uses data from an international cyber defense exercise, derived from network, attacker, observer logs, and vulnerability scans of over 100 participants. The study finds that CVSS data alone poorly predicts time‑to‑compromise, but metrics incorporating more CVSS information show stronger correlation, and weakest‑link models are less effective.

Abstract

The Common Vulnerability Scoring System (CVSS) is a widely used and well-established standard for classifying the severity of security vulnerabilities. For instance, all vulnerabilities in the US National Vulnerability Database (NVD) are scored according to this method. As computer systems typically have multiple vulnerabilities, it is often desirable to aggregate the score of individual vulnerabilities to a system level. Several such metrics have been proposed, but their quality has not been studied. This paper presents a statistical analysis of how 18 security estimation metrics based on CVSS data correlate with the time-to-compromise of 34 successful attacks. The empirical data originates from an international cyber defense exercise involving over 100 participants and were collected by studying network traffic logs, attacker logs, observer logs, and network vulnerabilities. The results suggest that security modeling with CVSS data alone does not accurately portray the time-to-compromise of a system. However, results also show that metrics employing more CVSS data are more correlated with time-to-compromise. As a consequence, models that only use the weakest link (most severe vulnerability) to compose a metric are less promising than those that consider all vulnerabilities.