State of the Art: Automated Black-Box Web Application Vulnerability Testing

TLDR

Black‑box web application vulnerability scanners are automated tools that probe web applications for security vulnerabilities. The study aims to assess the state of the art of such scanners by examining the classes of vulnerabilities they test, their effectiveness, and the relevance of those vulnerabilities to real‑world findings, while avoiding vendor comparisons. The authors evaluated eight leading tools using a custom vulnerable web application and earlier versions of popular applications that contain known vulnerabilities. Results indicate that automated scanners are generally effective but have notable limitations, particularly in detecting stored XSS and SQL injection vulnerabilities.

Abstract

Black-box web application vulnerability scanners are automated tools that probe web applications for security vulnerabilities. In order to assess the current state of the art, we obtained access to eight leading tools and carried out a study of: (i) the class of vulnerabilities tested by these scanners, (ii) their effectiveness against target vulnerabilities, and (iii) the relevance of the target vulnerabilities to vulnerabilities found in the wild. To conduct our study we used a custom web application vulnerable to known and projected vulnerabilities, and previous versions of widely used web applications containing known vulnerabilities. Our results show the promise and effectiveness of automated tools, as a group, and also some limitations. In particular, "stored" forms of Cross Site Scripting (XSS) and SQL Injection (SQLI) vulnerabilities are not currently found by many tools. Because our goal is to assess the potential of future research, not to evaluate specific vendors, we do not report comparative data or make any recommendations about purchase of specific tools.