Hackers vs. Testers: A Comparison of Software Vulnerability Discovery Processes

TLDR

Software vulnerability discovery is typically performed by testers before release and by white‑hat hackers afterward, yet the processes and interactions between these groups are poorly understood. This study aims to better understand and improve the vulnerability discovery ecosystem by interviewing 25 testers and hackers about their methods, skill development, and challenges. The authors conducted a semi‑structured interview study with 25 participants, comprising both testers and hackers, to examine how each group finds vulnerabilities, develops skills, and faces challenges. Findings show that testers and hackers follow similar processes but achieve different results because of varying experience and security knowledge, leading to recommendations for enhanced tester training, improved hacker–developer communication, and smarter bug‑bounty policies.

Abstract

Identifying security vulnerabilities in software is a critical task that requires significant human effort. Currently, vulnerability discovery is often the responsibility of software testers before release and white-hat hackers (often within bug bounty programs) afterward. This arrangement can be ad-hoc and far from ideal; for example, if testers could identify more vulnerabilities, software would be more secure at release time. Thus far, however, the processes used by each group - and how they compare to and interact with each other - have not been well studied. This paper takes a first step toward better understanding, and eventually improving, this ecosystem: we report on a semi-structured interview study (n=25) with both testers and hackers, focusing on how each group finds vulnerabilities, how they develop their skills, and the challenges they face. The results suggest that hackers and testers follow similar processes, but get different results due largely to differing experiences and therefore different underlying knowledge of security concepts. Based on these results, we provide recommendations to support improved security training for testers, better communication between hackers and developers, and smarter bug bounty policies to motivate hacker participation.