Using benchmarking to advance research: a challenge to software engineering

TLDR

Benchmarks are widely used in computer science to compare performance, and their adoption often drives rapid technical progress and community building, leading us to formulate a theory of benchmarking within scientific disciplines. Based on this theory, we challenge software engineering research to become more scientific and cohesive by working as a community to define benchmarks. We present a case study of the reverse engineering community that demonstrates how benchmarks can be employed to advance research. The case study shows that using benchmarks successfully advanced the state of research in reverse engineering.

Abstract

Benchmarks have been used in computer science to compare the performance of computer systems, information retrieval algorithms, databases, and many other technologies. The creation and widespread use of a benchmark within a research area is frequently accompanied by rapid technical progress and community building. These observations have led us to formulate a theory of benchmarking within scientific disciplines. Based on this theory, we challenge software engineering research to become more scientific and cohesive by working as a community to define benchmarks. In support of this challenge, we present a case study of the reverse engineering community, where we have successfully used benchmarks to advance the state of research.