The DaCapo benchmarks

TLDR

Benchmarks drive research and product development, yet Java’s dynamic compilation, garbage collection, and SPEC’s legacy methods inadequately capture its performance characteristics. The paper proposes new benchmarking selection and evaluation methods and introduces the DaCapo benchmark suite to better support Java and other managed languages. It employs novel value, time‑series, and statistical metrics for static and dynamic properties such as code complexity, size, heap composition, and pointer mutations to assess benchmarks. The metrics reveal that DaCapo outperforms SPEC Java by stressing more complex code, richer object behavior, and demanding memory systems, highlighting the need for more extensive evaluation of Java workloads.

Abstract

Since benchmarks drive computer science research and industry product development, which ones we use and how we evaluate them are key questions for the community. Despite complex runtime tradeoffs due to dynamic compilation and garbage collection required for Java programs, many evaluations still use methodologies developed for C, C++, and Fortran. SPEC, the dominant purveyor of benchmarks, compounded this problem by institutionalizing these methodologies for their Java benchmark suite. This paper recommends benchmarking selection and evaluation methodologies, and introduces the DaCapo benchmarks, a set of open source, client-side Java benchmarks. We demonstrate that the complex interactions of (1) architecture, (2) compiler, (3) virtual machine, (4) memory management, and (5) application require more extensive evaluation than C, C++, and Fortran which stress (4) much less, and do not require (3). We use and introduce new value, time-series, and statistical metrics for static and dynamic properties such as code complexity, code size, heap composition, and pointer mutations. No benchmark suite is definitive, but these metrics show that DaCapo improves over SPEC Java in a variety of ways, including more complex code, richer object behaviors, and more demanding memory system requirements. This paper takes a step towards improving methodologies for choosing and evaluating benchmarks to foster innovation in system design and implementation for Java and other managed languages.