2018 · 11 citations · 14 references
ComputingComputer ScienceComputer SystemsCompilersComputer ArchitectureOperating SystemsBenchmarkingPerformance ModelingPerformance OptimizationComputer Simulation FrameworksEvaluationEvaluation ObjectivesPerformance BenchmarkingPerformance PredictionPerformance ProfilingTrainingWorkload Characterization
A proper evaluation of techniques that require multiple training and evaluation executions of a benchmark, such as Feedback-Directed Optimization (FDO), requires multiple workloads that can be used to characterize variations on the behaviour of a program based on the workload. This paper aims to improve the performance evaluation of computer systems - including compilers, computer architecture simulation, and operating-system prototypes - that rely on the industrystandard SPEC CPU benchmark suite. A main concern with the use of this suite in research is that it is distributed with a very small number of workloads. This paper describes the process to create additional workloads for this suite and offers useful insights in many of its benchmarks. The set of additional workloads created, named the Alberta Workloads for the SPEC CPU 2017 Benchmark Suite1 is made freely available with the goal of providing additional data points for the exploration of learning in computing systems. These workloads should also contribute to ameliorate the hidden learning problem where a researcher sets parameters to a system during development based on a set of benchmarks and then evaluates the system using the very same set of benchmarks with the very same workloads.
14
Profiling a warehouse-scale computer
Svilen Kanev, Juan Pablo Darago, Kim Hazelwood et al. · 2015 · 386 citations · Full text
Measuring Program Similarity: Experiments with SPEC CPU Benchmark Suites
Aashish Phansalkar, Ajay Joshi, Lieven Eeckhout et al. · 2005 · 171 citations