2013 · 60 citations · 19 references
BusinessComputer ScienceComputer SystemsPerformance MeasurementSystems AnalysisVirtualizationBenchmarkingComparative Performance AnalysisPerformance EvaluationPerformance ModelingPerformance OptimizationPerformance TestingBenchmarking TestingExperimental MethodologyPerformance BenchmarkingServer BenchmarkingVirtual Machines
Experimental evaluation is key to systems research. Because modern systems are complex and non-deterministic, good experimental methodology demands that researchers account for uncertainty. To obtain valid results, they are expected to run many iterations of benchmarks, invoke virtual machines (VMs) several times, or even rebuild VM or benchmark binaries more than once. All this repetition costs time to complete experiments. Currently, many evaluations give up on sufficient repetition or rigorous statistical methods, or even run benchmarks only in training sizes. The results reported often lack proper variation estimates and, when a small difference between two systems is reported, some are simply unreliable.
19
The earth is round (p < .05).
Jacob Cohen · American Psychologist · 1994 · 3.9K citations
Stephen M. Blackburn, Robin Garner, Chris Hoffmann et al. · 2006 · 1.6K citations · Full text