ACM Transactions on Storage · 2008 · 205 citations · 148 references
ComputingComputer ScienceComputer SystemsExperimental PerformanceOperating SystemsStorage SystemsBenchmarkingCache Behavior AnalysisCachingFile System AnalysisFile SystemsEvaluation BenchmarksFile System PerformanceFile System Performance EvaluationMeasurement BenchmarksRead-modify-write OperationsStorage BenchmarkingStorage System Benchmarking
Benchmarking is critical when evaluating performance, but is especially difficult for file and storage systems. Complex interactions between I/O devices, caches, kernel daemons, and other OS components result in behavior that is rather difficult to analyze. Moreover, systems have different features and optimizations, so no single benchmark is always suitable. The large variety of workloads that these systems experience in the real world also adds to this difficulty. In this article we survey 415 file system and storage benchmarks from 106 recent papers. We found that most popular benchmarks are flawed and many research papers do not provide a clear indication of true performance. We provide guidelines that we hope will improve future performance evaluations. To show how some widely used benchmarks can conceal or overemphasize overheads, we conducted a set of experiments. As a specific example, slowing down read operations on ext2 by a factor of 32 resulted in only a 2--5% wall-clock slowdown in a popular compile benchmark. Finally, we discuss future work to improve file system and storage benchmarking.
148
Wide-area cooperative storage with CFS
Frank Dabek, M. Frans Kaashoek, David R. Karger et al. · 2001 · 1.4K citations
Ceph: a scalable, high-performance distributed file system
Sage A. Weil, Scott Brandt, Ethan L. Miller et al. · eScholarship (California Digital Library) · 2006 · 1.4K citations · Full text
Scale and performance in a distributed file system
John H. Howard, Michael L. Kazar, Sherri G. Menees et al. · ACM Transactions on Computer Systems · 1988 · 1.4K citations
Sanjay Ghemawat, Howard Gobioff, Shun-Tak A. Leung · 2003 · 1.4K citations