ACM Transactions on Storage · 2018 · 117 citations · 40 references
EngineeringComputer ScienceComputer SystemsComputer ArchitectureFault-tolerant ComputingFault-tolerant SystemsDevice ReliabilityFault DetectionFault ModelingFault-tolerant Storage SystemsHardware Fault ToleranceReliable System DesignFailure Mode AssessmentFailure Mode DetectionFault Propagation AnalysisFault-tolerant ArchitectureHardware Fault DetectionHardware Reliability AnalysisProcessor FaultsRoot Causes
Fail-slow hardware is an under-studied failure mode. We present a study of 114 reports of fail-slow hardware incidents, collected from large-scale cluster deployments in 14 institutions. We show that all hardware types such as disk, SSD, CPU, memory, and network components can exhibit performance faults. We made several important observations such as faults convert from one form to another, the cascading root causes and impacts can be long, and fail-slow faults can have varying symptoms. From this study, we make suggestions to vendors, operators, and systems designers.
40