2006 · 586 citations · 21 references
EngineeringComputer ScienceHigh-performance ComputingCluster ComputingFault ToleranceFault-tolerant ComputingDependability MetricsFault ModelingReliability ModelingReliable System DesignFailure Data AnalysisFailure StatisticsMean Time Between FailuresMean Time To RepairMean Time To Repair AnalysisRepair Time AnalysisTime-to-failure AnalysisWeibull Model
Designing highly dependable systems requires a good understanding of failure characteristics, yet little raw data on failures in large IT installations is publicly available. This paper analyzes failure data recently made publicly available by one of the largest high‑performance computing sites. The authors analyze 23,000 failures recorded over nine years at Los Alamos National Laboratory across more than 20 systems, studying root causes, mean time between failures, and mean time to repair. The study shows that average failure rates vary from 20 to 1000 per year, with inter‑failure times following a Weibull distribution with decreasing hazard, while mean repair times range from under an hour to over a day and fit a lognormal distribution.
Designing highly dependable systems requires a good understanding of failure characteristics. Unfortunately, little raw data on failures in large IT installations is publicly available. This paper analyzes failure data recently made publicly available by one of the largest high-performance computing sites. The data has been collected over the past 9 years at Los Alamos National Laboratory and includes 23000 failures recorded on more than 20 different systems, mostly large clusters of SMP and NUMA nodes. We study the statistics of the data, including the root cause of failures, the mean time between failures, and the mean time to repair. We find for example that average failure rates differ wildly across systems, ranging from 20-1000 failures per year, and that time between failures is modeled well by a Weibull distribution with decreasing hazard rate. From one system to another, mean repair time varies from less than an hour to more than a day, and repair times are well modeled by a lognormal distribution
21