A large-scale study of failures in high-performance computing systems

Bianca Schroeder, Garth A. Gibson

2006 · 586 citations · 21 references

Concepts

TL;DR

Designing highly dependable systems requires a good understanding of failure characteristics, yet little raw data on failures in large IT installations is publicly available. This paper analyzes failure data recently made publicly available by one of the largest high‑performance computing sites. The authors analyze 23,000 failures recorded over nine years at Los Alamos National Laboratory across more than 20 systems, studying root causes, mean time between failures, and mean time to repair. The study shows that average failure rates vary from 20 to 1000 per year, with inter‑failure times following a Weibull distribution with decreasing hazard, while mean repair times range from under an hour to over a day and fit a lognormal distribution.

Abstract

Designing highly dependable systems requires a good understanding of failure characteristics. Unfortunately, little raw data on failures in large IT installations is publicly available. This paper analyzes failure data recently made publicly available by one of the largest high-performance computing sites. The data has been collected over the past 9 years at Los Alamos National Laboratory and includes 23000 failures recorded on more than 20 different systems, mostly large clusters of SMP and NUMA nodes. We study the statistics of the data, including the root cause of failures, the mean time between failures, and the mean time to repair. We find for example that average failure rates differ wildly across systems, ranging from 20-1000 failures per year, and that time between failures is modeled well by a Weibull distribution with decreasing hazard rate. From one system to another, mean repair time varies from less than an hour to more than a day, and repair times are well modeled by a lognormal distribution

References

21