2007 · 608 citations · 25 references
ComputingComputer ScienceComputer SystemsComputer ArchitectureFault-tolerant ComputingOperating SystemsError DetectionFault DetectionFault DiagnosisLog AnalysisLog ManagementSystem MonitoringAnomaly Detection In LogsAutomated Log AnalysisFilteringLog CollectionLog Data CollectionLog-based DetectionOperational LogsSystem Log Analysis
Real deployed system logs are needed to detect and diagnose failures in large‑scale computer systems, but empirical data has been inaccessible, hindering progress. The study examines system logs from five supercomputers to provide insight and guide future research on log‑based failure detection. The authors collected logs from Blue Gene/L, Red Storm, Thunderbird, Spirit, and Liberty, identified alerts, and introduced a simpler filtering algorithm and an expanded operational context. This is the first study of raw system logs from multiple supercomputers.
If we hope to automatically detect and diagnose failures in large-scale computer systems, we must study real deployed systems and the data they generate. Progress has been hampered by the inaccessibility of empirical data. This paper addresses that dearth by examining system logs from five supercomputers, with the aim of providing useful insight and direction for future research into the use of such logs. We present details about the systems, methods of log collection, and how alerts were identified; propose a simpler and more effective filtering algorithm; and define operational context to encompass the crucial information that we found to be currently missing from most logs. The machines we consider (and the number of processors) are: Blue Gene/L (131072), Red Storm (10880), Thunderbird (9024), Spirit (1028), and Liberty (512). This is the first study of raw system logs from multiple supercomputers.
25
Critical event prediction for proactive management in large-scale computer clusters
R.K. Sahoo, Adam J. Oliner, Irina Rish et al. · 2003 · 268 citations
BlueGene/L Failure Analysis and Prediction Models
Yinglung Liang, Yanyong Zhang, Anand Sivasubramaniam et al. · 2006 · 258 citations
Failure data analysis of a large-scale heterogeneous server environment
R.K. Sahoo, Mark S. Squillante, Anand Sivasubramaniam et al. · 2004 · 238 citations