Online System Problem Detection by Mining Patterns of Console Logs

TLDR

We present a novel online system that automatically monitors and detects abnormal execution traces from console logs using data mining and statistical learning methods. The approach employs a two‑stage detection system: first, frequent‑pattern mining and distribution estimation capture dominant execution patterns, and second, PCA‑based anomaly detection flags actual problems. Evaluation on a 203‑node Hadoop cluster demonstrates highly accurate, fast problem detection and provides operators with deeper insight into system execution patterns.

Abstract

We describe a novel application of using data mining and statistical learning methods to automatically monitor and detect abnormal execution traces from console logs in an online setting. Different from existing solutions, we use a two stage detection system. The first stage uses frequent pattern mining and distribution estimation techniques to capture the dominant patterns (both frequent sequences and time duration). The second stage use principal component analysis based anomaly detection technique to identify actual problems. Using real system data from a 203-node Hadoop cluster, we show that we can not only achieve highly accurate and fast problem detection, but also help operators better understand execution patterns in their system.