DeepLog

TLDR

System logs, universally available in nearly all computer systems, provide valuable information for debugging, root‑cause analysis, and online monitoring, making anomaly detection a critical step toward secure and trustworthy systems. The authors propose DeepLog, an LSTM‑based model that treats system logs as natural language sequences. DeepLog learns normal log patterns with an LSTM, detects deviations as anomalies, supports online incremental updates, and builds diagnostic workflows to facilitate root‑cause analysis. Experiments on large log datasets demonstrate that DeepLog outperforms existing log‑based anomaly detection methods that rely on traditional data mining.

Abstract

Anomaly detection is a critical step towards building a secure and trustworthy system. The primary purpose of a system log is to record system states and significant events at various critical points to help debug system failures and perform root cause analysis. Such log data is universally available in nearly all computer systems. Log data is an important and valuable resource for understanding system status and performance issues; therefore, the various system logs are naturally excellent source of information for online monitoring and anomaly detection. We propose DeepLog, a deep neural network model utilizing Long Short-Term Memory (LSTM), to model a system log as a natural language sequence. This allows DeepLog to automatically learn log patterns from normal execution, and detect anomalies when log patterns deviate from the model trained from log data under normal execution. In addition, we demonstrate how to incrementally update the DeepLog model in an online fashion so that it can adapt to new log patterns over time. Furthermore, DeepLog constructs workflows from the underlying system log so that once an anomaly is detected, users can diagnose the detected anomaly and perform root cause analysis effectively. Extensive experimental evaluations over large log data have shown that DeepLog has outperformed other existing log-based anomaly detection methods based on traditional data mining methodologies.