2008 · 106 citations · 14 references
ComputingComputer ScienceComputer SystemsData MiningFeature EngineeringNatural Language ProcessingAnomaly DetectionFault DetectionLog AnalysisLog ManagementAnomaly Detection In LogsApplication Log AnalysisAutomated Log AnalysisLog ParsingRuntime Anomaly DetectionSource Code AnalysisSystem Log AnalysisText Mining Features
The console logs generated by an application contain messages that the application developers believed would be useful in debugging or monitoring the application. Despite the ubiquity and large size of these logs, they are rarely exploited in a systematic way for monitoring and debugging because they are not readily machine-parsable. In this paper, we propose a novel method for mining this rich source of information. First, we combine log parsing and text mining with source code analysis to extract structure from the console logs. Second, we extract features from the structured information in order to detect anomalous patterns in the logs using Principal Component Analysis (PCA). Finally, we use a decision tree to distill the results of PCA-based anomaly detection to a format readily understandable by domain experts (e.g. system operators) who need not be familiar with the anomaly detection algorithms. As a case study, we distill over one million lines of console logs from the Hadoop file system to a simple decision tree that a domain expert can readily understand; the process requires no operator intervention and we detect a large portion of runtime anomalies that are commonly overlooked. 1
14
Ian H. Witten, Eibe Frank · ACM SIGMOD Record · 2002 · 5.2K citations
Diagnosing network-wide traffic anomalies
Anukool Lakhina, Mark Crovella, Christophe Diot · 2004 · 1K citations · Full text
Ingo Mierswa, Michael Wurst, Ralf Klinkenberg et al. · 2006 · 1K citations