Log Parser with One-to-One Markup

Abstract

System logs are often used as the primary resource in data-driven methods to ensure system health and stability. The typical process of system log analysis is to first parse unstructured logs into structured data, and then apply data mining and machine learning techniques to analyze the data and build a workflow model. Existing log parsing methods focus on similar matching of log messages and log templates. We believe that the accuracy of log message parsing is the primary task of log parsing, so we propose One-to-One, a log parser that is marked one-to-one according to the rules duringthe matching process according to the token type and part of speech. Way to parse log messages online. We evaluated One-to-One on different log sets and compared them with the three most advanced log parsing methods. The results show that our method is similar to the results of the other three methods in parsing simple logs. However, when parsing complex OpenStack logs, the accuracy can reach 98%, which is 20% higher than the best. It can parse tens of thousands of log messages per second. This method shows high efficiency and precision for all three types of test logs, and is applicable to modern system logs.