An Efficient Log Parsing Algorithm Based on Heuristic Rules

Log files usually contain very rich running information of the software system, which can be used for anomaly detection, performance modeling, and failure diagnosis, etc. In a large-scale deployment system, log records are always unstructured and can not directly use for log analysis. Log parsing, as a key prerequisite for log analysis, converts unstructured log records into structured event templates by extracting the constant portion of the raw log. Traditionally, log parsing can be achieved by manually using the regular expression, which requires many experts knowledge and has very low efficiency. Therefore, the accuracy and efficiency of log parsing are very important, especially in large-scale distributed systems. In this paper, we propose an efficient algorithm namely CLF (Clustering based on Length and First token) for extracting log event templates from raw log based on heuristic rules. The CLF algorithm works through a 3-step process: clustering unstructured logs based on heuristic rules, clustering again according to specific separation rules and finally generating event templates. Finally, we used 7 data sets to evaluate the performance of CLF and compared with three state-of-the-art log parser algorithms, where CLF ranks higher on most of the data sets and also has advantages in execution time.