An Analysis of Burstiness and Causality of System Logs

System logs are important data to detect system faults and diagnose root causes of them in a large scale network system. However, due to a huge amount and wide diversity of logs, it is not easy and time consuming for network operators. This paper focuses on burstiness and causality of log time series data to extract meaningful information for troubleshooting. With Kleinberg's burst detection algorithm, we conduct three types of burstiness analysis depending on the combination of the log time series generated by 15 months syslog data obtained in an academic network in Japan: single, pair, and device-based burst detections. The contribution of this paper is as follows. In the single burst detection, we confirm our preprocessing can remove over 90% of trivial bursts. Next, in the pair burst analysis, we investigate causality of co-occurred bursts with causal inference results [9] and find that 99% of pair bursts are coincident; remaining 1% are causal pair bursts. Furthermore, our similarity analysis distinguishes two types of pair bursts depending on complexity of network event causality. In the device-based burst detection, we find 3,735 bursts that are only found by this multivariate analysis. In addition, we find some causal bursts missed in previous causal inference results. To combine these findings, we can extract meaningful log bursts from all the detected ones.