Automatic feature engineering from very high dimensional event logs using deep neural networks

As communication networks have grown, event logs have increased in both size and complexity at a very fast rate. Thus, mining event logs has become very challenging due to their high variety and volume. The traditional solution to model raw event logs is to transform the raw logs into features with fewer dimensions through manual feature engineering. However, feature engineering is very time-consuming, and its quality is highly dependent on data scientists' domain knowledge. Furthermore, repeatedly preprocessing event logs significantly delays the scoring process, which must scan all items in the logs. In this paper, we present our recent study on mining high-dimensional event logs using deep neural networks. We propose a Midway Neural Network (MNN) to avoid both manual feature engineering and the re-preprocessing of event logs. MNN embeds an input feature vector from a particular time window into a dense representation and memorizes these midway representations for incremental training and prediction. The experimental results demonstrated that the proposed method minimized human intervention, decreased the time for training and scoring, and decreased the memory and storage costs while maintaining a similar modeling performance compared to traditional solutions. We hope that our insights and knowledge can inspire colleagues who are working on similar problems.