Monitoring a CI/CD Workflow Using Process Mining

Process mining (PM) is a unique approach to extract workflow models of actual real-world activities, namely those related to software development. To be efficient and produce more reliable results, its algorithms require structured input data. However, actual real-world data originate from multiple heterogeneous sources; thus, integration and normalization are required preparatory steps before applying PM techniques. This problem is exacerbated by the need of performing this analysis in real time, rather than off-line in a batch-style approach. In this paper, we show how Apache Kafka pipelines can be used to support the integration and normalization of the event logs from multiple sources into data streams that feed the process mining algorithms in real-time. An application to the complex CI/CD pipeline of a major European e-commerce company is presented, showing that these techniques provide means to monitor and have higher observability of development processes.