Cinder — A log processing system

Cinder is a log processing system developed in Hrvatski Telekom with a primary purpose to handle and store logs produced by different telco platforms. Together with File Enqueuer system, Cinder offers scalable service capable of real-time processing millions of records each day. Every platform (MMS center, GGSN, IN‥) has its own syntax and log format and therefore the development had to cover all those formats and syntaxes — from well known as ASN.1 to the proprietary ones. Management and storage of such huge amount of data was additional challenge to cope with, having in mind that all the data has to be available for real-time retrieval. This paper summarizes our findings and describes the implementation and solutions to all encountered problems. Additionally, performance of the system is analyzed in detail and various statistics are represented.