Something Is Better Than Everything: A Distributed Approach to Audit Log Anomaly Detection

Computing systems produce large amounts of system log information at a scale wildly disproportionate to the growth of computing and bandwidth resources. This growth outpaces the ability of human auditors and administrators to digest such quantities of data via manual analysis. This situation is only expected to worsen over time as more and more data become available due to expanded technological reach and instrumentation. We propose a framework based around process models for analyzing the log data in near-real-time, by taking advantage of distributed resources and then pushing results to a central location for correlation and contextualization instead of relying on the movement of the full volume of raw data. These results will have already been algorithmically determined to be relevant to adversarial process models before being aggregated. Under this distributed philosophy, the computational workload of audit log event detection is shared among end point sensors, and not needing to immediately centralize all raw data reduces overhead and allows the detection framework to scale with advances and expansions in network technology. By only sharing analytical results instead of underlying logs, meaningful information is prioritized, reducing both the burden on the system administrator and the real-time transfer cost of moving information to centralized storage. This is illustrated using a proof-of-concept deployment of a process modeling tool for insider threat detection.