Near real-time intrusion alert aggregation using concept-based learning

Intrusion detection systems generate a large number of streaming alerts. It can be overwhelming for analysts to quickly and effectively find related alerts stemmed from correlated attack actions. What if fast arriving alerts could be automatically processed with no prior knowledge to find related actions in near real-time? The Concept Learning for Intrusion Event Aggregation in Realtime (CLEAR) system aims to learn and update an evolving set of temporal 'concepts,' each consisting of aggregates of related alerts that exhibit similar statistical arrival patterns. With no training data, the system constructs the concepts in near real-time from statistically similar alert aggregates. Tracked concepts are then applied to incoming alerts for fast and high-fidelity aggregation. The concepts learned by CLEAR are significantly more unique and invariant when compared to those learned by alternative drift detection methods. Furthermore, it provides insights for how specific individual, or co-occuring, alerts arrive with distinct and consistent temporal patterns.