A data mining analysis of RTID alarms

Abstract IBM's emergency response service provides real-time intrusion detection (RTID) services through the Internet for a variety of clients. As the number of clients increases, the volume of alerts generated by the RTID sensors becomes intractable. This problem is aggravated by the fact that some sensors may generate hundreds or even thousands of innocent alerts per day. With an eye towards managing these alerts more effectively, IBM's data mining services group analyzed a database of RTID reports. The first objective was an approach for characterizing the “normal” stream of alerts from a sensor. Using such models tuned to individual sensors, we then developed a methodology for detecting anomalies. In contrast to many popular approaches, the decision to filter an alarm out or not takes into consideration the context in which it occurred and the historical behavior of the sensor it came from. Our second objective was to identify all the different profiles of our clients. Based on their history of alerts, we discovered several different types of clients, with different alert behaviors and thus different monitoring needs. We present the issues encountered, solutions, and findings, and discuss how our results may be used in large-scale RTID operations.