Toward a monitoring and threat detection system based on stream processing as a virtual network function for big data

The late detection of security threats causes a significant increase in the risk of irreparable damages and restricts any defense attempt. In this paper, we propose a sCAlable TRAffic Classifier and Analyzer (CATRACA). CATRACA works as an efficient online Intrusion Detection and Prevention System implemented as a Virtualized Network Function. CATRACA is based on Apache Spark, a Big Data Streaming processing system, and it is deployed over the Open Platform for Network Functions Virtualization (OPNFV), providing an accurate real‐time threat‐detection service. The system presents a friendly graphical interface that provides real‐time visualization of the traffic and the attacks that occur in the network. Our prototype can differentiate normal traffic from denial of service (DoS) attacks and vulnerability probes over 95% accuracy under three different datasets. Moreover, CATRACA handles streaming data under concept drift detection with more than 85% of accuracy.