Integrating Short History for Improving Clustering Based Network Traffic Anomaly Detection

Traffic anomaly detection is of premier importance for network administrators as anomalies have a dramatic impact on network performances, and QoS perceived by users. It is, however, a very time consuming and costly task that often requires decision from network and security experts. For making anomaly detection autonomous, many research works started investigating the use of unsupervised machine learning techniques, and in most cases traffic clustering. Identifying the clusters corresponding to anomalous traffic classes among the full set of detected clusters still remains a challenge. This is mostly due to the nature of clustering techniques that work on traffic samples of a given duration, each cluster being classified after an uncertain post processing stage. In this paper, we show how anomaly detectors can benefit from keeping a temporal track of the clustering results along time. This improvement has been added to ORUNADA (Online Real-time Unsupervised Network Anomaly detection Algorithm) that aimed at providing efficient anomaly detection on high speed networks. This new ORUNADA version - called H-ORUNADA for History-ORUNADA - is then evaluated on a new ground truth, called SynthONTS, that is currently designed to provide a modern and complete dataset with labeled anomaly. H-ORUNADA has also been implemented on Spark Streaming for being able to work on very high speed networks (targeting several hundreds of Gbits/s), and evaluated on the Google Cloud Platform.