CEFF: An efficient approach for traffic anomaly detection and classification

Nowadays, there are two major challenges to detect traffic anomalies in a large scale network. One is how to handle huge amounts of traffic data when we detect traffic anomalies in a network, and the other is how to carry out fast and detailed detection and classification. To address these two challenges, we propose a Change based Effective Frequent flow Features approach (CEFF), which can quickly obtain the anomaly detection and classification results by scanning the flow data only once. We implement CEFF for both offline and online detection and classification in Spark, a popular big data processing platform. Besides, we evaluate CEFF using China Telecom NetFlow format data in experiments, and make comparisons between CEFF and Shannon entropy based method, which has been proved to be effective for traffic anomaly detection. The experiment results show that CEFF has excellent performance in traffic anomaly detection and classification.