A Fully Scalable Big Data Framework for Botnet Detection Based on Network Traffic Analysis

Abstract Many traditional Botnet detection methods have trouble scaling up to meet the needs of multi-Gbps networks. This scalability challenge is not just limited to bottlenecks in the detection process, but across all individual components of the Botnet detection system including data gathering, storage, feature extraction, and analysis. In this paper, we propose a fully scalable big data framework that enables scaling for each individual component of Botnet detection. Our framework can be used with any Botnet detection method - including statistical methods, machine learning methods, and graph-based methods. Our experimental results show that the proposed framework successfully scales in live tests on a real network with 5Gbps of traffic throughput and 50 millions IP addresses visits. In addition, our run time scales logarithmically with respect to the volume of the input for example, when the scale of the input data multiplies by 4x, the total run time increases by only 31 percent. This is significant improvement compared to schemes such as Botcluster in which run time increases by 86 percent under similar scale condition.