On the analysis of open source datasets: validating IDS implementation for well-known and zero day attack detection

Abstract This paper presents the implementation of an anomaly-based Intrusion Detection System (IDS), capable to detect well-known and zero-day attacks. First, we extend our previous work by generating the Machine Learning (ML) predictors based on KDD99, NSL-KDD and CIC-IDS2018 datasets, and providing the programming language evaluation and the final validation platform. We have built IDS detection solution in two phases. The first Training phase explores available datasets to generate the predictors. The second phase is composed of two processes. Extraction generates the statistical network traffic metrics from the PCAP files and processes them into commma separated values (CSV) files. The Prediction loads predictors in main memory and feeds them with CSV files to predict the well-known and zero-day attacks. The aforementioned initial datasets contain the statistical network traffic metrics of the well-known attacks, collected at runtime execution of the malicious software. Zero day attacks can generate a statistical network traffic metrics similar to well-known attacks. Therefore, to showcase the zero-day anomaly detection, we realise a validation platform. Six attacks (three Denial of Service (DoS) and three scanning), not recorded in the initial datasets, are executed in an isolated environment. The achieved result indicates a misclassification prediction error that inhibits the application of the automatic attack responses, although the misclassification errors were minimised, during the Training phase.