A Multi-Agent Approach to Advanced Persistent Threat Detection in Networked Systems

Advanced cyber threats that are well planned, funded and stealthy are an increasing issue facing secure networked systems. As our reliance on protected networked systems continues to grow, the motivation for developing new malicious techniques that cannot be easily detected by traditional signature-based systems, and that make use of previously unseen zero-day vulnerabilities, continues to grow. Lack of adaptivity, extended data-collection and generalised algorithms to detect stealthy attacks is contributing to the insecurity of modern networked systems. To protect these networks, new approaches that can monitor and respond to indicators of compromise in a reflective way that considers all of the available evidence rather than individual points of data is required. This thesis presents a novel approach to intrusion detection and specifically focuses on detecting advanced persistent threats which are characteristically stealthy and evasive attacks. This approach offers a multi-agent model for automatically collecting, analysing and classifying data in a distributed way that considers the context in which the data was found. Using a context-based classification that considers the likelihood of a data-point being a false alarm or legitimate is used to decrease the prevalence of erroneous classifications and regulate continuation of the data collection process. Using this architecture, a detection rate increase of up to 20% is achieved in false alarm environments and an efficiency increase of up to 50% made over traditional monolithic intrusion detection systems. Additionally, the shortcomings of algorithms to detect stealthy attacks are addressed by providing a generalised anomaly detection algorithm for detecting the initial traces of an attack and deploying the proposed multi-agent model to investigate the attack further. The generalised algorithms can detect a wide variety of network-based attacks at an average detection rate of 85% providing an accurate and scalable way to detect the initial traces of compromise. The main novelty of this thesis is providing systems for detecting attacks where the threat model is increasingly stealthy and assumed capable of bypassing traditional signature-based approaches. The multi-agent architecture is unique in its ability, and the generalised anomaly detection algorithm is novel in detecting a variety of different cyber attacks from the network-flow layer. The evidence from this research suggests that context-based evidence gathering can provide a more efficient approach to analysing data and the generalised anomaly detection algorithm can be applied widely to detect attack indicators.