Network defence strategy evaluation: Simulation vs. live network

A lot of research has been dedicated to finding an optimal strategy to defend network infrastructure. The proposed methods are usually evaluated using simulations, replayed attacks or testbed environments. However, these evaluation methods may give biased results, because in real life, attackers can follow a suboptimal strategy or react to a defence in an unexpected way. In this paper, we use a network of honeypots as a testing environment for evaluating network defence strategies. The honeypot network provides the opportunity to test a defence strategy against real attackers and is not as time and resource consuming as using white hat hackers. In our experiment, we use two different strategies to defend a group of honeypots in a live network and we compare these results to the results of a simulation with replayed attacks. We show that the results of the strategies in the simulation significantly differ from the results on the honeypot network which implies simulations are not sufficient for strategy evaluation. We also investigate how the attacker adapts to the responses taken by a defence strategy and how this change in behaviour affects the evaluation results.