Multi-layered intrusion detection and prevention in the SDN/NFV enabled cloud of 5G networks using AI-based defense mechanisms

Abstract Software defined networking (SDN), network function virtualization (NFV), and cloud computing are receiving significant attention in 5G networks. However, this attention creates a new challenge for security provisioning in these integrated technologies. Research in the field of SDN, NFV, cloud computing, and 5G has recently focused on the intrusion detection and prevention system (IDPS). Existing IDPS solutions are inadequate, which could cause large resource wastage and several security threats. To alleviate security issues, timely detection of an attacker is important. Thus, in this paper, we propose a novel approach that is referred to as multilayered intrusion detection and prevention (ML-IDP) in an SDN/NFV-enabled cloud of 5G networks. The proposed approach defends against security attacks using artificial intelligence (AI). In this paper, we employed five layers: data acquisition layer, switches layer, domain controllers (DC) layer, smart controller (SC) layer, and virtualization layer (NFV infrastructure). User authentication is held in the first layer using the Four-Q-Curve algorithm. To address the flow table overloading attack in the switches layer, the game theory approach, which is executed in the IDP agent, is proposed. The involvement of the IDP agent is to completely avoid a flow table overloading attack by a deep reinforcement learning algorithm, and thus, it updates the current state of all switches. In the DC layer, packets are processed and classified into two classes (normal and suspicious) by a Shannon Entropy function. Normal packets are forwarded to the cloud via the SC. Suspicious packets are sent to the VNF using a growing multiple self-organization map (GM-SOM). The proposed ML-IDP system is evaluated using NS3.26 for different security attacks, including IP Spoofing, flow table overloading, DDoS, Control Plane Saturation, and host location hijacking. From the experiment results, we proved that the ML-IDP with AI-based defense mechanisms effectively detects and prevents attacks.