Multi-Agent Deep Reinforcement Learning for Massive Access in 5G and Beyond Ultra-Dense NOMA System

With the rapid development of machine-type communications (MTC), the future communication architecture needs to provide services for both human-type communications (HTC) and MTC with unique characteristics. The huge connections from MTC bring serious challenges to the existing wireless network. Ultra-dense network (UDN), a promising candidate technology, can support massive device access through dense deployment of small base stations (SBSs). Different from the resource management in traditional wireless network with single base station (BS), the resource allocation problem at BS level is more prominent in UDN, and the diversity of devices will make this problem more complicated. In view of this, we investigate the joint optimization of massive access and resource management in the UDN where HTC and MTC coexist. Considering the computational complexity and scalability, we propose a multi-agent deep reinforcement learning based SBS state selection scheme, in which each SBS acts as an agent and selects the optimal state between active and idle by continuously interacting with the environment. In addition, we adopt the power-domain non-orthogonal multiple access to further improve system throughput, and use grant-based and grant-free access manners for HTC and MTC respectively, so as to meet their unique characteristics. Extensive numerical results demonstrate the superior performances of proposed scheme in multiple perspectives.