Dynamic Multichannel Sensing in Cognitive Radio: Hierarchical Reinforcement Learning

Efficient use of spectral resources is critical in wireless networks and has been extensively studied in recent years. Dynamic spectrum access (DSA) is one of the key techniques on utilizing the spectral resources. Among them, reinforcement learning (RL) for DSA has gained great attention due to the excellent performance. Limited by the large state space in RL, obtaining the best solution to the spectrum access problem is often computationally expensive. Besides, it is hard to balance multiple objectives of the reward function in RL. To tackle these problems, we explore deep reinforcement learning in a layered framework and propose a hierarchical deep Q-network (h-DQN) model for DSA. The proposed approach divides the original problem into separate “sub problems”, each of which is solved using its own reinforcement learning agent. This partitioning simplifies each individual problem, enables modularity, and reduces the complexity of the whole optimization process in the multi-objective case. The performance of Q-learning for dynamic sensing(QADS), deep reinforcement learning for dynamic access (DRLDA), and the proposed h-DQN model is evaluated through simulations. The simulation results show that h-DQN yields better performance with the faster convergence and higher channel utilization than the other two compared methods.