Deep Q-Learning Based Node Positioning for Throughput-Optimal Communications in Dynamic UAV Swarm Network

In this paper, we study the communication-oriented unmanned air vehicle (UAV) placement issue in a typical manned-and-unmanned (MUM) airborne network. The MUM network consists of a few powerful aircraft nodes in the higher layer and high-density UAVs in the lower layer. While the aircraft network is relatively stable, the UAVs can form different swarm network topologies. Some UAVs are selected as gateway nodes to aggregate the received UAV data and send to a nearby aircraft which acts as a control node for the UAVs in a swarm. Assume a source UAV has data to be sent to its gateway node by using a route which may have broken links. Our goal is to guide the position of one or more relay UAVs to make up for the broken wireless links under the dynamic swarm topology. The placement of the relay node is determined by both traffic quality-of-service (QoS) requirements and the link conditions. We design a new queueing model, called multi-hop priority queue, to analyze the achievable QoS performance through multi-hop queue-to-queue accumulation modeling. To handle dynamic swarm topology and time-varying link conditions, we design a deep ${Q}$ -learning (DQN) model to determine the optimal link between two UAV nodes, and then use an optimization algorithm to locally fine-tune the position of the UAV node to optimize the overall network performance. The DQN-based UAV link selection is computed in the powerful aircraft (control node) which maintains the graphs of the swarm topology, where the optimization is implemented at the UAV. Our simulation results validate the throughput efficiency of our DQN-based UAV positioning scheme.