A Deep Learning Framework of Autonomous Pilot Agent for Air Traffic Controller Training

In this work, a deep learning-based framework is proposed to implement an autonomous pilot agent (APA), which serves as a human pseudo-pilot to assist air traffic controller (ATCO) training. A novel paradigm, including speech recognition, language understanding, pilot repetition generation (PRG), and text-to-speech (TTS), is designed to formulate the framework pipeline, which also incorporates a simulation system interface. We mainly focus on the PRG and TTS models to address the ATC specificities in this work. The neural architecture is proposed to generate the text repetition instruction by using a sequence-to-sequence text mapping. The Transformer block is improved to implement a high-efficient TTS model, in which the nonautoregressive mechanism is applied to achieve the parallel synthesis. A dedicated phoneme vocabulary is designed to cope with the multilingual issue in the ATC domain and address the out-of-vocabulary problem. With the APA framework, a virtual training mode is proposed to complete the training task without the limitation of time and location. Experimental results on a real-world dataset show that the proposed APA framework replaces the human pilot with considerable high confidence in a real-time manner during the simulation training. Most importantly, the APA framework and the virtual training system are able to cope with the dilemma of physical attendance (like COVID-19) and improve the equipment utilization capacity for the ATCO training.