Data Management and Analytics System for Online Flight Conformance Monitoring and Anomaly Detection

Air Navigation Service Providers (ANSP) worldwide have been making a considerable effort for the development of a better method to monitor conformance to the planned routes and detect anomalies within a particular airspace. Conformance monitoring and anomaly detection are crucial for a better managed airspace, both strategically and tactically, yielding a higher level of automation and thereby reducing the air traffic controller's workload. Although the prior approaches with limited amount of static air traffic data have been able to address the problem to some extent, data management and query processing of ever-increasing vast volume of streaming air traffic data at high rates for online conformance monitoring and anomaly detection still remain a challenge. In this paper, we present a novel data management and analytics system to continuously conformance monitor flights and accurately detect anomalies within the National Airspace System (NAS). The incoming Traffic Flow Management (TFM) data is streaming, big, uncorrelated and noisy. In the overall data pipeline, the system monitors flights and detects anomalies in 3 steps: In the preprocessing step, the system continuously processes the incoming raw flight data and makes it available for the next step where an interim Key-Value data store is created and maintained for efficient query processing. In the final step, the system learns from historical trajectories and pertinent weather parameters and builds a Long Short-Term Memory (LSTM) model. As the flights progress, the non-conforming trajectory segments as part of the live data stream are raised as anomalies. Evaluations on real air traffic and weather data in the U.S. verify that our system efficiently and accurately detects anomalies.