Controller-Pilot Voice Communication and Intent Monitoring for Future Aviation Systems Safety

In this study, a set of models and software are developed to support Aviation System Safety Analysis tasks. Three primary models are utilized to conduct the following analyses: (1) Speech Attribute analysis, (2) Speech-to-Text analysis, and (3) Intent Inference analysis. We analyze air traffic controller-pilot speech attributes and transcribed texts of voice messages to gain additional insights into controller-pilot communication dynamics. Such analyses are integrated with situation assessment modules to identify any anomalies in flight trajectories. Key speech attribute patterns are demonstrated with LiveATC audios, including a bimodal distribution of Spectral Flatness, Onset Rate, and Onset Strength and clear distinguishing patterns across different sectors. An IsolationForest Outlier classification model is developed to classify off-nominal air traffic controllers’ utterances. Additionally, the Speech-to-Text (STT) model showed a 54% increase in accuracy, with improved model performance (increase in callbacks, commands identified, fragments, and a decrease in run-on words). The intent inference models have successfully identified common pilot intents without STT input. Future work will focus on improving safety by comparing expected and actual words spoken, updating word lists, and examining voice attributes and conditions related to STT issues and intent mismatches. These distinct modules are developed to improve future aviation systems safety, and the final phase of this work will integrate the three models to develop a full system with the capability to detect aviation safety risk.