Identifying and managing risks of AI-driven operations: A case study of automatic speech recognition for improving air traffic safety

Yi LIN, Min RUAN, Kunjie CAI, Dn LI, Ziqing ZENG,Fn LI, Bo YANG,*

aCollege of Computer Science, Sichuan University, Chengdu 610045, China

bSouthwest Air Traffic Management Bureau, Civil Aviation Administration of China, Chengdu 610000, China

cBusiness School, Sichuan University, Chengdu 610045, China

dKey Laboratory of Flight Techniques and Flight Safety,CAAC,Civil Aviation Flight University of China,Guanghan 618307,China

KEYWORDS

AbstractIn this work, the primary focus is to identify potential technical risks of Artificial Intelligence (AI)-driven operations for the safety monitoring of the air traffic from the perspective of speech communication by studying the representative case and evaluating user experience.The case study is performed to evaluate the AI-driven techniques and applications using objective metrics,in which several risks and technical facts are obtained to direct future research.Considering the safety–critical specificities of the air traffic control system, a comprehensive subjective evaluation is conducted to collect user experience by a well-designed anonymous questionnaire and a faceto-face interview.In this procedure,the potential risks obtained from the case study are confirmed,and the impacts on human working are considered.Both the case study and the evaluation of user experience provide compatible results and conclusions: (A) the proposed solution is promising to improve the traffic safety and reduce the workload by detecting potential risks in advance; (B)the AI-driven techniques and whole diagram are suggested to be enhanced to eliminate the possible distraction to the attention of air traffic controllers.Finally, a variety of strategies and approaches are discussed to explore their capability to advance the proposed solution to industrial practices.?2022 Chinese Society of Aeronautics and Astronautics.Production and hosting by Elsevier Ltd.This is an open access article under the CC BY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).

1.Introduction

1.1.Air traffic control

Air traffic is an extremely complicated system depending on sophisticated collaborations of various concerned stakeholders, such as state departments, airports, airline companies,air traffic management, etc.With the booming of the civil aviation industry, air traffic safety is experiencing a critical challenge1.The concerned departments studied policies,management measures,and assistive systems to enhance operational safety.As the key to air traffic safety,a dedicated position, Air Traffic Controller (ATCO), is designed to direct aircraft to prevent collisions and improve operational efficiency2.

This procedure is called Air Traffic Control (ATC), with the duty of providing services to the flight operation, which is also a core part of Air Traffic Management(ATM).During the ATM procedure, the Air-Ground Collaboration (AGC) is continuously attracting management attention since it is always regarded as a Human-In-The-Loop(HITL)risk procedure3.The double-post regulation,i.e.,two ATCOs for a single sector simultaneously, was built to eliminate the safety risk caused by human psychological and physiological conditions.

Specifically, since the aircrew is not capable of efficiently perceiving the surrounding environment in the air, they are only allowed to perform the aircraft operation with the guidance of the ground service provided by ATC departments.In other words, the aircraft safety in the air is highly dependent on the ATCO decision.In current practices, AGC negotiation depends on speech communication via the radio transmission in an analog manner, i.e., Very High-Frequency (VHF)communication.Although enormous efforts have been made to improve the Controller Pilot Data Link Communications(CPDLC) for digital transmission4, it still only serves as an emergency measure for the air-ground communication for the following limitations:

(1) The CPDLC requires a certain onboard device to create communication with ground devices,which results in an upgrade for the aircraft system.

(2) The CPDLC is based on satellite communication,which needs an extra operational expense for stakeholders,such as airline companies and airports.

Fig.1 Procedure of ATC work (human and system).

According to the Standard Operational Procedure (SOP)published by the International Civil Aviation Organization(ICAO), the AGC lies in a four-step communication loop,i.e., issue, readback, execution, and monitoring, as illustrated in Fig.1.In the issue and readback stages, any speechrelated misunderstanding or error between the ATCO and aircrew may impact the air traffic operation, which even further causes some incident risks (i.e., in execution and monitoring stages).In this way, introducing speech communication into the current ATC systems is an initial step to monitor the HITL factors,based on which the ATC-related applications are capable of achieving safety enhancement in an automatic manner.

1.2.Motivations of solution

In the AGC procedure,speech communication is the most vulnerable process (susceptible to transmission interference) due to HITL factors.It is believed that the HITL factors deserve to pay more attention to detect the potential risks before the real accidents occurred5.In recent years, increasing incidents were caused by speech-based AGC communication, and the HITL factors always continually generate safety risks6.Based on the aforementioned descriptions, clear and efficient speech communication between the ATCO and the aircrew is a prerequisite to ensure the operational safety of air traffic, which is also the most important procedure to be monitored in this proposal.

Therefore, automatically detecting potential risks from the AGC procedure is the key to improving air traffic safety,which benefits the management of the HITL factors and further the enhancement of air traffic safety7.In this work, considering the mentioned contexts, a new solution (called AISafe) is proposed to monitor the AGC procedure between the ATCO and aircrew, with the ultimate goal of improving the safety and efficiency of the air traffic operation.

To achieve the mentioned goals, it can be seen that Automatic Speech Recognition (ASR) is a fundamental technique to introduce the analog speech signal into the current ATC system, which can formulate an automated closed-loop processing pipeline.The primary purpose of the ASR technique is to convert the speech signal into text instruction that can be fed into an automatic processing system.In succession, the Spoken Instruction Understanding (SIU) technique is also implemented to further extract structured elements (including intent and required parameter items) from text instructions to support the subsequent ATC-related applications.Based on the ASR and SIU techniques,it is feasible to capture values from real-time AGC dynamics concerning ATC contexts (text and structured element) that can be further applied to achieve risk detection8.

Fortunately, thanks to the available data storage of the existing ATC-related systems5, it is practical to tackle the AGC issues by the data-driven approaches to achieve Artificial Intelligence (AI)-driven innovations in the ATC domain.In the AISafe solution, the ASR and SIU techniques in the ATC context are achieved by improving state-of-the-art deep learning models, leveraging real-world samples to fit the data distribution to support the mentioned tasks5.To be specific,for each stakeholder, a corresponding information system is required to be created to support its operation,which produces a large amount of data to record the real-time traffic situation.In general, the recorded data comprises structured data (such as flight trajectory) and unstructured data (such as spoken instruction).It is believed that big data has the ability to create and capture values to advance industrial developments9,which further generates AI-driven operations in real industrial practices.The historical data is of great potential to improve the quality of transportation services, such as operational safety and efficiency10.Increasing academic studies attempted to mine the historical data to find significant transition patterns11–12and further facilitate air traffic safety13–14.

The AISafe comprises the following modules: data sensing(spoken instructions, flight trajectory, etc.), traffic prediction,and risk detection (regulation, readback, potential conflicts,etc.).The details of the proposed AISafe solution are provided in Section 2,and a software prototype is developed to validate the proposed solution.By the proposed solution, we can cope with the human factors of the ATC procedure in an automatic manner.In addition, since the information sensing from the AGC procedure is achieved before the aircraft performs the required operation, the proposed solution is able to provide more look-ahead time than current ATC systems, which benefits the emergency disposal for ATCO.The post analysis based on the AI-driven techniques is also developed to detect possible risks from historical data, which is further applied to optimize the real-time working procedure.

1.3.Research scheme

As a HITL management system, it is important to study the potential impacts of introducing a completely new AI-driven practice into a safety–critical domain15, on both the human factors and the ATC procedure16.To this end, in this work,we mainly focus on evaluating the effectiveness and efficacy of the new prototype, as well as the actual experience of front-line users17.The primary purpose is not only to clarify the advantages of the proposed solution on air traffic safety but also to identify the potential management risks (human and ATC operations) from current practices.The primary research diagram of this work is shown in Fig.2, following a five-step workflow.

Firstly, based on the aforementioned descriptions, we tackle the AGC procedure to reduce human-related risks,which benefits the formulation of problem definition and the clarification of safety–critical applications.In this stage, we also investigate the domain-specific characteristics to emphasize the technical difficulties that need to be addressed for this solution.Secondly, core techniques are proposed and implemented to achieve the proposed solution, including the datarelated work, data-driven approaches, and diagram of risk detection for the pre-defined tasks.After completing the solution design, a software prototype is developed to serve as an entity to perform the whole solution for real-world applications,which also provides the raw case for solution evaluation.Based on the solution practice,we investigate the concept and prototype evaluation by case studies and user experience,with the purpose of identifying the potential adverse effects on current ATC work.Based on the identified management risks,we also attempt to explore the potential measures to eliminate those risks and provide further improvements on the proposed solution and prototype.

Fig.2 Research diagram in this work.

Finally, by revisiting the system functionalities and user feedback18, key suggestions about the ATC procedures and technical management are expected to be achieved to improve the applicability of the proposed solution (i.e., enhancing the safety of air traffic by reducing human-related errors),and further advance it to real practice in the industry.In general, we evaluate the proposed AISafe solution and its prototype in the following three steps:

(1) A comprehensive case study is conducted to evaluate the AI-empowered techniques and ATC-related safety applications by corresponding objective metrics.The Chengdu area control center is selected as the case since it is the first representative application site of the proposed solution.

(2) To consider the subjective evaluation of system performance from front-line users, we also employ the questionnaire and interview to discuss the system usage and applicability, for both the merits and limitations.Specifically, we prefer to collect the shortages and limitations of the proposed system on the human (ATCOs)and ATC procedure(safety and efficiency),which can be applied to further improve the system performance.

(3) Based on both the objective and subjective evaluation of the proposed AISafe solution, key findings of the proposed solution are expected to be achieved to indicate further improvements, for both ATC procedure and technical management.

1.4.Contributions

All in all,this work contributes to the AI-based innovations of the ATC domain in the following ways:

(1) Considering the importance of the AGC procedure in the ATC domain, an AI-driven innovation, called AISafe, is proposed to monitor the AGC speech, with the goal of reducing the potential risks caused by HITL factors.The new solution enhances the safety from the perspective of speech communication during the AGC procedure, which is able to detect risks in advance(i.e.,before performing required aircraft actions)to provide more prewarning time to an emergency.Meanwhile, the ATCO workload can also be reduced by automatically monitoring the aircrew readback and flight behavior.

(2) A comprehensive evaluation of the proposed AISafe solution is conducted based on a case study (Chengdu ACC) and evaluation of user experience (questionnaire and interview), concerning the AI-based techniques and core safety applications.Most importantly, certain potential risks of the AI-driven operations are inferred by the system evaluation, in which some key facts are also found to enlighten the research topics for future system improvements.

(3) Based on the detected technical risks and facts,a variety of strategies and approaches are considered and explored to enhance the performance of each module in this proposed solution, concerning speech quality,digit recognition, advanced model architecture, contextual awareness, human interface, working procedure,etc.All the improvements aim to enhance the applicability and reliability of the proposed AISafe solution in a real ATC environment, on both safety enhancement for air traffic operations and workload reduction for ATCOs.

1.5.Organization of this paper

The rest of this paper is organized as follows: The proposed solution is described in detail in Section 2.In Section 3, we report and discuss the results of the case study in the Chengdu area control center.The design of the subjective evaluation is sketched in Section 4.The results of the subjective evaluation are listed in Section 5,where we also provide the potential risks of the proposed solution and measures for further improvements.The conclusions are drawn in Section 6.

2.Solutions

To enhance air traffic safety by reducing human-related factors from the perspective of the AGC procedure,based on the fourstep ATC communication procedure, a systematic solution is designed to implement the mentioned modules.The new solution is named as AISafe, i.e., artificial intelligence empowered safety enhancement system, which is a completely innovative AI-driven operation to monitor the spoken instructions in the ATC domain.The details of the new practice are described in this section.

2.1.Basic assumptions

As a new attempt of using AI techniques to enhance air traffic safety,the following assumptions are pre-designed to complete the concept design of the new solution.

(1) In view of safety incidents in the long civil aviation history, over 99.9% of spoken instructions are issued in a safe traffic situation19.As an early warning system,only the common instructions are monitored to detect potential risks in advance.Once encountering an exact emergency, more comprehensive procedures will be initiated to cope with the safety risks, which is not the primary goal of the proposed AISafe solution.

(2) As a safety–critical system, we believe that any possible symptom for safety incidents should be detected to avoid missing alarms.In other words, when designing the system functionalities and detection diagrams, the system prefers to provide warnings to the ATCO as much as possible in an over-safe criterion (i.e., generate an alert message with more temporal and spatial margins), which eliminates any possible missing alarm, but may cause unexpected false alarms.

(3) As a new practice,to validate its core functionalities,we prefer to design an independent system to reduce the coupling degree with the current ATC system, which is also able to speed up the deployment and integration of the prototype system.

(4) To confirm the new concept and solution, the data consistency between the AISafe solution and the current ATC system is a precondition.In our opinion, any failure caused by inconsistent information cannot be attributed to the new solution.In addition, the flow for data access is a key indicator to analyze the applicability of the proposed solution.

2.2.Preliminary techniques

In this section,a comprehensive review is provided to describe the previous works on ASR, SIU, and risk detection in the ATC domain, which benefits the understanding of the implementation of the whole solution.

2.2.1.Studies on ASR and SIU

The ASR can be traced back to the 1950s,and achieved desired performance by different technical frameworks, including the statistic models,hybrid deep learning models,end-to-end models, and sequence-to-sequence architecture20.In common fields, the SIU task is a core application of the natural language processing,including the intent detection and slot filling,which can be achieved by separate models and joint models using Conditional Random Forest (CRF) and neural architectures21.

In the ATC domain, the challenges, applications, and future research topics of the ASR research were comprehensively reviewed22–23.A challenge was held by the Airbus company in 2018, concerning the ASR task and the callsign detection in the ATC domain and over 20 teams joined this challenge and submitted their results24.Considering the realtime traffic dynamics, the contextual information (such as call signs,waypoints,etc.)was applied to improve the ASR performance25, which was also injected into the language model to improve the applicability of the ASR technique26.Since the ASR technique usually suffers from the small sample problems in the ATC domain, the semi-supervised training strategy was studied to improve the final performance27, and a knowledge extraction algorithm was also applied to obtain prior information to enhance the model ability28.A benchmark was conducted to achieve the ASR task in the ATC domain for English speeches29.

Regarding the multilingual ASR task in the ATC domain(i.e., Chinese and English), a unified framework was constructed based on a cascaded pipeline of acoustic model, and pronunciation model30.An end-to-end framework was designed to achieve the multilingual ASR task based on Chinese characters and English letters31or English sub-words30.The feature representation was also studied to enhance the ASR performance by providing discriminative input features32.The transfer learning scheme was presented to train the ASR model for different controlling centers3.

As the SIU task in the ATC domain, a comparative study was conducted to achieve the role identification task based on both acoustic features and text instructions33.The intent detection and slot filling were implemented by deep learning models6,34.The BERT was also studied to achieve speaker role identification35.The grammar of the text instructions was considered to achieve the SLU-related tasks for enhancing the ASR performance36.

2.2.2.Studies on ASR-based risk detection

As to the risk detection in the ATC domain, the ASR technique was regarded as a fundamental technique to enhance the safety and efficiency of air traffic operation6,37.To detect the instruction concerning the closed runway,the key ATC elements were also extracted by the ASR model to detect potential violations for Dulles International Airport38.The assistant-based ASR served as another sensor to obtain the exchange between the ATCO and aircrew about the deviation of the planned sequence, which supported the management of the flight arrival13.Similarly,the ASR model was proposed to confirm the aircrew readback39.The ASR technique was studied to maintain radar label information,which further reduced the ATCO workload40.The ASR technique was applied to achieve the ATCO training, which further enhanced the qualified ATCO for the real-time environment41.The operations of the national airspace system were characterized by extracting the situational context information from historical data using ATC models42.To our best knowledge, all the mentioned works focused on only a certain safety-related task in the ATC domain, and no systematic solution is presented until now, which is also the motivation of this work.

2.3.Designs of core applications

In practice, by analyzing the SOP of air traffic control, we believe that a total of four steps may eventually result in potential risks for traffic operations, which are also regarded as the primary functions of this new solution.The core applications of this new system are summarized in Table 1 by integrating the existing techniques and approaches12,43–44.

In addition to the mentioned applications, by investigating the potential safety risks from the daily work of the ATCO position, the following modules are also engaged to assist the ATC activities, and further improve the applicability of the AISafe solution.

(1) Runway Incursion Alert (RIA): considering that the runway-related resources are limited in airport surface,the RIA is to detect resource conflicts for aerodrome tower control based on the spoken instruction, before performing certain instructions.

(2) Similar Call-sign Alert (SCA): considering that the flights with similar call-signs in the same sector are the potential cause for incorrect readback response (such as CCA9747 and CCA9745), the SCA module is designed to notify the ATCO to focus on the readback confirmation from the corresponding aircrew once a spoken readback instruction is received.

(3) Speech Rate Measurement (SRM): as to the spoken instruction during the AGC procedure,considering that a higher speech rate may impose unnecessary burdens for human understanding, the purpose of the SRM module is to evaluate the speech rate for each spoken instruction, which is further expected to subconsciously force the ATCO to adjust their speech rate in a proper manner.

Table 1 Description of core applications in proposed solution.

(4) Scene Data Storage (SDS) and Post-Scene Replay(PSR): the two modules are designed to record scene data and replay ATC scenes to support the post-data analysis,respectively.They can also serve as replay tools for the quality evaluation of the ATC working.

2.4.System architecture

Based on core techniques and applications, the proposed new solution is developed into a prototype(software system)to validate its applicability.The architecture of the new system is illustrated in Fig.3,which is generally categorized into the following two pipelines:

(1) Offline training: in the offline training pipeline, the core idea is to produce data-driven models that can be applied to create and capture values (human intents and traffic dynamics)from real-time multi-source traffic data for improving air traffic safety.The raw data is firstly collected from current ATC systems,which is further produced for the training samples.In succession,the data-driven (specifically deep neural network) models are constructed to achieve the core techniques,which are trained on the built dataset.Finally, the optimal models are saved to be deployed for supporting the online application.Based on the mechanism of generating the AGC speech, a total of 6 technical specificities should be considered to achieve the ASR technique in the ATC domain, including the background noise,multilingual, unstable speech rate, ATC terminologies,sample scarcity,and context-depedent entities45.Considering the requirements of the ATC procedure,the task of the SIU module concerns the speaker role identification (ATCO or aircrew), intent detection, and slot filling21,23.The core techniques of the proposed solution have been achieved in our previous work6,32,45–46, which serve as the foundation in this work.Readers who are interested in our techniques can review the detailed implementations in the mentioned references, including the ASR specificities, AIempowered deep learning models, detection diagram,etc.

Fig.3 Architecture of proposed prototype system.

(2) Online application: in the online application pipeline,the primary purpose is to design the whole risk detection diagram and implement safety monitoring applications to further assist the ATC work based on core datadriven techniques47.The system architecture is illustrated as two parts: the current ATC system and the new AISafe system, in which the current ATC system provides the required information to the AISafe system,while the AISafe system contributes the alert message to the current ATC system.In addition to the aforementioned modules (Section 2.3), the SIU module is designed to extract required ATC elements from the text instructions generated by the ASR model, which is also implemented by data-driven approaches.Furthermore,the Multi-source Information Processing (MIP) and Traffic Situation Prediction (TSP) modules are also designed to obtain significant real-time traffic dynamics by referring to Refs.43,48–50.Finally, the whole modules are combined with the core techniques to formulate the whole prototype system to serve as an entity to validate the proposed solution.

2.5.Practice sites

After the prototype of the AISafe solution was developed, it was deployed in various controlling centers in China, such as Chengdu, Changsha, Taiyuan, Chongqing, Lijiang, etc.The system concerns different flight phases, including aerodrome tower (TWR), approach (APP), and Area Control Center(ACC).The AISafe system serves as a support system,working with the current ATC system collaboratively.Taking Chengdu ACC as an example, a total of 25 working positions, i.e., 25 airspace sectors, are equipped with this system.In each day,up to 6000 flights are monitored with over 100,000 spoken ATC instructions.Currently, the new system is confirming the possibility of regarding as an SOP to check the operational safety of air traffic.

Based on real-world deployed systems in several practice sites, the effectiveness and efficacy of the proposed solution and prototype are evaluated to identify the potential risks by the case study and the evaluation of user experience.The potential risks are expected to be managed by addressing the underlying causes of the proposed solution and the ATC management system, such as concepts, techniques, and measures.

3.Case study

In this section,the Chengdu ACC is selected as a case to evaluate the system performance since it is the first representative project of the new system.Considering the proposed solution,the following core functionalities and corresponding objective metrics are applied to detect new findings of the proposed AISafe system.

(1) As the core AI-driven operation, the ASR technique is applied to obtain real-time traffic dynamics, which is the fundamental of the proposed solution.Therefore, the performance of the ASR technique is first evaluated in terms of Character Error Rate (CER)27.

in which I, D and S save the required number of insert,deletion, and substitution operations that convert the predicted sentence into the ground truth.LoG denotes the length of the ground truth.Note that the character refers to Chinese characters and English letters due to the nature of the ATC speech in the multilingual language.30

(2) Similarly, the performance of the SIU module is also considered in terms of the Accuracy of the Intent Detection(AID), the Accuracy of Role Identification (ARI), and the F1 score of the slot filling.The metrics are illustrated as follows:

In the upper equations,N saves the number of test samples.The operator a=b is 1 if a is equal to b,otherwise it is 0.The IDpand IDgare the predicted and ground truth for the label of the intent detection task,and the ARI has a same meaning.As to the F1 score, the detailed descriptions of the notations can be found in Ref.41.

(3)The warning applications are evaluated in terms of precision.As shown in Eq.(5),the TP and FP denote the number of the true and false samples in all detected warnings, respectively51.The false alarm rate is measured by 1-Precision.

3.1.Case statistics

As mentioned before, the raw data items for the case study were collected from a real-world industrial AISafe system deployed in Chengdu ACC.The data period is from 1 Sep.2020, to 30 Nov.2020, i.e., about a total of three months.The reason why we select this period as the case is that a comprehensive system upgrade was performed on 7 Oct.2020 based on the user feedback.Originally, the AISafe system was deployed in Chengdu ACC from 02 Jul.2020, which is operated to collect the raw traffic data, including ATC speeches, flight trajectories, flight plans, etc.The primary system changes in this upgrade are summarized below:

(1) AI-driven models:the proposed system allows us to collect raw data from a real ATC environment to enhance the AI-driven models.In this upgrade, the collected ATC speeches were annotated to achieve the model enhancement (i.e., ASR and SIU model) using training samples in the same scheduled ATC season.Meanwhile,some special strategies are also applied to improve the SIU performance, such as the keyword matching for the query-related intents.

(2) The traffic prediction application is updated to consider the custom flight behaviors in Chengdu ACC,for example, using a localized climb rate in different sectors to predict the altitude changes.

(3) The risk detection diagram comprehensively undergoes a systematic test and is updated to consider the commonly used settings for the ATCOs in Chengdu ACC,including the intents to be confirmed for the NIC application, the whitelist and blacklist for the SCA application, and dedicated data storage strategies for incidents.

(4) Configurable parameters for risk detection are updated to accommodate the ATCO expectation by testing the original configurations, concerning the allowed readback delay for the ARC application, the horizontal and vertical interval for the PCD application,etc.

By taking this period as the case, we can consider the system performance before and after the system upgrade.To obtain the actual case statistics, we are requested to complete the following pre-works:

(1) Manually annotate the text sequence for each speech utterance to evaluate the ASR performance, concerning different sectors, languages, and special terminologies about the ATC rules.

(2) Manually annotate the intent and ATC-related parameters of the spoken instruction to evaluate the SIU performance,in which the samples are the same as those in the ASR evaluation.

(3) Replay the real-time ATC working scenes(by using SDS and PSR) to confirm the effectiveness of the raised typical warnings.

(4) Analyze the underlying causes for incorrect warnings to provide insights to identify and address the potential risks,such as instruction segmentation,traffic dynamics,risk detection diagram, and other system-level factors.

Since confirming the effectiveness of the case statistics is highly expert-dependent and requires heavy human-resource cost, we apply the sampling strategy to obtain the final statistics from a certain day52.Specifically, in the following case study, every Wednesday is selected as the case for each week to analyze the overall performance since it is the day with the highest flight density in Chengdu ACC.The number of flights and the number of utterances for each day are considered to illustrate the representability of the data statistics on Wednesday.

3.1.1.ASR-related statistics

As a fundamental technique of this new practice,we first evaluate the ASR performance in terms of the mentioned CER measurement.As shown in Fig.4, the results are grouped by different dates, and both the total utterance and duration of the collected speeches are reported.As to the CER metrics,based on the multilingual facts of the ATC speech in China,they are divided into Chinese and English to illustrate the overall performance of the data-driven ASR model on the AGC speeches.

It can be seen that the total utterance and duration of ATC speech in Fig.4(a)fluctuate to a narrow extent during the consideration period, while the CER metric achieves a slight promotion in a steady manner after the system upgrade.Based on the CER measurements in Fig.4(b), the highest accuracy for Chinese and English speeches reach over 97% and 96%,respectively, which can be regarded as a superior performance than the ability of human beings (i.e., 95%)53.

3.1.2.SLU-related statistics

Similar to the ASR model, the SLU model is to convert the text instructions into ATC-related structured data to support the following risk detection applications.Therefore, the SLU performance is also regarded as a part of the case study to illustrate the AI-driven operations, in terms of the AID,ARI, and F1 score measurements.

Fig.4 Case statistics of ASR technique.

Fig.5 Case statistics of SLU-related techniques.

As shown in Fig.5, we can see that the SLU-related tasks are achieved with considerable high performance, i.e., over 95%AID for the intent detection task,95.5%ARI for speaker role identification task, and about 96 F1 score for the slot filling task.In addition, the underlying causes of the incorrect recognition of the SLU tasks are also considered to enlighten future research topics.Based on our analysis, we found that the primary incorrect recognitions are caused by the ASR errors since the SLU module is cascaded to the ASR module,which results in confused intent for the text instruction and further impacts the performance of slot filling task.Meanwhile, we also consider the performance of the keywordmatching-based intent detection strategy, and we found that keyword matching is also a feasible tool to cope with the rarely seen samples (such as go-around, mayday, etc.) except for learning-based SLU models.The results indicate that a combination of the rule-based and learning-based methods can achieve higher performance for the SLU tasks in the ATC domain due to the issue of sample scarcity.

3.1.3.Warning-related statistics

Fig.6 Case statistics of ARC and PCD applications.

To validate the applicability of the AISafe system, among the designed functions, the most critical ones, including the ARC and PCD, are selected as the cases to illustrate the warning performance, in terms of precision measurement.As shown in Fig.6, both the number of warnings and corresponding Warning Precision (WP) are reported to illustrate the overall performance of the risk detection.From the results, we can see that the proposed solution is able to detect possible risks from the perspective of the spoken instruction in advance,which further reduces the ATCO workload of confirming readback instructions.To be specific, the number of warnings reported by the proposed prototype decreases in this period using the updated rules and configurable parameters requested by front-line users, while the warning precision increases after the system upgrade.

3.2.ARC analysis

As the AI-driven operations,the ARC is the most critical part of the AGC procedure,which is also the core application from the perspective of speech communication.In this section, we further analyze the underlying reasons for the incorrect ARC warnings (i.e., false alarms).Based on the ATC rules and the system pipeline, the reasons are categorized into the following types:

(1) Speech-related factors: including poor speech quality,error segmentation for spoken instructions, etc.

(2) ASR-related factors: including ASR errors.

(3) SIU-related factors: including speaker role, intent, and slot filling errors.

(4) Warning detection factors: including improper warning rules, calculation steps, risk detection diagram, etc.

(5) Other factors:including the information error,information delay, or other system malfunctions.The final results are reported in Fig.7,in which the vertical dimension denotes the number of warnings for each day corresponding to the ARC statistics in Fig.6.The total number of warnings in Fig.7 is identical to the number of incorrect warnings in Fig.6,which can be obtained by multiplying the warning precision and the total number of warnings.As can be seen from the results, the AI-driven techniques (ASR and SIU) are the main contributors to the false alarm for the ARC warnings, which also indicates the research topic and technical issues to be addressed in the future.

3.3.Case discussion

Fig.7 Number of contributors for incorrect ARC warning.

As can be seen from the case statistics, the proposed AISafe system achieves the data-driven tasks with considerable high confidence and the corresponding warnings (considering realtime ATC dynamics) are able to improve the operational safety by detecting possible safety risks in advance,i.e.,providing more pre-warning time to the ATCO for coping with the emergency.However,based on the case study,we also identify the following potential risks:

Potential risk 1.Considering the technical framework, the errors caused by AI-driven techniques can be identified as the dominant contributors to the system failure since it is the foundation of the proposed AISafe solution.Although the recognition accuracy is superior to human ability, the 3%CER in up to 95,000 utterances will inevitably reduce the robustness of the following ATC-related applications, which is unfavorable to the safety warning systems in the ATC domain.

Potential risk 2.By analyzing the precision of selective warnings, it can be concluded that the current performance of warning applications is still required to be further improved to enhance the practicability of the proposed solution.Meanwhile, the incorrect warnings may be a potential factor affecting the daily work of the ATCO by introducing a new prototype system into the ATC procedure.

After clarifying the factors for incorrect warnings, we further collect data items that provide insights for improving the AI-driven approaches (i.e., including the ASR and SIU technique), as described below:

(1) A performance paradox is encountered by analyzing the ASR-related false alarm.In general, the overall ASR accuracy for Chinese speeches is higher than that of English speeches.However, ASR errors for Chinese instructions probably result in more incorrect warnings due to the following facts:

Fact 1 (high speech rate).Since Chinese is the native language for most ATCOs in China, they tend to speak the Chinese instructions at a higher speech rate45.Meanwhile, as it is well-known that Chinese is a monosyllable language, a higher speech rate can easily lead to the overlapped phonemes for continuous words, which imposes extra burdens to recognize the speeches with the higher rate.

Fact 2 (digit recognition).As a basic descriptor in the ATC domain, the digits are widely applied to denote the aircraft identification, speed, altitude, heading, radio frequency, etc.For Chinese digits, the ASR outputs can be confused by near-phonetic words (such as ‘‘liu and jiu”), and the language model will also fail to correct the results since the distribution of digits is almost uniform in the training corpus.On the contrary, the English letters for English speeches are able to provide valuable contexts (i.e., words) to improve the ASR performance due to its polysyllable essence.

Fact 3(ASR accuracy).Considering that the incorrectly recognized digits are the main contributors of the ASR errors,we also compare their performance (accuracy of recognizing the digits)with the overall ASR accuracy.Surprisingly,the results demonstrate that the performance of recognizing digits is at the same level as the overall performance.In other words,instead of the low ASR accuracy, the incorrectly recognized digits are regarded as the main factor of the false alarm only due to their widespread usage.This is a kind of survivorship bias or survival bias54, i.e., evaluate the ASR performance by concentrating on errors from the reported warnings.

(2) A certain percentage of incorrect warnings can be attributed to the errors of the speaker role identification in the SIU module.In current practice,speaker role identification is achieved by learning the text format of the ATCO or aircrew instruction36.Once the certain instruction format is broken or in an emergency, the current approach may be failed in this situation.In succession,as the starting point of the AGC procedure, the risk detection is performed on an incorrect speaker role,which further causes false alarms.

(3) By further reviewing other factors for the false alarms,the speech quality deserves to be improved to provide high-quality inputs for the AI-driven operations.In addition, we found that the information consistency between the current ATC system and the AISafe system is also a major obstacle due to the delay of the information transmission.

In the next section, we will focus on identifying the potential risks of the new solution, from the AI-driven techniques,engineering facets to ATC procedure, which is expected to indicate the research topics for future improvements of the proposed system.

4.User experience evaluation

In this section, based on the aforementioned case, a comprehensive study is further conducted to consider the actual user experience, mainly concerning the influence of system failures on the ATC daily work,which is particularly important to validate the applicability, the robustness, the influence on procedure or training, etc.The evaluation of user experience is performed in two steps: an online questionnaire and a faceto-face interview.

4.1.Online questionnaire

4.1.1.Questionnaire design

To investigate the user experience of the proposed new solution,a questionnaire is firstly required to present certain items to the respondents with standardized answers.Focusing on the risks in the case study, the management of the technical risks for AI-driven innovations is considered in this work, mainly concerning the technical performance and technology use55–56.The detailed contents of the questionnaire are listed in Table 2.

It is noted that the runway incursion also serves as an item in the questionnaire,denoting a sub-category of the flight conflict on the airport surface for the TWR phase.Meanwhile,since the workload of the ATCOs is also objectively caused by traffic situation(usually high flight intensity or peak hours),the ‘‘Workload”and ‘‘Traffic situation”are combined to present a unified question to respondents.In this way, the questions in the questionnaire for ‘‘Safety-critical impacts in ATC”are divided into two categories: (A) AGC-related questions:the‘‘Air-ground communication”in Table 2;(B)trafficrelated questions: the ‘‘Workload”and ‘‘Traffic situation”in Table 2.

Except for the mentioned items, in each category, the open-ended issues are also allowed to collect extra insightful comments from respondents57.Except for the basic information, all other items are evaluated by the five score degrees to generate the final measurements.Note that a prepreprocessing procedure is performed to unify the inherent meanings from score 1 (negative feedback) to score 5 (positive feedback)58.

After completing the initial design of the questionnaire, we first conduct an online pre-investigation to collect suggestions to formulate a formal questionnaire.In this procedure, only selective experienced users are invited to complete the investigation questionnaire.A total of 16 professional ATCOs, covering different flight phases and controlling centers, are invited to review the questionnaire and provide insightful comments.A three-round revision is performed to determine the final questionnaire, as shown below:

(1) We design the initial questionnaire and invite selective front-line users to provide their comments on the design of the questionnaire.

(2) We revise the questionnaire based on the comments of each invitee independently and return to them to confirm the revision.

(3) We invite all the selective front-line users to attend an online meeting to determine the final items of the questionnaire, aiming to cross-validate the comments from different invitees in a concurrent and collaborative manner.

To conduct an extensive investigation,front-line users from different departments(flight phases including TWR,APP,and ACC), and areas (Chengdu, Changsha, and so on) are invited to perform the questionnaire to improve the diversity of collected raw data and contribute their actual experience in an anonymous manner.Based on current application sites,a total of 194 respondents from 10 controlling centers are invited to complete the questionnaire.All the respondents are requested to complete all the items of the designed questionnaire,and the open-end issues are highly appreciated to improve the diversity of the questionnaire.

4.1.2.Discussion on questionnaire results

(1) Basic information

Among the designed items, based on the consultant with invited ATCOs, the following items are reported to illustrate the coverage of the questionnaire and validate its effectiveness to contribute further key insights for ATC safety, as shown in Fig.8.From the reported results,we can come to the following summaries:

(A) As shown in Fig.8(a),the respondents come from different areas of our representative deployment sites, i.e.,toward no particular biases to the user area, which is able to illustrate the overall performance of the proposed solution and its prototype.It is noted that the Chengdu, Shuangliu, and Tianfu in the reported results denote the flight phase of ACC, TWR, and APP,respectively.

Table 2 Design of questionnaire.

(B) Based on the basic information for the daily role in Fig.8(b), nearly 90% of all the respondents are frontline users of air traffic systems(i.e.,ATCOs and supervisors), which are expected to contribute actual evaluations of the proposed solution and prototype.

(C) As shown in Fig.8(c), the respondents devote the ATC work for different flight phases (TWR, APP,and ACC), which validates the applicability of the proposed solution to the whole procedure of the flight execution.

Fig.8 Selective results for basic information.

(D) Based on the positional level in Fig.8(d) and working years in Fig.8(e), we can see that the respondents are mainly distributed at a sophisticated level, which indicates that the questionnaires imply the professional viewpoints from the system users.

In addition, about 81.4% of respondents believe that they are familiar with the functionalities and can make a comprehensive evaluation of the new solution and the prototype.To further validate the questionnaire, a total of 19 items of the questionnaire results measured by the five-degree approach are analyzed to confirm its reliability and validity, in the following which is twofold:

(A) For the reliability analysis, Cronbach’s alpha reliability is 0.876, which can be regarded as a high-reliability and internal consistency between 0.70 and 0.9559for evaluating a new innovative practice.

(B) As to the validity analysis, the indicator of the Kaiser-Meyer-Olkin (KMO) test is 0.838, which is meritorious for factor analysis60.In addition, the results for Bartlett’s Test of Sphericity are 2045.051 (Approximate Chi-Square), 171 (Degree of Freedom), and 0(Significance).

In summary, based on the questionnaire results about the basic information, it can be concluded that the questionnaire results are capable of representing the whole evaluation of the proposed solution, which indicates that we can come to the conclusions from the questionnaire to provide key insights for further technical improvements.

(2) Safety-critical impacts in ATC

To validate the concept design of the proposed solution,we investigate the most critical impacts of the ATC work on the operational safety of air traffic, which facilitates the formulation of the proposed solution and prototype in this work.In the questionnaire, each interviewee can select up to 4 items in this category.As reported in Fig.9,a total of two categories,including the traffic-operation- and speech-communicationrelated issues, are investigated in details to consider the dedicated specificities of the ATC work, mainly focusing on operational safety.It can be seen that four indicators are regarded as primary safety-critical impacts for each category, which should be considered to improve operational safety in a proper manner.

For traffic operation, the objective traffic-related issues,including bad weather, flight intensity, and flight conflict, are regarded as primary contributors to impact the operational safety of air traffic.Similarly, the subjective human-related issues,concerning the continuous working hours and the practices for ATCO trainees,are regarded as the influential factors for traffic safety.For the bad weather and flight conflict, the proposed solution creates traffic prediction and risk detection tools to assist the ATCO work, which aims to provide more prewarning time by detecting potential risks in advance.As to the flight intensity and human-related issues, we design the safety-enhance applications from the perspective of airground communication, such as the readback confirmation,which can greatly reduce ATCO workloads by automatically monitoring the traffic situation.

Fig.9 Results for safety–critical impacts in ATC.

For speech communication,the readback error,radio communication jamming, instruction compliance, and language type are selected as the primary factors for traffic safety.As the core technique of this solution,the corresponding modules and functions are designed to address the mentioned issues,such as the NIC,ARC,and CIB.For the language issue,the SIU module can convert English spoken instruction into pre-defined ATC elements in the form of letters and numbers,which eliminates the misunderstandings caused by language issues for international flights.

In summary, the primary factors selected by respondents shed light on the requirements of a safety monitoring system in the ATC context.Fortunately, the corresponding functions are designed in the proposed solution and prototype to address the mentioned issues,which validate the motivation of the new solution, further improve the traffic safety and reduce the ATCO workload.

(3) Influence of ASR technique on ATC safety

In this section, we mainly report the evaluation results for the core functionalities of the proposed solution, which can be generally divided into the following three categories:

(A) As shown in Fig.10, all the top-3 popular applications(each interview can select up to 4 applications) are derived from the core AI-driven techniques in this work(ASR and SIU).The ARC and SCA are ATC-related applications based on the ASR and SIU technique,which benefits the reduction of the ATCO workload by monitoring the readback errors and misunderstandings caused by similar call-signs.

Fig.10 Popular applications of the proposed solution.

(B) As to the evaluations for the designed risk detection,94.3%of respondents believe that it is credible to regard the raised warnings from the proposed prototype as a key safety indicator of the traffic operation.About 93.8%of respondents declare to handle the raised warnings in a prompt manner in their daily work, in which over 78.0% of them choose to confirm the safety of the traffic operation as soon as an alert message is provided.

(C) For the safety enhancement for air traffic, 92.3% of respondents witness the safety enhancement of the traffic operation after the proposed prototype is deployed.To be specific, 93.8% of them reveal that the proposed solution benefits the issuing of the normative ATC instructions by using AI-driven techniques.Meanwhile,95.9%of them demonstrate that the SRM module plays a positive role in forcing the ATCO to adjust their speech rate of the spoken instructions.

As shown in Fig.11, the results of 6 selective applications for the risk detection are reported as a boxplot.In the boxplot,the whole distributions of the data items are depicted as follows: the upper and lower quarter values are formulated as a box; the median value for each category is denoted by the dashed line; the mean values are presented as the square dots.The upper and lower boundaries are also illustrated in the figure,as well as outliners.It can be seen that most scores for all applications are densely distributed at score levels 3 and 4,indicating a positive review of the application performance.In addition, score 1 is regarded as outliners, which means that only a few samples are in the resulting scores.Meanwhile,we can also see that the median value and the lower quarter value are exactly overlapped, indicating that the third score level is the dominant selection in the results.

Fig.11 Score distribution of core applications in proposed solution.

(D) We investigate the influence on the workload by working with a new assistive system in the ATC procedure.Unexpectedly,about 60.3%of respondents declare that they are plagued by extra workloads to confirm the raised warnings.Only 17% of them reveal a workload reduction when working with this new system, while the remaining 22.7% keep a neutral viewpoint about the workload.

(E) It can be seen from the results that 92.8% of respondents give positive feedback, and 86.6% of them would like to recommend the new solution and prototype to their colleagues (ATCO, safety, etc.).This result indicates that although the ATCOs may experience extra workloads, they are still willing to contribute this positive feedback as long as the proposed prototype can detect potential risks to assist the ATC work,which confirms the fact that safety is a one-vote veto factor in ATC domain.

In summary, the proposed solution contributes following advantages to the ATC procedure, i.e., improve safety by detecting potential risks from the speech communication;reduce the workload by confirming the readback instructions;enhance the ATC procedure by forcing the ATCO to issue normative instructions in a proper speech rate.However,an obvious potential risk can be identified from the questionnaire results:

Potential risk 3.Although the proposed solution is capable of reducing the ATCO workload by monitoring the speech communication, the ATCOs suffer from extra workloads to confirm the raised warnings with the new assistive system.This fact results in two adverse aspects: distract the attention of ATCO (further impact the flight safety) and discourage the user interest in this new practice.

Following the identified risk,we also consider the key metrics of warning systems in the ATC domain to evaluate the user tolerance to the system failures,i.e.,false alarm and missing alarm rate.The questionnaire results demonstrate that most respondents (over 70%) believe that only less than 20% of a false alarm- and missing alarm-rate are acceptable for their daily work, which requires a rigorous tradeoff between false alarms and missing alarms in the future design.

4.2.Face-to-face interview

After completing the formal investigation,we invite 19 ATCOs from Chengdu ACC to conduct a face-to-face interview to consider the identified potential risks from the case study and questionnaire.Firstly, the interview topics lie in the core evaluation items from the questionnaire, and the results are shown in Table 3.As can be seen from the result comparison,the face-to-face interview basically obtains a consistent evaluation with that obtained by questionnaire,concerning the feedback and suggestions.

Table 3 Comparison between questionnaire and face-to-face interview.

In addition, we also focus on cross-validating the impacts possibly affecting the performance of the new practice,as summarized in Table 4.The responses from the interviewees are summarized below:

(1) 15 of the interviewees confirm the difference of speech quality,and the rest 4 interviewees claim no obvious difference.They also recognize that it may be a practical way to improve the system performance since the speech is the input of the proposed prototype.

(2) All the 19 interviewees verify this point, and some of them admit that the SRM can benefit the speech rate adjustment.

Fact 4(Contextual awareness).Although the ATCOs recognize the speech quality and the rate as an influential factor for ASR performance (especially for Chinese instructions), they still believe that the communication between the ATCO and aircrew will not be affected by the two factors.The ATCOs think that the contextual awareness from the traffic dynamics enhances the exchange and understanding between the ATCOs and aircrews during the ATC procedure.

For example,even though the high speech rate may lead to ignoring the overlapped words (such as ‘‘san san”->‘‘san”in Chinese), the participant (either the ATCO or the aircrew)can understand each other in a task-oriented ATC dialog environment.As shown in Fig.8, the speech rate is only the 6th factor selected by the respondents for impacting air traffic safety.

Table 4 Open-ended issues for face-to-face interview.

(3) 17 of the interviewees believe that the incorrectly recognized digits finally result in the false alarm,which should be properly addressed to improve the applicability of the proposed solution in the future.

(4) For incorrect speaker role identification, 14 of the invited interviewees frankly confirm that this issue cannot only depend on the system improvement.The normative instructions should also be ensured to eliminate human errors in the ATC procedure.

(5) About 11 interviewees suffer from the extra workloads caused by the new practices (9 for slight burden and 2 for heavy), and three of them are concerned about the negative impacts on traffic safety.

(6) Fortunately,no actual alarm is ignored by the proposed solution (i.e., no missing alarm until now) until now,which also supports our assumptions to detect the potential risks in an over-safe manner.

In summary, we yield positive feedback about the overall evaluation of the system from the face-to-face interview, and underlying impacts analyzed from our case study and user experience evaluation are validated by interviewees, including all the core techniques, applications, and related management system, which are expected to provide key insights for further system improvements.

5.Discussion

5.1.Insights

In this work, the available data storage in the ATC systems enables us to develop AI-driven innovations to enhance traffic operation.To be specific, the ASR and SIU techniques are innovatively proposed to obtain real-time traffic dynamics,which further serves as a powerful means to support subsequent ATC-related applications.Based on the case study and the evaluation of user experience, we can draw the following conclusions:

(1) Based on the core techniques, the proposed solution is promising to improve the operational safety and efficiency of air traffic, and also reduce the ATCO workload to a certain extent.The proposed solution has the ability to automatically detect potential safety risks(such as readback error, tactical conflict, etc.) before performing the instructions, thereby providing more prewarning time for ATCOs to deal with possible emergencies in the ATC work.Fig.12 illustrates some selective examples of the detected warnings reported by the proposed solution.

(2) The case study and the evaluation of user experience provide compatible insights, and both of them validate the effectiveness and efficiency of the proposed solution.Based on the results of the questionnaire, the safetycritical impacts (selected by interviewees) also support the motivation of our design for system functionalities,corresponding to different applications of the risk detection diagram.

However, any technological innovation has pros and cons.Based on the case study and subjective evaluation, a total of three potential risks are identified and several major causes are also analyzed from cases and collected from interviewees.Specifically, we can conclude that the primary risk chain of the proposed solution can be represented as: failures of AIdriven techniques->false alarms of ATC warnings->extra workload for the ATCO->potential human risks for air traffic operation.The risk chain can be regarded as the most important basis to conduct future research for system improvements.

As a safety-critical management system, in the ATC domain, it is not recommended to address the existing issues at the expense of bringing new problems,unless its extra effects can be controlled properly.Considering that the ATC work is a typical HITL procedure,the most urgent task is to reduce the negative impact of false alarms on human ATCOs.Since the proposed solution is a multi-step pipeline, it is better to improve the performance of each module/step to further eliminate cascaded errors of the whole system.Certainly,we prefer to focus on addressing the performance issues of the pipeline as front as possible (i.e., AI-driven ASR techniques), so that the downstream applications are not affected by failures of upstream tasks.Only in this way can the proposed solution achieve the required tasks in higher confidence, and the false alarms are expected to be reduced to further assist the ATCO work in an efficient manner.

5.2.Improvements

In this section, we first consider the items in the ‘‘further improvements”of the questionnaire results to explore the possible measures from front-line users.As shown in Fig.13, the detailed explanations for the ASR, RFA, IUI, EPA, PDE,VSI, IAS and IAT are improving the ASR accuracy, reducing the false alarm,improving user interface,enhancing post analysis, providing data export tools, voice speaker identification,integration with ATC system and integration with ATCO training, respectively.It can be seen that the dominant demands from front-line users are to improve the ASR performance and reduce the false alarm rate,which indicates that the obtained summaries from system evaluation are consistent with the user requirements.In addition, the analysis of underlying causes for system failures can also provide insightful trails to the scheme of the system improvements.

5.2.1.Technical improvements

Based on the aforementioned discussions (potential risks and technical facts), as the foundation of AI-driven innovations,improving the ASR accuracy is an effective way to reduce the false alarm.In our future works, the following technical strategies are expected to be applied to the ASR study to further improve the ASR accuracy.

Fig.12 Selective examples for improving the traffic safety reported by the proposed prototype.(Currently,since it is a prototype for the ATC department in China, the characters for human–machine interface in the figures are in Chinese.).

Fig.13 Further improvements selected by interviewees.

(1) Speech quality

As the input of the new practice,a high-quality ATC speech is of great significance to improve the ASR performance,which can be achieved in the following ways:

(A) To address issues in Fact 4: Based on the face-to-face interview, the quality of the ATC speech received by the proposed solution is obviously inferior to that heard by ATCO.Currently, the primary concern of introducing the speech signal from ATC communication equipment is that the connection hardware cable possibly results in the failure of the electronic equipment,thereby affecting speech communication and air traffic safety.The most straightforward way to match the speech quality for the proposed prototype is to adjust the routing scheme of the speech signal, which is expected to enhance its quality and improve the final ASR performance.An intuitive approach is to utilize the software transmission, e.g., developing network sockets to transmit the speech from the equipment to the proposed solution.

Attempts and exploration.We attempt to collect the ATC speech directly from communication equipment (called clean CQ speech), which is further applied to test the ASR performance.A total of 32 min speeches are collected from the 2nd airspace sector in Chengdu ACC on 13 Dec.2021.A preliminary result is that the CQ speech can slightly enhance the ASR performance,from 3.29%CER to 3.05%.More comprehensive experiments should be considered to validate the effectiveness and efficacy of the new scheme.

(B) To address issues in Fact 4: The speech enhancement techniques should be considered to remove the noise background of the ATC speech, which improves the quality and intelligibility of the ATC speech.To this end, the noise model of the ATC environment is the key to separating the clear ATC speech.However, it is usually intractable in the ATC domain due to various domain specificities45,23.

Attempts and exploration.A deep learning based speech enhancement model is studied to improve the speech quality for better ASR performance.Currently, on a 5 h test set(2nd airspace sector in Chengdu ACC), we yield an improvement of the PESQ score from 1.9 to 3.5(the higher the better,and 3.8 is the accept score for telephone communication)45.The CER obtained by the proposed ASR model also harvests a prominent improvement,from 5.40%to 3.96%.We will further evaluate this scheme and report more quantitative results by scientific papers.

(C) To address issues in Fact 1:The ATCOs are requested to issue their spoken instructions at a normal speech rate to improve the speech intelligibility,not only for improving the system performance but also for eliminating the misunderstanding between the ATCO and aircrew.

Attempts and exploration.By referring to our previous research45, extensive experiments are conducted to measure the speech rate corresponding to the best ASR performance.The results demonstrate that about 3.9 words per second(w/s) is a preferred speech rate for both human understanding and the ASR model recognition.This means that the ATCOs are requested to reduce their speech rate from 5.7 w/s to 3.9 w/s.

(2) Digit recognition

To address issues in Fact 2: As the widely used digits, the aircraft identification, speed, altitude, frequency, or other ATC-related resources can be accessed by multi-source contextual information.Therefore, the contextual-awareness characteristic of the ATC procedure should be considered to improve the ASR performance by correcting the situational ATC elements.

Attempts and exploration.A preliminary attempt to enhance the recognition of the aircraft identification is reported in Ref.26.We will further investigate the approaches to integrate more contextual information into the ASR model.

Note that this strategy is a double-edged sword for a realtime safety–critical monitoring system.Although this strategy has the ability to improve the ASR performance to greatly reduce the false alarm and further relieve the extra ATCO workload caused by the new prototype, it is also possible to miss some actual potential risks by correcting the real speech instructions using the contextual ATC information.

For instance, if an actual incorrect readback instruction is corrected by the ASR technique using contextual knowledge,the extract warning message will be ignored, which may further result in real incidents to drop the safety level of air traffic.Therefore, the tradeoff between the false alarm and missing alarm is always a major issue for the safety monitoring system,which also supports the motivation of the design of the questionnaire about the acceptable false alarm and missing alarm rate.

(3) Overall ASR performance

To address issues in Fact 3: Considering the overall ASR performance, we will attempt to enhance the AI-driven techniques since a CER of 97%is not qualified to achieve the tasks in a safety–critical ATC system.We decide to explore advanced architectures, models, and frameworks to reimplement the ASR task in the ATC domain.By referring to the current proposed solution, we can improve the ASR technique in a targeted manner to enhance the applicability in the ATC domain.For an AI-driven innovation in the ATC domain, a long upgrade interval may be required to validate any new improvement about the technical framework.

Attempts and exploration.We have introduced the selfsupervised learning mechanism and transformer architecture to implement the ASR technique in the ATC domain, and we will report the related results in our future works.Based on the ASR study in common applications61, the selfsupervised learning mechanism is capable of utilizing the unlabeled data to enhance the ASR performance.Once the selfsupervised learning mechanism is practically applied to enhance the ASR technique in the ATC domain, it is believed that the AI-driven and big-data-driven innovations are really promising to address the existing issues in the ATC industry,without causing extra workloads.

(4) Speaker role identification

To address issues in Potential risk 1: Although the ICAO requests the instruction format for both ATCO and aircrew,in practice, the speaker usually breaks the rules in an emergency, which will confuse text-dependent role recognition approaches and further invalidate the subsequent tasks.In the future, we will explore the speaker verification approach to determine the speaker role from the acoustic features.The speaker verification is not only able to identify the speaker role(ATCO or aircrew), but also able to determine the human identity for certain ATCOs, which supports the quality evaluation of ATCO work, information retrieval for a certain ATCO.In common areas, speaker verification is widely achieved by data-driven approaches,which is expected to facilitate anther AI-driven innovations in the ATC domain.

Attempts and exploration.Currently, we are collecting the raw speech data for certain individuals to implement datadriven speaker verification models.It can be predicted that the speaker verification is promising to address the issue of role identification by acoustic features, even though the ASR outputs an incorrect aircraft identification.

5.2.2.ATC-related measures

In addition to reducing the false alarm by improving the ASR accuracy, other false alarms also deserve to be relieved by proposing a proper risk detection diagram.In the future, the following measures are considered to improve the applicability of the proposed solution, from both the techniques and management systems.

(A) To address issues in Potential risk 3: As a new practice,the ATCO may be influenced or distracted by any unseen behavior,such as the display of the ASR results,visual and auditory presentations of the raised warnings.Based on the user experience, it is highly recommended to integrate the AISafe solution into current ATCO training systems, which helps to improve the familiarity of ATCO on the new prototype.The ATCO can be trained to cope with any malfunction to reduce the negative impacts of the ATCO’s attention.

Attempts and exploration.Related studies to achieve the ATCO training are reported in Refs.2,41.The proposed solution can be integrated into the ATCO training system via simple adaption.After this measure is evaluated and approved by concerned departments, it can be performed to enhance the applicability of the proposed solution.

(B) To address issues in Potential risk 2: A delayed strategy deserves to be considered to report the real warnings to the ATCO after a potential warning can be confirmed in a dynamic temporal period.In this procedure,the effectiveness of the warning information is required to be cross-validated by referring to the multi-source data and the emerging trends among the temporal intervals.Only the warning information can be continuously and consistently confirmed in several temporal periods(maybe a manually set threshold), and can be reported to the ATCO.

(C) To address issues in Potential risk 3:It is better to refine the definition of the warning level to distinguish the importance of different warning messages for ATC work.In this way, the proposed solution only presents warning information to ATCO about the high safety–critical ones, which is conducive to reducing the extra workload, and further eliminates the operational risks caused by distracting the ATCO attention.

Attempts and exploration.The above two measures are carefully discussed by cooperating with the front-line users.A preliminary upgrade version of the software for the proposed prototype is implemented to evaluate its effectiveness and efficacy on real-time applications.

(D) To address issues in Potential risk 3: The management measures and SOPs should be developed to complete the working procedures with a new safety monitoring system.This approach lies in the concerned management departments and should be fully evaluated by a variety of ATC-related stakeholders about the adverse changes in the daily work6.

Attempts and exploration.This measure is to undergo a review for ATC-related stakeholders, which is expected to encourage the front-line users to work with the new prototype to further validate its effectiveness.

In summary, although great progress has been achieved to enhance air traffic safety by leveraging AI-driven innovations, the proposed solution and prototype are required to further improve to meet the requirements of the ATC practice.On the one hand, the core techniques should be highly improved to reduce the false alarm to lay a solid foundation for the proposed solution.On the other hand, based on the assumptions of the concept design, we would like to optimize our strategy to balance the tradeoff between the false alarm and the missing alarm.And we will also conduct cooperation with concerned ATC stakeholders on the engineering construction of the proposed solution and new management measures, which will further improve the system applicability in the ATC domain.

6.Conclusions

In this work, a novel framework is proposed to monitor the human factors by detecting potential risks from air-ground speech communication for air traffic control.Thanks to the available data storage and artificial intelligence algorithms, automatic speech recognition and spoken instruction understanding are achieved to deal with speech communication by leveraging data-driven mechanisms.A software prototype is developed to evaluate the proposed solution, which allows us to detect human-related risks in an automatic manner.Both the objective (case study) and subjective (user experience evaluation) are conducted to identify the potential risks and provide insights for improving the proposed solution.The results demonstrate that the proposed solution has the ability to improve the operational safety of air traffic, and also reduce the ATCO workload by monitoring the readback instructions.Nevertheless, the system performance should be continuously enhanced to improve its applicability in real practices, concerning the core techniques and safety applications.The most urgent task is to relieve the extra workload caused by confirming the raised warning information.Based on the investigation results, several strategies and approaches are prepared to further explore the capability of data-driven mechanisms.

In the future, the core techniques, including the ASR and SIU,deserve to be enhanced to reduce the false alarm.We also plan to cooperate with the concerned management departments and experts to improve the concept design, which is expected to meet the requirements of front-line ATCO users.

Declaration of Competing Interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Acknowledgments

This work was supported by the National Natural Science Foundation of China (Nos.62001315, 71971150, and U20A20161), the Open Fund of Key Laboratory of Flight Techniques and Flight Safety, Civil Aviation Administration of China (No.FZ2021KF04), and Fundamental Research Funds for the Central Universities of China (No.2021SCU12050).