Measurement and Evaluation of Pilot Mental Workload Based on Flight Simulation System

WU Xu（吳 旭），WANYAN Xiaoru（完顏笑如），ZHUANG Damin（莊達民）

School of Aeronautics Science and Engineering，Beihang University，Beijing 100191，China

Introduction

It was necessary for pilots to monitor visual information from the multiple cockpit display as well as the outside view，which required pilots to allocate their attention reasonably based on the priority level of the visual information in order to ensure the comprehensiveness，preciseness and timeliness of the information collection［1］.During the monitoring tasks，attention level and mental workload state of pilots were closely related，so that any ultralow evoking or overloading of mental resources may cause difficulties to maintain attention at the appropriate level and then lead to human error involved with information leakage，misuse and misjudgment［2-3］.Time pressure model purposed by Rice et al.provided a prediction method to evaluated mental workload level［4］.In this model，time pressure was defined as the ratio of the required time to complete the task and the given period of time.The time pressure model has been adopted and used by Boeing airplane design to study the possibility of task performance since the computer display and control system equipped with Boeing737.

As the important element of human-computer interaction（HCI）system in aviation，cockpit display instrument interface design was directly related to the flight task performance as well as the flight efficiency and safety［5］.Therefore，the behavior performance and physiological state of pilots during the instrument monitoring task were measured to assess the mental workload level and evaluate the display interface design.Unusual attitude recovery task was one of the most important worldwide methods for objective evaluation of flight display instrument interface design［6］.The U.S.A.Military Standard MIL-STD-1787Cclearly stated that the flight display instrument was designed to ensure the performance of unusual attitude recovery task with response time less than 1sas well as the correct rate greater than 95%［7］.Moreover，this requirement was still a valuable guideline for flight display design in civil aviation.

Considering the security and economy of design，the current digital simulation technology was widely applied in aircraft design and manufacturing engineering outside the factory［8］.The simulation method of flight display interface and fight task based on computer program made it possible to realize the cockpit scenarios easily and quickly and provided convenience to interface design evaluation［9］.Based on digital simulation technology，the present study established a set of digital simulation methods of flight environment，scenario and flight tasks.Then，ergonomics experiment was carried out in a flight simulation system to evaluate pilot mental workload and optimize flight display design.Participants were required to performance the unusual attitude recovery tasks during the flight simulation task under different time pressure conditions.Behavior performances including correct rate（CR）and response time（RT）were recorded by digital video camera.Physiological parameters involved eye-movement tracking were measured by SmartEyePro 5.9system and others such as heart rate（HR）and respiration rate（RR）were measured by Tongfang Shenhuo THP physiological tester.The analysis of variance（ANOVA）and correlation analysis were adopted to process the original data.

1 Digital Flight Simulation Methods

1.1 Flight environment simulation platform system

The platform hardware included primary flight display（PFD）and navigation display（ND），engine indication and crew alerting system or electronic centralized aircraft monitor（EICAS／ECAM），visual display，auto-pilot or mode control panel（MCP），flight management system （FMS），steering wheel，seats，platform server and windshield glass as shown in Fig.1.The cockpit layout was designed based on Boeing 737with reference to Collins's advanced flight deck simulator design.The flight display panel stitched by liquid crystal displays could easily switch the display interface，which realized the various types of cockpit display interface［10］.In addition，experimental console was also set up in the cockpit to facilitate the control of experimental parameters and processes during the human-computer interaction（HCI）experiment.

Fig.1 Hardware of theflight simulation environment

The platform software was mainly composed of four parts：flight control server，display server，HCI experiment model and experimental console，as shown in Fig.2.The flight control server received all kinds of control inputs and computed real-time flight data.Display server accepted flight data and drove virtual instrument interface on each display.HCI experiment model generated unusual attitude information for the interface evaluation and recorded behavior performances of recovery manipulations.Experimental console was responsible for start，stop，and configuration parameters of each system.

Fig.2 Software structure of the flight simulation environment

A set of computer-based flight display instrument construction methods was developed in order to truly realize the display interface driven by flight data of display server in cockpit instrument［11］.First，Adobe Photoshop was used to image process the existing flight information display instrument interface and generate the required simulation texture.And then DiSTI GL Studio and Microsoft Visual Studio were used to integrate environment attribute and data interface，and generate simulation procedures.Three models of primary flight display such as Boeing737，Airbus320and ARJ21were taken as the templates and simulated as three sets of flight instrument display after proper simplification and abstraction，as shown in Fig.3.

Fig.3 Flight instrument panels based on Boeing737（a），Airbus320（b）and ARJ21（c）

1.2 Flight task simulation

According to the characteristics of flight simulation environment and scenario，the unusual attitude recovery tasks were used to evaluate flight performances and pilot mental workload.The simulation tasks focused on the cruise phase based on the cognitive characteristics in unusual attitude recovery task.In the HCI experiments，participants were required to monitor the status of target information in cockpit display interface and ensure each indicator in the pre-determined safe threshold.Once a target indicator exceeded the normal range，the unusual attitude appeared and then the corresponding response in steering wheel was expected to be taken to recover and eliminate the unusual attitude.When the unusual attitude occurred in barometric altitude and pitching angle，participants were asked to pull or push the wheel to response.And when it occurred in rolling and heading angle，rotating the wheel was needed to response.The reaction time of unusual attitude recovery tasks was calculated by the manipulation of information input from the display server to the experimental console.

In order to input the HCI parameters and control the experiment，the control interface of HCI experiment model was established inside the experimental console as shown in Fig.4.The top menu“Display interface options”made it possible to select and present any of the flight instrument display interfaces mentioned in Fig.3.The middle menu“Presentation time options”was used to set up the presenting time and refreshing time of unusual attitude information.The menu “Upper limit of normal attitude”and “Lower limit of normal attitude” were used to define the corresponding range of normal attitude of flight information while the menu“Upper limit of total attitude”and“Lower limit of total attitude”were used to define the corresponding range of all flight information attitudes.The menu“Information weight”was used to set up the occurring probability of the corresponding unusual attitude，and the menu“Normal information weight”was used to define to the occurring probability of the normal attitude.Moreover，the button“Start ergonomics test”was pressed to initiate the whole ergonomics experiment of unusual attitude recovery tasks according to the settings mentioned above.The button“Stop ergonomics test”was pressed to end the unusual attitude recovery tasks.The bottom buttons“Total freeze”，“System reset”and“System shutdown”were used to freeze，reset and shutdown the entire flight simulation platform system.

Fig.4 Control interface of the HCI experiment

2 Ergonomics Experiment

2.1 Participants

Forty-eight participants from Beihang University were recruited in this study.All participants（36 males，12females；ranging from 22to 28years old）were familiar with the basic computer operations and aeronautics knowledge.All participants were right-handed with normal or correct to normal vision.

2.2 Apparatus

The experiment was conducted in a high-fidelity Boeing737 flight simulation system as shown in Fig.5.The experiment interface was designed as the three sets of PFD simulation interface.SmartEyePro 5.9 was used to track the participant's eye-movement in a nature way，and the blink rate（BR），pupil diameter（PD）and eyelid opening （EO）were recorded.Meanwhile，Tongfang Shenhuo TH-P physiological tester was adopted to measure the participant's skin electrodermal activity（EDA），HR as well as RR through sensors attached to the finger and wrist of the participant［12］.Besides，a Sony video camera was set to record the participant's response to the experiment task.

Fig.5 Experiment environment and apparatus

2.3 Experiment design

Participants were required to practice the experiment task operations until they were familiar with the flight simulator.Four indicators of PFD were selected as the monitoring task target，including barometric altitude（1），pitching angle（2），rolling angle（3），and heading angle（4）［13］.Unusual attitude of the four indicators appeared randomly by computer program and only one indicator was allowed to turn abnormal simultaneously.A 3×4 two-factor mixed design experiment was carried out.The between-subject factor was PDF type with three levels：（1）Boeing737，（2）Airbus320and（3）ARJ21.The within-subject factor was informativeness condition with four levels as shown in Table 1.

Table 1 Informativeness design of time pressure and unusual attitude probability

2.4 Experiment procedure

All participants were randomly and equally divided into three groups according to the experiment interface type.All participants were required to perform four flight simulation experiments under different informativeness conditions.Each experiment lasted for 10min with interval of 5 min.The order of experiment was counterbalanced in a Latin Square Design fashion.Once the unusual attitude of an indicator occurred，the corresponding information would suddenly change and exceed the normal range associated with a warning color.So the participant was asked to make a response to recovery it by pulling／pushing or rotating the wheel under the given time pressure.The SmartEyePro 5.9，TH-P physiological tester and Sony video camera kept real-time tracking.

3 Experiment Results

3.1 ANOVA of behavior performances

The repeated measure ANOVA showed that the main effects of PFD type were not significant in neither CR nor RT：F（2，38）＝0.644，p＝0.531；F（2，38）＝0.716，p＝0.495.However，the average CRs of Boeing737，Airbus320and ARJ21were 0.837，0.879and 0871while the average RTs were 846，866and 842ms.

The repeated measure ANOVA showed that the main effects of informativeness were significant in both CR and RT：F（3，114）＝31.615，p＜0.01；F（3，114）＝42.064，p＜0.01.The post hoc test showed significant differences between informativeness conditions 1，3and conditions 2，4in both CR and RT （p＜0.01）.The average CRs in four conditions of informativeness were 0.762，0.931，0.792and 0.962while the average RTs were 916，769，911and 810ms.

The repeated measure ANOVA showed that the main effects of flight indicator were significant in both CR and RT：F（3，114）＝24.745，p＜0.01；F（3，114）＝13.705，p＜0.01.The post hoc test showed significant differences between indicators 1，2and indicators 3，4in both CR and RT（p＜0.01）.The average CRs of four flight indicators were 0.828，0.802，0.898and 0.919 while the average RTs were 880，878，828and 820ms.

The ANOVA showed the interaction effects between PFD type and informativeness were insignificant in both CR and RT：F（6，114）＝1.061，p＝0.390；F（6，114）＝0.703，p＝0.648.The interaction effects between PFD type and flight indicator were still insignificant in CR（F（6，114）＝1.124，p＝0.353）but critical significant in RT（F（6，114）＝2.163，p＝0.052）.Moreover，the interaction effects between informativeness and flight indicator were significant in both CR and RT：F（9，342）＝7.916，p＜0.01；F（9，342）＝3.444，p＜0.01，as shown in Table 2.

Table 2 Behavior performances of four flight indicators of three PDF tpyes under four informativeness conditions

3.2 ANOVA of physiological parameters

For eye-movement tracking results， ANOVA showed insignificant main effect of PFD type in BR，PD and EO：F（2，37）＝0.474，p＝0.626；F（2，37）＝0.606，p＝0.551；F（2，37）＝0.508，p＝0.606.The average BRs of Boeing737，Airbus320and ARJ21were 16.62，19.62and 16.91times／min while the average PDs were 3.99，4.12and 3.81mm and the average EOs were 10.32，9.87and 10.50mm.The ANOVA showed significant main effect of informativeness in BR，PD and EO：F（3，111）＝4.652，p＝0.004；F（3，111）＝17.572，p＜0.01；F（3，111）＝4.492，p＝0.005.The average BRs in four conditions of informativeness were 16.14，18.32，17.86 and 18.55times／min while the average PDs were 4.18，3.97，3.98and 3.76 mm and the average EOs were 10.45，10.16，10.23and 10.07 mm.Moreover，the interaction effects between PFD type and informativeness were insignificant in BR，PD and EO：F（6，111）＝0.625，p＝0.71；F（6，111）＝1.572，p＝0.162；F（6，111）＝1.331，p＝0.249，as shown in Table 3.

Table 3 Physiological parameters measured in different conditions of time pressure and informativeness

The post hoc test of informativeness showed the most significant differences between informativeness conditions 1，2and conditions 3，4in BR （p12＜0.01，p34＝0.712），PD（p12＜0.01，p34＜0.01）and EO （p12＝0.032，p34＝0.061）.

For physiological tester results， ANOVA showed insignificant main effect of PFD type in HR，RR and EDA：F（2，37）＝0.07，p＝0.932；F（2，37）＝2.553，p＝0.091；F（2，37）＝2.205，p ＝0.125.The average HRs of Boeing737，Airbus320and ARJ21 were 81.37，78.96and 79.80times／min while the average RRs were 21.42，19.78 and 21.28times／min and the average EDAs were 2.50，2.46and 2.51mV.The ANOVA showed partially significant main effects of informativeness in HR，RR and EDA：F（3，111）＝11.81，p＜0.01；F（3，111）＝22.696，p＜0.01；F（3，111）＝2.018，p＝0.115.The average HRs in four conditions of informativeness were 81.86，78.39，81.06and 79.13times／min while the average RRs were 21.87，19.93，21.65and 19.84times／min and the average EDAs were 2.53，2.45，2.52and 2.45 mV.Moreover，the interaction effects between PFD type and informativeness were insignificant in BR，PD and EO：F（6，111）＝0.319，p＝0.926；F（6，111）＝0.562，p＝0.760；F（6，111）＝0.266，p＝0.952，as shown in Table 3.

The post hoc testof informativeness also showed most significant differences between informativeness conditions 1，2and conditions 3，4in HR（p12＜0.01，p34＝0.004），RR（p12＜0.01，p34＜0.01）and EDA（p12＝0.171，p34＝0.037）.

3.3 Correlation analysis of physiological parameters

To research the connection of mental workload related indices，partial correlation analysis was adopted to process physiological parameters.Taking PDF type and informativeness as controlled variables，the correlation analysis showed significant correlations between HR and RR（r＝0.245，p＜0.01），PD and BR（r＝0.288，p＜0.01），PD and EO （r ＝0.254，p ＝0.001）.In addition，the correlation analysis still showed some critical significant correlations between HR and EDA（r＝0.123，p＝0.107），EDA and RR（r＝0.147，p＝0.053），RR and EO（r＝0.121，p＝0.112）.

4 Discussion

Based on digital simulation technology，the current study established a flight simulation system，which was involved with flight environment platform and flight task simulation.Then，HCI experiment was carried out and unusual attitude recovery task was performed to evaluate both pilot mental workload and display interface design.Time pressure and informativeness conditions were applied to controlling the mental workload level and behavior performances along with physiological parameters were measured as the indices of mental workload.

The ANOVA results showed insignificant main effect of PFD type in most indices.The descriptive statistics results revealed slightly higher CR and slower RT in Airbus320.Meanwhile，the physiological parameters in Airbus320 indicated a relatively lower level of mental workload with lower HR，smaller PD and so on.However，PFD type，as a between-subject designed factor，was manipulated as three interfaces of Boring737，Airbus320and ARJ21，which were well designed and validated through a series of sophisticated procedures.Besides，the influence of individual differences was quite unignorable， especially in physiological measurement［14］.

On the contrary，the main effect of informativeness was totally significant and the post hoc test verified time pressure control as a reasonable method to set up different levels of mental workload.There were obvious discrepancies in CR and RT between higher and lower levels of time pressure，as shown in Fig.5.And the physiological parameters described the consistent tendency as shown in Table 3.It was coincided with the viewpoint in Lavie's perceptual load theory［15-16］.The cognitive resource overflew could be used to process the irrelevant information under the low time pressure，and thus the target information processing was higher in CR but slower in RT.Moreover，the correlation analysis illustrated certain significant relevance of HR and RR as well as eye-movement indices.

In addition，the interaction effect of PFD type and informativeness condition was mainly insignificant，which expressed a sensitive connection between unusual attitude recovery task and time pressure rather than specified PFD interface［17-18］.

5 Conclusions

This study proposed a digital simulation method of flight environment and task simulation system in order to further ergonomics evaluation and design of cockpit display interface.Therefore，unusual attitude recovery task was performed in the HCI experiment and evaluation indices were recorded by eye-movement tracking and physiological tester devices.The ANOVA results showed significant sensitivity between mental workload and those indices.The flight simulation system and ergonomics experiment provided an effective method to measure physiological and psychological status and evaluate cockpit display interface design.

［1］Wu X，Wanyan X R，Zhuang D M.Attention Allocation Modeling under Multi-factor Condition［J］.Journal of Beijing University of Aeronautics and Astronautics，2013，39（8）：1086-1090.（in Chinese）

［2］Wiegmann D A，Shappell S A.Human Error Perspectives in Aviation［J］.International Journal of Aviation Psychology，2001，11（4）：341-357.

［3］Wanyan X R，Zhuang D M，Liu W.Analysis on Effect of Mental Workload on Preattentive Processing［J］.Journal of Beijing University of Aeronautics and Astronautics，2012，38（4）：497-501.（in Chinese）

［4］Rice S，Keller D，Trafimow D，et al.Retention of a Time Pressure Heuristic in a Target Identification Task［J］.Journal of General Psychology，2010，137（3）：239-255.

［5］Wickens C D，Small R L， Andre T S.Multisensory Enhancement of Command Displays for Unusual Attitude Recovery［J］.International Journal of Aviation Psychology，2008，18（4）：255-267.

［6］Wickens C D，Self B P，Andre T S.Unusual Attitude Recoveries with a Spatial Disorientation Icon ［J］.International Journal of Aviation Psychology，2007，17（2）：153-165.

［7］Zhuang D M， Wanyan X R.Pilot Attention Allocation Theory and Applicationis［M］.Beijing：Science Press，2013：31-45.

［8］Campbell C B.Advanced Integrated General Aviation Primary Flight Display User Interface Design， Development and Assessment ［C］.The 21st Digital Avionics Systems Conference，Ivrine，USA，2002：1-12.

［9］Chen T L，Pritchett A R.Development and Evaluation of a Cockpit Decision-Did for Emergency Trajectory Generation［J］.Journal of Aircraft，2001，38（5）：935-943.

［10］Liggett K K，Gallimore J J.An Analysis of Control Reversal Errors during Unusual Attitude Recoveries Using Helmet-Mounted Display Symbology ［J］.Aviation Space and Environmental Medicine，2002，73（2）：102-111.

［11］Wei H Y，Zhuang D M，Wanyan X R，et al.An Experimental Analysis of Situation Awareness for Cockpit Display Interface Evaluation Based on Flight Simulation［J］.Chinese Journal of Aeronautics，2013，26（4）：884-889.

［12］Langan-Fox J，Sankey M J，Canty J M.Human Factors Measurement for Future Air Traffic Control Systems ［J］.Journal of the Humam Factors and Ergonomics Society，2009，51（5）：595-637.

［13］Wanyan X R，Zhuang D M，Wei H Y，et al.Pilot Attention Allocation Model Based on Fuzzy Theory［J］.Computers ＆Mathemetics with Applications，2011，62（7）：2727-2735.

［14］Wickens C D，McCarley J S.Applied Attention Theory［M］.Boca Raton：CRC Press，2008：300-310.

［15］Lavie N.Attention，Distraction，and Cognitive Control under Load［J］.Current Direction in Psychological Science，2010，19（3）：143-148.

［16］Lavie N，Hirst A，de Fockert J W，et al.Load Theory of Selective Attention and Cognitive Control［J］.Journal of Experimental Psychology General，2004，133（3）：339-354.

［17］Sarter N B，Mumaw J B，Wickens C D.Pilots'Monitoring Strategies and Performance on Automated Flight Decks：an Empirical Study Combining Behavioral and Eye-Tracking Data［J］.Journal of the Human Factors and Ergonomics Socirty，2007，49（3）：347-357.

［18］Rice S，Keller D，Trafimow D.Time Pressure Heuristics Can Improve Performance due to Increased Consistency ［J］.Journal of General Psychology，2012，139（4）：273-288.