Cats second experiment results


Published on

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Before starting with the goal, I would like to stress that this experiment is a proof of concept study. We’re at the beginning of the concept development and then at an early stage in the EOCVM level of maturity. Regarding the Level of Maturity, pre-requisite for developing the validation strategy/activities according to the E-OCVM, we cover the level V1 and V2. V1The goal there is to describe the proposed solution and highlight benefits foreseen, through the validation strategy V2The aim of this task is the evaluation of the concept proposed. It’s an iterative process to show the operational feasibility of the proposed solution.
  • evaluate the impact of the Contract of Objectives and Target Windows on the work of the controllers To scope the experiment, we evaluate this impact between two ANSPs, environment restricted to …
  • When building our validation strategy, and to answer to stakeholders expectations, we had in minds some hypothesis for this first expe.
  • To ensure that CATS results map to the SESAR Performance Framework, we used the same metrics than SESAR & EP3, to have common language. From the stakeholders concerns and performance framework [§4], 4 of the SESAR KPA have been identified as potentially improved by CoO and associated TWs introduction, mainly: Safety (SAF): the implementation of CoO will not degrade safety. Capacity (CAP): CoO is still manageable even with increase of traffic foreseen in 2020. Efficiency (EFF): CoO implementation affects positively the aircraft outputs in sector (flight duration ...). Predictability (PRED): implementation of TWs will ensure the respect of schedule. This first objective (system performances) is a key aspect of the validation. The aim is to assess if the benefits are delivered as proposed. The aim of the second objective (human performances) is to see if the contribution of the human to overall system performance is within expected capabilities (workload, stress, SA...) and not reaching the human limits. The human performance could be seen as an enabler to reach the system performance.
  • From the catalogue of Performance Indicators, delivered by EP3, CATS extracted some Local PI relative to its concerns and added also some indicators relative to the concept. As CATS Operational assessments will evaluate the operational acceptability of the CoO and associated TWs concepts from the various actors' point of view between two ACCs and not the ECAC wide performance of En-Route, we will only focus on Local Performance Indicators (PI) layer.
  • Specify 2 from Roma ACC and 2 from Brindisi
  • At departure TW should be larger, no problem At arr should be in atube… Should be evaluated in case of emergency…. For the ATCO no increase of workload even if it was perceived as an additional task during the debriefing For pilot TW slightly impact the workload but it was found acceptable during the debriefings.
  • SA was not impacted by TW implementation , but the controllers have the feelings that TW information increase their traffic pictures, by the data allowing them to better manage FPL. For pilots SA better but need to more anticipate the following TW on display. NB: no tool equivalent for pilot, SASHA validated for ATCO, some qu changed for pilots with same idea….
  • Observations allowed to better analyse the samples (false alarms, Why TW out…) to have more sense.
  • Two following Superimposed Target Windows should be enough larger to allow room of manoeuvre for operators as well as optimum for aircraft navigation. Superimposed Target Windows should not be on the main crossing points. Superimposed Target Windows have to be far from the sector vertical boundaries, to avoid potentially four sectors responsible for the same aircraft. A buffer zone is really needed. As explain in HIL1, TWs should not be at the same exit point, same FL envelop and same time envelop
  • No particular changes in the way the controller work, except direct order less diverging from the planned route. In 2020 more FL than GOTO even if not significant. Safety remains the first priority
  • Here we have the % of RT communication per flight duration. The main problem was that the simulation platform used the current SkyGuide voice recorder , usefull in case of sound tape analysis but not really for simulation analysis purposes… Collaboration between ATCO and Pilot was not impacted They all have the feeling that the TW implementation will increase the common awareness , as the aircraft intent is shared. No particular needs for specific phraseo, DUE TO TW was instinctively found to start comm regarding TW.
  • More awareness for PLN, focusing on what will happen in the sector , helping the EXE by suggesting solutions, using what if tool as collaborative tool.
  • The separation performance tool provides the actual and predicted flight times (in minutes) for different separation bands for all the flights. The actual separation represents the total flight time that the aircraft have flown during the exercises, grouped by the closest separation distance between aircraft. Predicted flight time is based on aircraft trajectories without controllers' interventions, and the closest separation distances according to the intended trajectories. The separation above the 10 Nm and 1000 ft are grouped together with the traffic that did not risk loss of separation. The bands of separation are: Loss of separation is when aircraft is < 5 NM and 800 ft.
  • Safety not impacted, as well as in questionnaires and debriefing where the level of Safety was judged as equivalent as today..
  • Even if they apply more DCT , some flight are shorter but they are no predictable, so when a conflict occurs, the resolution increase globally the flight duration.
  • TW does not impact
  • We use a potentiality offered by Flight Simulator. Was not validated through ….(env) Was also confirmed by the three pilots, qualified on this aircraft.
  • NASA TLX & ISA Learning effect, just 2 weeks of simulation Same platform to be used for HIL2 with further improvements
  • Cats second experiment results

    1. 1. CONTRACT-BASED AIR TRANSPORTATION SYSTEM (CATS) Operational Assessment A Human in the Loop experiment to assess the CoO concept from ATCO & Pilot point of view Second CATS Workshop – Geneva – January 26th, 2010 Page: Presented by Sandrine Guibert EUROCONTROL
    2. 2. Goal <ul><li>Proof of Concept Study, following E-OCVM </li></ul><ul><li>The CATS operational assessment is focused on the En-route part and covers the V1 level ( Scope) and V2 level ( Feasibility) of the Level of Maturity. </li></ul>
    3. 3. Goal <ul><li>Second operational assessment to e valuate the operational acceptability of the CoO and associated TWs concepts from the controllers' and pilots’ point of view . </li></ul><ul><ul><li>Operational acceptability will be evaluated in the context of the transfer of responsibility area between two ANSPs, and the relationship between ATCOs and Pilots. </li></ul></ul><ul><ul><li>Evaluation environment is restricted to </li></ul></ul><ul><ul><ul><li>2 en-route controller working positions ( CWP: EXE + PLN ) which manage the traffic and handover the aircraft </li></ul></ul></ul><ul><ul><ul><li>and two cockpit positions </li></ul></ul></ul><ul><li>The objectives of the HIL2 experiment is twofold: </li></ul><ul><ul><li>Analyse t he collaboration process between controllers and aircrew regarding the TW management. </li></ul></ul><ul><ul><li>Analyse the impact of the TW management on the aircrew's activity in the cockpit. </li></ul></ul>
    4. 4. HIL2 Hypothesis <ul><li>CoO implementation allows safe operations. </li></ul><ul><li>CoO is still manageable even with increase of traffic as foreseen in 2020. </li></ul><ul><li>CoO implementation affects positively the flight within the sector (flight duration ...). </li></ul><ul><li>Implementation of TWs ensures the respect of schedule. </li></ul><ul><li>TWs integrate flexibility to cope with uncertainty. </li></ul><ul><li>The working methods offered to ATCOs and pilots, as a result of the CoO implementation, are feasible and acceptable (task sharing, role and responsibility, as well as the offered support tools). </li></ul><ul><li>Implementation of CoO does not impose significant additional workload to ATCOs or pilots. </li></ul>
    5. 5. Methodology <ul><li>Evaluation principles: </li></ul><ul><ul><li>SESAR KPAs (Key Performance Area): system performance </li></ul></ul><ul><ul><ul><li>4 KPAs are applicable for CATS </li></ul></ul></ul><ul><ul><ul><ul><li>Safety </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Capacity </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Efficiency </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Predictability </li></ul></ul></ul></ul><ul><ul><li>Objectives relating to human performance </li></ul></ul>
    6. 6. Operational issues evaluation Number of TWs fulfilled PRED.LOCAL.ER. PI (1 & 2) EFF.LOCAL.ER. PI (1, 7, 8, 9, 10, 11) CAP.LOCAL.ER. PI (2, 8, 10, 11, 12 & 13) <ul><li>Workload: ISA, NASA-TLX, Interviews, Observations, Performance outcomes, Questionnaire </li></ul><ul><li>Situation Awareness: SASHA_Q, Interviews, Observations, Performance outcomes, questionnaire </li></ul><ul><li>Error production and management: Observations, Questionnaire, Interviews, Performance outcomes </li></ul><ul><li>Operator's activity: Cognitive processes, Decision making, Risk management, Constraints, etc. </li></ul><ul><li>Collaborative and R/T activity: Communications (number, time, content, speaker and receiver, etc.) </li></ul>SAF.LOCAL.ER. PI (1, 2, 3, 5, 6 & 8) Indicators Indicators PREDICTABILITY EFFICIENCY CAPACITY <ul><li>Feasibility and acceptability of the aircrew & ATCos' working methods due to the CoO execution </li></ul><ul><li>Impact of CoO execution on aircrew & ATCOs' performance </li></ul><ul><li>Impact of CoO on aircrew & ATCOs' activity </li></ul>SAFETY Objectives relating to human performance Objectives relating to system performance
    7. 7. Experimental environment <ul><li>Traffic : </li></ul><ul><ul><li>Real traffic </li></ul></ul><ul><ul><li>Traffic load has been adapted to the airspace </li></ul></ul><ul><ul><ul><li>Current level of traffic (2008) </li></ul></ul></ul><ul><ul><ul><li>2020 traffic load (EUROCONTROL STATFOR) </li></ul></ul></ul><ul><li>Simulation facilities </li></ul><ul><ul><li>SkySoft platform (rapid prototyping adapted from SkyGuide simulator) </li></ul></ul><ul><ul><li>2 sectors (MI1 & KL1) </li></ul></ul><ul><ul><li>EXE and PLN on each CWP </li></ul></ul><ul><ul><li>2 cockpits (Microsoft Flight Simulator) </li></ul></ul><ul><ul><li>No pseudo-pilot: </li></ul></ul><ul><ul><ul><li>Automatic order execution by aircraft </li></ul></ul></ul><ul><ul><ul><li>Automatic hand over/ assume by feed sectors </li></ul></ul></ul>
    8. 8. Experimental environment KL1 FL275 – FL345 MI1 FL275 – FL345 Measured Sectors FL345 FL275 LIRRMI1 LIRR FL345 FL275 LSAGKL1 LSAG Max FL Min FL Sector ACC Exe KL MI Crew1 Pln Exe Pln ATCos Pilot Pilot Crew2
    9. 9. Conduct of the experiment <ul><li>Duration: 10 days </li></ul><ul><li>Dates: </li></ul><ul><ul><li>19th. to the 23th. October 2009. </li></ul></ul><ul><ul><li>26th. to the 30th. October 2009. </li></ul></ul><ul><li>Location: Skysoft premises – Geneva </li></ul><ul><li>Four controllers experienced in En-route ATC </li></ul><ul><li>Two experimented pilots </li></ul><ul><li>One on site pre-training session at Roma ACC and in Airfrance </li></ul>
    10. 10. Conduct of the experiment <ul><li>Timetable: </li></ul>Spare day 30 th . October Experimental runs #15 & 16 – Final Debriefing 29 th . October Experimental runs #12, 13 & 14 28 th . October Experimental runs #9, 10 & 11 27 th . October Experimental runs #7 & 8 Familiarization and Training 26 th . October Experimental runs #4, 5 & 6 23 th . October Experimental runs #1, 2 & 3 22 th . October Operational training (session 3) Operational training (session 2) 21 th . October Operational training (session 1) Familiarization 20 th . October Familiarization Simulation devices presentation 19 th . October Afternoon Morning Day
    11. 11. Conduct of the experiment <ul><li>Daily Programme </li></ul>SkySoft arrival 0830 Leave 1630 Exercise 3 + Debriefing & Questionnaires 1415 – 1600 Lunch 1245 – 1415 Exercise 2 + Debriefing & Questionnaires 1100 – 1245 Break 1045 – 1100 Exercise 1 + Debriefing & Questionnaires 0900 – 1045 Set-up in Operations Room 0845 Daily Programme
    12. 12. Experimental plan (1) <ul><li>Independent variables: </li></ul><ul><ul><li>With and without TWs </li></ul></ul><ul><ul><li>2 levels of traffic (2008/expected in 2020) </li></ul></ul><ul><ul><li>Disruptions (nominal & non nominal scenario) </li></ul></ul><ul><li>4 similar traffic scenarios </li></ul>X X disruptions X Traffic load X X TWs Pilot ATCO Independent variables
    13. 13. Experimental Plan (2) <ul><li>Dependent variables : </li></ul><ul><ul><li>Observations </li></ul></ul><ul><ul><ul><li>ATCOs performance (Overt The Shoulder – OTS rating scales, FAA) </li></ul></ul></ul><ul><ul><ul><li>Spontaneous verbalisations </li></ul></ul></ul><ul><ul><li>Performance measurements </li></ul></ul><ul><ul><ul><li>STCA, Separation Performance Indicators ( Intervention type to resolve predicted conflict, Time of intervention before potential LoS, Distance before potential LoS, Localisation of controllers’ interventions) </li></ul></ul></ul><ul><ul><ul><li>ATCO orders </li></ul></ul></ul><ul><ul><ul><li>Flight duration </li></ul></ul></ul><ul><ul><ul><li>Fulfilled TW </li></ul></ul></ul><ul><ul><li>Self assessment </li></ul></ul><ul><ul><ul><li>Workload : ISA & NASA-TLX </li></ul></ul></ul><ul><ul><ul><li>Situation awareness : SASHA_Q </li></ul></ul></ul><ul><ul><ul><li>Safety feeling : questionnaire on risks & hazards </li></ul></ul></ul><ul><ul><li>Post run interviews & Post experimental questionnaires (concept, performance, cooperation, simulation) </li></ul></ul>
    14. 14. Experimental plan (3) <ul><li>16 runs and 32 measures: 2 traffic loads per run </li></ul><ul><ul><li>8 runs “with TW” for 2008 & 2020 traffic loads for ATCO </li></ul></ul><ul><ul><li>8 runs “without TW” for 2008 & 2020 traffic loads for ATCO </li></ul></ul><ul><ul><li>The 4 experimental conditions were compared for each ATCOs team (EXE-PLN for MI1 & EXE-PLN for KL1 ) </li></ul></ul><ul><ul><li>16 x 4 measured flight segments for pilots </li></ul></ul>
    15. 15. ATCOs Workload (ISA) <ul><li>Not significant between “with TW” and “without TW” </li></ul><ul><li>Significant between 2008 and 2020 traffic load </li></ul>
    16. 16. ATCOs Workload self assessment (NASA TLX) <ul><li>Not significant between “with TW” and “without TW </li></ul><ul><li>Significant between 2008 and 2020 traffic load </li></ul>
    17. 17. Pilots Workload self assessment (NASA TLX) <ul><li>Significant impact of TW implementation. </li></ul><ul><li>Pilot evaluated this impact as fully acceptable with the workload induced by the other cockpit tasks during the cruise phase. </li></ul><ul><li>However, experiments should be done to validate the workload induced by the TWs in emergency situations or in high workload conditions. </li></ul>P=0,00008 Impact With-TW / Without-TW Wilcoxon test: Variable impact results if p<0,05
    18. 18. ATCOs Situation Awareness (SASHA-Q) <ul><li>Not significant between “with TW” and “without TW”, except MI PLN. </li></ul><ul><li>Significant between 2008 and 2020 traffic load </li></ul>
    19. 19. Pilots Situation Awareness (SASHA-QP) <ul><li>Not significant between “with TW” and “without TW” </li></ul>P=0,2425 No impact With-TW / Without-TW Wilcoxon test: Variable impact results if p<0,05
    20. 20. OTS Performance Not significant between “with TW” and “without TW”
    21. 21. Acceptability and Usability <ul><li>Concept has been described by pilots and ATCOs as: </li></ul><ul><ul><li>Feasible & Acceptable </li></ul></ul><ul><ul><li>TW was manageable, for 2008 as well as 2020, even if TW management is an additional task </li></ul></ul><ul><ul><li>Traffic volume constrained ATCOs more than the TW </li></ul></ul><ul><li>Training proposed provided sufficient knowledge on the concept , platform, HMI and tools </li></ul><ul><ul><li>Pilots and ATCOs find the concept easy to learn and to use </li></ul></ul><ul><li>Potential improvements: </li></ul><ul><ul><li>For ATCOs, some aspects regarding the TWs positions within the sector have to be changed in TWs calculation </li></ul></ul><ul><ul><li>For Pilots, there is a need to more anticipate the following TWs on display. </li></ul></ul>
    22. 22. ATCOs Orders <ul><li>Not significant between “with TW” and “without TW” </li></ul><ul><li>Significant between 2008 and 2020 traffic load for KL </li></ul><ul><li>Detailed analysis shows: </li></ul><ul><ul><li>no impact of TW implementation on FL and speed orders </li></ul></ul><ul><ul><li>impact on Go To orders, as expected, significant in 2008, not in 2020. </li></ul></ul>
    23. 23. RT Communication <ul><li>Controller communication duration with piloted aircraft shows no significant difference between “with TW” and “without TW” whatever the traffic load conditions </li></ul><ul><li>2020 traffic condition generates longer communication duration, even if it is not significant. </li></ul>P=0,1441 No impact 2008TW/2020TW P=0,1441 No impact 2008-2020 P=0,2733 No impact 2020-2020-TW P=0,0678 No impact 2008/2008-TW Wilcoxon test: Variable impact results if p<0,05
    24. 24. Working Methods <ul><li>Pilots and ATCOs affirmed that TWs implementation induces no particular changes in their working methods </li></ul><ul><ul><li>TW management was consider as a task mainly for PLN, for traffic preparation. </li></ul></ul><ul><ul><li>TW management was consider as a PF task </li></ul></ul><ul><ul><ul><li>But a need to have party line and maybe TWs on FMS </li></ul></ul></ul><ul><li>Safety remains the first priority </li></ul><ul><ul><li>Conflict resolution first then TW achievement </li></ul></ul><ul><li>Collaboration </li></ul><ul><ul><li>Aircraft’s intentions more shared between operators </li></ul></ul><ul><ul><li>not impacted between ATCOs </li></ul></ul><ul><ul><ul><li>PLN able to help the EXE through collaborative tool </li></ul></ul></ul><ul><ul><li>pilots have the feeling TW implementation could increase slightly the communication between the aircrew. </li></ul></ul>
    25. 25. Safety: Aircraft Separation 2008 <ul><li>No loss of separation </li></ul><ul><li>Majority of the traffic is maintained of more than 10Nm and 1000ft </li></ul>
    26. 26. Safety: Aircraft Separation 2020 <ul><li>No loss of separation </li></ul><ul><li>A high safety level is maintained whatever the traffic load conditions and/or the TWs conditions </li></ul><ul><li>The TWs do not impact the controllers anticipation to manage safety </li></ul><ul><li>Pilots and ATCOs reported in questionnaires that their safety feeling was not impaired by the TWs use </li></ul>
    27. 27. Efficiency: Flight duration <ul><li>Significant between “with TW” and “without TW” in both sectors in 2008 conditions </li></ul><ul><li>Significant between “with TW” and “without TW” in both sectors in 2020 condition for Milano only </li></ul><ul><li>The flight durations &quot;with TW&quot; are closer to the reference than &quot;without TW&quot; whatever the traffic load conditions. </li></ul>
    28. 28. Efficiency: TWs OUT <ul><li>No significant difference (p<0,05) between the two traffic load conditions whatever the controlled sector or event- very low level of TWs OUT </li></ul>
    29. 29. Efficiency: Fuel Burned <ul><li>No significant difference (p<0,05) for the fuel burned by the aircraft whatever the traffic load conditions and the TWs implementation </li></ul>
    30. 30. Conclusions <ul><li>CoO concept is manageable and acceptable with the 2008 & 2020 traffic loads, without any impact on Safety. </li></ul><ul><li>CoO improve the predictability in times and routing </li></ul><ul><li>ATCOs and pilots agreed CoO could optimise ATM system. </li></ul><ul><ul><li>However, the forecasted increase of traffic remains a limitation. New tools are required to support the traffic load. </li></ul></ul><ul><li>The number of TWs not fulfilled predicts positive prospect for HIL3. </li></ul>
    31. 31. <ul><li>Thank you for your attention! </li></ul>