This document proposes a model-based approach for the automatic evaluation of aircraft human-machine interfaces (AHMIs). It suggests that AHMIs can be modeled using existing interaction techniques and evaluated for workload, execution time, and adherence to usability guidelines. The approach involves representing any AHMI as a model that can be automatically evaluated, allowing traditional metrics as well as more complex evaluation of how the interface adheres to design standards. The goal is to integrate evaluation into the development process for AHMIs.

1. *
Université catholique de Louvain (UCL)
Belgian Laboratory of Computer-Human Interaction (BCHI)
{juan.m.gonzalez, jean.vanderdonckt}@uclouvain.be
†
OFFIS e.V., Escherweg 2, 26127,
Oldenburg, Germany
{luedtke, osterloh}@offis.de
Towards Model-Based AHMI Automatic
Evaluation
Juan Manuel Gonzalez-Calleros*
, Jean Vanderdonckt*
,
Andreas Lüdtke†
, Jan-Patrick Osterloh†

2. • Existing work on automated evaluation of AHMI
• Some studies suggested that AHMI and software user interfaces
share several similarities:
• In terms of interaction techniques for input/output
Williges, R.C., Williges, B.H., and Fainter, R.G., Software interfaces for aviation systems, in
Human Factors in Aviation, E.L. Wiener and D.C. Nagel (Eds.) 1988, Academic Press: San
Diego: pp. 463-493.
• In terms of automatic analysis of workload
Eldredge, D., S. Mangold, and R.S. Dodd, A Review and Discussion of Flight Management
System Incidents Reported to the Aviation Safety Reporting System 1992, Battelle/U.S.
Dept. of Transportation.
• In terms of automatic evaluation based on goal models
Irving, S., Polson, P., Irving, J.E., A GOMS Analysis of the Advanced Automated Cockpit, Proc.
of ACM Conf. CHI’94, ACM Press, New York, 1994, pp. 344-350
Introduction
8 April 2015 Page 2

3. • Several challenges for Aircraft cockpit design, such as:
• Introduction of new technologies break previous user experience.
Introduction
8 April 2015 Page 3

4. • New challenges for AHMI: analysis, design, implementa-
tion, and evaluation
• Integrating evaluation in the loop of AHMI life cycle
involves the use of pilots and a physical simulator.
• Shortcomings: availability, cost, time,…
• Human project: new ways to substitute pilots and physical
simulator are explored by coupling them to virtual simulator
• Our goal: to introduce UI automated evaluation (workload,
execution time, usability guidelines) in the life cycle
Introduction
8 April 2015 HMAT Workshop, June 2010 Page 4

5. • Interactive Cooperative Objects (ICOs)
• Used to model aircraft interactive systems (air traffic workstations,
civil aircraft cockpit and military aircraft cockpit)
• Formal description for interactive cockpit applications using Petri
nets
• Behavioural aspects of systems in the cockpit are modelled
• Recent work on usability, reliability, and scalability
• Navarre, D., Palanque, Ph., Ladry, J.-F., Barboni, E., ICOs: A model-based user interface description
technique dedicated to interactive systems addressing usability, reliability and scalability, November 2009,
Transactions on Computer-Human Interaction (TOCHI) , Volume 16 Issue 4
• Problems
• There is a limited used of widgets, UI models and guidelines.
State of the Art
8 April 2015 HMAT Workshop, June 2010 Page 5

6. • ARINC 661 Standard
• Defines protocols to communicate the dialogue and the functional core
of cockpit display system:
• a user application is defined as a system that has two-way
communication with the CDS (Cockpit Display System):
• Transmission of data to the CDS, possibly displayed to the flight deck crew
• Reception of input from interactive items managed by the CDS
• A set of widgets is included as a recommendation
• Problems
• No design guidelines for the widgets,
• No method to design UIs are considered in the standard
• Each manufacturer is free to implement their own understanding about
the standard
• The ARINC standard is not used for primary cockpit applications
(AHMI)
• Only deals with secondary applications involved in the management of
the flight such as the ones allocated to the Multiple Control Display Unit
State of the Art
8 April 2015 HMAT Workshop, June 2010 Page 6

7. • A Methodology is suggested that is composed of:
• Models (Traditional Widgets + AHMI specific UI elements)
• Language (User Interface Description Language)
• Method (Model-Based UI Design applied to the AHMI)
• Software support
• A Language Engineering approach
• Semantics as meta-models (UML class diagrams)
• Syntax as XML Schemas
• Stylistics is the visual syntax, in our context not used.
• A Structured Method compliant to the Cameleon Reference
Framework for UI development
• Just focus in the layer that concerns to the concrete description
model.
Model-Based AHMI Design
8 April 2015 HMAT Workshop, June 2010 Page 7

8. • The Concrete UI Model (CUI) allows:
• Specification AHMI presentation and behaviour independently of
any programming toolkit
Model-Based AHMI Design
8 April 2015 HMAT Workshop, June 2010 Page 8
X3D OpenGL

9. • Automated evaluation of
• Static aspects (UI layout, position of objects)
• Dynamic concepts (state of a button during the interaction, color of
the label).
• Any UI is represented in a UsiXML model submitted to
automated evaluation (automatic or manual)
• Usability guidelines over the UI objects (distribution of the
widgets composing the UI) are evaluated.
• Special attention was paid to those guidelines for standard
certification and quality assurance and to express them in
the Guideline Definition Language (GDL)
Evaluating the AHMI User
Interface
8 April 2015 HMAT Workshop, June 2010 Page 9
http://www.usixml.org
http://www.w3.org/2005/Incubator/model-based-ui/

10. Integrating UI Evaluation in a
Simulation Environment
8 April 2015 HMAT Workshop, June 2010 Page 10

11. UI timeline
8 April 2015 HMAT Workshop, June 2010 Page 11

12. • Evaluating the AHMI User Interface
• Execution Time
• Workload
• Guidelines
Outline
8 April 2015 Page 12

13. Evaluation-Execution time

14. 8 April 2015 Page 14
Evaluation-Workload

15. 8 April 2015 Page 15
Evaluation-Execution Workload
1.2 + 3x(1.5 + 0.075) + 1.5 = 7.4254.6 + 4x(4.6 + 2.2) = 31.8
Workload
Remember date
Select day
Select month
Select year
Execution Time
Select day
Select month
Select year
Validate
(Use mouse to point at object on screen
1.5 second)+Execute a mental “step”
.075 second
Retrieve a simple item from a long-term
memory
1.2 second
4.6
Evaluation/Judgment (consider single
aspect) + 2.2
Discrete Actuation (button, toggle, trigger)

16. Compilation
UsiXML Evaluation
Evaluation report
Formal rules file
Original rule
Usability guidelines
Smith & Mosier
Rule interpretation
WCAG 2.0
Natural language
Formal language
…
A Rule
A Header
Contrainte A
Variable AVariable AA Variable
Contrainte AA constraint
Formal des-
cription of A
A Widget
A Quantifier
UA: evaluation process
Compiled evaluation
rules

17. Compilation
UsiXML Evaluation
Evaluation report
Formal rules file
Source file
Lexical analysis
Syntactic tree
Sequence of
Lexical units
Syntactic
Analysis
Header:1, “White and yellow should
not be used together for
text“,high,partial, "Legibility and
perception",must,
enabled
rule:no inputtext i: i has_properties
(bgcolor == "#ffffff") and
i has_properties(fgcolor ==
"#ffff00")
UA: evaluation process
Compiled evaluation
rules

18. Compilation
UsiXML Evaluation
Evaluation report
Formal rules file
Création d’algorithmes
testant le contenu de
l’information des
“GraphicalWidget”
InputText AInputText A
InputText BInputText B
InputText C
UA: evaluation process
Compiled evaluation
rules

19. Compilation
UsiXML Evaluation
Evaluation report
Formal rules file
InputText C
UA: evaluation process
Compiled evaluation
rules

20. Some guidelines
• Cockpit display systems should at least be consistent with
systems of our daily life [Singer 2002]
• Usability guidelines from ISO 9126 could be evaluated
• Messages should follow always the nomenclature: first letter in
capital and the rest in lower case
• AHMI display systems such as the consistency in the roll index in
the compass rose [Singer 2001]
8 April 2015 HMAT Workshop, June 2010 Page 20

21. • A model-based method for the development of the AHMI
was presented allowing:
• Structured development process (making design more explicit)
• UI evaluation
• Traditional measurements can be assessed like UI workload and
execution time,
• More complex automated evaluation based on guideines.
• Explore design options, for instance, modality of interaction of the
UI can be object of evaluation.
• The original 2D rendering can be equally rendered in 3D.
• A future plan is to automatically generate the AHMI from its model
and to submit it to run-time analysis.
Conclusion
8 April 2015 HMAT Workshop, June 2010 Page 21

22. 8 April 2015 HMAT Workshop, June 2010 Page 22