Model-Driven Engineering of User Interfaces for Multimodal Web Applications Jean Vanderdonckt Université catholique de Louvain (UCL) Louvain School of Management (LSM) Information Systems Unit (ISYS) Belgian Laboratory of Computer-Human Interaction (BCHI) http://www.isys.ucl.ac.be/bchi

What is a multimodal user interface? Some terminology: notion of mode We have 5 Modes corresponding to our 5 human senses The human body has five major senses which operate to gather information from the world around us, sight, hearing, smell, taste, and touch. Any stimulus to one of the sense areas is detected by sensory nerves and is sent to the brain for interpretation.

Some terminology: notion of mode

We have 5 Modes corresponding to our 5 human senses

The human body has five major senses which operate to gather information from the world around us, sight, hearing, smell, taste, and touch. Any stimulus to one of the sense areas is detected by sensory nerves and is sent to the brain for interpretation.

What is a multimodal user interface? Combining several modes simultaneously: some are very hard! [Tison,2003]

Combining several modes simultaneously: some are very hard!

What is a multimodal user interface? Some terminology: notion of modality Definition: a particular form of a mode Example: for auditive mode, we have: speech, music, voice, chorals, noise User interface for each mode thus involves one or many modalities A system is said to be Monomodal iff only one modality is involved Multimodal iff many modalities are involves Some modalities are still in their infancy: Olfactive user interfaces Aromatic user interfaces [Bodnar,2004]

Some terminology: notion of modality

Definition: a particular form of a mode

Example: for auditive mode, we have: speech, music, voice, chorals, noise

User interface for each mode thus involves one or many modalities

A system is said to be

Monomodal iff only one modality is involved

Multimodal iff many modalities are involves

Some modalities are still in their infancy:

Olfactive user interfaces

Aromatic user interfaces

What is a multimodal user interface? Some terminology: notion of media Definition: is a hardware/software device allowing interaction between a user and a system according to a given set of modalities Example: for auditive mode, we have several media possible: speech, music, voice, chorals, noise User interface for each mode thus involves one or many modalities [Bellik,2002]

Some terminology: notion of media

Definition: is a hardware/software device allowing interaction between a user and a system according to a given set of modalities

Example: for auditive mode, we have several media possible: speech, music, voice, chorals, noise

User interface for each mode thus involves one or many modalities

Why are multimodal user interfaces important? Graphical user interfaces – classical human-computer interface paradigm Users with limited literacy and typing skills Handheld devices Need for hands and eyes free Difficult to support mouse and keyboard interaction Web access anyplace, anywhere, anytime Need for newer, better interface paradigm Multimodal user interface is a possible answer… Adaptation to the context of use (user, platform, environment) More natural and efficient human-computer interaction [IntuiLab,2006]

Graphical user interfaces – classical human-computer interface paradigm

Users with limited literacy and typing skills

Handheld devices

Need for hands and eyes free

Difficult to support mouse and keyboard interaction

Web access anyplace, anywhere, anytime

Need for newer, better interface paradigm

Multimodal user interface is a possible answer…

Adaptation to the context of use (user, platform, environment)

More natural and efficient human-computer interaction

Why are multimodal user interfaces challenging? For Web Engineering Since presentation and navigation are no longer visual only, how can we design vocal, tactile and multimodal presentation and navigation within a page or across pages? For Model-Driven Approach So far, little research has been devoted to modelling the various aspects involved in multimodal web applications. Therefore, there is a need for identifying the appropriate implementation characteristics which will be the turned into high level design options to be supported by model-to-model and model-to-code transformations For Human-Computer Interaction: Developing a user interface for any combination of graphical, vocal, and tactile interaction modalities remains a challenge because of synchronization issues in time and space For Usability Engineering It is not because we are able to develop a user interface with any combination of any modalities that the resulting user interface is usable or tractable for a end user. We know little about the usability guidelines to be respected in this area

For Web Engineering

Since presentation and navigation are no longer visual only, how can we design vocal, tactile and multimodal presentation and navigation within a page or across pages?

For Model-Driven Approach

So far, little research has been devoted to modelling the various aspects involved in multimodal web applications. Therefore, there is a need for identifying the appropriate implementation characteristics which will be the turned into high level design options to be supported by model-to-model and model-to-code transformations

For Human-Computer Interaction:

Developing a user interface for any combination of graphical, vocal, and tactile interaction modalities remains a challenge because of synchronization issues in time and space

For Usability Engineering

It is not because we are able to develop a user interface with any combination of any modalities that the resulting user interface is usable or tractable for a end user. We know little about the usability guidelines to be respected in this area

Concepts: The CARE Properties From Amodeus European Project (1995!) Complementarity Assignment Redundancy Equivalence Put that there [Bolt,1979]

From Amodeus European Project (1995!)

Complementarity

Assignment

Redundancy

Equivalence

What are the appropriate abstractions? Design space that supports effectively and efficiently the progress of the development life cycle towards a final system while assuring some form of quality Design space consists of an n-dimensional space where each dimension is denoted by a single design option Each design option has a finite set of design values E.g., label location (left-aligned, centered, right-aligned)

Design space that supports effectively and efficiently the progress of the development life cycle towards a final system while assuring some form of quality

Design space consists of an n-dimensional space where each dimension is denoted by a single design option

Each design option has a finite set of design values

E.g., label location (left-aligned, centered, right-aligned)

Why a design space? Advantages: Clarifies the development process in terms of options Facilitates the incorporation in the development life cycle of design options as an abstraction covered by a software Supports the tractability of more complex design problems Shortcomings: In theory: design options could be very numerous (even infinite) In practice: impossible to consider a large amount of design options Too complex or expensive to implement Design options do not necessarily address users’ needs and requirements Out of the designer’s scope of understanding, imagination or background Design decision is not always clear (may violate some usability principles or guidelines)

Advantages:

Clarifies the development process in terms of options

Facilitates the incorporation in the development life cycle of design options as an abstraction covered by a software

Supports the tractability of more complex design problems

Shortcomings:

In theory: design options could be very numerous (even infinite)

In practice: impossible to consider a large amount of design options

Too complex or expensive to implement

Design options do not necessarily address users’ needs and requirements

Out of the designer’s scope of understanding, imagination or background

Design decision is not always clear (may violate some usability principles or guidelines)

Multimodal Web Applications The languages in which they are implemented (e.g., XHTML, VoiceXML, X+V): Restrict the amount of possible UIs to obtain Set directly the CARE properties to Assignment, Equivalence and Redundancy Limited to graphic, voice, and tactile modalities [W3C,2006]

The languages in which they are implemented (e.g., XHTML, VoiceXML, X+V):

Restrict the amount of possible UIs to obtain

Set directly the CARE properties to Assignment, Equivalence and Redundancy

Limited to graphic, voice, and tactile modalities

Solution space Cameleon reference framework UsiXML language: Graphical interaction Vocal interaction Multimodal interaction Design option-based transformational approach: Step 1: Construct the Task & Domain Models Step 2: From Task & Domain to Abstract User Interface (AUI) Step 3: From AUI to Concrete User Interface (CUI) Step 4: From CUI to FUI [Stanciulescu et al.,2005]

Cameleon reference framework

UsiXML language:

Graphical interaction

Vocal interaction

Multimodal interaction

Design option-based transformational approach:

Step 1: Construct the Task & Domain Models

Step 2: From Task & Domain to Abstract User Interface (AUI)

Step 3: From AUI to Concrete User Interface (CUI)

Step 4: From CUI to FUI

Design options for graphical web UIs Design options for graphical UIs Sub-task presentation Sub-task navigation Concretization of navigation & control Navigation concretization Control concretization

Design options for graphical web UIs Sub-task presentation: specifies the appearance of each sub-task in the final user interface. Sub-task presentation separated combined one by one combined box tabbed dialog box float window many at once float window all in one areas with separators group boxes bulleted list numbered list Abstract level Concrete level

Sub-task presentation: specifies the appearance of each sub-task in the final user interface.

Sub-task presentation – example Separated sub-task presentation Combined sub-task presentation T11 T12 T13 T1 AC11 AC12 AC13 AC1 T11 T12 T13 T1 AC11 AC12 AC13 NAC LHS RHS NAC LHS RHS

Separated sub-task presentation

Combined sub-task presentation

Design options for graphical web UIs Navigation concretization: identifies the placement and the cardinality of the objects/logically grouped set of objects that ensure the navigation. Navigation concretization [Stanciulescu,2006] placement cardinality local global simple multiple

Navigation concretization: identifies the placement and the cardinality of the objects/logically grouped set of objects that ensure the navigation.

Sub-task navigation - example Local placement for navigation objects Global placement for navigation object AC1 T11 T12 T13 T1 AC11 AC12 AC13 AIC111 AIC121 AIC131 AC1 T11 T12 T13 T1 AC11 AC12 AC13 AIC11 NAC LHS RHS NAC LHS RHS

Local placement for navigation objects

Global placement for navigation object

Design options for graphical web UIs Sub-task navigation : specifies the way in which the dialog control is transferred from one sub-task to another. Sub-task navigation asynchronous sequential (synchronous)

Sub-task navigation : specifies the way in which the dialog control is transferred from one sub-task to another.

Design options for graphical web UIs Concretization of navigation and control : specifies if the navigation and control are ensured or not by the same object. Concretization of navigation & control separated combined

Concretization of navigation and control : specifies if the navigation and control are ensured or not by the same object.

Design options for graphical web UIs Control concretization : specifies the placement and the cardinality of the objects that ensure the control. Control concretizations placement cardinality local global multiple simple

Control concretization : specifies the placement and the cardinality of the objects that ensure the control.

Design options for multimodal web UIs Consider the ergonomic criteria for the evaluation of human-computer interfaces Compatibility, consistency, work load, adaptation, dialog control, guidance, error management Associate a corresponding CARE property to each design value. Design options for multimodal UIs [Scapin & Bastien,1997] [Vanderdonckt,1995] Prompt Grouping / Distinction of items Immediate feedback Guidance

Consider the ergonomic criteria for the evaluation of human-computer interfaces

Compatibility, consistency, work load, adaptation, dialog control, guidance, error management

Associate a corresponding CARE property to each design value.

Design options for multimodal web UIs Prompt : refers to the interaction channels available in order to lead the users to take specific actions whether it be data entry or other tasks. vocal (A) graphical (A) multimodal (E, C, R)

Prompt : refers to the interaction channels available in order to lead the users to take specific actions whether it be data entry or other tasks.

vocal (A)

graphical (A)

multimodal (E, C, R)

Design options for multimodal web UIs Grouping/Distinction of items : concerns the organization of information in relation to one another. Input : any information input from the user to the system vocal (A) graphical (A) multimodal (E, C, R) Output : any information output from the system to the user vocal (A) graphical (A) multimodal (E, C, R)

Grouping/Distinction of items : concerns the organization of information in relation to one another.

Input : any information input from the user to the system

vocal (A)

graphical (A)

multimodal (E, C, R)

Output : any information output from the system to the user

vocal (A)

graphical (A)

multimodal (E, C, R)

Design options for multimodal web UIs Immediate feedback : concerns the system responses to users’ action vocal (A) graphical (A) multimodal (E, C, R)

Immediate feedback : concerns the system responses to users’ action

vocal (A)

graphical (A)

multimodal (E, C, R)

Design options for multimodal web UIs Guidance : refers to the means available to advise, orient, inform, instruct and guide the users throughout their interactions with the system Guidance for input: any guidance offered to the user in order to guide him with the input Textual (A), Iconic (A), Acoustic (A), Speech (A) Multimodal (E, C, R) Guidance for output: any guidance offered to the user in order to guide him with the feedback Textual (A), Iconic (A), Acoustic (A), Speech (A) Multimodal (E, C, R)

Guidance : refers to the means available to advise, orient, inform, instruct and guide the users throughout their interactions with the system

Guidance for input: any guidance offered to the user in order to guide him with the input

Textual (A), Iconic (A), Acoustic (A), Speech (A)

Multimodal (E, C, R)

Guidance for output: any guidance offered to the user in order to guide him with the feedback

Textual (A), Iconic (A), Acoustic (A), Speech (A)

Multimodal (E, C, R)

Multimodal example Multimodal text field Prompt: multimodal (R) Grouping for input: multimodal (E) Immediate feedback: multimodal (R) Guidance: Input: iconic (A) Output: iconic (A) [Stanciulescu,2006] Prompt: multimodal Please say your name (vocal + graphical) 1 Input: multimodal Juan Gonzalez 2 (vocal + graphical) Juan Gonzalez Guidance for input: iconic Guidance for feedback: iconic Feedback:multimodal Your answer was: Juan Gonzalez 3 (graphical + vocal)

Multimodal text field

Prompt: multimodal (R)

Grouping for input: multimodal (E)

Immediate feedback: multimodal (R)

Guidance:

Input: iconic (A)

Output: iconic (A)

Vocal Concrete Interaction Objects VocalContainers: VocalGroup VocalForm VocalMenu VocalConfirmation VocalIndividualComponents: VocalOutput: VocalFeedback VocalPrompt VocalMenuItem VocalInput VocalNavigation Break Exit EventTypes: error, help, noInput, noMatch

VocalContainers:

VocalGroup

VocalForm

VocalMenu

VocalConfirmation

VocalIndividualComponents:

VocalOutput:

VocalFeedback

VocalPrompt

VocalMenuItem

VocalInput

VocalNavigation

Break

Exit

EventTypes: error, help, noInput, noMatch

Vocal Concrete Interaction Objects Example: virtual ice cream shop

Example: virtual ice cream shop

Multimodal Concrete Interaction Objects MultimodalCIO: combination of graphical CIO and vocalCIO outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance groupBox + radioButtons vocalMenu + vocalItems + vocalInput groupBox + radioButtons + vocalMenu + vocalItems + vocalInput + guidance Graphical Vocal Multimodal

MultimodalCIO: combination of graphical CIO and vocalCIO

Concrete UI Relationships GraphicalRelationships GraphicalTransition GraphicalAdjacency GraphicalContainment VocalRelationships VocalTransition VocalAdjacency VocalContainment Synchronization: between vocalCIOs and graphicalCIOs

GraphicalRelationships

GraphicalTransition

GraphicalAdjacency

GraphicalContainment

VocalRelationships

VocalTransition

VocalAdjacency

VocalContainment

Synchronization: between vocalCIOs and graphicalCIOs

Tool support MultimodaliXML – five software modules: IdealXML TransformiXML GrafiXML CFB Generator XHTML+Voice Generator tool

MultimodaliXML – five software modules:

IdealXML

TransformiXML

GrafiXML

CFB Generator

XHTML+Voice Generator tool

Multimodal Concrete Interaction Objects Syntax example: multimodal text field with equivalent input and redundant output

Syntax example: multimodal text field

with equivalent input and redundant output

Specification of transformation Graph-based patterns Syntactically typed patterns Textual concrete syntax of patterns: UsiXML syntax

Graph-based patterns

Syntactically typed patterns

Textual concrete syntax of patterns: UsiXML syntax

How transformation rules work… LHS: Find an occurrence of LHS into G NAC: Check that NAC does not match into G. If there is a match then skip to another occurrence of LHS RHS: Replace LHS by RHS [Limbourg,2004] Transformation rule Initial Model Resulting Model

LHS: Find an occurrence of LHS into G

NAC: Check that NAC does not match into G. If there is a match then skip to another occurrence of LHS

RHS: Replace LHS by RHS

UsiXML syntax of transformation rules NAC NAC RHS mappings

Transformation rule NAC: LHS: RHS: Multimodal prompt Guidance for input Multimodal input Guidance for output Vocal feedback

Case studies 2 case studies: Virtual Polling system and Car rental system

2 case studies: Virtual Polling system and Car rental system

Conclusion Model-driven Engineering of Multimodal Web applications is possible Requirements fulfillment: Support for multimodal input/output CARE properties support for input/output modalities Approach based on design space composed of 13 design options Design space covers a subset of the total number of UIs which can be developed with X+V Methodology covers a subset of the UIs which can be developed with X+V

Model-driven Engineering of Multimodal Web applications is possible

Requirements fulfillment:

Support for multimodal input/output

CARE properties support for input/output modalities

Approach based on design space composed of 13 design options

Design space covers a subset of the total number of UIs which can be developed with X+V

Methodology covers a subset of the UIs which can be developed with X+V

Conclusion Advantages of combining design options into a design space: Vertues of a design space Descriptive Generative Comparative [Stanciulescu & Vanderdonckt,2006] [Beaudouin-Lafon,2004]

Advantages of combining design options into a design space:

Vertues of a design space

Descriptive

Generative

Comparative

Future work Investigate the implementation of a new transformation engine: any suggestion? Reinforce existing vocal components Extend the ontology with new vocal components Design space improvement Extend transformation catalogs Adaptation of multimodal web user interface to the context of use

Investigate the implementation of a new transformation engine: any suggestion?

Reinforce existing vocal components

Extend the ontology with new vocal components

Design space improvement

Extend transformation catalogs

Adaptation of multimodal web user interface to the context of use

References See videos and papers on MultimodaliXML at: http://www.usixml.org/index.php?view=page&idpage=51 Stanciulescu, A., Vanderdonckt, J., Design Options for Multimodal Web Applications, Proc. of 6th Int. Conf. on Computer-Aided Design of User Interfaces CADUI’2006 (Bucharest, 6-8 June 2006), Chapter 4, Springer-Verlag, Berlin, 2006, pp. 41-56 Stanciulescu, A., Limbourg, Q., Vanderdonckt, J., Michotte, B., Montero, F., A Transformational Approach for Multimodal Web User Interfaces based on UsiXML, Proc. of 7th ACM Conf. on Multimodal Interfaces ICMI’2005 (Trento, 4-6 October, 2005), ACM Press, New York, 2005, pp. 259-266

See videos and papers on MultimodaliXML at: http://www.usixml.org/index.php?view=page&idpage=51

Stanciulescu, A., Vanderdonckt, J., Design Options for Multimodal Web Applications, Proc. of 6th Int. Conf. on Computer-Aided Design of User Interfaces CADUI’2006 (Bucharest, 6-8 June 2006), Chapter 4, Springer-Verlag, Berlin, 2006, pp. 41-56

Stanciulescu, A., Limbourg, Q., Vanderdonckt, J., Michotte, B., Montero, F., A Transformational Approach for Multimodal Web User Interfaces based on UsiXML, Proc. of 7th ACM Conf. on Multimodal Interfaces ICMI’2005 (Trento, 4-6 October, 2005), ACM Press, New York, 2005, pp. 259-266

Thank you very much for your attention For more information and downloading, http://www.isys.ucl.ac.be/bchi http://www.usixml.org User Interface eXtensible Markup Language http://www.similar.cc European network on Multimodal UIs Special thanks to all members of the team!