Model-Driven Engineering of User Interfaces for Multimodal Web Applications Jean Vanderdonckt Université catholique de Lou...
What is a multimodal user interface? <ul><li>Some terminology:  notion of mode </li></ul><ul><ul><li>We have 5 Modes corre...
What is a multimodal user interface? <ul><li>Combining several modes simultaneously: some are very hard! </li></ul>[Tison,...
What is a multimodal user interface? <ul><li>Some terminology:  notion of modality </li></ul><ul><ul><li>Definition: a par...
What is a multimodal user interface? <ul><li>Some terminology:  notion of media </li></ul><ul><ul><li>Definition: is a har...
Why are multimodal user interfaces important? <ul><li>Graphical user interfaces – classical human-computer interface parad...
Why are multimodal user interfaces challenging? <ul><li>For Web Engineering </li></ul><ul><ul><li>Since presentation and n...
Concepts: The CARE Properties <ul><li>From Amodeus European Project (1995!) </li></ul><ul><ul><li>Complementarity </li></u...
What are the appropriate abstractions? <ul><li>Design space  that supports effectively and efficiently the progress of the...
Why a design space? <ul><li>Advantages: </li></ul><ul><ul><li>Clarifies the development process in terms of options </li><...
Multimodal Web Applications <ul><li>The languages in which they are implemented (e.g., XHTML, VoiceXML, X+V): </li></ul><u...
Solution space <ul><li>Cameleon reference framework </li></ul><ul><li>UsiXML language: </li></ul><ul><ul><li>Graphical int...
Design options for graphical web UIs Design options for graphical UIs Sub-task  presentation Sub-task  navigation Concreti...
Design options for graphical web UIs <ul><li>Sub-task presentation: specifies the appearance of each sub-task in the final...
Sub-task presentation – example <ul><li>Separated sub-task presentation </li></ul><ul><li>Combined sub-task presentation <...
Design options for graphical web UIs <ul><li>Navigation concretization: identifies the placement and the cardinality of th...
Sub-task navigation - example <ul><li>Local placement for navigation objects </li></ul><ul><li>Global placement for naviga...
Design options for graphical web UIs <ul><li>Sub-task navigation : specifies the way in which the dialog control is transf...
Design options for graphical web UIs <ul><li>Concretization of navigation and control : specifies if the navigation and co...
Design options for graphical web UIs <ul><li>Control concretization : specifies the placement and the cardinality of the o...
Design options for multimodal web UIs <ul><li>Consider the ergonomic criteria for the evaluation of human-computer interfa...
Design options for multimodal web UIs <ul><li>Prompt : refers to the interaction channels available in order to lead the u...
Design options for multimodal web UIs <ul><li>Grouping/Distinction of items : concerns the organization of information in ...
Design options for multimodal web UIs <ul><li>Immediate feedback : concerns the system responses to users’ action </li></u...
Design options for multimodal web UIs <ul><li>Guidance : refers to the means available to advise, orient, inform, instruct...
Multimodal example <ul><li>Multimodal text field </li></ul><ul><ul><li>Prompt: multimodal (R) </li></ul></ul><ul><ul><li>G...
Vocal Concrete Interaction Objects <ul><li>VocalContainers: </li></ul><ul><ul><li>VocalGroup </li></ul></ul><ul><ul><li>Vo...
Vocal Concrete Interaction Objects <ul><li>Example: virtual ice cream shop </li></ul>
Multimodal Concrete Interaction Objects  <ul><li>MultimodalCIO: combination  of graphical CIO and vocalCIO </li></ul>outpu...
Concrete UI Relationships <ul><li>GraphicalRelationships </li></ul><ul><ul><li>GraphicalTransition </li></ul></ul><ul><ul>...
Tool support <ul><li>MultimodaliXML – five software modules: </li></ul><ul><ul><li>IdealXML </li></ul></ul><ul><ul><li>Tra...
Multimodal Concrete Interaction Objects <ul><li>Syntax example: multimodal text field  </li></ul><ul><li>with equivalent i...
Specification of transformation <ul><li>Graph-based patterns </li></ul><ul><li>Syntactically typed patterns </li></ul><ul>...
How transformation rules work… <ul><li>LHS: Find an occurrence of LHS into G </li></ul><ul><li>NAC: Check that NAC does no...
UsiXML syntax of transformation rules NAC NAC RHS mappings
Transformation rule NAC: LHS: RHS: Multimodal prompt Guidance for input Multimodal input Guidance for output Vocal feedback
Case studies <ul><li>2 case studies: Virtual Polling system and Car rental system </li></ul>
Conclusion <ul><li>Model-driven Engineering of Multimodal Web applications is possible </li></ul><ul><li>Requirements fulf...
Conclusion <ul><li>Advantages of combining design options into a design space: </li></ul><ul><ul><li>Vertues of a design s...
Future work <ul><li>Investigate the implementation of a new transformation engine: any suggestion? </li></ul><ul><li>Reinf...
References <ul><li>See videos and papers on MultimodaliXML at: http://www.usixml.org/index.php?view=page&idpage=51 </li></...
Thank you very much for your attention For more information and downloading, http://www.isys.ucl.ac.be/bchi   http://www.u...
Upcoming SlideShare
Loading in...5
×

Model-driven engineering of multimodal user interfaces

2,626

Published on

This presentation explains to use model-driven engineering techniques for producing multimodal user interfaces for the Web.

Published in: Business, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,626
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
158
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Model-driven engineering of multimodal user interfaces

  1. 1. Model-Driven Engineering of User Interfaces for Multimodal Web Applications Jean Vanderdonckt Université catholique de Louvain (UCL) Louvain School of Management (LSM) Information Systems Unit (ISYS) Belgian Laboratory of Computer-Human Interaction (BCHI) http://www.isys.ucl.ac.be/bchi
  2. 2. What is a multimodal user interface? <ul><li>Some terminology: notion of mode </li></ul><ul><ul><li>We have 5 Modes corresponding to our 5 human senses </li></ul></ul><ul><ul><li>The human body has five major senses which operate to gather information from the world around us, sight, hearing, smell, taste, and touch. Any stimulus to one of the sense areas is detected by sensory nerves and is sent to the brain for interpretation. </li></ul></ul>
  3. 3. What is a multimodal user interface? <ul><li>Combining several modes simultaneously: some are very hard! </li></ul>[Tison,2003]
  4. 4. What is a multimodal user interface? <ul><li>Some terminology: notion of modality </li></ul><ul><ul><li>Definition: a particular form of a mode </li></ul></ul><ul><ul><li>Example: for auditive mode, we have: speech, music, voice, chorals, noise </li></ul></ul><ul><ul><li>User interface for each mode thus involves one or many modalities </li></ul></ul><ul><ul><li>A system is said to be </li></ul></ul><ul><ul><ul><li>Monomodal iff only one modality is involved </li></ul></ul></ul><ul><ul><ul><li>Multimodal iff many modalities are involves </li></ul></ul></ul><ul><ul><li>Some modalities are still in their infancy: </li></ul></ul><ul><ul><ul><li>Olfactive user interfaces </li></ul></ul></ul><ul><ul><ul><li>Aromatic user interfaces </li></ul></ul></ul>[Bodnar,2004]
  5. 5. What is a multimodal user interface? <ul><li>Some terminology: notion of media </li></ul><ul><ul><li>Definition: is a hardware/software device allowing interaction between a user and a system according to a given set of modalities </li></ul></ul><ul><ul><li>Example: for auditive mode, we have several media possible: speech, music, voice, chorals, noise </li></ul></ul><ul><ul><li>User interface for each mode thus involves one or many modalities </li></ul></ul>[Bellik,2002]
  6. 6. Why are multimodal user interfaces important? <ul><li>Graphical user interfaces – classical human-computer interface paradigm </li></ul><ul><ul><li>Users with limited literacy and typing skills </li></ul></ul><ul><ul><li>Handheld devices </li></ul></ul><ul><ul><ul><li>Need for hands and eyes free </li></ul></ul></ul><ul><ul><ul><li>Difficult to support mouse and keyboard interaction </li></ul></ul></ul><ul><ul><li>Web access anyplace, anywhere, anytime </li></ul></ul><ul><li>Need for newer, better interface paradigm </li></ul><ul><ul><li>Multimodal user interface is a possible answer… </li></ul></ul><ul><ul><li>Adaptation to the context of use (user, platform, environment) </li></ul></ul><ul><ul><li>More natural and efficient human-computer interaction </li></ul></ul>[IntuiLab,2006]
  7. 7. Why are multimodal user interfaces challenging? <ul><li>For Web Engineering </li></ul><ul><ul><li>Since presentation and navigation are no longer visual only, how can we design vocal, tactile and multimodal presentation and navigation within a page or across pages? </li></ul></ul><ul><li>For Model-Driven Approach </li></ul><ul><ul><li>So far, little research has been devoted to modelling the various aspects involved in multimodal web applications. Therefore, there is a need for identifying the appropriate implementation characteristics which will be the turned into high level design options to be supported by model-to-model and model-to-code transformations </li></ul></ul><ul><li>For Human-Computer Interaction: </li></ul><ul><ul><li>Developing a user interface for any combination of graphical, vocal, and tactile interaction modalities remains a challenge because of synchronization issues in time and space </li></ul></ul><ul><li>For Usability Engineering </li></ul><ul><ul><li>It is not because we are able to develop a user interface with any combination of any modalities that the resulting user interface is usable or tractable for a end user. We know little about the usability guidelines to be respected in this area </li></ul></ul>
  8. 8. Concepts: The CARE Properties <ul><li>From Amodeus European Project (1995!) </li></ul><ul><ul><li>Complementarity </li></ul></ul><ul><ul><li>Assignment </li></ul></ul><ul><ul><li>Redundancy </li></ul></ul><ul><ul><li>Equivalence </li></ul></ul>Put that there [Bolt,1979]
  9. 9. What are the appropriate abstractions? <ul><li>Design space that supports effectively and efficiently the progress of the development life cycle towards a final system while assuring some form of quality </li></ul><ul><li>Design space consists of an n-dimensional space where each dimension is denoted by a single design option </li></ul><ul><li>Each design option has a finite set of design values </li></ul><ul><li>E.g., label location (left-aligned, centered, right-aligned) </li></ul>
  10. 10. Why a design space? <ul><li>Advantages: </li></ul><ul><ul><li>Clarifies the development process in terms of options </li></ul></ul><ul><ul><li>Facilitates the incorporation in the development life cycle of design options as an abstraction covered by a software </li></ul></ul><ul><ul><li>Supports the tractability of more complex design problems </li></ul></ul><ul><li>Shortcomings: </li></ul><ul><ul><li>In theory: design options could be very numerous (even infinite) </li></ul></ul><ul><ul><li>In practice: impossible to consider a large amount of design options </li></ul></ul><ul><ul><ul><li>Too complex or expensive to implement </li></ul></ul></ul><ul><ul><ul><li>Design options do not necessarily address users’ needs and requirements </li></ul></ul></ul><ul><ul><ul><li>Out of the designer’s scope of understanding, imagination or background </li></ul></ul></ul><ul><ul><ul><li>Design decision is not always clear (may violate some usability principles or guidelines) </li></ul></ul></ul>
  11. 11. Multimodal Web Applications <ul><li>The languages in which they are implemented (e.g., XHTML, VoiceXML, X+V): </li></ul><ul><ul><li>Restrict the amount of possible UIs to obtain </li></ul></ul><ul><ul><li>Set directly the CARE properties to Assignment, Equivalence and Redundancy </li></ul></ul><ul><ul><li>Limited to graphic, voice, and tactile modalities </li></ul></ul>[W3C,2006]
  12. 12. Solution space <ul><li>Cameleon reference framework </li></ul><ul><li>UsiXML language: </li></ul><ul><ul><li>Graphical interaction </li></ul></ul><ul><ul><li>Vocal interaction </li></ul></ul><ul><ul><li>Multimodal interaction </li></ul></ul><ul><li>Design option-based transformational approach: </li></ul><ul><ul><li>Step 1: Construct the Task & Domain Models </li></ul></ul><ul><ul><li>Step 2: From Task & Domain to Abstract User Interface (AUI) </li></ul></ul><ul><ul><li>Step 3: From AUI to Concrete User Interface (CUI) </li></ul></ul><ul><ul><li>Step 4: From CUI to FUI </li></ul></ul>[Stanciulescu et al.,2005]
  13. 13. Design options for graphical web UIs Design options for graphical UIs Sub-task presentation Sub-task navigation Concretization of navigation & control Navigation concretization Control concretization
  14. 14. Design options for graphical web UIs <ul><li>Sub-task presentation: specifies the appearance of each sub-task in the final user interface. </li></ul>Sub-task presentation separated combined one by one combined box tabbed dialog box float window many at once float window all in one areas with separators group boxes bulleted list numbered list Abstract level Concrete level
  15. 15. Sub-task presentation – example <ul><li>Separated sub-task presentation </li></ul><ul><li>Combined sub-task presentation </li></ul>T11 T12 T13 T1 AC11 AC12 AC13 AC1 T11 T12 T13 T1 AC11 AC12 AC13 NAC LHS RHS NAC LHS RHS
  16. 16. Design options for graphical web UIs <ul><li>Navigation concretization: identifies the placement and the cardinality of the objects/logically grouped set of objects that ensure the navigation. </li></ul>Navigation concretization [Stanciulescu,2006] placement cardinality local global simple multiple
  17. 17. Sub-task navigation - example <ul><li>Local placement for navigation objects </li></ul><ul><li>Global placement for navigation object </li></ul>AC1 T11 T12 T13 T1 AC11 AC12 AC13 AIC111 AIC121 AIC131 AC1 T11 T12 T13 T1 AC11 AC12 AC13 AIC11 NAC LHS RHS NAC LHS RHS
  18. 18. Design options for graphical web UIs <ul><li>Sub-task navigation : specifies the way in which the dialog control is transferred from one sub-task to another. </li></ul>Sub-task navigation asynchronous sequential (synchronous)
  19. 19. Design options for graphical web UIs <ul><li>Concretization of navigation and control : specifies if the navigation and control are ensured or not by the same object. </li></ul>Concretization of navigation & control separated combined
  20. 20. Design options for graphical web UIs <ul><li>Control concretization : specifies the placement and the cardinality of the objects that ensure the control. </li></ul>Control concretizations placement cardinality local global multiple simple
  21. 21. Design options for multimodal web UIs <ul><li>Consider the ergonomic criteria for the evaluation of human-computer interfaces </li></ul><ul><ul><li>Compatibility, consistency, work load, adaptation, dialog control, guidance, error management </li></ul></ul><ul><li>Associate a corresponding CARE property to each design value. </li></ul>Design options for multimodal UIs [Scapin & Bastien,1997] [Vanderdonckt,1995] Prompt Grouping / Distinction of items Immediate feedback Guidance
  22. 22. Design options for multimodal web UIs <ul><li>Prompt : refers to the interaction channels available in order to lead the users to take specific actions whether it be data entry or other tasks. </li></ul><ul><ul><li>vocal (A) </li></ul></ul><ul><ul><li>graphical (A) </li></ul></ul><ul><ul><li>multimodal (E, C, R) </li></ul></ul>
  23. 23. Design options for multimodal web UIs <ul><li>Grouping/Distinction of items : concerns the organization of information in relation to one another. </li></ul><ul><ul><li>Input : any information input from the user to the system </li></ul></ul><ul><ul><ul><li>vocal (A) </li></ul></ul></ul><ul><ul><ul><li>graphical (A) </li></ul></ul></ul><ul><ul><ul><li>multimodal (E, C, R) </li></ul></ul></ul><ul><ul><li>Output : any information output from the system to the user </li></ul></ul><ul><ul><ul><li>vocal (A) </li></ul></ul></ul><ul><ul><ul><li>graphical (A) </li></ul></ul></ul><ul><ul><ul><li>multimodal (E, C, R) </li></ul></ul></ul>
  24. 24. Design options for multimodal web UIs <ul><li>Immediate feedback : concerns the system responses to users’ action </li></ul><ul><ul><li>vocal (A) </li></ul></ul><ul><ul><li>graphical (A) </li></ul></ul><ul><ul><li>multimodal (E, C, R) </li></ul></ul>
  25. 25. Design options for multimodal web UIs <ul><li>Guidance : refers to the means available to advise, orient, inform, instruct and guide the users throughout their interactions with the system </li></ul><ul><ul><li>Guidance for input: any guidance offered to the user in order to guide him with the input </li></ul></ul><ul><ul><ul><li>Textual (A), Iconic (A), Acoustic (A), Speech (A) </li></ul></ul></ul><ul><ul><ul><li>Multimodal (E, C, R) </li></ul></ul></ul><ul><ul><li>Guidance for output: any guidance offered to the user in order to guide him with the feedback </li></ul></ul><ul><ul><ul><li>Textual (A), Iconic (A), Acoustic (A), Speech (A) </li></ul></ul></ul><ul><ul><ul><li>Multimodal (E, C, R) </li></ul></ul></ul>
  26. 26. Multimodal example <ul><li>Multimodal text field </li></ul><ul><ul><li>Prompt: multimodal (R) </li></ul></ul><ul><ul><li>Grouping for input: multimodal (E) </li></ul></ul><ul><ul><li>Immediate feedback: multimodal (R) </li></ul></ul><ul><ul><li>Guidance: </li></ul></ul><ul><ul><ul><li>Input: iconic (A) </li></ul></ul></ul><ul><ul><ul><li>Output: iconic (A) </li></ul></ul></ul>[Stanciulescu,2006] Prompt: multimodal Please say your name (vocal + graphical) 1 Input: multimodal Juan Gonzalez 2 (vocal + graphical) Juan Gonzalez Guidance for input: iconic Guidance for feedback: iconic Feedback:multimodal Your answer was: Juan Gonzalez 3 (graphical + vocal)
  27. 27. Vocal Concrete Interaction Objects <ul><li>VocalContainers: </li></ul><ul><ul><li>VocalGroup </li></ul></ul><ul><ul><li>VocalForm </li></ul></ul><ul><ul><li>VocalMenu </li></ul></ul><ul><ul><li>VocalConfirmation </li></ul></ul><ul><li>VocalIndividualComponents: </li></ul><ul><ul><li>VocalOutput: </li></ul></ul><ul><ul><ul><li>VocalFeedback </li></ul></ul></ul><ul><ul><ul><li>VocalPrompt </li></ul></ul></ul><ul><ul><ul><li>VocalMenuItem </li></ul></ul></ul><ul><ul><li>VocalInput </li></ul></ul><ul><ul><li>VocalNavigation </li></ul></ul><ul><ul><li>Break </li></ul></ul><ul><ul><li>Exit </li></ul></ul><ul><li>EventTypes: error, help, noInput, noMatch </li></ul>
  28. 28. Vocal Concrete Interaction Objects <ul><li>Example: virtual ice cream shop </li></ul>
  29. 29. Multimodal Concrete Interaction Objects <ul><li>MultimodalCIO: combination of graphical CIO and vocalCIO </li></ul>outputText + inputText vocalPrompt + vocalInput outputText + inputText + vocalInput + vocalFeedback + guidance groupBox + radioButtons vocalMenu + vocalItems + vocalInput groupBox + radioButtons + vocalMenu + vocalItems + vocalInput + guidance Graphical Vocal Multimodal
  30. 30. Concrete UI Relationships <ul><li>GraphicalRelationships </li></ul><ul><ul><li>GraphicalTransition </li></ul></ul><ul><ul><li>GraphicalAdjacency </li></ul></ul><ul><ul><li>GraphicalContainment </li></ul></ul><ul><li>VocalRelationships </li></ul><ul><ul><li>VocalTransition </li></ul></ul><ul><ul><li>VocalAdjacency </li></ul></ul><ul><ul><li>VocalContainment </li></ul></ul><ul><li>Synchronization: between vocalCIOs and graphicalCIOs </li></ul>
  31. 31. Tool support <ul><li>MultimodaliXML – five software modules: </li></ul><ul><ul><li>IdealXML </li></ul></ul><ul><ul><li>TransformiXML </li></ul></ul><ul><ul><li>GrafiXML </li></ul></ul><ul><ul><li>CFB Generator </li></ul></ul><ul><ul><li>XHTML+Voice Generator tool </li></ul></ul>
  32. 32. Multimodal Concrete Interaction Objects <ul><li>Syntax example: multimodal text field </li></ul><ul><li>with equivalent input and redundant output </li></ul>
  33. 33. Specification of transformation <ul><li>Graph-based patterns </li></ul><ul><li>Syntactically typed patterns </li></ul><ul><li>Textual concrete syntax of patterns: UsiXML syntax </li></ul>
  34. 34. How transformation rules work… <ul><li>LHS: Find an occurrence of LHS into G </li></ul><ul><li>NAC: Check that NAC does not match into G. If there is a match then skip to another occurrence of LHS </li></ul><ul><li>RHS: Replace LHS by RHS </li></ul>[Limbourg,2004] Transformation rule Initial Model Resulting Model
  35. 35. UsiXML syntax of transformation rules NAC NAC RHS mappings
  36. 36. Transformation rule NAC: LHS: RHS: Multimodal prompt Guidance for input Multimodal input Guidance for output Vocal feedback
  37. 37. Case studies <ul><li>2 case studies: Virtual Polling system and Car rental system </li></ul>
  38. 38. Conclusion <ul><li>Model-driven Engineering of Multimodal Web applications is possible </li></ul><ul><li>Requirements fulfillment: </li></ul><ul><ul><li>Support for multimodal input/output </li></ul></ul><ul><ul><li>CARE properties support for input/output modalities </li></ul></ul><ul><ul><li>Approach based on design space composed of 13 design options </li></ul></ul><ul><ul><li>Design space covers a subset of the total number of UIs which can be developed with X+V </li></ul></ul><ul><li>Methodology covers a subset of the UIs which can be developed with X+V </li></ul>
  39. 39. Conclusion <ul><li>Advantages of combining design options into a design space: </li></ul><ul><ul><li>Vertues of a design space </li></ul></ul><ul><ul><li>Descriptive </li></ul></ul><ul><ul><li>Generative </li></ul></ul><ul><ul><li>Comparative </li></ul></ul>[Stanciulescu & Vanderdonckt,2006] [Beaudouin-Lafon,2004]
  40. 40. Future work <ul><li>Investigate the implementation of a new transformation engine: any suggestion? </li></ul><ul><li>Reinforce existing vocal components </li></ul><ul><li>Extend the ontology with new vocal components </li></ul><ul><li>Design space improvement </li></ul><ul><li>Extend transformation catalogs </li></ul><ul><li>Adaptation of multimodal web user interface to the context of use </li></ul>
  41. 41. References <ul><li>See videos and papers on MultimodaliXML at: http://www.usixml.org/index.php?view=page&idpage=51 </li></ul><ul><li>Stanciulescu, A., Vanderdonckt, J., Design Options for Multimodal Web Applications, Proc. of 6th Int. Conf. on Computer-Aided Design of User Interfaces CADUI’2006 (Bucharest, 6-8 June 2006), Chapter 4, Springer-Verlag, Berlin, 2006, pp. 41-56 </li></ul><ul><li>Stanciulescu, A., Limbourg, Q., Vanderdonckt, J., Michotte, B., Montero, F., A Transformational Approach for Multimodal Web User Interfaces based on UsiXML, Proc. of 7th ACM Conf. on Multimodal Interfaces ICMI’2005 (Trento, 4-6 October, 2005), ACM Press, New York, 2005, pp. 259-266 </li></ul>
  42. 42. Thank you very much for your attention For more information and downloading, http://www.isys.ucl.ac.be/bchi http://www.usixml.org User Interface eXtensible Markup Language http://www.similar.cc European network on Multimodal UIs Special thanks to all members of the team!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×