Designing and Executing Multimodal Interfaces for the Web based on State Chart XML

570 views

Published on

Our daily life is full of activities: working, cooking, cleaning, driving are just some of them. While performing these activities our communicating capabilities are limited since most of our senses are blocked by the main activity. Thus we cannot benefit from accessing the computer or using our hands and eyes for driving or our voice for talking to other people. In this paper we invest into enabling and designing human-computer interaction in situations where common modes of communication fail. Therefore we allow controlling an interface just by using head movements and propose a model-based interface design that enables to design and implement such interactions. As a proof of concept we explain our approach based on a natural interface for a musician that we have implemented. It enables to turn music sheets just by head movements.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Designing and Executing Multimodal Interfaces for the Web based on State Chart XML

  1. 1. Designing and Executing Multimodal Interfaces for the Web based on State Chart XML Sebastian Feuerstack , Jessica H. Colnago and César R. De Souza 18. November 2011 Dr.-Ing. Sebastian Feuerstack Universidade Federal de São Carlos Departamento de Computacão
  2. 2. <ul><li>Basic Challenge </li></ul><ul><li>How to enable Multimodal Interfaces for the Web ? </li></ul>18. November 2011
  3. 3. <ul><li>Multi-Touch </li></ul><ul><li>Server-sided speech processing </li></ul><ul><ul><li>Google Voice Search, SIRI, Nuance Speech API </li></ul></ul><ul><li>Sound API </li></ul><ul><li>Native API </li></ul><ul><li>CSS3 Speech, Media Query, … </li></ul><ul><li>Device Orientation </li></ul>18. November 2011 What`s the new Hype?
  4. 4. <ul><li>Distribution </li></ul><ul><li>Migration </li></ul><ul><li>Adaptation </li></ul><ul><ul><li>(Ok, there is Content Selection & Device Discovery – but who uses more than the Browser UA?) </li></ul></ul><ul><li>Meta-UI </li></ul><ul><li>=> Natural Interaction </li></ul>18. November 2011 What is missing?
  5. 5. <ul><li>Developing Methods </li></ul><ul><ul><li>How do I design and implement complex Interactions ? (Native API…?) </li></ul></ul><ul><li>Free Tools </li></ul><ul><ul><li>What supports implementation and Debugging (Javascript Console?) </li></ul></ul><ul><li>Open Source Implementation </li></ul><ul><ul><li>Apache.org success </li></ul></ul><ul><li>Missing Standards </li></ul><ul><ul><li>W3C Model-Based User Interfaces (MBUI) Working Group </li></ul></ul>18. November 2011 What`s the barrier?
  6. 6. 18. November 2011 Multimodal Interaction Framework
  7. 7. <ul><li>Implemented </li></ul><ul><li>Use Cases </li></ul>18. November 2011
  8. 8. <ul><li>Rapid model-based Design and Comparison of three variants </li></ul><ul><ul><li> Published: HCII 2011, MoBe 2011, I-Com Journal </li></ul></ul>The Augmented “Drag-and-Drop 18. November 2011 Use Case I: Test and Evaluation Gesture-based Interface Navigation
  9. 9. Gesture-based Interface Navigation 18. November 2011
  10. 10. <ul><li>Furniture Drag and Drop </li></ul><ul><ul><li>Published: SVR 2011 </li></ul></ul>18. November 2011 Use Case II: Paradigm Design Augmented Drag and Drop Cam Cam Pointing Command Monitor Jessica
  11. 11. Gesture-based Drag and Drop Control 18. November 2011
  12. 12. <ul><li>Evaluation of controling a music sheet by using head movements </li></ul>18. November 2011 Use Case III: Test and Evaluation Head-based Interface Control
  13. 13. <ul><li>Two concepts </li></ul><ul><ul><li>Interactors </li></ul></ul><ul><ul><li>Mappings </li></ul></ul><ul><li>3 steps </li></ul><ul><ul><li>Widget Design </li></ul></ul><ul><ul><li>Interacton Design </li></ul></ul><ul><ul><li>Mapping </li></ul></ul>18. November 2011 SCXML-based Interactor Design
  14. 14. 1) Interactive Music Sheet Widget 18. November 2011 The Augmented “Drag-and-Drop
  15. 15. 2) Interaction Design 18. November 2011
  16. 16. 3) Mapping 18. November 2011
  17. 17. Thanks for you interest! <ul><li>On our website : </li></ul><ul><ul><li>All articles </li></ul></ul><ul><ul><li>Videos </li></ul></ul><ul><ul><li>Open Source Software </li></ul></ul><ul><ul><li>MINT Framework </li></ul></ul><ul><li>Focus on: </li></ul><ul><ul><li>Fusion </li></ul></ul><ul><ul><li>Paradigm Design </li></ul></ul><ul><ul><li>Formalization </li></ul></ul><ul><ul><li>Tools </li></ul></ul><ul><ul><li>MINT Framework </li></ul></ul>Design of Multimodal Interaction 18. November 2011 http://www.multi-access.de Visit our website!
  18. 18. 18. November 2011

×