Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Modelling of User Interfaces based on State- Charts to Ease the Test and Evaluation of Multimodal Interactions

678 views

Published on

New forms of interactions such as the gesture-based control of interfaces could enable interaction in situations where hardware controls are missing and support impaired people where other controls fails. The rich spectrum of combining hand postures with movements offers great interaction possibilities but requires extensive user testing to figure out a user interface navigation control with a sufficient performance and a low error rate. In this paper we describe a model-based interface design based on assembling interactors and multimodal mappings to design multimodal interfaces. We propose to use state charts for the rapid generation of different user interface controls to accelerate user testing. First we describe how our approach can be applied to design direct manipulation interfaces that rely on a mouse-based interface control. Then we present how we applied the interactor based design to quickly generate several variants of a gesture-based interface navigation control to demonstrate our approach. We comparethe three most promising variants in a user test and report about the test results.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Modelling of User Interfaces based on State- Charts to Ease the Test and Evaluation of Multimodal Interactions

  1. 1. Sebastian Feuerstack I Sebastian@Feuerstack.org29. Januar 2015 Modelling of User Interfaces based on State- Charts to Ease the Test and Evaluation of Multimodal Interactions Sebastian Feuerstack, Mauro Dos Santos Anjo, Jessica Colnago und Ednaldo Pizzolato Dr.-Ing. Sebastian Feuerstack Universidade Federal de São Carlos Departamento de Computacão
  2. 2. 229. Januar 2015 Basic Question How to model Interactions that span Modes and Media ?
  3. 3. 329. Januar 2015 Why modeling? • One (visual) language that – is declarative and precise to discuss & execute – Is located between Tool and Code – Supports different forms of interaction and flexible multimodal prototyping – Supports paradigm design
  4. 4. 429. Januar 2015 What is Out of scrope ? • No transformations, no process ! – Anticipation, long processes, complex transformations, foreign process – Instead: Interactor assembly like GUI building – System Design vs. Application Design • Tools – Visual language -> Tool support
  5. 5. The Augmented “Drag-and-Drop 529. Januar 2015 Use Case: Test and Evaluation Gesture-based Interface Navigation • Rapid model-based Design and Comparison of three variants :IN:hand_gestures NoHands [one_hand] wait_one previous select OneHand [no_hands] selected confirmed [confirm] select next next Navigation CommandPredecessor Ticker started [1s]/tick previous Successor previous next tick tick previous / closer / farer next / closer /farer select confirm start_p tick start_n tick
  6. 6. 629. Januar 2015 How to model? • Models – Interactors (Abstract & Concrete Media, Mode) • Static: class diagram, Behavior: state-chart – Mappings Mode-To-Media Synchronization • Custom Notation
  7. 7. 729. Januar 2015 Abstract Media Model
  8. 8. 829. Januar 2015 Abstract Media Model Single Choice Aggregates a set of Entities from that only one can be chosen at a time Examples: Direction (left or right), Shopping Cart Further Properties: Container -> Aggregation, Discrete, Output to the User Contains Single Choice Elements that are Inputs
  9. 9. 929. Januar 2015 Abstract Behavior Model AUI:AIO:AIChoiceElement: AISingleChoiceElement initialized listed focus suspended organize defocus organized suspend focused organize Presenting chosen unchosen drag dragging unchoose drop H [in(focused)] choose / aios=find(parent.childs.chosen); aios.all.unchoose next||prev||parent /aio=find(act); aio.focus Selectionpresent
  10. 10. 1029. Januar 2015 IR:IN:HandGesture:FlexibleTicker NoHands right_hand_appeared wait_one select Right Hand right_hand_disappeared selected confirmed confirm select Navigation Command SpeedAdjustment fastest entry/ t = 800ms; restart_ticker faster entry/ t = 1000ms; restart_ticker closer farer previous nextnormal entry/ t = 1200ms; restart_ticker closer farer previous next next Predecessor previous Successor tick tick start_p tick start_n tick Speed started entry/start_ticker exit/stop_ticker [1s]/tick IR:IN:HandGesture:FlexibleTicker NoHands right_hand_appeared Right Hand right_hand_disappeared IR:IN:HandGesture:FlexibleTicker NoHands right_hand_appeared Right Hand right_hand_disappeared Navigation Command IR:IN:HandGesture:FlexibleTicker NoHands right_hand_appeared wait_one select Right Hand right_hand_disappeared selected confirmed confirm Navigation Command IR:IN:HandGesture:FlexibleTicker NoHands right_hand_appeared wait_one select Right Hand right_hand_disappeared selected confirmed confirm select Navigation Command previous next previous next next Predecessor previous Successor tick tick start_p tick start_n tick IR:IN:HandGesture:FlexibleTicker NoHands right_hand_appeared wait_one select Right Hand right_hand_disappeared selected confirmed confirm select Navigation Command previous next previous next next Predecessor previous Successor tick tick start_p tick start_n tick Speed started entry/start_ticker exit/stop_ticker [t]/tick IR:IN:HandGesture:FlexibleTicker NoHands right_hand_appeared wait_one select Right Hand right_hand_disappeared selected confirmed confirm select Navigation Command SpeedAdjustment fastest entry/ t = 800ms; restart_ticker faster entry/ t = 1000ms; restart_ticker closer farer previous nextnormal entry/ t = 1200ms; restart_ticker closer farer previous next next Predecessor previous Successor tick tick start_p tick start_n tick Speed started entry/start_ticker exit/stop_ticker [t]/tick Mode Model (Example: Gesture Interactor)
  11. 11. 1129. Januar 2015 Multimodal Mapping (Combining Mode with concrete Media)
  12. 12. 1229. Januar 2015 Multimodal Mapping The Drag-and-Drop (abstract) C dst=SingleChoice.focused C Tw<0,3s sc.drag dst.drop(aios) sc=SingleChoiceElement.focused Tw<0,3sLeftButton.pressed LeftButton.released sc.parent.drop fail
  13. 13. 1329. Januar 2015 Conclusions & Future Work What’s the advantage? • Detailed Modeling of Multimodal Interactions (Media, Mode) including Behavior • Paradigm Design • No gap between Design- and Runtime • No foreign process
  14. 14. Whats next? Design of Multimodal Interaction 1429. Januar 2015 Visit our website – All papers – Videos – Open Source Software – MINT Framework http://www.multi-access.de Focus on: – Fusion – Paradigm Design – Formalization – Tools – MINT Framework
  15. 15. 1529. Januar 2015

×