1. Model-based Design and Generation of a
Gesture-based User Interface Navigation
Control
Sebastian Feuerstack, Mauro dos Santos Anjo, and Ednaldo Pizzolato
Dr.-Ing. Sebastian Feuerstack
Universidade Federal de São Carlos
Departamento de Computacão
4. November 2011 Sebastian Feuerstack I Sebastian@Feuerstack.org

2. Basic Question
How to model and run
multimodal interaction
prototype variants ?
4. November 2011 2

3. Why modeling?
• One (visual) language that
– is declarative and precise to discuss &
execute
– Is located between a tool and source code
– Supports different forms of interaction and
flexible prototyping of multimodal interfaces
4. November 2011 3

4. What is state of the art ?
1. Task AUI CUI FUI
CUI FUI
2. Widget
W1 W2
Alg.
A1 D1
Driver
4. November 2011 4

5. Use Case: Test and Evaluation
Gesture-based Interface Navigation
:IN:hand_gestures
NoHands
[one_hand] [no_hands]
OneHand
Navigation previous / next /
Predecessor Command
closer / farer closer /farer
Ticker tick previous wait_one
start_p
started tick previous
select confirmed
next
[1s]/tick previous
next selected [confirm]
Successor
tick next start_n
tick select select confirm
• Rapid model-based Design and Comparison
of three variants
4. November 2011 The Augmented “Drag-and-Drop 5

6. How to model?
• Models
– Interactors (Abstract & Concrete Media, Mode)
• Static: class diagram, Behavior: state-chart
– Mappings Mode-To-Media Synchronization
• Custom Notation
4. November 2011 6

7. Abstract Media Model
4. November 2011 7

8. Abstract Media Model
Single Choice
Aggregates a set of
Entities from that only
one can be chosen at
a time
Examples: Direction (left or right), Shopping Cart
Further Properties:
Container -> Aggregation, Discrete, Output to the User
Contains Single Choice Elements that are Inputs
4. November 2011 8

9. Abstract Behavior Model
AUI:AIO:AIChoiceElement:
AISingleChoiceElement
Presenting
drop listed
initialized next||prev||parent
focus
dragging defocus /aio=find(act);
organize aio.focus
drag focused
organized
present Selection
H
organize unchosen
[in(focused)] choose
suspend unchoose / aios=find(parent.childs.chosen);
aios.all.unchoose
chosen
suspended
4. November 2011 9

10. Mode Model
(Example: Gesture Interactor)
IR:IN:HandGesture:FlexibleTicker
NoHands
right_hand_appeared right_hand_disappeared
Right Hand
Navigation
Speed Predecessor Command
started [1s]/tick
[t]/tick
entry/start_ticker
tick previous wait_one
exit/stop_ticker previous start_p
tick
select confirmed
next
SpeedAdjustment previous
closer closer next selected confirm
normal faster fastest
entry/ entry/ entry/ Successor
t = 1200ms; farer t = 1000ms; farer t = 800ms;
restart_ticker restart_ticker restart_ticker tick next start_n select
tick
4. November 2011 10

11. Multimodal Mapping
(Combining Mode with concrete Media)
4. November 2011 11

12. Conclusions & Future Work
What’s the advantage?
• Detailed Modeling of
Multimodal
Interactions
including behavior
• Existing tools
• No gap between
design- and runtime
4. November 2011 12

13. Whats next?
Focus on: Visit our website
– Fusion – All papers
– Paradigm Design – Videos
– Formalization – Open Source
– Tools Software
– MINT Framework – MINT Framework
http://www.multi-access.de
4. November 2011 Design of Multimodal Interaction 13

14. 4. November 2011 14