With the introduction of new modes to control user interfaces like speech, gesture and multi-touch Post-WIMP interfaces increasingly substitute classical WIMP (windows, icons, menus, pointer) user interfaces. In this paper, we describe a toolchain of three connected tools to design and monitor custom Post-WIMP interactors using model-based user interface design (MBUID) methods. We use state charts and mappings as the basic modeling technique. Different to other MBUID approaches our models are get created to specify the behavior of existing widgets. These models are then used to attach further control modes and connect interactors using mappings. We explain the toolchain by an interactive music sheet web application that can be controlled by head movements to turn the sheets.

1. MINT-Composer – A Toolchain for the Model-
based Specification of Post-WIMP
Interactors
Sebastian Feuerstack
Dr.-Ing. Sebastian Feuerstack
Universidade Federal de São Carlos
Departamento de Computacão
18. Oktober 2012 Sebastian Feuerstack I Sebastian@Feuerstack.org

2. Basic Question
How to model and run
multimodal Post-WIMP
interactions?
18. Oktober 2012 2

3. Post-WIMP Interacton
• WIMP interaction style:
Window, Icon, Menu, Pointing device
• Post-WIMP Interfaces ”…attempt to appear
more natural and less obtrusive. Ideally this
kind of post-WIMP interaction feels as
familiar and fluid as using simple nondigital
artifacts” [Jetter 2012]
18. Oktober 2012 3

4. Interactive music sheet
18. Oktober 2012 4

5. Why modeling?
• Visual language that
– is declarative and precise to discuss &
execute
– Is located between a tool and source code
– Supports different forms of interaction and
flexible prototyping of multimodal interfaces
18. Oktober 2012 5

6. How to model?
AUI
Widget Mapping CUI
CUI
W W
Modality
D1
• Models
– Interactors (Abstract & Concrete Media, Mode)
• Static: class diagram, Behavior: state-chart
– Mappings
• Custom Notation
18. Oktober 2012 6

7. Development Process
18. Oktober 2012 7

8. Interactor Modeling
18. Oktober 2012 8

9. Mode Modeling
IR:Head
NoHead
discover
lost
Head Captured
right_tilted center
right
centered
left
left_tilted center
18. Oktober 2012 9

10. Demonstration
Home Network
MINT- Head
Platform Recognition
App
Download
WebServer Home Gateway End-Device
WebServer Webbrowser
18. Oktober 2012 The Augmented “Drag-and-Drop 10

11. Conclusions & Future Work
What’s the advantage?
• Detailed Modeling of
Multimodal
Interactions
including behavior
• Existing tools
• No gap between
design- and runtime
18. Oktober 2012 11

12. Whats next?
Focus on:
– Fusion
– Paradigm Design
– Formalization
– Tools
– W3C Standardization
http://www.multi-access.de
All papers, Videos, MINT Framework as Open Source
18. Oktober 2012 Design of Multimodal Interaction 12

13. 18. Oktober 2012 13