1. User Interface Engineering Team
CLIPS-IMAG Laboratory, University of Grenoble
Mixed Reality:
A Model of Mixed Interaction
Céline Coutrix, Laurence Nigay

2. Introduction: Mixed Systems
• Merging
Physical and
Digital
Worlds
• In many
application
fields
1

3. Introduction: Designers Needs
A framework for :
Understanding and capitalizing on
Descriptive power
existing Mixed Systems
Creating designs
Generative power
Assessing multiple design
Comparative power
alternatives and increasing usability
2

4. Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
3

5. Outline
• Mixed Interaction Model
• Mixed object
• Mixed interaction
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
4

6. Example
5

7. Mixed Interaction Model: Mixed Object
mixed object
tool
task object
6

8. Mixed Interaction Model: Mixed Object
physical object
mixed object
digital object
physical
physical
digital
digital
properties
properties
properties
?
properties
• Interaction modality m = (d, l)
• speech = (microphone, natural language)
7

9. Mixed Interaction Model: Mixed Object
• Linking modality
mixed object
input
acquired
physical
input
device
language
data
physical
digital
properties
properties
output
generated
output
device
physical
language
data
8

10. Mixed Interaction Model: Mixed Object
mixed drawing
camera
image
computer
vision
physical
properties
shapes
projector
image
selection
9

11. Mixed Interaction Model: Mixed Object
a music bottle
lighting
pitch and
music
physical
screen
colored
volume
part
properties
table
sequence
analysis
loud
audio
identity
speakers
sequence
10

12. Mixed Interaction Model: Mixed Object
mixed object
physical
digital
properties
properties
Linking
modalities
tool
task object
11

13. Mixed Interaction Model: Mixed Interaction
tool
elementary
task
action
response
reaction
task object
feedback
12

14. Mixed Interaction Model: Mixed Interaction
• Interaction modality m = (d, l)
13

15. Mixed Interaction Model: Mixed Interaction
mixed tool
input
acquired
input
device
physical
language
data
physical
digital
properties
properties
generated
output
output
physical
language
device
data
interaction
interaction
output
input
language
language
task object
14

16. Mixed Interaction Model: Mixed Interaction
15

17. Mixed Interaction Model: Mixed Interaction
mixed tool
computer
camera
image
button
mixed tool
vision
pressed
paper
word,
button
place
on drawing
translation
drawing
16

18. Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
17

19. Descriptive Power
• Schemes for classification and better
understanding of mixed systems
18

20. Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
19

21. RAZZLE: Case Study
20

22. RAZZLE: Case Study
mixed piece
location
numeric
geometric
sensor
values
transforma°
Identifier
physical
orienta°
numeric
geometric
Location
properties
sensor
values
transforma°
Orientation
head
3D
mounted
image
projection
display
21

23. Generative Power
mixed tool
?
?
?
?
?
22

24. RAZZLE: Gestural Modality
23

25. RAZZLE: Gestural Modality
mixed tool
computer
camera
image
vision
physical
recognized
properties
gestures
translation
24

26. RAZZLE: Tactile Modality
25

27. RAZZLE: Tactile Modality
mixed tool
input
touchpad
data
language
physical
click
properties
translation
26

28. Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
27

29. Comparative Power
• Criteria within the model for assessing
multiple design alternatives
• Observability
• Spatial continuity
28

30. Comparative Power: Observability
mixed tool
input
acquired
input
device
physical
language
data
physical
digital
properties
properties
generated
output
output
physical
language
device
data
interaction
interaction
output
input
language
language
task object
input
acquired
input
device
physical
language
data
physical
digital
properties
properties
generated
output
output
physical
language
device
data
29

31. Comparative power: Observability
mixed tool
input
camera
image
language
physical
recognized
properties
gestures
translation
30

32. Comparative Power: Observability
mixed tool
input
touchpad
data
language
physical
click
properties
translation
31

33. Comparative Power: Spatial Continuity
mixed tool
input
acquired
input
device
physical
language
data
physical
digital
properties
properties
generated
output
output
physical
language
device
data
interaction
interaction
output
input
language
language
task object
input
acquired
input
device
physical
language
data
physical
digital
properties
properties
generated
output
output
physical
language
device
data
32

34. Comparative Power: Spatial Continuity
mixed tool
computer
camera
image
vision
physical
recognized
properties
gestures
translation
mixed piece
location
numeric
geometric
sensor
values
transforma°
Identifier
physical
orienta°
numeric
geometric
Location
properties
sensor
values
transforma°
Orientation
head
3D
mounted
image
projection
display
33

35. Comparative Power: Spatial Continuity
mixed tool
input
touchpad
data
language
physical
click
properties
translation
mixed piece
location
numeric
geometric
sensor
values
transforma°
Identifier
physical
orienta°
numeric
geometric
Location
properties
sensor
values
transforma°
Orientation
head
3D
mounted
image
projection
display
34

36. Outline
• Mixed Interaction Model
• Descriptive Power
• Generative and Comparative Powers
• Conclusion and Future Work
35

37. Conclusion
• Definition of a Mixed Interaction Model which
• Capitalizes existing approaches to describe mixed
systems
• Allows designers to create new interaction
modalities
• Allows them to choose between multiple design
alternatives
36

38. Future Work
• Improving comparative power by identifying more
criteria
37

39. Future Work
• Providing a development environment
based on ICARE
38

40. User Interface Engineering Team
CLIPS-IMAG Laboratory, University of Grenoble
Mixed Reality:
A Model of Mixed Interaction
Céline Coutrix, Laurence Nigay