6. Real life multimodal use
Sequential: Exclusive
-Sequential use of modalities that are not linked together in the same task. For example, recording speech and
then closing all the unnecessary windows with a mouse.
Sequential: Alternate
The use of modalities is sequential, but they are linked together in the same task. For example, pointing an
object with a mouse and giving a voice command to copy it.
Parallel: Concurrent
The use of modalities is parallel, but they are not linked together. Usually the hardest kind of multimodal
systems, but may result in a better efficiency with some training. For example, driving a car while speaking
in a mobile phone
Parallel: Synergistic
The use of modalities is parallel and they are linked together. For example, pointing and speech command at
the same time.
[Nigay and Coutaz]
7. Multimodal versus Multimedia
Multimodal systems use more than one sense/mode of interaction e.g.
visual and aural senses: a text processor may speak the words as well
as printing them to the screen
Multimedia systems use a number of different media to communicate
information e.g. an e-learning system may use video, animation, text
and still images and use speech and non-speech
Interaction with these systems The combination of human output
channels effectively increases the bandwidth of the human-
machine channel (e.g., Oviatt, 1999)
8. “Bandwith” in computer game history
Examples
text based adventures
adding graphics
print
Sound and space
3 d space
3d audio
haptic
9. Multimodal output
-Visual: Text, pictures, signs, color
-Audio: Verbal, non-verbal, signals
-Olfactory: Smell
-Haptic (combination of movement and tactile)
12. Taal van Geluid
Vanuit Film en Tekenfilm is er een
“geluidstaal” ontstaan, waar rekening mee
gehouden moet worden...
http://www.youtube.com/watch?v=7EeE2kWkFqA&fmt=18
http://www.youtube.com/watch?v=AQPSb2Piznk
13. Beeld en Geluid kunnen
-Informatie aanvullen
-ontkrachten
-emoties “versterken”
http://www.youtube.com/watch?v=9blFYZArEwA&fmt=18
18. Multimodalinteraction and multisenory feedback can exert significant cognitive demands on users. Attention is
selective and has limited capacity. Short term memory also has limitations. Consider human ability when designing
the interface, essential for control and learning affordances
Classic example
Confusion...
