Are Sensory Effects ready for the World Wide Web?

Are Sensory EffectsReady for the World Wide Web?
Christian Timmerer, Markus Waltl, and Hermann Hellwagner
Klagenfurt University (UNIKLU)  Faculty of Technical Sciences (TEWI)
Department of Information Technology (ITEC)  Multimedia Communication (MMC)
http://research.timmerer.com  http://blog.timmerer.com  mailto:christian.timmerer@itec.uni-klu.ac.at
20 May 2010
Acknowledgments. This work was supported in part by the European Commission in the context of the NoE INTERMEDIA (NoE 038419), the P2P-Next project (FP7-ICT-216217), and the ALICANTE project (FP7-ICT-248652).

Outline
Background / Introduction
MPEG-V Media Context and Control
Concept of MPEG-V Sensory Information
Sensory Effect Description Language (SEDL)
Software/Hardware components: SEVino, SESim, SEMP, and amBX+SDK
Are Sensory Effects Ready for the World Wide Web?
How to embed SEM into Web documents?
How to interpret SEM within a browser?
How to synchronize with audio/video content of the Web document?
How to extract color information automatically?
Conclusions and Future Work
2010/05/20
2
Christian Timmerer, Klagenfurt University, Austria

MPEG-V: Media Context and Control (MCC)
2010/05/20
3
System Architecture
Pt. 1: Architecture
Pt. 2: Control Information
Pt. 3: Sensory Information
Pt. 4: Virtual World Object Characteristics
Pt. 5: Data Formats for Interaction Devices
Pt. 6: Common Types and Tools
Pt. 7: Conformance and Reference Software
http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V

Concept of MPEG-V Sensory Information
Consumption of multimedia content may stimulate also other senses
Vision or audition
Olfaction, mechanoreception, equilibrioception, thermoception, …
Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects
2010/05/20
4
… giving her/him the sensation of being part of the particular media
➪ worthwhile, informative user experience

Sensory Effect Description Language (SEDL)
XML Schema-based language for describing sensory effects
Basic building blocks to describe, e.g., light, wind, fog, vibration, scent
MPEG-V Part 3, Sensory Information
Adopted MPEG-21 DIA tools for adding time information (synchronization)
Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV)
Extensibility: additional effects can be added easily w/o affecting SEDL
Flexibility: each application domain may define its own sensory effects
Description conforming to SEDL :== Sensory Effect Metadata (SEM)
May be associated to any kind of multimedia content (e.g., movies, music, Web sites, games)
Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate mediation device
➪ Increase the experience of the user
➪ Worthwhile, informative user experience
2010/05/20
5

Sensory Effect Description Language (cont’d)
2010/05/20
6
SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects|Effect|ReferenceEffect)+
Declarations ::= (GroupOfEffects|Effect|Parameter)+
GroupOfEffects ::= timestamp EffectDefinitionEffectDefinition (EffectDefinition)*
Effect ::= timestamp EffectDefinition
EffectDefinition ::= [activate][duration][fade][alt] [priority][intensity][position]
[adaptability]

SEVino, SESim, SEMP, and amBX
amBX (Ambient Experience) system + SDK
Two fan devices, a wrist rumbler, two sound speakers, a subwoofer, two lights, and a wall washer
Everything controlled by SEM descriptionsexcept light effect
2010/05/20
7
Annotation Tool: SEVino
Simulator: SESim
Player: SEMP

Are Sensory Effects Ready for the World Wide Web?
How to embed SEM into Web documents?
How to interpret/parse SEM within a browser?
Directly via JavaScript
Browser plug-in
How to synchronize with audio/video content of the Web document?
HTML5’s video and audio elements and corresponding DOM attributes
2010/05/20
8
<link rel="alternate" type="text/xml”
href="http://server.org/sem.xml”media="light, wind, vibration"/>

Are Sensory Effects Ready for the World Wide Web? (cont’d)
Synchronization with audio/video (cont’d)
currentTimeDOM attribute: current playback position in seconds
timeupdateevent: indicates that the current playback position changed as part of normal playback
2010/05/20
9

Are Sensory Effects Ready for the World Wide Web? (cont’d)
How to extract color information automatically?
Used for controlling ambient lights
currentFramedoes neither exist nor is foreseen in HTML5
canvas element allows for manipulating the display’s content
(+) Pro: extract the pixel information from the display
(-) Con: needs to be added explicitly during authoring
2010/05/20
10

Conclusions
Introduction to MPEG-V Sensory Effects
Are Sensory Effects Ready for the World Wide Web? – Yes, they are!
Future work
Implementation is ongoing
Subjective quality assessments
2010/05/20
11

Thank you for your attention
... questions, comments, etc. are welcome …
Ass.-Prof. Dipl.-Ing. Dr. Christian Timmerer
Klagenfurt University, Department of Information Technology (ITEC)
Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA
christian.timmerer@itec.uni-klu.ac.at
http://research.timmerer.com/
Tel: +43/463/2700 3621 Fax: +43/463/2700 3699
© Copyright: Christian Timmerer
12
2010/05/20

http://multimediacommunication.blogspot.com	28
http://www.slideshare.net	14
http://hghltd.yandex.net	2
http://www.lmodules.com	1
http://multimediacommunication.blogspot.com.br	1
http://translate.googleusercontent.com	1

Are Sensory Effects ready for the World Wide Web?

by Alpen-Adria-Universität Klagenfurt on May 20, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

6 Embeds 47

Statistics

Are Sensory Effects ready for the World Wide Web? — Presentation Transcript