Interfacing with Virtual Worlds An Introduction to MPEG-V Christian Timmerer Klagenfurt University (UNIKLU)  Faculty of Technical Sciences (TEWI) Department of Information Technology (ITEC)  Multimedia Communication (MMC) http://research.timmerer.com  http://blog.timmerer.com  mailto:christian.timmerer@itec.uni-klu.ac.at Authors : Christian Timmerer, Jean Gelissen, Markus Waltl, and Hermann Hellwagner Slides available at http://www.slideshare.net/christian.timmerer

Outline Introduction Part 1: System Architecture Overview of MPEG-V Parts 2 and 4 Part 3: Sensory Information Concept Sensory Effect Description Language Sensory Effect Vocabulary + Usage Examples (cf. paper) Conclusions (Demo Video) 2009/09/30 Christian Timmerer, Klagenfurt University, Austria

Introduction

Part 1: System Architecture

Overview of MPEG-V Parts 2 and 4

Part 3: Sensory Information

Concept

Sensory Effect Description Language

Sensory Effect Vocabulary + Usage Examples (cf. paper)

Conclusions

(Demo Video)

Introduction Multi-user online virtual worlds (NVE, MMOG) reached mainstream popularity e.g., World of Warcraft, Second Life, Lineage Boost real world economy by connecting virtual and real world? – Not only Gaming Entertainment, education, training, getting information, social interaction, work, virtual tourism, etc. For fast adoption of virtual worlds we need a better understanding of their internal economics , rules and regulations 2009/09/30 Christian Timmerer, Klagenfurt University, Austria

Multi-user online virtual worlds (NVE, MMOG) reached mainstream popularity

e.g., World of Warcraft, Second Life, Lineage

Boost real world economy by connecting virtual and real world? – Not only Gaming

Entertainment, education, training, getting information, social interaction, work, virtual tourism, etc.

For fast adoption of virtual worlds we need a better understanding of their internal economics , rules and regulations

Introduction (cont’d) Finally, interoperability achieved through standardization MPEG-V (ISO/IEC 23005) :== system architecture + associated information representations Interoperability between virtual worlds E.g., digital content provider of a virtual world (serious) gaming, simulation, DVD And real world E.g., sensors, actuators, vision and rendering, robotics (e.g. for revalidation), (support for) independent living, social and welfare systems, banking, insurance, travel, real estate, rights management 2009/09/30 Christian Timmerer, Klagenfurt University, Austria

Finally, interoperability achieved through standardization

MPEG-V (ISO/IEC 23005) :== system architecture + associated information representations

Interoperability between virtual worlds

E.g., digital content provider of a virtual world (serious) gaming, simulation, DVD

And real world

E.g., sensors, actuators, vision and rendering, robotics (e.g. for revalidation), (support for) independent living, social and welfare systems, banking, insurance, travel, real estate, rights management

MPEG-V System Architecture 2009/09/30 Christian Timmerer, Klagenfurt University, Austria Media context and control Pt. 1: Architecture Pt. 3: Sensory Information Pt. 4: Avatar Information Pt. 2: Control Information

Part 2: Control Information 2009/09/30 Christian Timmerer, Klagenfurt University, Austria Sensory Device Capabilities as ext. of dia:TerminalCapability unit, max/minIntensity, numOfLevels, delay, position light (color, flash), heating, cooling, wind, vibration scent, fog, water sprayer, color correction kinestetic, tactile User Sensory Preferences as ext. of dia:UserCharacteristics adaptability, max/minIntensity light (color, flash), heating, cooling, wind, vibration scent, fog, water sprayer, color correction kinestetic, tactile Fundamental Input to any Control Device (aka Adaptation Engine)

Sensory Device Capabilities as ext. of dia:TerminalCapability

unit, max/minIntensity, numOfLevels, delay, position

light (color, flash), heating, cooling, wind, vibration

scent, fog, water sprayer, color correction

kinestetic, tactile

User Sensory Preferences as ext. of dia:UserCharacteristics

adaptability, max/minIntensity

light (color, flash), heating, cooling, wind, vibration

scent, fog, water sprayer, color correction

kinestetic, tactile

Part 4: Avatar Characteristics Appearance Contains the high level description of the appearance and may refer a media containing the exact geometry and texture Haptics Properties Contains the high level description of the haptics properties Animation Contains the description of a set of animation sequences that the avatar is able to perform and may refer to several medias containing the exact (geometric transformations) animation parameters Communication Skills Contains a set of descriptors providing information on the different modalities an avatar is able to communicate Personality Contains a set of descriptors defining the personality of the avatar Control Contains a set of descriptors defining possible place-holders for sensors on body skeleton and face feature points 2009/09/30 Christian Timmerer, Klagenfurt University, Austria

Appearance

Contains the high level description of the appearance and may refer a media containing the exact geometry and texture

Haptics Properties

Contains the high level description of the haptics properties

Animation

Contains the description of a set of animation sequences that the avatar is able to perform and may refer to several medias containing the exact (geometric transformations) animation parameters

Communication Skills

Contains a set of descriptors providing information on the different modalities an avatar is able to communicate

Personality

Contains a set of descriptors defining the personality of the avatar

Control

Contains a set of descriptors defining possible place-holders for sensors on body skeleton and face feature points

Part 3: Sensory Information Universal Multimedia Access (UMA) Anywhere, anytime, any device + technically feasible Main focus on devices and network connectivity issues Universal Multimedia Experience (UME) Take the user into account Multimedia Adaptation and Quality Models/Metrics Single modality (i.e., audio, image, or video only) or a simple combination of two modalities (i.e., audio and video) Triple user characterization model Sensorial , e.g., sharpness, brightness Perceptual , e.g., what/where is the content Emotional , e.g., feeling, sensation Ambient Intelligence Add’l light effects are highly appreciated for both audio and visual content Calls for a scientific framework to capture , measure , quantify , judge , and explain the user experience 2009/09/30 Christian Timmerer, Klagenfurt University, Austria F. Pereira, “A triple user characterization model for video adaptation and quality of experience evaluation,” Proc. of the 7th Workshop on Multimedia Signal Processing , Shanghai, China, October 2005, pp. 1 – 4. B. de Ruyter, E. Aarts. “Ambient intelligence: visualizing the future”, Proceedings of the Working Conference on Advanced Visual Interfaces, New York, NY, USA, 2004, pp. 203–208. E. Aarts, B. de Ruyter, “New research perspectives on Ambient Intelligence”, Journal of Ambient Intelligence and Smart Environments, IOS Press, vol. 1, no. 1, 2009, pp. 5–14.

Universal Multimedia Access (UMA)

Anywhere, anytime, any device + technically feasible

Main focus on devices and network connectivity issues

Universal Multimedia Experience (UME)

Take the user into account

Multimedia Adaptation and Quality Models/Metrics

Single modality (i.e., audio, image, or video only) or a simple combination of two modalities (i.e., audio and video)

Triple user characterization model

Sensorial , e.g., sharpness, brightness

Perceptual , e.g., what/where is the content

Emotional , e.g., feeling, sensation

Ambient Intelligence

Add’l light effects are highly appreciated for both audio and visual content

Calls for a scientific framework to capture , measure , quantify , judge , and explain the user experience

Concept of MPEG-V Sensory Information Consumption of multimedia content may stimulate also other senses Vision or audition Olfaction , mechanoreception , equilibrioception , thermoception , … Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects 2009/09/30 Christian Timmerer, Klagenfurt University, Austria … giving her/him the sensation of being part of the particular media ➪ worthwhile, informative user experience

Consumption of multimedia content may stimulate also other senses

Vision or audition

Olfaction , mechanoreception , equilibrioception , thermoception , …

Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects

Sensory Effect Description Language (SEDL) XML Schema-based language for describing sensory effects Basic building blocks to describe, e.g., light, wind, fog, vibration, scent MPEG-V Part 3, Sensory Information Adopted MPEG-21 DIA tools for adding time information (synchronization) Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV) Extensibility : additional effects can be added easily w/o affecting SEDL Flexibility : each application domain may define its own sensory effects Description conforming to SEDL :== Sensory Effect Metadata (SEM) May be associated to any kind of multimedia content (e.g., movies, music, Web sites, games) Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate mediation device ➪ Increase the experience of the user ➪ Worthwhile, informative user experience 2009/09/30 Christian Timmerer, Klagenfurt University, Austria

XML Schema-based language for describing sensory effects

Basic building blocks to describe, e.g., light, wind, fog, vibration, scent

MPEG-V Part 3, Sensory Information

Adopted MPEG-21 DIA tools for adding time information (synchronization)

Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV)

Extensibility : additional effects can be added easily w/o affecting SEDL

Flexibility : each application domain may define its own sensory effects

Description conforming to SEDL :== Sensory Effect Metadata (SEM)

May be associated to any kind of multimedia content (e.g., movies, music, Web sites, games)

Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate mediation device

➪ Increase the experience of the user

➪ Worthwhile, informative user experience

Sensory Effect Description Language (cont’d) 2009/09/30 Christian Timmerer, Klagenfurt University, Austria EffectDefinition ::= [activate][duration][fade][alt] [priority][intensity][position] [adaptability] SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects| Effect|ReferenceEffect)+ Declarations ::= (GroupOfEffects|Effect|Parameter)+ GroupOfEffects ::= timestamp EffectDefinition EffectDefinition (EffectDefinition)* Effect ::= timestamp EffectDefinition

Example 2009/09/30 Christian Timmerer, Klagenfurt University, Austria < sedl:GroupOfEffects si:pts=&quot;3240000&quot; duration =&quot;100&quot; fade =&quot;15&quot; position =&quot; urn:mpeg:mpeg-v:01-SI-PositionCS-NS:center:*:front &quot; > < sedl:Effect xsi:type=&quot; sev:WindType&quot; intensity=&quot;0.0769&quot;/> < sedl:Effect xsi:type=&quot; sev:VibrationType &quot; intensity=&quot;0.56&quot;/> < sedl:Effect xsi:type=&quot; sev:LightType&quot; intensity=&quot;0.0000077&quot;/> </sedl:GroupOfEffects>

Conclusions MPEG-V: Media Context and Control Information exchange between Virtual Worlds Information exchange between Virtual and Real Worlds Currently comprises four parts (more to come, e.g., refsw, conf) MPEG-V Part 3: Sensory Information Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects ➪ enhanced , worthwhile , and informative user experience , giving the user the sensation of being part of the actual media Future work Standardization: currently at CD level & going to FCD in October 2009 Research & Development: Optimized and efficient delivery framework for MPEG-V enabled content New Quality of Service/Experience metrics Mechanism for (semi-)automatic generation of MPEG-V metadata End-to-end reference implementation of MPEG-V 2009/09/30 Christian Timmerer, Klagenfurt University, Austria

MPEG-V: Media Context and Control

Information exchange between Virtual Worlds

Information exchange between Virtual and Real Worlds

Currently comprises four parts (more to come, e.g., refsw, conf)

MPEG-V Part 3: Sensory Information

Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects

➪ enhanced , worthwhile , and informative user experience , giving the user the sensation of being part of the actual media

Future work

Standardization: currently at CD level & going to FCD in October 2009

Research & Development:

Optimized and efficient delivery framework for MPEG-V enabled content

New Quality of Service/Experience metrics

Mechanism for (semi-)automatic generation of MPEG-V metadata

End-to-end reference implementation of MPEG-V

References M. Waltl, C. Timmerer, and H. Hellwagner, “A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects”, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009) , San Diego, USA, July 29-31, 2009. C. Timmerer, J. Gelissen, M. Waltl, and H. Hellwagner, “Interfacing with Virtual Worlds”, accepted for publication in the Proceedings of the 2009 NEM Summit , Saint-Malo, France, September 28-30, 2009. C. Timmerer, “MPEG-V: Media Context and Control”, 89th ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Meeting , London, UK, June 2009. https://www-itec.uni-klu.ac.at/mmc/blog/2009/07/08/mpeg-v-media-context-and-control/ MPEG-V: http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V MPEG-V reflector: http://lists.uni-klu.ac.at/mailman/listinfo/metaverse 2009/09/30 Christian Timmerer, Klagenfurt University, Austria

M. Waltl, C. Timmerer, and H. Hellwagner, “A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects”, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009) , San Diego, USA, July 29-31, 2009.

C. Timmerer, J. Gelissen, M. Waltl, and H. Hellwagner, “Interfacing with Virtual Worlds”, accepted for publication in the Proceedings of the 2009 NEM Summit , Saint-Malo, France, September 28-30, 2009.

C. Timmerer, “MPEG-V: Media Context and Control”, 89th ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Meeting , London, UK, June 2009. https://www-itec.uni-klu.ac.at/mmc/blog/2009/07/08/mpeg-v-media-context-and-control/

MPEG-V: http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V

MPEG-V reflector: http://lists.uni-klu.ac.at/mailman/listinfo/metaverse

2009/09/30 Christian Timmerer, Klagenfurt University, Austria Demo & Video

Thank you for your attention ... questions, comments, etc. are welcome … Ass.-Prof. Dipl.-Ing. Dr. Christian Timmerer Klagenfurt University, Department of Information Technology (ITEC) Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA [email_address] http://research.timmerer.com/ Tel: +43/463/2700 3621 Fax: +43/463/2700 3699 © Copyright: Christian Timmerer 2009/09/30 Christian Timmerer, Klagenfurt University, Austria