MPEG-V: Media Context and Control

MPEG-VMedia Context and Control
N107{82,83,85,86} – Text of ISO/IEC 23006-[1-4] CD
Christian Timmerer
Klagenfurt University (UNIKLU)  Faculty of Technical Sciences (TEWI)
Department of Information Technology (ITEC)  Multimedia Communication (MMC)
http://research.timmerer.com  http://blog.timmerer.com  mailto:christian.timmerer@itec.uni-klu.ac.at

MPEG-V System Architecture
2009/07/03
Christian Timmerer, Klagenfurt University, Austria
2
Media context and control
Pt. 1: Architecture
Pt. 2: Control Information
Pt. 3: Sensory Information
Pt. 4: Avatar Information

Control Information
2009/07/03
3
Sensory Device Capabilities asext. of dia:TerminalCapability

unit, max/minIntensity, numOfLevels, delay, position
light (color, flash), heating, cooling, wind, vibration
scent, fog, water sprayer, color correction
kinestetic, tactile

Fundamental Input to any Control Device (aka Adaptation Engine)
User Sensory Preferences asext. of dia:UserCharacteristics

adaptability, max/minIntensity
light (color, flash), heating, cooling, wind, vibration
scent, fog, water sprayer, color correction
kinestetic, tactile

Avatar Characteristics
Appearance
Contains the high level description of the appearance and may refer a media containing the exact geometry and texture
Haptics Properties
Contains the high level description of the haptics properties
Animation
Contains the description of a set of animation sequences that the avatar is able to perform and may refer to several medias containing the exact (geometric transformations) animation parameters
Communication Skills
Contains a set of descriptors providing information on the different modalities an avatar is able to communicate
Personality
Contains a set of descriptors defining the personality of the avatar
Control
Contains a set of descriptors defining possible place-holders for sensors on body skeleton and face feature points
2009/07/03
4

What is MPEG Sensory Information?
UMA = Universal Multimedia Access
Access to content anywhere, anytime, any device == addressing QoS issues
Technical feasible thanks to standards ➙ MPEG-21 DIA
UME = Universal Multimedia Experience
User-centric approach – takes into account the user
Quality of Multimedia Experience
Sensorial, e.g., sharpness, brightness
Perceptual, e.g., what/where is the content
Emotional, e.g., feeling, sensation
MPEG-V Sensory Information
Consumption of multimedia assets may stimulate also other senses, e.g., olfaction, mechanoreception, or thermoception
Achieved by annotating the media resources with sensory effects
➙enhanced, worthwhile, and informative user experience, giving the user the sensation of being part of the actual media
2009/07/03
5
N9113
N10785

Concept of MPEG-V Sensory Information
2009/07/03
6

MPEG-V Sensory InformationSystem Architecture
2009/07/03
7
Native representation➙ standardized representation
Authoring
Packaging
Scope of Standardization
Delivery
Standardized representation➙ device capabilities &
user preferences
Adaptation

Committee Draft Structure
Sensory Effect Description Language
Basic building blocks: GeneralInformation, Declaration, Effect, GroupOfEffects, Parameter, ReferenceEffect
Common attributes: XSI (timing), activate, duration, fade, alt, priority, intensity, adaptType, adaptRange
Sensory Effect Vocabulary
Light (colored, flash), temperature, wind, vibration
Water sprayer, scent, fog, shadow, color correction
Kinesthetic, tactile
2009/07/03
8

Demo Setup
2009/07/03
9
Wall-washer / Light
Display
Light
Light
Fan
Fan
Speaker
Speaker
Vibration Bar
Subwoofer

Demo Video with Effects
2009/07/03
10

Kinesthetic / Tactile Effects
2009/07/03
11
♬♩♪♫
abcdefg
Haptic Texture
Force
Haptic interaction
Audio
Viewers
Television
Touch or exploration
Manipulation
Following movements
Dynamics
Feeling haptic effects
Video
Text
3D CG model

Kinesthetic / Tactile Effects (cont’d)
2009/07/03
12
Active interactions
Passive interactions

MPEG-V SI References
N10782, Text of ISO/IEC 23006-1 CD
Ad-hoc Group
metaverse@lists.uni-klu.ac.at
http://lists.uni-klu.ac.at/mailman/listinfo/metaverse
Some selected publications
J. Cha, I. Oakley, Y.-S. Ho, Y. Kim, J. Ryu, “A Framework of Haptic Broadcasting”, IEEE Multimedia, in print, Sept-Oct. 2009
J. Cha, S.-Y. Kim, Y.-S. Ho, J. Ryu, “3D Video Player System with Haptic Interaction based on Depth Image-Based Representation”, IEEE Transactions on Consumer Electronics, Vol. 52, No.2, pp.477-484, May 2006
M. Waltl, C. Timmerer, H. Hellwagner, A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009), San Diego, USA, July 29-31, 2009.
2009/07/03
13

Thank you for your attention
... questions, comments, etc. are welcome …
Ass.-Prof. Dipl.-Ing. Dr. Christian Timmerer
Klagenfurt University, Department of Information Technology (ITEC)
Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA
christian.timmerer@itec.uni-klu.ac.at
http://research.timmerer.com/
Tel: +43/463/2700 3621 Fax: +43/463/2700 3699
© Copyright: Christian Timmerer
14
2009/07/03