Your SlideShare is downloading. ×
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Immersive Future Media Technologies: Sensory Experience
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Immersive Future Media Technologies: Sensory Experience

1,875

Published on

Published in: Technology, Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,875
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
33
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Immersive Future Media Technologies: Sensory Experience
    Christian Timmerer
    Alpen-Adria-Universität Klagenfurt (AAU)  Faculty of Technical Sciences (TEWI) Department of Information Technology (ITEC)  Multimedia Communication (MMC)  Sensory Experience Lab (SELab)
    http://research.timmerer.com http://blog.timmerer.com  http://selab.itec.aau.at/mailto:christian.timmerer@itec.uni-klu.ac.at
    MobiMedia2011, Cagliari, Italy
    5th September, 2011
    Acknowledgments. This work was supported in part by the European Commission in the context of the NoE INTERMEDIA (NoE 038419), the P2P-Next project (FP7-ICT-216217), the ALICANTE project (FP7-ICT-248652), and the COST Action IC1003 QUALINET.
  • 2. Quality of Experience
    Factors impacting Quality of Experience
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    2
    Network
    Device
    Content Format
    Technical Factors
    Quality of Experience (QoE)
    Context
    TaskApplication
    User
    Social andPsychological
    Factors
    Environment
    User Expectation
    Content
    T. Ebrahimi, “Quality of Multimedia Experience: Past, Present and Future”, Keynote at ACM Multimedia 2009, Beijing, China, Oct 22, 2010. http://www.slideshare.net/touradj_ebrahimi/qoe
  • 3. Sensory Experience
    Consumption of multimedia content may stimulate also other senses
    Vision or hearing
    Olfaction, mechanoreception, thermoception, …
    Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    3
    … giving her/him the sensation of being part of the particular media
    ➪ worthwhile, informative user experience
  • 4. Outline
    Background / Introduction
    MPEG-V Media Context and Control featuring Sensory Information (SEDL+SEV)
    Demo/Software/Hardware components: SEVino, SESim, SEMP, amBX+SDK, and AmbientLib
    Improving the QoE through Sensory Effects ➪ Sensory Experience
    Motivation towards Sensory Experience
    Results from Subjective Tests: Experiment I + II
    Sensory Experience on mobile devices?
    2011/09/05
    4
    Christian Timmerer, Alpen-Adria-Universität, Austria
  • 5. Part One: Background / Introduction
    MPEG-V Media Context and Control
    Concept of MPEG-V Sensory Information
    Sensory Effect Description Language (SEDL) and Sensory Effect Vocabulary (SEV)
    Demo/Software/Hardware components: SEVino, SESim, SEMP, amBX+SDK, and AmbientLib
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    5
  • 6. Introduction to MPEG-V
    MPEG-V: Media Context and Control (ISO/IEC 23005) :== system architecture + associated information representations
    Interoperability between virtual worlds
    E.g., digital content provider of a virtual world (serious) gaming, simulation, DVD
    And real world
    E.g., sensors, actuators, vision and rendering, robotics (e.g. for revalidation), (support for) independent living, social and welfare systems, banking, insurance, travel, real estate, rights management
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    6
  • 7. MPEG-V: Media Context and Control
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    7
    System Architecture
    Pt. 1: Architecture
    Pt. 2: Control Information
    Pt. 3: Sensory Information
    Pt. 4: Virtual World Object Characteristics
    Pt. 5: Data Formats for Interaction Devices
    Pt. 6: Common Types and Tools
    Pt. 7: Conformance and Reference Software
    http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V
  • 8. Sensory Effect Description Language (SEDL)
    XML Schema-based language for describing sensory effects
    Basic building blocks to describe, e.g., light, wind, fog, vibration, scent
    MPEG-V Part 3, Sensory Information: Effects, GroupOfEffects
    Adopted MPEG-21 DIA tools for adding time information (synchronization)
    Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV)
    Extensibility: additional effects can be added easily w/o affecting SEDL
    Flexibility: each application domain may define its own sensory effects
    Description conforming to SEDL :== Sensory Effect Metadata (SEM)
    May be associated to any kind of multimedia content (e.g., movies, music, Web sites, games)
    Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate mediation device
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    8
  • 9. Sensory Effect Description Language (cont’d)
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    9
    SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects|Effect|ReferenceEffect)+
    Declarations ::= (GroupOfEffects|Effect|Parameter)+
    GroupOfEffects ::= timestamp EffectDefinitionEffectDefinition (EffectDefinition)*
    Effect ::= timestamp EffectDefinition
    EffectDefinition ::= [activate][duration][fade][alt] [priority][intensity][location]
    [adaptability][xsi:type][si:{pts,…}]
  • 10. Example
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    10
    <sedl:GroupOfEffectssi:pts="3240000"
    duration="100" fade="15"
    location="urn:mpeg:mpeg-v:01-SI-PositionCS-NS:center:*:front">
    <sedl:Effectxsi:type="sev:WindType" intensity="0.0769"/>
    <sedl:Effectxsi:type="sev:VibrationType" intensity="0.56"/>
    <sedl:Effectxsi:type="sev:LightType" intensity="0.0000077"/>
    </sedl:GroupOfEffects>
  • 11. Demo Setup
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    11
    Wall-washer / Light
    Display
    Light
    Light
    Fan
    Fan
    Speaker
    Speaker
    Vibration Bar
    Subwoofer
  • 12. Demo Video with Effects
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    12
  • 13. SEVino, SESim, SEMP, and amBX
    amBX (Ambient Experience) system + SDK
    Two fan devices, a wrist rumbler, two sound speakers, a subwoofer, two lights, and a wall washer
    Everything controlled by SEM descriptionsexcept light effect
    Web browser plug-in (AmbientLib)
    Various browser support, HTML5, Flash
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    13
    Annotation Tool: SEVino
    Simulator: SESim
    Player: SEMP
  • 14. Part Two:Improving the QoE through Sensory Effects
    Motivation towards Sensory Experience
    Results from Subjective Tests
    Experiment I: sensory effects a vital tool for enhancing QoE?
    Experiment II: relationship of the QoE to various video bit-rates
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    14
  • 15. Introduction (cont’d)
    Universal Multimedia Access (UMA)
    Anywhere, anytime, any device + technically feasible
    Main focus on devices and network connectivity issues ➪ QoS
    Universal Multimedia Experience (UME)
    Take the user into account ➪ QoE
    Multimedia Adaptation and Quality Models/Metrics
    Single modality (i.e., audio, image, or video only) or a simple combination of two modalities (i.e., audio and video)
    Triple user characterization model
    Sensorial, e.g., sharpness, brightness
    Perceptual, e.g., what/where is the content
    Emotional, e.g., feeling, sensation
    Ambient Intelligence
    Add’llight effects are highly appreciated for both audio and visual content
    Calls for a scientific framework to capture, measure, quantify, judge, and explain the user experience
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    15
    B. de Ruyter, E. Aarts. “Ambient intelligence: visualizing the future”, Proceedings of the Working Conference on Advanced Visual Interfaces, New York, NY, USA, 2004, pp. 203–208.
    E. Aarts, B. de Ruyter, “New research perspectives on Ambient Intelligence”, Journal of Ambient Intelligence and Smart Environments, IOS Press, vol. 1, no. 1, 2009, pp. 5–14.
    F. Pereira, “A triple user characterization model for video adaptation and quality of experience evaluation,” Proc. of the 7th Workshop on Multimedia Signal Processing, Shanghai, China, October 2005, pp. 1–4.
  • 16. Experiment I
    Aim: demonstrate that sensory effects is a vital tool for enhancing the user experience depending on the actual genre
    Tools
    Sensory Effect Media Player (SEMP)
    Test sequences annotated with sensory effects: action (Rambo 4, Babylon A.D.), news (ZIB Flash), documentary (Earth), commercials (Woist Klaus), and sports (Formula 1)
    Double Stimulus Impairment Scale (DSIS) also known as Degradation Category Rating (DCR)
    Five-level impairment scale ➪ new five-level enhancement scale
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    16
  • 17. Experiment I (cont’d)
    Test Procedure
    Show test sequences in random order
    Two sequences presented twice but not directly one after the other – test the reliability of the subjects
    First, show reference sequence w/o sensory effects
    Second, the same sequence enriched with sensory effects with a two second break in between
    Finally, subjects to rate the overall opinion of the audio/video resource and sensory effect quality
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    17
    Test Sequences
    Dur (sec) Genre #Effects
    Rambo 4 58.11 Action 10
    ZIB Flash 83.04 News 1
    Babylon A.D. 118.42 Action 28
    Woist Klaus? 59.16 Comm. 16
    Earth 66 Docu. 24
    Formula 1 116.2 Sports 43
    New Five-Level Enhancement Scale
    5 Big enhancement
    4 Little enhancement
    3 Imperceptible
    2 Annoying
    1 Very annoying
  • 18. Experiment I (cont’d)
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    18
  • 19. Major Findings from Experiment I
    Sensory effects is a vital tool for enhancing the user experience
    Action, sports, and documentary genres benefit more from these additional effects
    Rambo 4 and Babylon A.D. are from the same genre, the results differ slightly
    Commercial genre can also profit from the additional effects but not at the same level as documentary
    News genre will not profit from these effects
    Those videos presented twice differ in the results ➪ test method may not have functioned properly
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    19
    M. Waltl, C. Timmerer, H. Hellwagner, "Increasing the User Experience of Multimedia Presentations with Sensory Effects", 11th Int’l Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10), Desenzano del Garda, Italy, Apr. 2010.
  • 20. Experiment II
    Aim: investigate the relationship of the QoE to various video bit-rates of multimedia contents annotated with sensory effects.
    Subjective quality gap between video resources annotated with and without sensory effects at different bit-rates
    Tools
    Sensory Effect Media Player (SEMP)
    Test sequences annotated with sensory effects: action (Babylon A.D.) and documentary (Earth)
    Absolute Category Rating with Hidden Reference (ACR-HR) method using a five-point discrete scale from excellent to bad
    Novel voting device for continuous voting
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    20
    with
  • 21. Experiment II (cont’d)
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    21
    Test Sequences
    Voting Devices
    Sequence Babylon A.D. Earth
    Duration 35s 21s
    Resolution 1280 x 544 1280 x 720
    Motion High Low
    Nr. of Effects W: 7; V: 9 W: 8; V: 1
    Bit-rates Kbit/s PSNR Kbit/s PSNR
    Low Quality 2154 38.93 2204 38.11
    Medium Quality 3112 41.27 3171 40.65
    High Quality 4044 42.95 4116 42.27
    Highest Quality 6315 N/A 6701 N/A
    Procedure
  • 22. Experiment II (cont’d)
    MOS vs. PSNR/bit-rate for Babylon A.D.
    MOS vs. PSNR/bit-rate for Earth.
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    22
  • 23. Major Findings from Experiment II
    Results confirm the observations from the previous experiment
    MOS of the lowest bit-rate version with sensory effectsis (always) higher than the MOS of all higher bit-rate variants without sensory effects
    Sequence enriched with sensory effects is 0.5 MOS points higher on average than without sensory effects
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    23
    M. Waltl, C. Timmerer, H. Hellwagner, "Improving the Quality of Multimedia Experience through Sensory Effects", Proc. 2nd Int’l. Workshop on Quality of Multimedia Experience (QoMEX’10), Trondheim, Norway, Jun. 2010.
  • 24. Part ThreeSensory Experience on Mobile Devices?
    Sensory effects on traditional mobile devices
    Haptic – vibration
    Tactile (passive) – position/location, motion/rotation
    Light – backlight, ambient
    Scent, the Smellophone?
    How about other mobile devices?
    E.g., in transportation (bus, train, car, plane)
    Open issues
    Tradeoff with battery consumption (optimization problem)
    Applications and subjective quality assessments
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    24
  • 25. Conclusions
    MPEG-V Overview (ISO/IEC 23005)
    Pt. 1: Architecture
    Pt. 3: Sensory Information
    Sensory Information and HW/SW components
    Sensory Effect Description Language
    Sensory Effect Vocabulary
    SEVino, SESim, SEMP, and amBX+SDK
    How to improve QoE?➪ Results from experiments so far …
    … to be continued!
    Towards a Utility Model for Sensory Experiences
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    25
  • 26. Sensory Experience Lab
    http://selab.itec.aau.at/
    Software and Services
    Standardization
    Publications
    Media
    Funding
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    26
  • 27. Acknowledgments
    EC projects for funding this activity
    NoE INTERMEDIA (NoE 038419)
    http://intermedia.miralab.ch/
    P2P-Next project (FP7-ICT-216217)
    http://www.p2p-next.eu
    ALICANTE project (FP7-ICT-248652)
    http://www.ict-alicante.eu
    COST ICT Action IC1003
    QUALINET – European Network on Qualityof Experience in Multimedia Systems and Services
    http://www.qualinet.eu/
    Markus Waltl for implementing, preparing, conducting, evaluating almost all the experiments
    Benjamin Rainer for implementing the browser plug-in + Web tests
    Hermann Hellwagner for his advice and feedback
    ISO/IEC MPEG and its participating members for their constructive feedback during the standardization process
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    27
  • 28. References
    Markus Waltl, Christian Timmerer, Hermann Hellwagner, “A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects”, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009), San Diego, USA, July 29-31, 2009.
    C. Timmerer, J. Gelissen, M. Waltl, and H. Hellwagner, “Interfacing with Virtual Worlds”, Proceedings of the NEM Summit 2009, Saint-Malo, France, September 28-30, 2009.
    M. Waltl, Enriching Multimedia with Sensory Effects, VDM Verlag Dr. Müller, February, 2010.
    M. Waltl, C. Timmerer and H. Hellwagner, “Increasing the User Experience of Multimedia Presentations with Sensory Effects”, Proceedings of the 11th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10), Desenzano del Garda, Italy, April 12-14, 2010.
    C. Timmerer, M. Waltl, and H. Hellwagner, “Are Sensory Effects Ready for the World Wide Web?”, Proceedings of the Workshop on Interoperable Social Multimedia Applications (WISMA 2010), Barcelona, Spain, May 19-20, 2010.
    M. Waltl, C. Timmerer, and H. Hellwagner, “Improving the Quality of Multimedia Experience through Sensory Effects”, Proceedings of the 2nd International Workshop on Quality of Multimedia Experience (QoMEX2010), Trondheim, Norway, June 21-23, 2010.
    M. Waltl, C. Raffelsberger, C. Timmerer, and H. Hellwagner, “Metadata-based Content Management and Sharing System for Improved User Experience”, Proc. of the 4th InterMedia Open Forum (IMOF 2010), Palma de Mallorca, Spain, September 1, 2010.
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria
    28
  • 29. Thank you for your attention
    ... questions, comments, etc. are welcome …
    Ass.-Prof. Dipl.-Ing. Dr. Christian Timmerer
    Klagenfurt University, Department of Information Technology (ITEC)
    Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA
    christian.timmerer@itec.uni-klu.ac.at
    http://research.timmerer.com/
    Tel: +43/463/2700 3621 Fax: +43/463/2700 3699
    © Copyright: Christian Timmerer
    29
    2011/09/05
    Christian Timmerer, Alpen-Adria-Universität, Austria

×