• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Immersive Future Media Technologies: Sensory Experience
 

Immersive Future Media Technologies: Sensory Experience

on

  • 2,006 views

 

Statistics

Views

Total Views
2,006
Views on SlideShare
1,997
Embed Views
9

Actions

Likes
0
Downloads
25
Comments
0

2 Embeds 9

http://paper.li 7
http://selab.itec.aau.at 2

Accessibility

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Immersive Future Media Technologies: Sensory Experience Immersive Future Media Technologies: Sensory Experience Presentation Transcript

    • Immersive Future Media Technologies: Sensory Experience
      Christian Timmerer
      Alpen-Adria-Universität Klagenfurt (AAU)  Faculty of Technical Sciences (TEWI) Department of Information Technology (ITEC)  Multimedia Communication (MMC)  Sensory Experience Lab (SELab)
      http://research.timmerer.com http://blog.timmerer.com  http://selab.itec.aau.at/mailto:christian.timmerer@itec.uni-klu.ac.at
      MobiMedia2011, Cagliari, Italy
      5th September, 2011
      Acknowledgments. This work was supported in part by the European Commission in the context of the NoE INTERMEDIA (NoE 038419), the P2P-Next project (FP7-ICT-216217), the ALICANTE project (FP7-ICT-248652), and the COST Action IC1003 QUALINET.
    • Quality of Experience
      Factors impacting Quality of Experience
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      2
      Network
      Device
      Content Format
      Technical Factors
      Quality of Experience (QoE)
      Context
      TaskApplication
      User
      Social andPsychological
      Factors
      Environment
      User Expectation
      Content
      T. Ebrahimi, “Quality of Multimedia Experience: Past, Present and Future”, Keynote at ACM Multimedia 2009, Beijing, China, Oct 22, 2010. http://www.slideshare.net/touradj_ebrahimi/qoe
    • Sensory Experience
      Consumption of multimedia content may stimulate also other senses
      Vision or hearing
      Olfaction, mechanoreception, thermoception, …
      Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      3
      … giving her/him the sensation of being part of the particular media
      ➪ worthwhile, informative user experience
    • Outline
      Background / Introduction
      MPEG-V Media Context and Control featuring Sensory Information (SEDL+SEV)
      Demo/Software/Hardware components: SEVino, SESim, SEMP, amBX+SDK, and AmbientLib
      Improving the QoE through Sensory Effects ➪ Sensory Experience
      Motivation towards Sensory Experience
      Results from Subjective Tests: Experiment I + II
      Sensory Experience on mobile devices?
      2011/09/05
      4
      Christian Timmerer, Alpen-Adria-Universität, Austria
    • Part One: Background / Introduction
      MPEG-V Media Context and Control
      Concept of MPEG-V Sensory Information
      Sensory Effect Description Language (SEDL) and Sensory Effect Vocabulary (SEV)
      Demo/Software/Hardware components: SEVino, SESim, SEMP, amBX+SDK, and AmbientLib
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      5
    • Introduction to MPEG-V
      MPEG-V: Media Context and Control (ISO/IEC 23005) :== system architecture + associated information representations
      Interoperability between virtual worlds
      E.g., digital content provider of a virtual world (serious) gaming, simulation, DVD
      And real world
      E.g., sensors, actuators, vision and rendering, robotics (e.g. for revalidation), (support for) independent living, social and welfare systems, banking, insurance, travel, real estate, rights management
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      6
    • MPEG-V: Media Context and Control
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      7
      System Architecture
      Pt. 1: Architecture
      Pt. 2: Control Information
      Pt. 3: Sensory Information
      Pt. 4: Virtual World Object Characteristics
      Pt. 5: Data Formats for Interaction Devices
      Pt. 6: Common Types and Tools
      Pt. 7: Conformance and Reference Software
      http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V
    • Sensory Effect Description Language (SEDL)
      XML Schema-based language for describing sensory effects
      Basic building blocks to describe, e.g., light, wind, fog, vibration, scent
      MPEG-V Part 3, Sensory Information: Effects, GroupOfEffects
      Adopted MPEG-21 DIA tools for adding time information (synchronization)
      Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV)
      Extensibility: additional effects can be added easily w/o affecting SEDL
      Flexibility: each application domain may define its own sensory effects
      Description conforming to SEDL :== Sensory Effect Metadata (SEM)
      May be associated to any kind of multimedia content (e.g., movies, music, Web sites, games)
      Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate mediation device
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      8
    • Sensory Effect Description Language (cont’d)
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      9
      SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects|Effect|ReferenceEffect)+
      Declarations ::= (GroupOfEffects|Effect|Parameter)+
      GroupOfEffects ::= timestamp EffectDefinitionEffectDefinition (EffectDefinition)*
      Effect ::= timestamp EffectDefinition
      EffectDefinition ::= [activate][duration][fade][alt] [priority][intensity][location]
      [adaptability][xsi:type][si:{pts,…}]
    • Example
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      10
      <sedl:GroupOfEffectssi:pts="3240000"
      duration="100" fade="15"
      location="urn:mpeg:mpeg-v:01-SI-PositionCS-NS:center:*:front">
      <sedl:Effectxsi:type="sev:WindType" intensity="0.0769"/>
      <sedl:Effectxsi:type="sev:VibrationType" intensity="0.56"/>
      <sedl:Effectxsi:type="sev:LightType" intensity="0.0000077"/>
      </sedl:GroupOfEffects>
    • Demo Setup
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      11
      Wall-washer / Light
      Display
      Light
      Light
      Fan
      Fan
      Speaker
      Speaker
      Vibration Bar
      Subwoofer
    • Demo Video with Effects
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      12
    • SEVino, SESim, SEMP, and amBX
      amBX (Ambient Experience) system + SDK
      Two fan devices, a wrist rumbler, two sound speakers, a subwoofer, two lights, and a wall washer
      Everything controlled by SEM descriptionsexcept light effect
      Web browser plug-in (AmbientLib)
      Various browser support, HTML5, Flash
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      13
      Annotation Tool: SEVino
      Simulator: SESim
      Player: SEMP
    • Part Two:Improving the QoE through Sensory Effects
      Motivation towards Sensory Experience
      Results from Subjective Tests
      Experiment I: sensory effects a vital tool for enhancing QoE?
      Experiment II: relationship of the QoE to various video bit-rates
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      14
    • Introduction (cont’d)
      Universal Multimedia Access (UMA)
      Anywhere, anytime, any device + technically feasible
      Main focus on devices and network connectivity issues ➪ QoS
      Universal Multimedia Experience (UME)
      Take the user into account ➪ QoE
      Multimedia Adaptation and Quality Models/Metrics
      Single modality (i.e., audio, image, or video only) or a simple combination of two modalities (i.e., audio and video)
      Triple user characterization model
      Sensorial, e.g., sharpness, brightness
      Perceptual, e.g., what/where is the content
      Emotional, e.g., feeling, sensation
      Ambient Intelligence
      Add’llight effects are highly appreciated for both audio and visual content
      Calls for a scientific framework to capture, measure, quantify, judge, and explain the user experience
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      15
      B. de Ruyter, E. Aarts. “Ambient intelligence: visualizing the future”, Proceedings of the Working Conference on Advanced Visual Interfaces, New York, NY, USA, 2004, pp. 203–208.
      E. Aarts, B. de Ruyter, “New research perspectives on Ambient Intelligence”, Journal of Ambient Intelligence and Smart Environments, IOS Press, vol. 1, no. 1, 2009, pp. 5–14.
      F. Pereira, “A triple user characterization model for video adaptation and quality of experience evaluation,” Proc. of the 7th Workshop on Multimedia Signal Processing, Shanghai, China, October 2005, pp. 1–4.
    • Experiment I
      Aim: demonstrate that sensory effects is a vital tool for enhancing the user experience depending on the actual genre
      Tools
      Sensory Effect Media Player (SEMP)
      Test sequences annotated with sensory effects: action (Rambo 4, Babylon A.D.), news (ZIB Flash), documentary (Earth), commercials (Woist Klaus), and sports (Formula 1)
      Double Stimulus Impairment Scale (DSIS) also known as Degradation Category Rating (DCR)
      Five-level impairment scale ➪ new five-level enhancement scale
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      16
    • Experiment I (cont’d)
      Test Procedure
      Show test sequences in random order
      Two sequences presented twice but not directly one after the other – test the reliability of the subjects
      First, show reference sequence w/o sensory effects
      Second, the same sequence enriched with sensory effects with a two second break in between
      Finally, subjects to rate the overall opinion of the audio/video resource and sensory effect quality
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      17
      Test Sequences
      Dur (sec) Genre #Effects
      Rambo 4 58.11 Action 10
      ZIB Flash 83.04 News 1
      Babylon A.D. 118.42 Action 28
      Woist Klaus? 59.16 Comm. 16
      Earth 66 Docu. 24
      Formula 1 116.2 Sports 43
      New Five-Level Enhancement Scale
      5 Big enhancement
      4 Little enhancement
      3 Imperceptible
      2 Annoying
      1 Very annoying
    • Experiment I (cont’d)
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      18
    • Major Findings from Experiment I
      Sensory effects is a vital tool for enhancing the user experience
      Action, sports, and documentary genres benefit more from these additional effects
      Rambo 4 and Babylon A.D. are from the same genre, the results differ slightly
      Commercial genre can also profit from the additional effects but not at the same level as documentary
      News genre will not profit from these effects
      Those videos presented twice differ in the results ➪ test method may not have functioned properly
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      19
      M. Waltl, C. Timmerer, H. Hellwagner, "Increasing the User Experience of Multimedia Presentations with Sensory Effects", 11th Int’l Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10), Desenzano del Garda, Italy, Apr. 2010.
    • Experiment II
      Aim: investigate the relationship of the QoE to various video bit-rates of multimedia contents annotated with sensory effects.
      Subjective quality gap between video resources annotated with and without sensory effects at different bit-rates
      Tools
      Sensory Effect Media Player (SEMP)
      Test sequences annotated with sensory effects: action (Babylon A.D.) and documentary (Earth)
      Absolute Category Rating with Hidden Reference (ACR-HR) method using a five-point discrete scale from excellent to bad
      Novel voting device for continuous voting
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      20
      with
    • Experiment II (cont’d)
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      21
      Test Sequences
      Voting Devices
      Sequence Babylon A.D. Earth
      Duration 35s 21s
      Resolution 1280 x 544 1280 x 720
      Motion High Low
      Nr. of Effects W: 7; V: 9 W: 8; V: 1
      Bit-rates Kbit/s PSNR Kbit/s PSNR
      Low Quality 2154 38.93 2204 38.11
      Medium Quality 3112 41.27 3171 40.65
      High Quality 4044 42.95 4116 42.27
      Highest Quality 6315 N/A 6701 N/A
      Procedure
    • Experiment II (cont’d)
      MOS vs. PSNR/bit-rate for Babylon A.D.
      MOS vs. PSNR/bit-rate for Earth.
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      22
    • Major Findings from Experiment II
      Results confirm the observations from the previous experiment
      MOS of the lowest bit-rate version with sensory effectsis (always) higher than the MOS of all higher bit-rate variants without sensory effects
      Sequence enriched with sensory effects is 0.5 MOS points higher on average than without sensory effects
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      23
      M. Waltl, C. Timmerer, H. Hellwagner, "Improving the Quality of Multimedia Experience through Sensory Effects", Proc. 2nd Int’l. Workshop on Quality of Multimedia Experience (QoMEX’10), Trondheim, Norway, Jun. 2010.
    • Part ThreeSensory Experience on Mobile Devices?
      Sensory effects on traditional mobile devices
      Haptic – vibration
      Tactile (passive) – position/location, motion/rotation
      Light – backlight, ambient
      Scent, the Smellophone?
      How about other mobile devices?
      E.g., in transportation (bus, train, car, plane)
      Open issues
      Tradeoff with battery consumption (optimization problem)
      Applications and subjective quality assessments
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      24
    • Conclusions
      MPEG-V Overview (ISO/IEC 23005)
      Pt. 1: Architecture
      Pt. 3: Sensory Information
      Sensory Information and HW/SW components
      Sensory Effect Description Language
      Sensory Effect Vocabulary
      SEVino, SESim, SEMP, and amBX+SDK
      How to improve QoE?➪ Results from experiments so far …
      … to be continued!
      Towards a Utility Model for Sensory Experiences
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      25
    • Sensory Experience Lab
      http://selab.itec.aau.at/
      Software and Services
      Standardization
      Publications
      Media
      Funding
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      26
    • Acknowledgments
      EC projects for funding this activity
      NoE INTERMEDIA (NoE 038419)
      http://intermedia.miralab.ch/
      P2P-Next project (FP7-ICT-216217)
      http://www.p2p-next.eu
      ALICANTE project (FP7-ICT-248652)
      http://www.ict-alicante.eu
      COST ICT Action IC1003
      QUALINET – European Network on Qualityof Experience in Multimedia Systems and Services
      http://www.qualinet.eu/
      Markus Waltl for implementing, preparing, conducting, evaluating almost all the experiments
      Benjamin Rainer for implementing the browser plug-in + Web tests
      Hermann Hellwagner for his advice and feedback
      ISO/IEC MPEG and its participating members for their constructive feedback during the standardization process
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      27
    • References
      Markus Waltl, Christian Timmerer, Hermann Hellwagner, “A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects”, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009), San Diego, USA, July 29-31, 2009.
      C. Timmerer, J. Gelissen, M. Waltl, and H. Hellwagner, “Interfacing with Virtual Worlds”, Proceedings of the NEM Summit 2009, Saint-Malo, France, September 28-30, 2009.
      M. Waltl, Enriching Multimedia with Sensory Effects, VDM Verlag Dr. Müller, February, 2010.
      M. Waltl, C. Timmerer and H. Hellwagner, “Increasing the User Experience of Multimedia Presentations with Sensory Effects”, Proceedings of the 11th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10), Desenzano del Garda, Italy, April 12-14, 2010.
      C. Timmerer, M. Waltl, and H. Hellwagner, “Are Sensory Effects Ready for the World Wide Web?”, Proceedings of the Workshop on Interoperable Social Multimedia Applications (WISMA 2010), Barcelona, Spain, May 19-20, 2010.
      M. Waltl, C. Timmerer, and H. Hellwagner, “Improving the Quality of Multimedia Experience through Sensory Effects”, Proceedings of the 2nd International Workshop on Quality of Multimedia Experience (QoMEX2010), Trondheim, Norway, June 21-23, 2010.
      M. Waltl, C. Raffelsberger, C. Timmerer, and H. Hellwagner, “Metadata-based Content Management and Sharing System for Improved User Experience”, Proc. of the 4th InterMedia Open Forum (IMOF 2010), Palma de Mallorca, Spain, September 1, 2010.
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria
      28
    • Thank you for your attention
      ... questions, comments, etc. are welcome …
      Ass.-Prof. Dipl.-Ing. Dr. Christian Timmerer
      Klagenfurt University, Department of Information Technology (ITEC)
      Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA
      christian.timmerer@itec.uni-klu.ac.at
      http://research.timmerer.com/
      Tel: +43/463/2700 3621 Fax: +43/463/2700 3699
      © Copyright: Christian Timmerer
      29
      2011/09/05
      Christian Timmerer, Alpen-Adria-Universität, Austria