Immersive Future Media Technologies: Sensory Experience<br />Christian Timmerer<br />Alpen-Adria-Universität Klagenfurt (A...
Quality of Experience<br />Factors impacting Quality of Experience<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Un...
Sensory Experience<br />Consumption of multimedia content may stimulate also other senses<br />Vision or hearing<br />Olfa...
Outline<br />Background / Introduction<br />MPEG-V Media Context and Control featuring Sensory Information (SEDL+SEV)<br /...
Part One: Background / Introduction<br />MPEG-V Media Context and Control<br />Concept of MPEG-V Sensory Information<br />...
Introduction to MPEG-V<br />MPEG-V: Media Context and Control (ISO/IEC 23005) :== system architecture + associated informa...
MPEG-V: Media Context and Control<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />7<br />Sy...
Sensory Effect Description Language (SEDL)<br />XML Schema-based language for describing sensory effects<br />Basic buildi...
Sensory Effect Description Language (cont’d)<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br ...
Example<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />10<br /><sedl:GroupOfEffectssi:pts=...
Demo Setup<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />11<br />Wall-washer / Light<br /...
Demo Video with Effects<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />12<br />
SEVino, SESim, SEMP, and amBX<br />amBX (Ambient Experience) system + SDK<br />Two fan devices, a wrist rumbler, two sound...
Part Two:Improving the QoE through Sensory Effects<br />Motivation towards Sensory Experience<br />Results from Subjective...
Introduction (cont’d)<br />Universal Multimedia Access (UMA)<br />Anywhere, anytime, any device + technically feasible<br ...
Experiment I<br />Aim: demonstrate that sensory effects is a vital tool for enhancing the user experience depending on the...
Experiment I (cont’d)<br />Test Procedure<br />Show test sequences in random order<br />Two sequences presented twice but ...
Experiment I (cont’d)<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />18<br />
Major Findings from Experiment I<br />Sensory effects is a vital tool for enhancing the user experience<br />Action, sport...
Experiment II<br />Aim: investigate the relationship of the QoE to various video bit-rates of multimedia contents annotate...
Experiment II (cont’d)<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />21<br />Test Sequenc...
Experiment II (cont’d)<br />MOS vs. PSNR/bit-rate for Babylon A.D.<br />MOS vs. PSNR/bit-rate for Earth. <br />2011/09/05<...
Major Findings from Experiment II<br />Results confirm the observations from the previous experiment<br />MOS of the lowes...
Part ThreeSensory Experience on Mobile Devices?<br />Sensory effects on traditional mobile devices<br />Haptic – vibration...
Conclusions<br />MPEG-V Overview (ISO/IEC 23005)<br />Pt. 1: Architecture<br />Pt. 3: Sensory Information<br />Sensory Inf...
Sensory Experience Lab<br />http://selab.itec.aau.at/<br />Software and Services<br />Standardization<br />Publications<br...
Acknowledgments<br />EC projects for funding this activity<br />NoE INTERMEDIA (NoE 038419)<br />http://intermedia.miralab...
References<br />Markus Waltl, Christian Timmerer, Hermann Hellwagner, “A Test-Bed for Quality of Multimedia Experience Eva...
Thank you for your attention<br />... questions, comments, etc. are welcome …<br />Ass.-Prof. Dipl.-Ing. Dr. Christian Tim...
Upcoming SlideShare
Loading in...5
×

Immersive Future Media Technologies: Sensory Experience

1,952

Published on

Published in: Technology, Health & Medicine
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,952
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
33
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Immersive Future Media Technologies: Sensory Experience

  1. 1. Immersive Future Media Technologies: Sensory Experience<br />Christian Timmerer<br />Alpen-Adria-Universität Klagenfurt (AAU)  Faculty of Technical Sciences (TEWI) Department of Information Technology (ITEC)  Multimedia Communication (MMC)  Sensory Experience Lab (SELab)<br />http://research.timmerer.com http://blog.timmerer.com  http://selab.itec.aau.at/mailto:christian.timmerer@itec.uni-klu.ac.at<br />MobiMedia2011, Cagliari, Italy<br />5th September, 2011<br />Acknowledgments. This work was supported in part by the European Commission in the context of the NoE INTERMEDIA (NoE 038419), the P2P-Next project (FP7-ICT-216217), the ALICANTE project (FP7-ICT-248652), and the COST Action IC1003 QUALINET. <br />
  2. 2. Quality of Experience<br />Factors impacting Quality of Experience<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />2<br />Network<br />Device<br />Content Format<br />Technical Factors<br />Quality of Experience (QoE)<br />Context<br />TaskApplication<br />User<br />Social andPsychological<br />Factors<br />Environment<br />User Expectation<br />Content<br />T. Ebrahimi, “Quality of Multimedia Experience: Past, Present and Future”, Keynote at ACM Multimedia 2009, Beijing, China, Oct 22, 2010. http://www.slideshare.net/touradj_ebrahimi/qoe<br />
  3. 3. Sensory Experience<br />Consumption of multimedia content may stimulate also other senses<br />Vision or hearing<br />Olfaction, mechanoreception, thermoception, …<br />Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />3<br />… giving her/him the sensation of being part of the particular media<br />➪ worthwhile, informative user experience <br />
  4. 4. Outline<br />Background / Introduction<br />MPEG-V Media Context and Control featuring Sensory Information (SEDL+SEV)<br />Demo/Software/Hardware components: SEVino, SESim, SEMP, amBX+SDK, and AmbientLib<br />Improving the QoE through Sensory Effects ➪ Sensory Experience<br />Motivation towards Sensory Experience<br />Results from Subjective Tests: Experiment I + II<br />Sensory Experience on mobile devices?<br />2011/09/05<br />4<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />
  5. 5. Part One: Background / Introduction<br />MPEG-V Media Context and Control<br />Concept of MPEG-V Sensory Information<br />Sensory Effect Description Language (SEDL) and Sensory Effect Vocabulary (SEV)<br />Demo/Software/Hardware components: SEVino, SESim, SEMP, amBX+SDK, and AmbientLib<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />5<br />
  6. 6. Introduction to MPEG-V<br />MPEG-V: Media Context and Control (ISO/IEC 23005) :== system architecture + associated information representations<br />Interoperability between virtual worlds<br />E.g., digital content provider of a virtual world (serious) gaming, simulation, DVD<br />And real world<br />E.g., sensors, actuators, vision and rendering, robotics (e.g. for revalidation), (support for) independent living, social and welfare systems, banking, insurance, travel, real estate, rights management<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />6<br />
  7. 7. MPEG-V: Media Context and Control<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />7<br />System Architecture<br />Pt. 1: Architecture<br />Pt. 2: Control Information<br />Pt. 3: Sensory Information<br />Pt. 4: Virtual World Object Characteristics<br />Pt. 5: Data Formats for Interaction Devices<br />Pt. 6: Common Types and Tools<br />Pt. 7: Conformance and Reference Software<br />http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-V<br />
  8. 8. Sensory Effect Description Language (SEDL)<br />XML Schema-based language for describing sensory effects<br />Basic building blocks to describe, e.g., light, wind, fog, vibration, scent<br />MPEG-V Part 3, Sensory Information: Effects, GroupOfEffects<br />Adopted MPEG-21 DIA tools for adding time information (synchronization)<br />Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV)<br />Extensibility: additional effects can be added easily w/o affecting SEDL<br />Flexibility: each application domain may define its own sensory effects<br />Description conforming to SEDL :== Sensory Effect Metadata (SEM)<br />May be associated to any kind of multimedia content (e.g., movies, music, Web sites, games)<br />Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate mediation device<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />8<br />
  9. 9. Sensory Effect Description Language (cont’d)<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />9<br />SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects|Effect|ReferenceEffect)+<br />Declarations ::= (GroupOfEffects|Effect|Parameter)+<br />GroupOfEffects ::= timestamp EffectDefinitionEffectDefinition (EffectDefinition)*<br />Effect ::= timestamp EffectDefinition<br />EffectDefinition ::= [activate][duration][fade][alt] [priority][intensity][location]<br />[adaptability][xsi:type][si:{pts,…}]<br />
  10. 10. Example<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />10<br /><sedl:GroupOfEffectssi:pts="3240000"<br />duration="100" fade="15"<br />location="urn:mpeg:mpeg-v:01-SI-PositionCS-NS:center:*:front"><br /> <sedl:Effectxsi:type="sev:WindType" intensity="0.0769"/><br /> <sedl:Effectxsi:type="sev:VibrationType" intensity="0.56"/><br /> <sedl:Effectxsi:type="sev:LightType" intensity="0.0000077"/><br /></sedl:GroupOfEffects><br />
  11. 11. Demo Setup<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />11<br />Wall-washer / Light<br />Display<br />Light<br />Light<br />Fan<br />Fan<br />Speaker<br />Speaker<br />Vibration Bar<br />Subwoofer<br />
  12. 12. Demo Video with Effects<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />12<br />
  13. 13. SEVino, SESim, SEMP, and amBX<br />amBX (Ambient Experience) system + SDK<br />Two fan devices, a wrist rumbler, two sound speakers, a subwoofer, two lights, and a wall washer<br />Everything controlled by SEM descriptionsexcept light effect<br />Web browser plug-in (AmbientLib)<br />Various browser support, HTML5, Flash<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />13<br />Annotation Tool: SEVino<br />Simulator: SESim<br />Player: SEMP<br />
  14. 14. Part Two:Improving the QoE through Sensory Effects<br />Motivation towards Sensory Experience<br />Results from Subjective Tests<br />Experiment I: sensory effects a vital tool for enhancing QoE?<br />Experiment II: relationship of the QoE to various video bit-rates<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />14<br />
  15. 15. Introduction (cont’d)<br />Universal Multimedia Access (UMA)<br />Anywhere, anytime, any device + technically feasible<br />Main focus on devices and network connectivity issues ➪ QoS<br />Universal Multimedia Experience (UME)<br />Take the user into account ➪ QoE<br />Multimedia Adaptation and Quality Models/Metrics<br />Single modality (i.e., audio, image, or video only) or a simple combination of two modalities (i.e., audio and video)<br />Triple user characterization model<br />Sensorial, e.g., sharpness, brightness<br />Perceptual, e.g., what/where is the content<br />Emotional, e.g., feeling, sensation<br />Ambient Intelligence<br />Add’llight effects are highly appreciated for both audio and visual content<br />Calls for a scientific framework to capture, measure, quantify, judge, and explain the user experience <br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />15<br />B. de Ruyter, E. Aarts. “Ambient intelligence: visualizing the future”, Proceedings of the Working Conference on Advanced Visual Interfaces, New York, NY, USA, 2004, pp. 203–208.<br />E. Aarts, B. de Ruyter, “New research perspectives on Ambient Intelligence”, Journal of Ambient Intelligence and Smart Environments, IOS Press, vol. 1, no. 1, 2009, pp. 5–14. <br />F. Pereira, “A triple user characterization model for video adaptation and quality of experience evaluation,” Proc. of the 7th Workshop on Multimedia Signal Processing, Shanghai, China, October 2005, pp. 1–4.<br />
  16. 16. Experiment I<br />Aim: demonstrate that sensory effects is a vital tool for enhancing the user experience depending on the actual genre<br />Tools<br />Sensory Effect Media Player (SEMP)<br />Test sequences annotated with sensory effects: action (Rambo 4, Babylon A.D.), news (ZIB Flash), documentary (Earth), commercials (Woist Klaus), and sports (Formula 1)<br />Double Stimulus Impairment Scale (DSIS) also known as Degradation Category Rating (DCR)<br />Five-level impairment scale ➪ new five-level enhancement scale <br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />16<br />
  17. 17. Experiment I (cont’d)<br />Test Procedure<br />Show test sequences in random order<br />Two sequences presented twice but not directly one after the other – test the reliability of the subjects<br />First, show reference sequence w/o sensory effects<br />Second, the same sequence enriched with sensory effects with a two second break in between<br />Finally, subjects to rate the overall opinion of the audio/video resource and sensory effect quality<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />17<br />Test Sequences<br />Dur (sec) Genre #Effects<br />Rambo 4 58.11 Action 10<br />ZIB Flash 83.04 News 1<br />Babylon A.D. 118.42 Action 28<br />Woist Klaus? 59.16 Comm. 16<br />Earth 66 Docu. 24<br />Formula 1 116.2 Sports 43<br />New Five-Level Enhancement Scale<br />5 Big enhancement<br />4 Little enhancement<br />3 Imperceptible<br />2 Annoying<br />1 Very annoying<br />
  18. 18. Experiment I (cont’d)<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />18<br />
  19. 19. Major Findings from Experiment I<br />Sensory effects is a vital tool for enhancing the user experience<br />Action, sports, and documentary genres benefit more from these additional effects<br />Rambo 4 and Babylon A.D. are from the same genre, the results differ slightly<br />Commercial genre can also profit from the additional effects but not at the same level as documentary<br />News genre will not profit from these effects<br />Those videos presented twice differ in the results ➪ test method may not have functioned properly<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />19<br />M. Waltl, C. Timmerer, H. Hellwagner, "Increasing the User Experience of Multimedia Presentations with Sensory Effects", 11th Int’l Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10), Desenzano del Garda, Italy, Apr. 2010.<br />
  20. 20. Experiment II<br />Aim: investigate the relationship of the QoE to various video bit-rates of multimedia contents annotated with sensory effects.<br />Subjective quality gap between video resources annotated with and without sensory effects at different bit-rates <br />Tools<br />Sensory Effect Media Player (SEMP)<br />Test sequences annotated with sensory effects: action (Babylon A.D.) and documentary (Earth)<br />Absolute Category Rating with Hidden Reference (ACR-HR) method using a five-point discrete scale from excellent to bad<br />Novel voting device for continuous voting <br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />20<br />with<br />
  21. 21. Experiment II (cont’d)<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />21<br />Test Sequences<br />Voting Devices<br />Sequence Babylon A.D. Earth<br />Duration 35s 21s<br />Resolution 1280 x 544 1280 x 720<br />Motion High Low<br />Nr. of Effects W: 7; V: 9 W: 8; V: 1<br />Bit-rates Kbit/s PSNR Kbit/s PSNR<br />Low Quality 2154 38.93 2204 38.11<br />Medium Quality 3112 41.27 3171 40.65<br />High Quality 4044 42.95 4116 42.27<br />Highest Quality 6315 N/A 6701 N/A<br />Procedure<br />
  22. 22. Experiment II (cont’d)<br />MOS vs. PSNR/bit-rate for Babylon A.D.<br />MOS vs. PSNR/bit-rate for Earth. <br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />22<br />
  23. 23. Major Findings from Experiment II<br />Results confirm the observations from the previous experiment<br />MOS of the lowest bit-rate version with sensory effectsis (always) higher than the MOS of all higher bit-rate variants without sensory effects<br />Sequence enriched with sensory effects is 0.5 MOS points higher on average than without sensory effects<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />23<br />M. Waltl, C. Timmerer, H. Hellwagner, "Improving the Quality of Multimedia Experience through Sensory Effects", Proc. 2nd Int’l. Workshop on Quality of Multimedia Experience (QoMEX’10), Trondheim, Norway, Jun. 2010.<br />
  24. 24. Part ThreeSensory Experience on Mobile Devices?<br />Sensory effects on traditional mobile devices<br />Haptic – vibration<br />Tactile (passive) – position/location, motion/rotation<br />Light – backlight, ambient<br />Scent, the Smellophone?<br />How about other mobile devices?<br />E.g., in transportation (bus, train, car, plane)<br />Open issues<br />Tradeoff with battery consumption (optimization problem)<br />Applications and subjective quality assessments<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />24<br />
  25. 25. Conclusions<br />MPEG-V Overview (ISO/IEC 23005)<br />Pt. 1: Architecture<br />Pt. 3: Sensory Information<br />Sensory Information and HW/SW components<br />Sensory Effect Description Language<br />Sensory Effect Vocabulary<br />SEVino, SESim, SEMP, and amBX+SDK<br />How to improve QoE?➪ Results from experiments so far …<br />… to be continued!<br />Towards a Utility Model for Sensory Experiences <br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />25<br />
  26. 26. Sensory Experience Lab<br />http://selab.itec.aau.at/<br />Software and Services<br />Standardization<br />Publications<br />Media<br />Funding<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />26<br />
  27. 27. Acknowledgments<br />EC projects for funding this activity<br />NoE INTERMEDIA (NoE 038419)<br />http://intermedia.miralab.ch/<br />P2P-Next project (FP7-ICT-216217)<br />http://www.p2p-next.eu<br />ALICANTE project (FP7-ICT-248652)<br />http://www.ict-alicante.eu<br />COST ICT Action IC1003<br />QUALINET – European Network on Qualityof Experience in Multimedia Systems and Services<br />http://www.qualinet.eu/<br />Markus Waltl for implementing, preparing, conducting, evaluating almost all the experiments<br />Benjamin Rainer for implementing the browser plug-in + Web tests<br />Hermann Hellwagner for his advice and feedback<br />ISO/IEC MPEG and its participating members for their constructive feedback during the standardization process<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />27<br />
  28. 28. References<br />Markus Waltl, Christian Timmerer, Hermann Hellwagner, “A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects”, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009), San Diego, USA, July 29-31, 2009.<br />C. Timmerer, J. Gelissen, M. Waltl, and H. Hellwagner, “Interfacing with Virtual Worlds”, Proceedings of the NEM Summit 2009, Saint-Malo, France, September 28-30, 2009.<br />M. Waltl, Enriching Multimedia with Sensory Effects, VDM Verlag Dr. Müller, February, 2010.<br />M. Waltl, C. Timmerer and H. Hellwagner, “Increasing the User Experience of Multimedia Presentations with Sensory Effects”, Proceedings of the 11th International Workshop on Image Analysis for Multimedia Interactive Services (WIAMIS’10), Desenzano del Garda, Italy, April 12-14, 2010.<br />C. Timmerer, M. Waltl, and H. Hellwagner, “Are Sensory Effects Ready for the World Wide Web?”, Proceedings of the Workshop on Interoperable Social Multimedia Applications (WISMA 2010), Barcelona, Spain, May 19-20, 2010.<br />M. Waltl, C. Timmerer, and H. Hellwagner, “Improving the Quality of Multimedia Experience through Sensory Effects”, Proceedings of the 2nd International Workshop on Quality of Multimedia Experience (QoMEX2010), Trondheim, Norway, June 21-23, 2010.<br />M. Waltl, C. Raffelsberger, C. Timmerer, and H. Hellwagner, “Metadata-based Content Management and Sharing System for Improved User Experience”, Proc. of the 4th InterMedia Open Forum (IMOF 2010), Palma de Mallorca, Spain, September 1, 2010.<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />28<br />
  29. 29. Thank you for your attention<br />... questions, comments, etc. are welcome …<br />Ass.-Prof. Dipl.-Ing. Dr. Christian Timmerer<br />Klagenfurt University, Department of Information Technology (ITEC)<br />Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA<br />christian.timmerer@itec.uni-klu.ac.at<br />http://research.timmerer.com/<br />Tel: +43/463/2700 3621 Fax: +43/463/2700 3699<br />© Copyright: Christian Timmerer<br />29<br />2011/09/05<br />Christian Timmerer, Alpen-Adria-Universität, Austria<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×