QoMEX’09A Test-bed For Quality of Multimedia Experience Evaluation of Sensory EffectsChristian Timmerer and Markus WaltlKlagenfurt University (UNIKLU)  Faculty of Technical Sciences (TEWI)Department of Information Technology (ITEC)  Multimedia Communication (MMC)http://research.timmerer.com  http://blog.timmerer.com  mailto:christian.timmerer@itec.uni-klu.ac.atCo-authors: Markus Waltl, Christian Timmerer, and Hermann HellwagnerAcknowledgment: This work is supported in part by the European Commission in the context of the InterMedia project. Further information is available at http://intermedia.miralab.unige.ch/.Slides available at http://www.slideshare.net/christian.timmerer

OutlineIntroduction / MotivationSensory Effect Description LanguageTest-Bed: Annotation Tool and SimulatorTest Environment and Preliminary ResultsConclusionsDemo & Video2009/07/312Christian Timmerer, Klagenfurt University, Austria

IntroductionUniversal Multimedia Access (UMA)Anywhere, anytime, any time + technically feasibleMain focus on devices and network connectivity issuesUniversal Multimedia Experience (UME)Take the user into accountMultimedia Adaptation and Quality Models/MetricsSingle modality (i.e., audio, image, or video only) or a simple combination of two modalities (i.e., audio and video)Triple user characterization modelSensorial, e.g., sharpness, brightnessPerceptual, e.g., what/where is the contentEmotional, e.g., feeling, sensationAmbient IntelligenceAdd’llight effects are highly appreciated for both audio and visual contentCalls for a scientific framework to capture, measure, quantify, judge, and explain the user experience 2009/07/31Christian Timmerer, Klagenfurt University, Austria3B. de Ruyter, E. Aarts. “Ambient intelligence: visualizing the future”, Proceedings of the Working Conference on Advanced Visual Interfaces, New York, NY, USA, 2004, pp. 203–208.E. Aarts, B. de Ruyter, “New research perspectives on Ambient Intelligence”, Journal of Ambient Intelligence and Smart Environments, IOS Press, vol. 1, no. 1, 2009, pp. 5–14. F. Pereira, “A triple user characterization model for video adaptation and quality of experience evaluation,” Proc. of the 7th Workshop on Multimedia Signal Processing, Shanghai, China, October 2005, pp. 1–4.

MotivationConsumption of multimedia content may stimulate also other sensesVision or auditionOlfaction, mechanoreception, equilibrioception, thermoception, …Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects2009/07/31Christian Timmerer, Klagenfurt University, Austria4… giving her/him the sensation of being part of the particular media➪ worthwhile, informative user experience

Sensory Effect Description Language (SEDL)XML Schema-based language for describing sensory effectsBasic building blocks to describe, e.g., light, wind, fog, vibration, scentMPEG-V Part 3, Sensory InformationAdopted MPEG-21 DIA tools for adding time information (synchronization)Actual effects are not part of SEDL but defined within the Sensory Effect Vocabulary (SEV)Extensibility: additional effects can be added easily w/o affecting SEDLFlexibility: each application domain may define its own sensory effectsDescription conforming to SEDL :== Sensory Effect Metadata (SEM)May be associated to any kind of multimedia content (e.g., movies, music, Web sites, games)Steer sensory devices like fans, vibration chairs, lamps, etc. via an appropriate mediation device➪ Increase the experience of the user➪ Worthwhile, informative user experience2009/07/31Christian Timmerer, Klagenfurt University, Austria5

Sensory Effect Description Language (cont’d)2009/07/31Christian Timmerer, Klagenfurt University, Austria6SEM ::=[DescriptionMetadata](Declarations|GroupOfEffects|Effect|ReferenceEffect)+Declarations ::= (GroupOfEffects|Effect|Parameter)+GroupOfEffects ::= timestamp EffectDefinitionEffectDefinition (EffectDefinition)*Effect ::= timestamp EffectDefinitionEffectDefinition ::= [activate][duration][fade][alt] [priority][intensity][position][adaptability]

Test-Bed: Annotation Tool and SimulatorAnnotation Tool: SEVinoSimulator: SESim2009/07/31Christian Timmerer, Klagenfurt University, Austria7

Test EnvironmentBased on amBX (Ambient Experience) system + SDKTwo fan devices, a wrist rumbler, two sound speakers, a subwoofer, two lights, and a wall washerEverything controlled by SEM descriptionsexcept light effect➪ automatic color calculation is deployedAdvantagesReduction of description sizeSpeeds up authoring stageDifferent automatic color calculation methods may lead to different user experiences(1) Average color in the RGB color space (2-4) Dominant color in the RGB, HSV, and HMMDRequires a lot of computational resources: (2-4) > (1) due to the management of color bins & amBX supports only RGB2009/07/31Christian Timmerer, Klagenfurt University, Austria8

Preliminary ResultsMore or less constant color patternA lot of different colors which change very rapidly2009/07/31Christian Timmerer, Klagenfurt University, Austria9Color calculation is performed only on every pth frame (p=5) for efficiency reasons

ConclusionsTest-bed for the QoMEX evaluation of sensory effectsSEVino: a video annotation tool for sensory effectsSESim: a corresponding simulation toolA real world test environment based on the amBX system and SDKMajor findingsAverage color for the automatic color calculation ➪ immediate reaction to color changes, appealing effects, low computational requirements, real-time applicableRGB, HSV, and HMMD dominant color ➪ smoother reaction to color changes, higher computational requirementsFuture workOptimization of the automatic color calculation (real-time)Subjective tests (already started & stay tuned)(Semi-)automatic extraction of sensory effect information2009/07/31Christian Timmerer, Klagenfurt University, Austria10

ReferencesM. Waltl, C. Timmerer, and H. Hellwagner, “A Test-Bed for Quality of Multimedia Experience Evaluation of Sensory Effects”, Proceedings of the First International Workshop on Quality of Multimedia Experience (QoMEX 2009), San Diego, USA, July 29-31, 2009.C. Timmerer, J. Gelissen, M. Waltl, and H. Hellwagner, “Interfacing with Virtual Worlds”, accepted for publication in the Proceedings of the 2009 NEM Summit, Saint-Malo, France, September 28-30, 2009.C. Timmerer, “MPEG-V: Media Context and Control”, 89th ISO/IEC JTC 1/SC 29/WG 11 (MPEG) Meeting, London, UK, June 2009. https://www-itec.uni-klu.ac.at/mmc/blog/2009/07/08/mpeg-v-media-context-and-control/MPEG-V: http://www.chiariglione.org/mpeg/working_documents.htm#MPEG-VMPEG-V reflector: http://lists.uni-klu.ac.at/mailman/listinfo/metaverse2009/07/31Christian Timmerer, Klagenfurt University, Austria11

2009/07/31Christian Timmerer, Klagenfurt University, Austria12Demo & Video

Thank you for your attention... questions, comments, etc. are welcome …Ass.-Prof. Dipl.-Ing. Dr. Christian TimmererKlagenfurt University, Department of Information Technology (ITEC)Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIAchristian.timmerer@itec.uni-klu.ac.athttp://research.timmerer.com/Tel: +43/463/2700 3621 Fax: +43/463/2700 3699© Copyright: Christian Timmerer132009/07/31Christian Timmerer, Klagenfurt University, Austria