Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life

2,348 views

Published on

In computing and communications systems, quality is often difficult to define. Attempts to understand this concept date back to Aristotle, who included quality as one of his 10 categories of human apprehension. ISO standard 8402:1986 defines quality as “the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs,” which embraces objective as well as subjective parameters. In practice, however, quality could be compared to the elephant in the famous Indian parable about a group of blind men who each feels a different part of the animal and, thus, they disagree as to what it looks like....

Published in: Technology
  • Many thanks for slide 109 ;)
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life

  1. 1. Quality of Experience in Multimedia Systems and Services: A Journey Towards the Quality of Life Christian Timmerer(AAU Klagenfurt, Austria) Fernando Pereira (IST-IT, Portugal) Touradj Ebrahimi (EPFL, Switzerland) IEEE International Conference on Multimedia& Expo (ICME) 11th July 2016, Seattle, WA, USA
  2. 2. Outline 1. Quality of Experience for Multimedia Systems and Services Fernando Pereira 2. Applications of QoE: Adaptive Video Streaming and Sensory Experience Christian Timmerer 3. Towards the Concept of Quality of Life Touradj Ebrahimi
  3. 3. Fernando Pereira: About Me … • Associate Professor at University of Lisbon, Portugal • Senior Researcher at Instituto de Telecomunicações, Lisbon, Portugal • More than 250 publications in international journals and conferences • One of the designers of the MPEG-4 and MPEG-7 standards • ICIP, PCS, VCIP, WIAMIS, QoMEX General or Technical Program Chair • Associate Editor of several journals • Editor-in-Chief of the IEEE Journal of Selected Topics in Signal Processing (2013-2015) • ISO/IEC Award for contributions to the MPEG-4 Visual Standard • SPS Distinguished Lecturer • IEEE Fellow in 2008 for “contributions to object-based digital video representation technologies and standards” • EURASIP Fellow in 2013 for “contributions to digital video representation technologies and standards” • IET Fellow in 2015 • IEEE SPS Board of Governors and EURASIP Board of Directors • Several Excellence Teaching Awards • JPEG (currently) and MPEG (past) Requirements Chair
  4. 4. 1. Quality of Experience for Multimedia Systems and Services A. What is Quality B. Quality of Service (QoS) C. Quality of Experience (QoE) D. Trends in QoE
  5. 5. A. What is Quality ?
  6. 6. Quality: a Simple yet Difficult Concept • Like many human sensations, quality is easy to understand but difficult to define. • According to Wikipedia: – A quality (from Latin - qualitas) is an attribute or a property. – Some philosophers assert that a quality cannot be defined. – In contemporary philosophy, the idea of qualities and especially how to distinguish certain kinds of qualities from one another remains controversial.
  7. 7. An Old, Largely Under-Investigated Concept Aristotle classified every object of human apprehension into 10 Categories – Substance – Quantity – Quality – Relation – Place – Time – Position – State – Action – Affection Aristotle, 384 BC – 322 BC, Greece
  8. 8. Quality: Some Definitions from the Dictionary (1) • Definition 1 – General : Measure of excellence or state of being free from defects, deficiencies, and significant variations. – ISO 8402-1986 standard defines quality as “the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs”. • Definition 2 – Manufacturing : Strict and consistent adherence to measurable and verifiable standards to achieve uniformity of output that satisfies specific customer or user requirements.
  9. 9. Quality: Some Definitions from the Dictionary (2) • Definition 3 – Objective : Measurable and verifiable aspect of a thing or phenomenon, expressed in numbers or quantities, such as lightness or heaviness, thickness or thinness, softness or hardness. • Definition 4 – Subjective : Attribute, characteristic, or property of a thing or phenomenon that can be observed and interpreted, and may be approximated (quantified) but cannot be measured, such as beauty, feel, flavor, taste.
  10. 10. Quality According to ISO 9000 • ISO 9000: a family of standards for quality management systems. • Quality of something can be determined by comparing a set of inherent characteristics with a set of requirements – High quality: if characteristics meet requirements – Low quality: if characteristics do not meet all requirements • Quality is a relative concept – Degree of quality
  11. 11. Quality is like an Elephant … The blind men and the elephant, poem by John Godfrey Saxe
  12. 12. Quality in QUALINET • Quality: Is the outcome of an individual’s comparison and judgment process. It includes perception, reflection about the perception, and the description of the outcome. • In contrast to definitions which see quality as “qualitas”, i.e. a set of inherent characteristics, QUALINET considers quality in terms of the evaluated excellence or goodness, of the degree of need fulfillment, and in terms of a “quality event” (see Martens & Martens, 2001, and Jekosch, 2005). • Event: An observable occurrence. An event is determined in space (i.e. where it occurs), time (i.e. when it occurs), and character (i.e. what can be observed). from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013
  13. 13. What is QUALINET ? • Group of institutions and companies interest in multimedia quality. • Coordination of multidisciplinary QoE research in Europe and beyond. • Strengthening dissemination efforts through already established, and new initiatives, e.g. QoMEX, special events, books, journals. • Strengthening interaction between academia and industry (industrial forum, STSM, …). • Strengthening educational efforts in QoE, e.g. summer schools, PhD events, exchange of young researchers. • Coordinated contribution to international standardization bodies, e.g. ISO/IEC, ITU-T, VQEG, MPEG, JPEG. In summary, group of researchers interested in (multimedia) QoE issues, both theoretical and practical … Open to new researchers … http://www.qualinet.eu/
  14. 14. 1. Quality of Experience for Multimedia Systems and Services A. What is Quality B. Quality of Service (QoS) C. Quality of Experience (QoE) D. Trends in QoE
  15. 15. B. Quality of Service
  16. 16. Quality of Service (QoS): in Theory “[The] Totality of characteristics of a telecommunications service that bear on its ability to satisfy stated and implied needs of the user of the service.” ITU-T Rec. E.800, 2008 • QoS is focused on telecommunications services. • The context of usage and the user characteristics are not comprehensibly addressed by QoS as defined by the ITU. from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013
  17. 17. Quality of Service (QoS): De-Facto • The QoS de-facto definition deals mostly with physical, measurable performance factors of networks and delivery platforms in general. • Sometimes, also application-level factors, such as encodings, and their effect on the underlying network's performance are addressed, e.g. by taking more of the available bandwidth. Quality of Service (QoS) refers to a collection of networking technologies and measurement tools that allow for the network to guarantee delivering predictable results. partly from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013
  18. 18. Quality in QoS Framework: Several Dimensions Network Quality Capacity Coverage Handoff Link Quality Bitrate Frame/Bit/Packet loss Delay User Quality Speech fidelity Audio fidelity Image fidelity Video fidelity The multimedia signal processing community is already often using concepts such as the Mean Opinion Score (MOS) which directly involves users …
  19. 19. QoS in Computer Networks and Communications • Quality of Service (QoS) – Resource reservation control mechanisms – Ability to provide different priority to different applications, users, or data flows – Guarantee a certain level of performance (quality) to a data flow • (Service) Provider-centric concept
  20. 20. QoS Boundaries • Scope: QoS typically focuses on telecommunications services. • Focus: QoS deals with performance aspects of physical systems. • Methods: QoS has a very technology-oriented approach, and it relies on analytic approaches and empirical or simulative measurements. from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013
  21. 21. User Quality: Mostly Signal Fidelity • Subjective Evaluation • Objective Evaluation
  22. 22. Subjective Evaluation • Subjective tests aim at producing User Opinion Scores as a delicate mixture of ingredients and choices: – Test & lab environment – Test material – Test methodology – Test subjects – Analysis of the data
  23. 23. What is Mean Opinion Score (MOS)? • Widely used in many fields: – Politics/Elections – Marketing/Advertisement – Food industry – Multimedia – … • The likely level of satisfaction of a specific service/product dimension, e.g. visual quality, as appreciated by an averageuser (from a provider point of view). • Should be performed such that it generates reliable and reproducible results – Subjective evaluation methodology – More complex and difficult that it a priori seems – Much used for (and limited to) video and audio subjective qualities
  24. 24. Objective Evaluation • Subjective tests are time consuming, expensive, and difficult to design … • Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired – Full reference metrics – No-reference metrics – Reduced reference metrics
  25. 25. FR, RR and NR Scenarios • Full Referenceapproach: • Reduced Reference approach: • No-Reference approach: Input/Reference signal Output/Processed signal Signal processing Input/Reference signal Output/Processed signal Signal processing FR METRIC NR METRIC Input/Reference signal Output/Processed signal Signal processing Features extraction RR METRIC
  26. 26. Automatic MOS Predictors are Essential … Full Reference scenario • Most automatic MOS predictors are based on fidelity measures • Metrics look at the fidelity of the signal when compared to an explicit ´perfect’ reference: processed signal = perfect quality reference signal + error signal • Examples: – Mean Square Error (MSE) – Peak Signal to Noise Ratio (PSNR) – Weighted PSNR – Masked PSNR – Structural SIMilarity (SSIM) – Multiscale Structural SIMilarity (MSSIM) – Visual Information Fidelity (VIF)
  27. 27. 1. Quality of Experience for Multimedia Systems and Services A. What is Quality B. Quality of Service (QoS) C. Quality of Experience (QoE) D. Trends in QoE
  28. 28. C. Quality of Experience
  29. 29. Changing Landscape
  30. 30. UHD, 4K HDR HFR 3D Light fields Point clouds …
  31. 31. 31
  32. 32. Many Events ... Building Experiences ... • Event: An observable occurrence. An event is determined in space (i.e. where it occurs), time (i.e. when it occurs), and character (i.e. what can be observed). – Sensation refers to the responses of sensory receptors and sense organs to environmental stimuli. – Perception is a process which involves the recognition and interpretation of stimuli which register our senses. – Emotion is any relatively brief conscious experience characterized by intense mental activity and a high degree of pleasure or displeasure. • Experience: An experience is an individual’s stream of perception and interpretation of one or multiple events. partly from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013
  33. 33. So, Users are More than Perception Engines …
  34. 34. Many Services Sell Emotions ...
  35. 35. Multimedia Nowadays … • Multimediais about sharing experiences (real or imaginary) with others. • In a way, it all started with story telling and wall drawing around the fire in the caves of early men. • Modern multimedia systemsare evolved versions of the good old story tellingand wall drawing, which hopefullyoffer increasingly richer experiences. • The degree of richness of the experience may be measured by Quality of Experience (QoE).
  36. 36. Evolving Quality Paradigms
  37. 37. What do People Talk about when they Talk about QoE ? • “The degree of fulfillment of an intended experienceon a given user.” by Touradj Ebrahimi, 2001 • “perceived user experienceis psychological in nature and changes in different environmental conditions and with different multimedia devices.” from QoMEX 2009 Call for Papers • “The overall acceptability of an application or service, as perceived subjectively by the end user.” as defined by the ITU The term ‘experience’ is appealing because it implicitly promises individual engagement … Look good, sound good, and feel good !
  38. 38. QUALINET QoE Definition • Quality of Experience (QoE) is the degree of delight or annoyance of the user (persona) of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and/or enjoyment of the application or service in the light of the user’s personality and current state (context). • Experience: An experience is an individual’s stream of perception and interpretation of one or multiple events. • QoE feature: A perceivable, recognized and namable characteristic of the individual’s experience of a service which contributes to its quality. In the context of communication services, QoE can be influenced by factors such as service, content, network, device, application, and context of use. from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013
  39. 39. Moving to Quality of Experience • Quality of Service: Value of the average user’s service richness estimated by a service/product/contentprovider • Quality of Experience: Value (estimated or actually measured) of a specific user’s experience richness Quality of Experience is the dual (and extended) view of Quality of Service ! QoS=provider-centric QoE=user-centric
  40. 40. Factors Impacting Quality of Experience Context • System/Techni cal Influence Factors • Human/User Influence Factors • Context Influence Factors • Content Influence Factors • Social and Psychological Influence Factors
  41. 41. Experiences are Individual ! • Applications and Services may have to be designed to provide individual experiences ... • This involves capabilities allowing the user to gain control, e.g. interaction, personalization, recommendation, etc. • In fact, the user contributes to build is own experience ... If the system/service allows ...
  42. 42. How Shall a Multimedia User Experience Be ? Depending on the specific application, it may have to be • Faithful - accuracy • Truthful – realistic if relevant, synchronization • Immersive – natural, multimodal consistency • Contextual - adaptive • Engaging – fun, intense, emotional • Effective – fast, recognition • Useful – task performing • Interactive – natural, short delay • Intuitive, Easy – interfaces • …
  43. 43. QoE Modeling QoE modelingmay consider more or less influence factors dependingon the service/application, each with a different weight on the overall assessment. QoE is multi-dimensional, multi-modal and multi-sensorial. User centered influence factors are expected to be dominating. • System factors – technical properties (as in QoS) • Human/User factors – individual properties – sensorial properties – perceptual properties – emotional properties • Context factors – environmental/physical properties – temporal properties – service properties – economic properties – social properties • Content factors • …
  44. 44. A Practical QoE Model Example: IPTV • Video quality • Audio quality • Audiovisual syncronization • Stall occurence • Error resilience • Random access • Channel hoppingdelay • Interface usability • Navigation capabilities • Personalization capabilities • Metadata quality • Immersion effectiveness • …
  45. 45. QoE: Not an Easy Target … Why Should it be ? QoS QoP QoE QoS/P/E: Quality of Service/Perception/Experience
  46. 46. Experiences are multisensorial ...
  47. 47. Building Multisensorial Immersion ... • To insert Marianna’s movie ... Feel-around, from Kentucky Fried Movie
  48. 48. QoE Assessment: Again Subjective and Objective
  49. 49. QoS versus QoE • Scope: QoS typically focuses on telecommunications services, whereas QoE covers a much broader domain, which sometimes does not even involve telecommunications. • Focus: QoS deals with performance aspects of physical systems, whereas QoE deals with the users' assessment of system performance, as colored by context, culture, the users' expectations with respect to the system or service and their fulfillment, socio- economic issues, and psychological profiles, among other factors. • Methods: QoS has a very technology-oriented approach, whereas QoE requires a multi-disciplinary and multi-methodological approach for its understanding. • But it is also important to remember that QoE is, in a large part of instances, highly dependent on QoS. from “Qualinet White Paper on Definitions of Quality of Experience”, March 2013
  50. 50. QoE is like a (Bigger) Elephant … The blind men and the elephant, Poem by John Godfrey Saxe
  51. 51. QoE in Networked Multimedia
  52. 52. QoE Related Standardization Efforts • Standardization efforts in quality assessment and metrics – ITU-T SG 12 (Performance, QoS and QoE) – MPEG/ITU-T (High Efficiency Video Coding, HEVC) – MPEG (3D video coding, FTV, HDR) – Video Quality Experts Group (VQEG) – JPEG (Advanced Image Coding, AIC) – … QUALINET established links and deep collaborations with all of them !
  53. 53. 1. Quality of Experience for Multimedia Systems and Services A. What is Quality B. Quality of Service (QoS) C. Quality of Experience (QoE) D. Trends in QoE
  54. 54. D. Trends in QoE
  55. 55. QoE is Becoming Inevitable … • Digital world has (re-)discovered the notion of quality – Lower quality content is less and less tolerated by end-users in some environments – However, other environments seem to accept much lower quality and still be successful • Increasing interest in QoE – Extending from device-centric and system-centric quality optimization to end-to-end and especially user-centric optimization
  56. 56. QoE Holistic Approach • Marketing • Business model, e.g. prices, fidelization • System factors • Context factors • Human factors • Personalization • Content (and metadata) • Interface • Client support • ...
  57. 57. NOS UMA: an Example • Ultra HD 4K • Portability accross terminals, i.e. follows you • Voice control (voice recognition ?), i.e. recognizes you • User profiles within same family, i.e. individualizes you • Recommendations based on user characterization, i.e. targets you • Complementary content for the favourite series, i.e. thinks on you • Time warping, i.e. helps you • ... A TV that knows you !
  58. 58. QoE in Industry • QoE is becoming mainstream. • Many companies now speak about QoE. • Personalization, interaction and recommendation capabilities empower the user to create more individual experiences! • However, QoE has a budget impact in terms of network and system design, dimensioning, operation, maintenance, etc. • But QoE is becoming more affordable in many application domains … • Embracing QoE principles may bring revenue, e.g. by increasing viewing times and reducing churn.
  59. 59. Challenges Ahead • Content-dependentqualityassessmentmethods and metrics • Context-dependentqualityassessmentmethods and metrics • Quality assessmentmethods and metrics beyond AV (haptics, smell, …) • Multi-modal quality assessmentmethods and metrics (AV, …) • 3D quality assessmentmethods and metrics (3D sound, 3D video, …) • New modalities content quality assessmentmethods and metrics • Interaction qualitymetrics (closely related to usability) • Presence/immersion qualitymetrics • Role of emotions • Virtual reality immersive experiences • …
  60. 60. New Sensors ...
  61. 61. A Light Field Image … Behind each microlens, a micro-image (MI) is formed …
  62. 62. Light Field Photography: Array of Cameras
  63. 63. New Displays Microsoft Hololens Oculus Rift Microsoft holographic display InnoVision Diamond Series holographic projector USC light field display Holografika HoloVizio light field display
  64. 64. QoE for Virtual Reality • Compelling immersive and realistic visual experiences ! • Provides visual depth cues, such as stereopsis, binocular occlusions, vergence, full motion parallax and natural view-dependent lighting. • High resolution and high frame rate. • Low latency spatial random access. • Low motion-to-photon latency. On current HMDs, the closest depth for an object of interest is recommended to be at 0.75m without causing excessive eyestrain.
  65. 65. Holoportation: Virtual 3D Teleportation Courtesy of P.Chou, Microsoft
  66. 66. What Does this all Mean ? • Era of user-centric multimedia has already started … User is King/Queen ! • It is not anymore sufficient to merely add new features and functionalities to multimedia systems. • True added value in terms of impact on user’s experience of such features and functions should be evaluated and demonstrated. • Quality of Experience plays a central role in this new game ! Already targeting revenue …
  67. 67. Assessing Quality of Experience … A Bit Like Measuring ‘Happiness’ …
  68. 68. Take-Home Messages • QoE is user-centric ! • QoE is individual, multidimensional and multisensorial. • Services and systems are increasingly designed to allow the users to maximize its QoE. • Industry is increasingly embracing QoE principles because they may bring revenue. • QoE assessment is costly and risky but worth doing it. • Int’l Conference on Quality of Multimedia Experience (QoMEX): http://qomex.org/
  69. 69. 2. Applications of QoE: Adaptive Video Streaming and Sensory Experience A. Adaptive Video Streaming Principles and QoE B. Quality of Sensory Experience (QuaSE)
  70. 70. Christian Timmerer: About Me … • Associate Professor at Alpen-Adria-Universität Klagenfurt, Austria (blog.timmerer.com, dash.itec.aau.at) • Chief Innovation Officer | Head of Standardization and Research at Bitmovin Inc., bitmovin.com • Geschäftsführer Förderverein Technische Fakultät, ftf.or.at • Lecturer | Carinthia University of Applied Sciences, www.fh-kaernten.at • Research interest: immersive multimedia communication, streaming, adaptation, Quality of Experience, and sensory experience • More than 170 publications in international journals and conferences • General chair: WIAMIS’08, QoMEX’13, QCMan’14, MMSys’16 • Associate editor/editorial board: IEEE Computer, IEEE Trans. on Multimedia, Signal Processing: Image Communication, MTAP, IEEE Computing Now, ACM SIGMM Records, ACM SIGMM OSSC • Vice chair of IEEE ComSoC MMTC, WG leader in QUALINET • Research projects: FP6-IST-DANAE (2004-2006), FP6-IST-ENTHRONE (2006- 2008), FP7- ICT-P2P-Next (2008-2012), FP7-ICT-ALICANTE (2010-2013), FP7-ICT-SocialSensor (2010- 2014), COST-IC1003-Qualinet (2010-2014), FFG-AdvUHD-DASH (2014-2016), and FP7-ICT- ICoSOLE (2013-2016) • MPEG: MPEG-21, MPEG-M, MPEG-V, MPEG-DASH • IEEE Senior member; ACM member
  71. 71. Applications of QoE: Adaptive Video Streaming and Sensory Experience Priv.-Doz. Dr. Christian Timmerer [Ack: Ali C. Begen, MediaMelon, Inc., OzyeginUniversity] Alpen-Adria-Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Information Technology (ITEC) w Multimedia Communication (MMC) w Sensory Experience Lab (SELab) http://blog.timmerer.com w http://selab.itec.aau.at/ w http://dash.itec.aau.at w christian.timmerer@itec.aau.at Chief Innovation Officer (CIO) at bitmovin GmbH http://www.bitmovin.comw christian.timmerer@bitmovin.com Tutorial @ ICME 2016, July 2016 http://www.slideshare.net/christian.timmerer
  72. 72. Importance of Multimedia Delivery • Multimedia is predominant on the Internet • Real-time entertainment – Streaming video and audio – More than 70% of Internet traffic at peak periods • Popular services – YouTube (17.85%), Netflix (37.05%), Amazon Video (3.11%), Hulu (2.58%) – All delivered over-the-top (OTT) July 2016 ICME 2016 Tutorial, C. Timmerer 2 Global Internet Phenomena Report: Dec 2015
  73. 73. Open Digital Media Value Chain July 2016 ICME 2016 Tutorial, C. Timmerer 3 Create Content Aggregate Monetize Distribute Content Consume Content Any Content Any Storefront Any Network Any Device CDNsMedia Protocols Internet Transport DRM Encoding Encapsulation Dynamic Ads Clients Happy User
  74. 74. Common Annoyances in Streaming • Wrong format • Wrong protocol • Plugin requirements • DRM issues • Long start-up delay • Poor quality • Frequent stalls • Quality oscillations • No seeking features July 2016 ICME 2016 Tutorial, C. Timmerer 4
  75. 75. Over-The-Top – Adaptive Media Streaming • In a nutshell… July 2016 ICME 2016 Tutorial, C. Timmerer 5 Adaptation logic is within the client, not normatively specified by the standard, subject to research and development
  76. 76. Multi-Bitrate Encoding and Representation Switching July 2016 ICME 2016 Tutorial, C. Timmerer 6 Contents on the Web Server Movie A – 200 Kbps Movie A – 400 Kbps Movie A – 1.2 Mbps Movie A – 2.2 Mbps . . . . . . Movie K – 200 Kbps Movie K – 500 Kbps Movie K – 1.1 Mbps Movie K – 1.8 Mbps . . . . . . Time (s) Start quickly Keep requesting Improve quality Loss/congestion detection Revamp quality . . . . . . Segments
  77. 77. Adaptive Streaming over HTTP July 2016 ICME 2016 Tutorial, C. Timmerer 7 … … … … HTTPGETs Client Buffer Media Player HTTP Server
  78. 78. Scope of DASH: what is specified? July 2016 ICME 2016 Tutorial, C. Timmerer 8 Media Presentation on HTTP Server DASH-enabled ClientMedia Presentation Description . . . Segment … . . .Segment … . . . Segment … . . .Segment … … Segments located by HTTP-URLs DASH Control Engine HTTP/1.1 HTTP Client MPD Parser Media Engine On-time HTTP requests to segments
  79. 79. Scope of DASH: what is specified? July 2016 ICME 2016 Tutorial, C. Timmerer 9 Media Presentation on HTTP Server DASH-enabled ClientMedia Presentation Description . . . Segment … . . .Segment … . . . Segment … . . .Segment … … Segments located by HTTP-URLs DASH Control Engine HTTP/1.1 HTTP Client MPD Parser Media Engine On-time HTTP requests to segments
  80. 80. DASH Data Model July 2016 ICME 2016 Tutorial, C. Timmerer 10 MPD Period id = 1 start = 0 s Period id = 3 start = 300 s Period id = 4 start = 850 s Period id = 2 start = 100 s Adaptation Set 0 subtitle turkish Adaptation Set 2 audio english Adaptation Set 1 BaseURL=http://abr.rocks.com/ Representation 2 Rate = 1 Mbps Representation 4 Rate = 3 Mbps Representation 1 Rate = 500 Kbps Representation 3 Rate = 2 Mbps Resolution = 720p Segment Info Duration = 10 s Template: 3/$Number$.mp4 Segment Access Initialization Segment http://abr.rocks.com/3/0.mp4 Media Segment 1 start = 0 s http://abr.rocks.com/3/1.mp4 Media Segment 2 start = 10 s http://abr.rocks.com/3/2.mp4 Adaptation Set 3 audio german Adaptation Set 1 video Period id = 2 start = 100 s Representation 3 Rate = 2 Mbps Selection of components/tracks Well-defined media format Selection of representations Splicing of arbitrary content like ads Chunks with addresses and timing
  81. 81. July 2016 ICME 2016 Tutorial, C. Timmerer 11 type=static typically, for on demand content Base URL of the segments Subtitles Audio adaptation set with different representations (bw) Video adaptation set with different representations (bw) Different codecs (profiles) Segment URL constructed with template and base URL
  82. 82. http://www.dash-player.com/demo/ July 2016 ICME 2016 Tutorial, C. Timmerer 12
  83. 83. Adaptive Streaming Content Workflow July 2016 ICME 2016 Tutorial, C. Timmerer 13 Source Transcoding Encapsulation Encryption Origin Server HelperDistribution Client Linear: Multicast VoD: FTP, RTMP, HTTP, etc. Unicast HTTP (PUSH), FTP, etc. HTTP GET small objects Single highest-bitrate stream Multiple streams at target bitrates Multiple streams at target encapsulation formats Large video/virtual files and manifests
  84. 84. Adaptive Streaming Content Workflow Simplified July 2016 ICME 2016 Tutorial, C. Timmerer 14 Standard Delivery Infrastructure (CDN) Source Transcoding Encapsulation Encryption Multiple streams: video [bitrate (32000-20000000), profile (baseline, main, high), preset (standard, professional, premium), height (128- 7680), width (96-4320), frame rate (1-120), codec (h264, hevc)], audio: [bitrate (8000-256000), sample rate (0, 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000)] Single highest-bitrate stream: HTTP, FTP, RTMP; mp4, ts; AVC, AAC, Subtitles Multiple streams at target encapsulation formats: DASH (MPD + mp4), HLS (m3u8, ts) Multiple streams with multiple DRM formats: MPEG-CENC, Widewine, PlayReady, PrimeTime, Fairplay Player Heterogeneous Clients e.g. Bitmovin HTML5 Adaptive Player DASH, HLS, HTML5, MSE, EME
  85. 85. QoE for DASH Services • Different application domains have different QoE requirements – Need to provide specializations of the general QoE definition – Take into account requirements formulated by means of influence factors and features of QoE • QoE influence factors for DASH – Initial/start-up delay (low) – Buffer underruns, stalls, freezes (zero) – Quality switches (low) – Media throughput (high) – … July 2016 ICME 2016 Tutorial, C. Timmerer 15
  86. 86. ! h t t p s : / / b i t m o v i n . c o m / QoE Evaluation for DASH-based Services • Test sequence – Many datasets available – Adopted Big Buck Bunny & DASHed it • Players – bitdash – Proprietary solutions (smooth, HLS, HDS) – YouTube, dash.js, DASH-JS – …and compare it with ten different adaptation algorithms • Objective evaluation – Common test setup using network emulation & bandwidth shaping – Predefined bandwidth trajectory (or real network traces) • Subjective evaluation – Lab [ITU-T B.500 / P.910] vs. crowdsourcing with special platforms or social networks July 2016 ICME 2016 Tutorial, C. Timmerer 16
  87. 87. Crowdsourced QoE Evaluation • Quality of Experience … – Mean Opinion Score [0..100] – [other objective metrics: start-up time, throughput, stalls] • … Web-based Adaptive HTTP Streaming Clients … – HTML5+MSE: DASH-JS (dash.itec.aau.at), dash.js (DASH-IF, v1.1.2), YouTube • … Real-World Environments … – DASH-JS, dash.js hosted at ITEC/AAU (~ 10Gbit/s) – YouTube hosted at Google data centers – Content: Tears of Steel @ 144p (250 kbit/s), 240p (380 kbit/s), 360p (740 kbit/s), 480p (1308 kbit/s), and 720p (2300 kbit/s); segment size: 2s – Users access content over the open Internet (i.e., real-world environment) • … Crowdsourcing – Campaign at Microworker platform (others also possible: Mechanical Turk, social networks) limited to Europe, USA/Canada, India – Screening Techniques: Browser fingerprinting, stimulus presentation time, QoE ratings and pre- questionnaire July 2016 ICME 2016 Tutorial, C. Timmerer 17 B. Rainer, C. Timmerer, “Quality ofExperience ofWeb-based Adaptive HTTP Streaming Clients in Real-World Environments using Crowdsourcing”, Proceedings of International Workshop on VideoNext: Design,Quality andDeployment of Adaptive Video Streaming, Sydney, Australia,Dec. 2014.
  88. 88. MOS and Average Bitrate • 288 microworkers, 33 screened (Fingerprinting: 20, presentation time: 6, QoE ratings and pre- questionnaire: 7), 175 male and 80 female, majority (80%) is aged between 18 and 37 July 2016 ICME 2016 Tutorial, C. Timmerer 18
  89. 89. Startup Time and Number of Stalls July 2016 ICME 2016 Tutorial, C. Timmerer 19
  90. 90. Results Summary • DASH-JS – High start-up time – Low number of stalls – Good throughput, QoE • dash.js – Low start-up time – High # stalls – Low throughput – Low QoE • YouTube – Low start-up time – Low number of stalls – Best throughput, QoE July 2016 ICME 2016 Tutorial, C. Timmerer 20
  91. 91. Now, 10 different adaptation logics … July 2016 ICME 2016 Tutorial, C. Timmerer 21 Adaptation logics well-known in research literature Predefined bandwidth trajectory and test setup Different segment sizes, RTTs, HTTP/2, etc. C. Timmerer, M. Maeiro, B. Rainer, “Which Adaption Logic? An Objective and Subjective Performance Evaluation of HTTP-basedAdaptiveMedia Streaming Systems”, arXiv cs.MM, June 2016, http://arxiv.org/abs/1606.00341.
  92. 92. July 2016 ICME 2016 Tutorial, C. Timmerer 22
  93. 93. July 2016 ICME 2016 Tutorial, C. Timmerer 23
  94. 94. DASH-JS vs. bitdash July 2016 ICME 2016 Tutorial, C. Timmerer 24 C. Timmerer, D. Weinberger, C. Mueller, and S. Lederer, “Ultra-High-Definition-Quality of Experience with MPEG-DASH”, Proceedings of the Broadcast EngineeringConference (BEC), NAB2015, Las Vegas, NV, USA, April 2015.
  95. 95. Objective Evaluations July 2016 ICME 2016 Tutorial, C. Timmerer 25 Stalls (lower is better)Average Bitrate (higher is better)
  96. 96. Stalls are really bad… July 2016 ICME 2016 Tutorial, C. Timmerer 26 Conviva: Viewer Experience Report. 2014
  97. 97. Conclusions (1) • MPEG-DASH defines formats only – Media Presentation Description (MPD) – Segment format: mp4, ts • MPEG-DASH is not – System, protocol, presentation, codec, interactivity, DRM, client specification – Other standards required for a complete ecosystem: e.g., DASH-IF, WAVE, HMTL5, MSE, EME • Do we need MPEG-DASH? (for adaptive media streaming) – Not necessarily: e.g., WebM + VPx + manifest & control end-to-end – Required to address heterogeneous environments • Role of standards sometimes overrated but often underestimated July 2016 ICME 2016 Tutorial, C. Timmerer 27
  98. 98. Conclusions (2) • QoE for DASH-based services (a rule of thumb) – Startup delay (low [but live vs. on-demand & short vs. long-tail content]) – Buffer underrun / stalls (zero) – Quality switches (low) and media throughput (high) – Energy- and cost-awareness (data plan) • No general applicable QoE model for DASH – (Too) many factors influencing / features of QoE for DASH-based services – Methodology for reproducible research is in place and well established – Ample research opportunities July 2016 ICME 2016 Tutorial, C. Timmerer 28 Main QoE factors for DASH Come up with our own QoE factor and design, conduct, analyze a small-scale experiment
  99. 99. Quality of Sensory Experience • Consumption of multimedia content may stimulate also other senses – Vision or hearing – Olfaction, mechanoreception, thermoception, … • Annotation with metadata providing so-called sensory effects that steer appropriate devices capable of rendering these effects July 2016 ICME 2016 Tutorial, C. Timmerer 29 … giving her/him the sensation of being part of the particular mulsemedia worthwhile, informative user experience
  100. 100. General Principle – Outline • General principle: there is a need for a scientific framework to capture, measure, quantify, judge, and explain the quality of (sensory) experience • Outline – [How to create, delivery, consume?] – How to capture and measure? – How to quantify? – How to judge and explain? July 2016 ICME 2016 Tutorial, C. Timmerer 30
  101. 101. How to create, delivery, consume? • Sensory Effect Description Language (SEDL) – Basic building blocks to describe, e.g., light, wind, fog, vibration, scent – MPEG-V Part 3, Sensory Information: Effects, GroupOfEffects – Adopted MPEG-21 DIA tools for adding time information (synchronization) • Description conforming to SEDL :== Sensory Effect Metadata (SEM) – Can be associated to any kind of multimedia content (e.g., movies, music, Web sites, games) – Support to be included in file (MP4) and transport (M2TS) formats • Tool support for creating (annotation tools) and consumption (players, Web plugins) selab.itec.aau.at • Devices: e.g., amBX (Ambient Experience) system + SDK, Gameskunk, Scentscape, etc. July 2016 ICME 2016 Tutorial, C. Timmerer 31
  102. 102. How to capture and measure? • Subjective quality assessments – Methodology: based on standard methods – Test content: different genres, manually annotated (cf. QUALINET DB) • Experiment I – Aim: Demonstrate sensory effects as a vital tool for enhancing the quality of experience depending on the actual genre • Experiment II – Aim: investigate the relationship of the QoE to various video bit-rates of multimedia contents annotated with sensory effects. – Subjective quality gap between video resources annotated with and without sensory effects at different bit-rates • [Experiment III] ambient lights & different color calculation settings • Experiment IV – Aim: investigate the enhancement of the QoE and how users’ emotions are elicited and influenced by Web videos annotated with and without sensory effects July 2016 ICME 2016 Tutorial, C. Timmerer 32
  103. 103. Experiment II: Results July 2016 ICME 2016 Tutorial, C. Timmerer 33 Sequence Babylon A.D. Earth Duration 35s 21s Resolution 1280 x 544 1280 x 720 Motion High Low Nr. of Effects W: 7; V: 9 W: 8; V: 1 Bit-rates Kbit/s PSNR Kbit/s PSNR Low Quality 2154 38.93 2204 38.11 Medium Quality 3112 41.27 3171 40.65 High Quality 4044 42.95 4116 42.27 Highest Quality 6315 N/A 6701 N/A Test Sequences MOS vs. PSNR/bit-rate for Earth.
  104. 104. How to quantify? • Experiment V – Aim: towards a quality/utility model for QuaSE July 2016 ICME 2016 Tutorial, C. Timmerer 34 • Stimuli with all combinations of sensory effects – Vibration higher impact than light & wind – Highest QoE with all effects present • General QuaSE model
  105. 105. How to judge and explain? • Experiment VI – Aim: understand QuaSE • Biosensor-based QoE evaluation system July 2016 ICME 2016 Tutorial, C. Timmerer 35 J. Donley, C. Ritz, M. Shujau, "Analysing the Quality of Experience of Multisensory Media from Measurements of Physiological Responses,” QoMEX2014, Singapore, Sep. 2014.
  106. 106. How to judge and explain? • Experiment VII – Aim: understand QuaSE • EEG Correlates of Pleasant and Unpleasant Odor Perception July 2016 ICME 2016 Tutorial, C. Timmerer 36 E. Kroupi, A. Yazdani, J.-M. Vesin, T. Ebrahimi, "EEG Correlates of Pleasant and Unpleasant Odor Perception," ACM TOMM, vol. 11, no. 1s, Sep. 2014.
  107. 107. How to judge and explain? • Experiment VIII – Aim: understand QuaSE • Multiple-Scent Enhanced Multimedia Synchronization July 2016 ICME 2016 Tutorial, C. Timmerer 37 N. Murray, B. Lee, Y. Qiao, and G.-M. Muntean, "Multiple-Scent Enhanced Multimedia Synchronization," ACM TOMM, vol. 11, no. 1s, Sep. 2014. General temporal boundaries: -10s to +15s are “in-sync”, skew values beyond are “out-of-sync”
  108. 108. Conclusions • From the need for a scientific framework to capture, measure, quantify, judge, and explain the quality of experience • To … – How to create, delivery, consume? – How to capture and measure? – How to quantify? – How to judge and explain? • Open issues? July 2016 ICME 2016 Tutorial, C. Timmerer 38 Many!
  109. 109. Open Issues / Challenges • QoE assessment is a delicate mixture of ingredients and choices – Test & lab environment – Test content – Test methodology – Data analysis • (Semi-)Automatic content creation/annotation • Towards large scale deployment – Lessons learnt from 3D (disaster) – 4D, 5D, xD – adding another dimension does not guarantee success • Holistic approach not feasible – Need for much more specialized QuaSE models • QUALINET Task Force: "Immersive Media Experiences (IMEx)” – https://www3.informatik.uni-wuerzburg.de/qoewiki/qualinet:imex July 2016 ICME 2016 Tutorial, C. Timmerer 39
  110. 110. Assessing Quality of Experience … A Bit Like Measuring ‘Happiness’ … © F. Pereira, Instituto Superior Técnico, Univ. Lisboa, PortugalJuly 2016 ICME 2016 Tutorial, C. Timmerer 40
  111. 111. 3. Towards the Concept of Quality of Life A. Beyond Quality of Experience B. The age of wearables C. Ingredients of a modern assessment of Quality of Life
  112. 112. Touradj Ebrahimi: About Me … • Professor of multimedia signal processing at EPFL • Active in image/video compression, media interpretation (segmentation, annotation, search, retrieval, quality assessment, brain computer interface, affective computing, etc.) and media security (privacy protection, copyright protection, media integrity verification, etc.) • Member of MPEG standardization committee since 1992 and active in many of its video standardization activities: MPEG-4, H.264/AVC, MVC, H.265/HEVC, MV-HEVC, 3D-HEVC, SCC, HDR extensions. • Member of JPEG standardization committee since 1994 and active in many of its image standardization activities: JPEG 2000, JPSearch, JPEG XR, JPEG AIC, JPEG XT, JPEG XS, JPEG PLENO. • Member of the Steering Committee of QoMEX and chair of its first edition in 2009 • Convener of JPEG Standardization Committee since 2014 • Chair of COST Action IC1003 Qualinet • First coined in February 2001 the term Quality of Experience (QoE) as a user-centric alternative to Quality of Service (QoS)
  113. 113. A. Beyond Quality of Experience
  114. 114. What future for Quality of Experience • Prediction is very difficult, especially about the future → Niels Bohr (1885-1962): Physics Nobel Prize Winner 1922
  115. 115. Trends in user-centric multimedia • Consequences of Moore’s law - Better and richer content - Larger bandwidth networks - Bigger storage capacities - More sophisticated codecs/processing • Better integration - Art, design - Psychology, psychophysics, neuroscience - Sociology, humanities - Technology and engineering
  116. 116. Trends in user-centric experiences • New media experiences • Personal well-being and personal health • Big data and social media
  117. 117. New media experiences • UHD, HDR, HFR, 3D, … • Light field imaging • Integral imaging • Holographic imaging • Haptics • Virtual, Augmented, Mixed reality • Immersive media • Multi-sensory media • …
  118. 118. Major trends in multimedia New Media Wearables Internet of Things Big data Social media ContentSensor Processing Multimedia experiences
  119. 119. Major trends in multimedia New Media Wearables Internet of Things Big data Social media ContentSensor Processing Multimedia experiences Life
  120. 120. B. The age of wearables
  121. 121. Generation-0 (smart) wearables
  122. 122. Mobile phones as wearables
  123. 123. Smart watches as wearables
  124. 124. Personal well-being wearables
  125. 125. Not any wearable should be smart!
  126. 126. Smart watches versus watches
  127. 127. Other wearables for user sensing
  128. 128. Other wearables for user sensing
  129. 129. Other wearables for environment sensing
  130. 130. Other variants of smart glasses
  131. 131. Sony’s recently announced smart glasses
  132. 132. Microsoft Hololens
  133. 133. Other wearables for environment sensing
  134. 134. Other wearables for environment sensing
  135. 135. Wearable data analysis
  136. 136. C. Ingredients of a modern assessment of Quality of Life
  137. 137. Quality of Life • Quality of life (QoL) is the general well-being of individuals and societies. QoL has a wide range of contexts, including the fields of international development, healthcare, politics and employment.
  138. 138. When past becomes future • Quality of Life: Meaning, Measurement, and Models – Elyse W. Kerce, Navy Personnel Research and Development Center (U.S.) – Navy Personnel Research and Development Center, 1992 • Origins of the concept date back to 1725! – Francis Hutcheson
  139. 139. Measuring Quality of Life
  140. 140. Ingredients of a modern QoL assessment • User sensing • Environment sensing • Context extraction • Big data analytics Users Context Content
  141. 141. Data is the King! • (A lot of ) Data from users and their environments is needed to carry out research on wearables: – Publicly available – Reliable – Rich data – Generated via crowdsourcing
  142. 142. Big data and social media 1 billion monthly active users5million photos added every day 175k tweets posted every second
  143. 143. Devices and sensors are Queens! • (Easily available) Wearable devices and sensors are needed to generate the data: – Affordable components and sensors to be purchased by interested individuals – Reliable – Easy to configure and calibrate – User friendly
  144. 144. Software is the President! • (Efficient) Software to control wearables and sensors – Open source – Extensible – Reliable – Easy to install on a wide variety of platforms
  145. 145. Data management is a must! • (Distributed) server architecture for data synchronization, storage and access: – Cloud based – Reliable – Scalable – Respectful of Privacy and Ethical issues
  146. 146. Interoperability is essential! • (Open) Standard solutions – Standard components – Standard data syntax – Standard Interface – Compliance/Certification
  147. 147. Compelling use cases are important! • Concrete use cases – Dietary assessment – Life log – …
  148. 148. International consortium is needed! • A seed consortium from US, Japan and Swiss universities in place and has initiated work on this topic around Multimedia Dietary Assessment Use case
  149. 149. Consumer Health Applications • Mobile device as a data collection tool for dietary assessment • Increasing demand in applications for mobile devices
  150. 150. Consumer Health Applications • Manual intake data entry – Tap & Track • Barcode based intake entry – Fooducate • Image based intake entry – MealSnap
  151. 151. Where do people look when they eat? Need to cover a wide angle
  152. 152. Did you finish the plate? Need to consider temporal aspects Before After
  153. 153. 360/omnidirectional video camera
  154. 154. Creation of a new database + …
  155. 155. Food recognition Food Deep learning Omnidirectional image
  156. 156. Swallowing detection Use EMG to detect swallowing and enable/disable camera • The sound of mastication (food crushing) has relation to physical properties of the food, but little relevance to energy content Chewing sensors O. Amft, M. Stäger, and G. Tröster, “Analysis of chewing sounds for dietary monitoring,” UbiComp 2005, pp. 56–72, 2005. S. Päßler, M. Wolff, and W.-J. Fischer, “Food intake monitoring: an acoustical approach to automated food intake activity detection and classification of consumed food,” Physiol. Meas., vol. 33, no. 6, pp. 1073–1093, 2012. • The sound of mastication (food crushing) has relation to physical properties of the food, but little relevance to energy content Chewing sensors O. Amft, M. Stäger, and G. Tröster, “Analysis of chewing sounds for dietary monitoring,” UbiComp 2005, pp. 56–72, 2005. S. Päßler, M. Wolff, and W.-J. Fischer, “Food intake monitoring: an acoustical approach to automated food intake activity detection and classification of consumed food,” Physiol. Meas., vol. 33, no. 6, pp. 1073–1093, 2012.
  157. 157. Integrate other contextual aspects Time Place Context User personality
  158. 158. Measuring QoL today …
  159. 159. Measuring QoL tomorrow …
  160. 160. Take-Home Messages • QoL is the natural step beyond QoE ! • QoL not a new concept but it can take advantage of modern technologies. • A federating project is needed in order to create the necessary critical mass (especially in data). • Multimedia Dietary Assessment is a compelling use case.
  161. 161. For Some of the Slides ...

×