Your SlideShare is downloading. ×
0
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Quality of Experience Past, Present and Future Trends
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Quality of Experience Past, Present and Future Trends

1,072

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,072
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1Quality of Experience Past, Present and Future Trends Touradj Ebrahimi Touradj.Ebrahimi@epfl.chColloquium on Quality of Experience Alpen-Adria-Universität Klagenfurtin Multimedia Systems and Services 23rd November 2012, Austria
  • 2. Today  we  will  talk  about… 2•What  is  “quality”?•Origins  of  “quality”  evalua7on  in  telecommunica7ons•How  is  “quality”  measured  in  mul7media  communica7ons   today  ?•What  are  the  trends  on  “quality”  in  mul7media   communica7on?•  What  are  the  challenges  ahead? Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 3. 3 Quality: a simple yet difficult concept• Like many human sensations, quality is easy to understand but difficult to define• Quality according to Wikipedia: – A quality (from Latin - qualitas) is an attribute or a property – Some philosophers assert that a quality cannot be defined Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 4. A fundamental, ancient, but largely under investigated concept 4 Aristotle classified every object of human apprehension into 10 Categories – Substance – Quantity – Quality – Relation – Place – Time – Position – State – Action – Affection Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 5. 5 Quality According to ISO 9000• ISO 9000: a family of standards for quality management systems• Quality of something can be determined by comparing a set of inherent characteristics with a set of requirements – High quality: if characteristics meet requirements – Low quality: if characteristics do not meet all requirements• Quality is a relative concept – Degree of quality Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 6. 6 Qualinet white paper• White Paper produced by COST Action IC1003 (Qualinet): – http://www.qualinet.eu/images/stories/ whitepaper_v1.1_dagstuhl_output_corrected.pdf• Several definitions of quality in multimedia systems and services and other related concepts Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 7. 7 Quality is like an elephant … The blind men and the elephant: Poem by John Godfrey SaxeColloquium on Quality of Experience Alpen-Adria-Universität Klagenfurtin Multimedia Systems and Services 23rd November 2012, Austria
  • 8. 8Quality intelecommunications Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 9. 9 Quality  in  telecommunica4onsNetwork Quality Link Quality Capacity Bitrate Coverage Frame/Bit/Packet loss Handoff DelayUser Quality Speech fidelity Audio fidelity Image fidelity Video fidelity Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 10. 10 Quality in telecommunicationsObjective: input = output Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 11. 11 QoS in telecommunications• Quality of Service (QoS) – Resource reservation control mechanisms – Ability to provide different priority to different applications or data flows – Guarantee a certain level of performance (quality) to a data flow• Requires measuring or estimating quality• (Service) Provider-centric concept – Tightly related to the concept of Mean Opinion Score (MOS) Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 12. 12 Mean Opinion Score (MOS)?• Widely used in many fields: – Politics/Elections – Marketing/Advertisement – Food industry – Multimedia – …• The likely level of satisfaction of a service or product as appreciated by an average user in a typical use• Should be performed such that it generates reliable and reproducible results – Subjective evaluation methodology – More complex and difficult that it seems Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 13. 13 What is behind a MOS?Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurtin Multimedia Systems and Services 23rd November 2012, Austria
  • 14. 14 What is wrong with MOS?• Ground truth – It is assumed to be the optimal solution (ground truth)• Defies user preference – It is assumed to be independent of specific subjects• Ignores key parameters – It is often assumed to be independent of content and context• Misuse as a comparison approach – It is often used to compare performance between alternative tools for a same task/operation• Impractical or expensive• Statistical assumptions not always realistic Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 15. 15 Ground truthColloquium on Quality of Experience Alpen-Adria-Universität Klagenfurtin Multimedia Systems and Services 23rd November 2012, Austria
  • 16. 16 Defies user preferenceColloquium on Quality of Experience Alpen-Adria-Universität Klagenfurtin Multimedia Systems and Services 23rd November 2012, Austria
  • 17. 17 Content p01 $$$$$$p06 $$$$$$$$$$p10 $$$$$$$$$$$$$$$bike$$$$$$$$$$$$$$$$$$cafe$$$$$$$$$$$$$$$woman$Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurtin Multimedia Systems and Services 23rd November 2012, Austria
  • 18. 18 ContextColloquium on Quality of Experience Alpen-Adria-Universität Klagenfurtin Multimedia Systems and Services 23rd November 2012, Austria
  • 19. Objec4ve  quality  metrics 19• Subjec7ve   tests   are   7me   consuming,   expensive,   difficult   to   design,  and  cannot  be  used  in  real-­‐7me  systems• Objec7ve  algorithms,  i.e.   metrics,   es7ma7ng   MOS   with  high   level  of  correla7on  are  desired • Full  reference  metrics • No  reference  metrics • Reduced  reference  metrics Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 20. Objec4ve  quality  metrics  based  on  fidelity 20• Full  Reference  scenario• Metrics   which   look   at   the   fidelity   of   the   signal   when   compared  to  an  explicit  reference:   processed  signal = perfect  quality  reference  signal + error  signal Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 21. Objec4ve  quality  metrics  based  on  fidelity 21• Examples   of   fidelity   metrics   for   image   quality   quality   assessment − Mean  Square  Error  (MSE) − Peak  Signal  to  Noise  Ra7o  (PSNR) − Maximum  Pixel  Devia7on  (Linf) − Weighted  PSNR   − Masked  PSNR − Structural  SIMilarity  (SSIM) − Mul7scale  Structural  Similarity  (MSSIM) − Visual  Informa7on  Fidelity  (VIF) − etc… Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 22. 22 What is wrong with objective quality metrics ?• Poor performance in predicting MOS – Especially in real life situations• How to deal with color components? – Even in PSNR!• How to deal with video? – Time versus spatial dimension• How to take into account context? – Display size/type – Environmental illumination – …• How to deal with user preference/expectation?• ... Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 23. MVC Coding Performance 23 MVC assessment using PSNR as metricSample comparison of simulcast vs inter-view prediction 8 views (640 480), consider rate for all views ~25% bit rate savings over all views Ballroom Race1 40 42 39 41 40 38 39 37 38 PSNR (db)PSNR (db) 36 37 35 36 34 35 33 Simulcast 34 Simulcast 32 MVC 33 MVC 31 32 0 200 400 600 800 1000 1200 1400 1600 1800 0 200 400 600 800 1000 1200 1400 1600 Bitrate (Kb/s) Bitrate (Kb/s) CORESA 24-25 mai 2012, Lille, MPEG 3D Video Coding Activity Vetro: Update on France
  • 24. MVC:assessment by subjective evaluation MVC Subjective Performance 24 • Main finding: MVC achieves comparable quality to simulcast with as little as 25% rate for dependent view 4.50Mean Opinion Score 4.00 3.50 3.00 2.50 2.00 1.50 1.00 al C ) c t c t c t c t c t c t c t in 0P 5P 5P 0P 5P 0P 5P r ig AV _ O + _5 _3 _2 _2 _1 _1 L VC 12 L 12 L 12 L 12 L 12 L 12 L 12 (A st ca ul m Si Base view fixed at 12Mbps Dependent view at varying percentage of base view rate CORESA Vetro: Update on MPEG 3D Video Coding Activity 24-25 mai 2012, Lille, France
  • 25. 25 Evolving Quality ParadigmsColloquium on Quality of Experience Alpen-Adria-Universität Klagenfurtin Multimedia Systems and Services 23rd November 2012, Austria
  • 26. 26 User centered evaluation• It is becoming increasing important to evaluate quality from a user perspective rather than from a provider/system perspective• Constrains the tests to potential users and stimuli and evaluates the acceptability with respect to the task and the context• Identify underlying quality factors beside the overall quality CORESA 24-25 mai 2012, Lille, France
  • 27. 27 What people mean when they say QoE ?• “The degree of fulfillment of an intended experience on a given user” as defined by Touradj Ebrahimi, 2001• “The overall acceptability of an application or service … by the end user” as defined by the ITU-T• “The degree of delight ... of the user of an application or service ... user’s personality and current state.” as defined in white paper by Qualinet The term ‘experience’ promises individual engagement … Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 28. Quality  of  Service  vs  Quality  of  Experience 28• Quality   of   Service:   Value   of   the   average   user’s   experience   es7mated   by   a   service/product/content   provider  • Quality   of   Experience:   Value   (es7mated   or   actually   measured)  of  a  specific  user’s  richness  of  experience• Quality  of  Experience  is  the  dual  (and  extended)  view  of   QoS  problem QoS=provider-­‐centric QoE=user-­‐centric Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 29. 29 A simple model for QoE• User attributes – individual attributes – expectation, age, sex, personality, background… – sensorial attributes – including limitations and deficiencies – perceptual attributes – emotional attributes• System attributes – technical attributes (as in QoS) QoE• Contextual attributes user QoS context – environmental attributes – device attributes – service attributes – content attributes Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 30. Personas and scenarios [Strohmeier2009] 30• Personas (user preference) – Archetypical user representing the needs, behaviors and goals of a particular group of users – Not real people but fictional characters derived from user requirements• Scenarios (context) – Realistic usage environment CORESA 24-25 mai 2012, Lille, France
  • 31. 31 Trends in QoE• Digital world has (re-)discovered the notion of quality – Lower quality content is increasingly less tolerated by end-users – Digital technology have surpassed barriers of attainable quality, while remaining cost effective• Increasing interest in QoE – Extending from device-centric and system-centric quality optimization to end-to-end and especially user-centric optimization Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 32. 32 Trends in QoE community building• Increased interest in workshops and conferences around the notion of quality assessment and metrics – QoMEX: International Workshop on Quality of Multimedia Experience (http://www.qomex.org) – VPQM: International Workshop on Video Processing and Quality Metrics for Consumer Electronics (http://www.vpqm.org) – …• QoE is one of the issues referred to in research programs funded by the EC – Various calls in the recent year include QoE issues – QoE is one of the top topics in future framework program – Already ongoing initiatives such as COST Action IC1003 (Qualinet) Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 33. 33 Trends in standardization• Standardization efforts in quality assessment and metrics – Video Quality Experts Group (VQEG) – ITU-T SG 12 (Performance, QoS and QoE) – JPEG (Advanced Image Coding - AIC) – MPEG (HEVC, 3DVC) – … Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 34. 34 Challenges ahead• Some key issues in QoE: – Content-dependent quality assessment methods and metrics – Context-dependent quality assessment methods and metrics – Quality assessment methods and metrics beyond AV (haptics, …) – Multi-modal quality assessment methods and metrics (AV, …) – 3D quality assessment methods and metrics (3D sound, 3D video, …) – HDR content quality assessment methods and metrics – Interaction quality metrics (closely related to usability) – Presence/immersion quality metrics – …• Need for Quality Certification Mechanisms of multimedia services and products – Similar in idea to ISO 9000 series Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 35. 35 What does this all mean to you?• Era of user-centric multimedia has already started – It is not anymore sufficient to merely add new features and functionalities to multimedia systems – True added value in terms of impact on user experience of such features and functions should be demonstrated – Quality of Experience plays a central role in this new game• Opportunities in research, technology, business, art and entertainment Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurt in Multimedia Systems and Services 23rd November 2012, Austria
  • 36. Thank you for your attention 36 Questions?Colloquium on Quality of Experience Alpen-Adria-Universität Klagenfurtin Multimedia Systems and Services 23rd November 2012, Austria

×