Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Quality of Multimedia Experience Past, Present and Future Prof. Dr. Touradj Ebrahimi [email_address]
<ul><li>What  is “ quality ” of  multimedia  content? </li></ul><ul><li>How   is multimedia content “quality”  measured  t...
Quality – a simple yet difficult concept <ul><li>Like many human sensations quality is easy to understand but difficult to...
A fundamental yet largely under-investigated concept <ul><li>Aristotle classified every object of human apprehension into ...
Some definitions according to dictionary <ul><li>Definition 1 </li></ul><ul><ul><li>General  : Measure of  excellence  or ...
Some definitions according to dictionary <ul><li>Definition 2 </li></ul><ul><ul><li>Manufacturing  : Strict and consistent...
Some definitions according to dictionary <ul><li>Definition 3 </li></ul><ul><ul><li>Objective  :  Measurable  and  verifia...
Some definitions according to dictionary <ul><li>Definition 4 </li></ul><ul><ul><li>Subjective  :  Attribute ,  characteri...
Definition according to ISO 9000 <ul><li>ISO 9000: a family of standards for  quality management systems </li></ul><ul><li...
Quality – is in fact an elephant The blind men and the elephant:  Poem by John Godfrey Saxe
Quality of Service in computer networks and communications <ul><li>Quality of Service  (QoS) refers to a collection of  ne...
Quality in QoS framework Network Quality Capacity Coverage Handoff Link Quality Bitrate Frame/Bit/Packet loss Delay User Q...
Quality of Service in computer networks and communications <ul><li>Quality of Service (QoS) </li></ul><ul><ul><li>Resource...
What is Mean Opinion Score (MOS)? <ul><li>Widely used in many fields: </li></ul><ul><ul><li>Politics/Elections </li></ul><...
What is behind a MOS?
<ul><ul><li>A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices: </li></ul></ul>Su...
Test/lab environment <ul><li>Type of Monitors/Speakers and other test equipments </li></ul><ul><li>Lighting /Acoustic cond...
Test material <ul><li>Meaningful content for the envisaged scenario/application </li></ul><ul><ul><li>Typical content </li...
Test methodology <ul><li>Subjects </li></ul><ul><ul><li>Naïve or Expert? </li></ul></ul><ul><li>Instructions </li></ul><ul...
<ul><ul><li>Test conditions and methodologies are specified in: </li></ul></ul><ul><ul><ul><li>Recommendation ITU-R BT. 50...
<ul><ul><li>Single Stimulus (SS) </li></ul></ul>Test methodology (I) <ul><ul><ul><li>Non-categorical adjectival or numeric...
<ul><ul><li>Double Stimulus Impairment Scale (DSIS) </li></ul></ul>Test methodology (II) <ul><ul><ul><li>Categorical impai...
<ul><ul><li>Double Stimulus Continuous Quality Scale (DSCQS) </li></ul></ul>Test methodology (III) Sample 1 Sample 2 <ul><...
<ul><ul><li>Stimulus Comparison (SC) </li></ul></ul>Test methodology (IV) <ul><ul><ul><li>Categorical adjectival compariso...
<ul><ul><li>Single Stimulus Continuous Quality Evaluation (SSCQE) </li></ul></ul>Test methodology (V) (Very annoying) (Imp...
<ul><ul><li>Simultaneous Double Stimulus for Continuous Evaluation (SDSCE)  </li></ul></ul>Test methodology (VI) (Much bet...
Analysis of the data <ul><li>Scores distributions across subjects is assumed to be close to  normal distribution </li></ul...
What is behind a MOS?
Relationship between estimated mean values <ul><li>Hypothesis test  to find out whether the  difference  between two MOS v...
MOS hypothesis test 0.25 bpp 0.50 bpp 0.75 bpp 1.00 bpp 1.25 bpp 1.50 bpp 6 5 4 3 2 1 0 Number of times H 0  is rejected J...
<ul><ul><li>Subjective tests are time consuming, expensive, and difficult to design </li></ul></ul><ul><ul><li>Objective a...
FR, RR and NR scenarios <ul><li>Full Reference approach: </li></ul><ul><li>Reduced Reference approach: </li></ul><ul><li>N...
<ul><ul><li>Full Reference scenario </li></ul></ul><ul><ul><li>Metrics which look at the  fidelity  of the signal when com...
<ul><ul><li>Examples </li></ul></ul><ul><ul><ul><li>Mean Square Error (MSE) </li></ul></ul></ul><ul><ul><ul><li>Peak Signa...
Peak Signal to Noise Ratio <ul><li>Widely used because of its simplicity and ease in formalizing optimization problems! </...
PSNR for color images/video (I) <ul><li>Several alternatives to compute PSNR for color images/video : </li></ul>WPSNR = w ...
PSNR for color images/video (II) <ul><li>Which color space to use?  </li></ul><ul><ul><li>RGB </li></ul></ul><ul><ul><li>Y...
PSNR for color images/video (III) on R component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) on G component: on B ...
on Y’ component: on Cb component: on Cr component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) PSNR for color image...
<ul><ul><li>Human Visual System (HVS) based metrics </li></ul></ul><ul><ul><ul><li>simulating properties of the early stag...
PSNR-HVS-M <ul><ul><li>where: </li></ul></ul><ul><ul><li>  M, N = image dimensions </li></ul></ul><ul><ul><li>  K = consta...
<ul><ul><li>Metrics  based on the hypothesis that the  HVS is highly adapted for extraction of structural information from...
Mean SSIM (MSSIM) <ul><ul><li>Luminance comparison function:   (C 1 =constant) </li></ul></ul><ul><ul><li>Contrast compari...
<ul><li>MSSIM vs PSNR </li></ul><ul><li>PSNR = 24.9 dB </li></ul><ul><li>for all the images </li></ul>Mean SSIM (MSSIM) MS...
<ul><ul><li>At times one is interested in  specific types of distortions  that occur in multimedia systems </li></ul></ul>...
Blur metric <ul><li>A perceptual quality blur metric without a reference image. </li></ul><ul><li>Example: </li></ul>Gauss...
NR blur metric
NR blur metric
Correlation subjective ratings / NR blur metrics 96% correlation 85% correlation
Multimedia communication  –  a definition <ul><li>Multimedia is about  sharing experience  (real or imaginary) with others...
Evolutions versus Revolutions in multimedia <ul><li>Evolution:  A given modality improves itself in terms of  Quality of E...
Some of the major milestones in multimedia <ul><li>Story telling and cave drawing </li></ul><ul><li>Books and written pres...
Quality of Service vs Quality of Experience <ul><li>Quality of Service: Value of the  average  user’s  experience richness...
Factors impacting Quality of Experience  Context
Quality of Experience in networked multimedia
Quality Wheel
Trends in QoE <ul><li>Digital world has (re-)discovered the notion of quality </li></ul><ul><ul><li>Lower quality content ...
Trends in QoE community building <ul><li>Increased interest in workshops and conferences around the notion of quality asse...
Trends in standardization <ul><li>Standardization efforts in quality assessment and metrics </li></ul><ul><ul><li>Video Qu...
Challenges ahead <ul><li>Some key issues in QoE need to be better addressed </li></ul><ul><ul><li>Content -dependent quali...
What does this all mean to you? <ul><li>Era of  user-centric multimedia  has already started </li></ul><ul><ul><li>It is n...
Thank you for your attention Questions?
Acknowledgements <ul><li>Some of the concepts and illustrations presented in this talk are the results of collaborations w...
Upcoming SlideShare
Loading in …5
×

Quality of Multimedia Experience: Past, Present and Future

3,440 views

Published on

Slides of my keynote at ACM Multimedia 2009 in Beijing, China on October 22nd, 2009

  • Be the first to comment

Quality of Multimedia Experience: Past, Present and Future

  1. 1. Quality of Multimedia Experience Past, Present and Future Prof. Dr. Touradj Ebrahimi [email_address]
  2. 2. <ul><li>What is “ quality ” of multimedia content? </li></ul><ul><li>How is multimedia content “quality” measured today? </li></ul><ul><li>What are trends in assessment of “quality” in multimedia? </li></ul><ul><li>What are the challenges ahead? </li></ul>Today we will talk about…
  3. 3. Quality – a simple yet difficult concept <ul><li>Like many human sensations quality is easy to understand but difficult to define </li></ul><ul><li>According to Wikipedia: </li></ul><ul><ul><li>A quality (from Latin - qualitas ) is an attribute or a property . </li></ul></ul><ul><ul><li>Some philosophers assert that a quality cannot be defined . </li></ul></ul><ul><ul><li>In contemporary philosophy, the idea of qualities and especially how to distinguish certain kinds of qualities from one another remains controversial . </li></ul></ul>
  4. 4. A fundamental yet largely under-investigated concept <ul><li>Aristotle classified every object of human apprehension into 10 Categories </li></ul><ul><ul><li>Substance, Quantity, Quality , Relation, Place, Time, Position, State, Action, Affection </li></ul></ul>Aristotle 384 BC – 322 BC
  5. 5. Some definitions according to dictionary <ul><li>Definition 1 </li></ul><ul><ul><li>General : Measure of excellence or state of being free from defects, deficiencies, and significant variations. </li></ul></ul><ul><ul><li>ISO 8402-1986 standard defines quality as &quot;the totality of features and characteristics of a product or service that bears its ability to satisfy stated or implied needs &quot; </li></ul></ul>
  6. 6. Some definitions according to dictionary <ul><li>Definition 2 </li></ul><ul><ul><li>Manufacturing : Strict and consistent adherence to measurable and verifiable standards to achieve uniformity of output that satisfies specific customer or user requirements . </li></ul></ul>
  7. 7. Some definitions according to dictionary <ul><li>Definition 3 </li></ul><ul><ul><li>Objective : Measurable and verifiable aspect of a thing or phenomenon, expressed in numbers or quantities, such as lightness or heaviness, thickness or thinness, softness or hardness. </li></ul></ul>
  8. 8. Some definitions according to dictionary <ul><li>Definition 4 </li></ul><ul><ul><li>Subjective : Attribute , characteristic , or property of a thing or phenomenon that can be observed and interpreted , and may be approximated (quantified) but cannot be measured, such as beauty, feel, flavor, taste. </li></ul></ul>
  9. 9. Definition according to ISO 9000 <ul><li>ISO 9000: a family of standards for quality management systems </li></ul><ul><li>Quality of something can be determined by comparing a set of inherent characteristics with a set of requirements </li></ul><ul><ul><li>High quality: if characteristics meet requirements </li></ul></ul><ul><ul><li>Low quality: if characteristics do not meet all requirements </li></ul></ul><ul><li>Quality is a relative concept </li></ul><ul><ul><li>Degree of quality </li></ul></ul>
  10. 10. Quality – is in fact an elephant The blind men and the elephant: Poem by John Godfrey Saxe
  11. 11. Quality of Service in computer networks and communications <ul><li>Quality of Service (QoS) refers to a collection of networking technologies and measurement tools that allow for the network to guarantee delivering predictable results </li></ul>
  12. 12. Quality in QoS framework Network Quality Capacity Coverage Handoff Link Quality Bitrate Frame/Bit/Packet loss Delay User Quality Speech fidelity Audio fidelity Image fidelity Video fidelity
  13. 13. Quality of Service in computer networks and communications <ul><li>Quality of Service (QoS) </li></ul><ul><ul><li>Resource reservation control mechanisms </li></ul></ul><ul><ul><li>Ability to provide different priority to different applications, users, or data flows </li></ul></ul><ul><ul><li>Guarantee a certain level of performance ( quality ) to a data flow </li></ul></ul><ul><li>(Service) Provide-centric concept </li></ul><ul><li>Tightly related to the concept of Mean Opinion Score (MOS) </li></ul>
  14. 14. What is Mean Opinion Score (MOS)? <ul><li>Widely used in many fields: </li></ul><ul><ul><li>Politics/Elections </li></ul></ul><ul><ul><li>Marketing/Advertisement </li></ul></ul><ul><ul><li>Food industry </li></ul></ul><ul><ul><li>Multimedia </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><li>The likely level of satisfaction of a service or product as appreciated by an average user </li></ul><ul><li>Should be performed such that it generates reliable and reproducible results </li></ul><ul><ul><li>Subjective evaluation methodology </li></ul></ul><ul><ul><li>More complex and difficult that it a priori seems </li></ul></ul>
  15. 15. What is behind a MOS?
  16. 16. <ul><ul><li>A subjective tests aiming at producing MOS is a delicate mixture of ingredients and choices: </li></ul></ul>Subjective evaluation <ul><ul><ul><li>Test/lab environment </li></ul></ul></ul><ul><ul><ul><li>Test material </li></ul></ul></ul><ul><ul><ul><li>Test methodology </li></ul></ul></ul><ul><ul><ul><li>Analysis of the data </li></ul></ul></ul>
  17. 17. Test/lab environment <ul><li>Type of Monitors/Speakers and other test equipments </li></ul><ul><li>Lighting /Acoustic conditions </li></ul><ul><li>Laboratory architecture, background, … </li></ul><ul><li>Viewing distance /Hearing position </li></ul><ul><li>… </li></ul>
  18. 18. Test material <ul><li>Meaningful content for the envisaged scenario/application </li></ul><ul><ul><li>Typical content </li></ul></ul><ul><ul><li>Worst case content </li></ul></ul><ul><ul><li>… </li></ul></ul>p01 p06 p10 bike cafe woman
  19. 19. Test methodology <ul><li>Subjects </li></ul><ul><ul><li>Naïve or Expert? </li></ul></ul><ul><li>Instructions </li></ul><ul><ul><li>Which questions to ask subjects and how </li></ul></ul><ul><ul><li>Training </li></ul></ul><ul><li>Presentation </li></ul><ul><ul><li>Single or double stimulus </li></ul></ul><ul><ul><li>Sequential or simultaneous </li></ul></ul><ul><li>Grading scale </li></ul><ul><ul><li>Numerical </li></ul></ul><ul><ul><li>Categorical </li></ul></ul><ul><li>… </li></ul>
  20. 20. <ul><ul><li>Test conditions and methodologies are specified in: </li></ul></ul><ul><ul><ul><li>Recommendation ITU-R BT. 500-11 “ Methodology for the subjective assessment of the quality of television pictures ” (1974-2002). </li></ul></ul></ul><ul><ul><ul><li>Recommendation ITU-T P. 910 “ Subjective video quality assessment methods for multimedia applications ” (1999). </li></ul></ul></ul><ul><ul><ul><li>Recommendation ITU-R BT. 1788 “ Methodology for the subjective assessment of video quality in multimedia applications ” (2007). </li></ul></ul></ul><ul><ul><li>Based on television scenario ! </li></ul></ul>ITU Recommendations for test methodologies
  21. 21. <ul><ul><li>Single Stimulus (SS) </li></ul></ul>Test methodology (I) <ul><ul><ul><li>Non-categorical adjectival or numerical grading scale </li></ul></ul></ul>100 0 Excellent Bad <ul><ul><ul><li>Categorical adjectival grading scale: </li></ul></ul></ul><ul><ul><ul><li>Categorical numerical grading scale: </li></ul></ul></ul><ul><ul><ul><li>“ Rate from 1 to 11” </li></ul></ul></ul>5 Excellent 4 Good 3 Fair 2 Poor 1 Bad 5 Imperceptible 4 Perceptible but not annoying 3 Slightly annoying 2 Annoying 1 Very annoying
  22. 22. <ul><ul><li>Double Stimulus Impairment Scale (DSIS) </li></ul></ul>Test methodology (II) <ul><ul><ul><li>Categorical impairment grading scale: </li></ul></ul></ul>5 Imperceptible 4 Perceptible but not annoying 3 Slightly annoying 2 Annoying 1 Very annoying
  23. 23. <ul><ul><li>Double Stimulus Continuous Quality Scale (DSCQS) </li></ul></ul>Test methodology (III) Sample 1 Sample 2 <ul><ul><ul><li>Non-categorical adjectival or numerical grading scale: </li></ul></ul></ul>100 0 Excellent Bad Sample 1 Sample 2 100 0 Excellent Bad
  24. 24. <ul><ul><li>Stimulus Comparison (SC) </li></ul></ul>Test methodology (IV) <ul><ul><ul><li>Categorical adjectival comparison scale: </li></ul></ul></ul><ul><ul><ul><li>“ same or different” </li></ul></ul></ul><ul><ul><ul><li>Non-categorical judgement: </li></ul></ul></ul>Much worse Much better much worse worse slightly worse the same slightly better better much better
  25. 25. <ul><ul><li>Single Stimulus Continuous Quality Evaluation (SSCQE) </li></ul></ul>Test methodology (V) (Very annoying) (Imperceptible)
  26. 26. <ul><ul><li>Simultaneous Double Stimulus for Continuous Evaluation (SDSCE) </li></ul></ul>Test methodology (VI) (Much better) (Much worse) (Reference) (Test sequence)
  27. 27. Analysis of the data <ul><li>Scores distributions across subjects is assumed to be close to normal distribution </li></ul><ul><li>Outlier detection and removal </li></ul><ul><li>Mean Opinion Scores (MOS) and 95% confidence intervals </li></ul>m ij = score by subject i for test condition j . N = number of subjects after outliers removal. t(1-α/2,N) = t-value corresponding to a two-tailed t-Student distribution with N-1 degrees of freedom and a desired significance level α (α=0.05 in our case). σ j = standard deviation of the scores distribution across subjects for test condition j .
  28. 28. What is behind a MOS?
  29. 29. Relationship between estimated mean values <ul><li>Hypothesis test to find out whether the difference between two MOS values are statistically significant </li></ul><ul><ul><li>Two-sided t-test: </li></ul></ul><ul><ul><li>t statistic: </li></ul></ul><ul><ul><li>Decision rule to reject H 0 : </li></ul></ul>
  30. 30. MOS hypothesis test 0.25 bpp 0.50 bpp 0.75 bpp 1.00 bpp 1.25 bpp 1.50 bpp 6 5 4 3 2 1 0 Number of times H 0 is rejected JPEG 2000 4:2:0 JPEG 2000 4:4:4 JPEG JPEG XR MS JPEG XR PS JPEG 2000 4:2:0 JPEG 2000 4:4:4 JPEG JPEG XR MS JPEG XR PS
  31. 31. <ul><ul><li>Subjective tests are time consuming, expensive, and difficult to design </li></ul></ul><ul><ul><li>Objective algorithms, i.e. metrics, estimating subjective MOS with high level of correlation are desired </li></ul></ul><ul><ul><ul><li>Full reference metrics </li></ul></ul></ul><ul><ul><ul><li>No reference metrics </li></ul></ul></ul><ul><ul><ul><li>Reduced reference metrics </li></ul></ul></ul>Objective quality metrics
  32. 32. FR, RR and NR scenarios <ul><li>Full Reference approach: </li></ul><ul><li>Reduced Reference approach: </li></ul><ul><li>No Reference approach: </li></ul>Input/Reference signal Output/Processed signal signal processing FR METRIC Input/Reference signal Output/Processed signal signal processing NR METRIC Input/Reference signal Output/Processed signal signal processing Features extraction RR METRIC
  33. 33. <ul><ul><li>Full Reference scenario </li></ul></ul><ul><ul><li>Metrics which look at the fidelity of the signal when compared to an explicit reference: </li></ul></ul><ul><ul><li>processed signal </li></ul></ul><ul><ul><li>= </li></ul></ul><ul><ul><li>perfect quality reference signal </li></ul></ul><ul><ul><li>+ </li></ul></ul><ul><ul><li>error signal </li></ul></ul>MOS predictors based on fidelity measures
  34. 34. <ul><ul><li>Examples </li></ul></ul><ul><ul><ul><li>Mean Square Error (MSE) </li></ul></ul></ul><ul><ul><ul><li>Peak Signal to Noise Ratio (PSNR) </li></ul></ul></ul><ul><ul><ul><li>Maximum Pixel Deviation (L inf ) </li></ul></ul></ul><ul><ul><ul><li>Weighted PSNR </li></ul></ul></ul><ul><ul><ul><li>Masked PSNR </li></ul></ul></ul><ul><ul><ul><li>Structural SIMilarity (SSIM) </li></ul></ul></ul><ul><ul><ul><li>Multiscale Structural Similarity (MSSIM) </li></ul></ul></ul><ul><ul><ul><li>Visual Information Fidelity (VIF) </li></ul></ul></ul><ul><ul><ul><li>etc… </li></ul></ul></ul>MOS predictors based on fidelity metrics
  35. 35. Peak Signal to Noise Ratio <ul><li>Widely used because of its simplicity and ease in formalizing optimization problems! </li></ul><ul><li>For image and video data (Y component), a correlation of circa 80% reported when compared to subjective MOS evaluation </li></ul><ul><ul><li>where: </li></ul></ul><ul><ul><li> </li></ul></ul><ul><ul><li> M, N = image dimensions </li></ul></ul><ul><ul><li> Im a , Im b = pictures to compare </li></ul></ul><ul><ul><li> B= bit depth </li></ul></ul>
  36. 36. PSNR for color images/video (I) <ul><li>Several alternatives to compute PSNR for color images/video : </li></ul>WPSNR = w 1 PSNR 1 + w 2 PSNR 2 + w 3 PSNR 3 WPSNR_MSE WPSNR_PIX
  37. 37. PSNR for color images/video (II) <ul><li>Which color space to use? </li></ul><ul><ul><li>RGB </li></ul></ul><ul><ul><li>Y’CbCr </li></ul></ul><ul><ul><li>other? </li></ul></ul><ul><li>Which weights to use? </li></ul><ul><ul><li>w1=w2=w3=1/3 </li></ul></ul><ul><ul><li>w1=0.8, w2=w3=0.1 </li></ul></ul><ul><ul><li>other? </li></ul></ul>
  38. 38. PSNR for color images/video (III) on R component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) on G component: on B component:
  39. 39. on Y’ component: on Cb component: on Cr component: bpp (bits/pixel) bpp (bits/pixel) bpp (bits/pixel) PSNR for color images/video (IV)
  40. 40. <ul><ul><li>Human Visual System (HVS) based metrics </li></ul></ul><ul><ul><ul><li>simulating properties of the early stages of the HVS </li></ul></ul></ul><ul><ul><ul><li>Examples: PSNR-HVS-M, etc… </li></ul></ul></ul><ul><ul><ul><li>simulating high level features of the HVS </li></ul></ul></ul><ul><ul><ul><li>Examples: Osberger’s metric, etc… </li></ul></ul></ul><ul><ul><ul><li> Better correlation with human perception. </li></ul></ul></ul><ul><ul><ul><li> High complexity. </li></ul></ul></ul>MOS predictors based on fidelity metrics Computational model of the visual system Visibility Map
  41. 41. PSNR-HVS-M <ul><ul><li>where: </li></ul></ul><ul><ul><li> M, N = image dimensions </li></ul></ul><ul><ul><li> K = constant </li></ul></ul><ul><ul><li> = visible difference between DCT coefficients of the original </li></ul></ul><ul><ul><li> and distorted based on a contrast masking </li></ul></ul><ul><ul><li> T c = matrix of correcting factors based on standard visually optimized </li></ul></ul><ul><ul><li>JPEG quantization tables </li></ul></ul><ul><ul><li> B= bit depth </li></ul></ul>
  42. 42. <ul><ul><li>Metrics based on the hypothesis that the HVS is highly adapted for extraction of structural information from the content of a still image or video. </li></ul></ul><ul><ul><ul><li>degradation of still images or video = perceived structural information variation </li></ul></ul></ul><ul><ul><ul><li>Structural Similarity by Wang et al. </li></ul></ul></ul>MOS predictors based on fidelity measures
  43. 43. Mean SSIM (MSSIM) <ul><ul><li>Luminance comparison function: (C 1 =constant) </li></ul></ul><ul><ul><li>Contrast comparison function: (C 2 =constant) </li></ul></ul><ul><ul><li>Structure comparison function: </li></ul></ul><ul><ul><li> </li></ul></ul><ul><ul><li> </li></ul></ul>(C 3 =constant)
  44. 44. <ul><li>MSSIM vs PSNR </li></ul><ul><li>PSNR = 24.9 dB </li></ul><ul><li>for all the images </li></ul>Mean SSIM (MSSIM) MSSIM=0.9168 MSSIM=0.6949 MSSIM=0.7052
  45. 45. <ul><ul><li>At times one is interested in specific types of distortions that occur in multimedia systems </li></ul></ul><ul><ul><li>Examples </li></ul></ul><ul><ul><ul><li>Bluriness </li></ul></ul></ul><ul><ul><ul><li>Blockiness </li></ul></ul></ul><ul><ul><ul><li>Ringing/Mosquito noise </li></ul></ul></ul><ul><ul><ul><li>Jerkiness </li></ul></ul></ul><ul><ul><ul><li>etc… </li></ul></ul></ul>Specific distortion metrics
  46. 46. Blur metric <ul><li>A perceptual quality blur metric without a reference image. </li></ul><ul><li>Example: </li></ul>Gaussian blurred image JPEG2000 compressed image
  47. 47. NR blur metric
  48. 48. NR blur metric
  49. 49. Correlation subjective ratings / NR blur metrics 96% correlation 85% correlation
  50. 50. Multimedia communication – a definition <ul><li>Multimedia is about sharing experience (real or imaginary) with others </li></ul><ul><li>In a way it all started with story telling and wall drawing around the fire in the caves of early men </li></ul><ul><li>Modern multimedia systems are evolved versions of the good old story telling and wall drawing, which hopefully offer increasingly richer experience </li></ul><ul><li>The degree of richness of the experience is measured by Quality of Experience (QoE) </li></ul>
  51. 51. Evolutions versus Revolutions in multimedia <ul><li>Evolution: A given modality improves itself in terms of Quality of Experience : </li></ul><ul><ul><li>B&W TV </li></ul></ul><ul><ul><li>Color TV </li></ul></ul><ul><ul><li>Stereo and CD quality audio TV </li></ul></ul><ul><ul><li>HDTV </li></ul></ul><ul><li>Revolution: New modalities and expression are introduced bringing new dimensions in Quality of Experience </li></ul><ul><ul><li>Photography </li></ul></ul><ul><ul><li>Cinema </li></ul></ul><ul><ul><li>Internet </li></ul></ul>
  52. 52. Some of the major milestones in multimedia <ul><li>Story telling and cave drawing </li></ul><ul><li>Books and written press </li></ul><ul><li>Photography </li></ul><ul><li>Telegraph </li></ul><ul><li>Telephone </li></ul><ul><li>Radio and music recording </li></ul><ul><li>Cinema </li></ul><ul><li>Television and video recording </li></ul><ul><li>Internet (including VOIP, IPTV, etc.) </li></ul><ul><li>Mobile communication </li></ul><ul><li>Social networking (Web 2.0) </li></ul><ul><li>(What is next?) </li></ul>
  53. 53. Quality of Service vs Quality of Experience <ul><li>Quality of Service: Value of the average user’s experience richness estimated by a service/product/content provider </li></ul><ul><li>Quality of Experience: Value (estimated or actually measured) of a specific user’s experience richness </li></ul><ul><li>Quality of Experience is the dual (and extended) view of QoS problem </li></ul><ul><ul><li>QoS=provider-centric </li></ul></ul><ul><ul><li>QoE=user-centric </li></ul></ul>
  54. 54. Factors impacting Quality of Experience Context
  55. 55. Quality of Experience in networked multimedia
  56. 56. Quality Wheel
  57. 57. Trends in QoE <ul><li>Digital world has (re-)discovered the notion of quality </li></ul><ul><ul><li>Lower quality content is less and less tolerated by end-users </li></ul></ul><ul><ul><li>Digital technology can now rival and even surpass the old analog systems performance while remaining cost effective </li></ul></ul><ul><li>Increasing interest in QoE </li></ul><ul><ul><li>Extending from device-centric and system-centric quality optimization to end-to-end and especially user-centric optimization </li></ul></ul>
  58. 58. Trends in QoE community building <ul><li>Increased interest in workshops and conferences around the notion of quality assessment and metrics </li></ul><ul><ul><li>QoMEX: International Workshop on Quality of Multimedia Experience (http://www.qomex.org) </li></ul></ul><ul><ul><li>VPQM: International Workshop on Video Processing and Quality Metrics for Consumer Electronics (http://www.vpqm.org) </li></ul></ul><ul><ul><li>IMQA: International Workshop on Image Media Quality and its Applications ( http://www.mi.tj.chiba-u.jp/imqa2008/) </li></ul></ul><ul><li>QoE is one of the issues referred to in future funding programs by the EC </li></ul><ul><ul><li>Workshop on “The Future of Networked Immersive Media” (http://cordis.europa.eu/fp7/ict/netmedia/workshop/ws20090610_en.html) </li></ul></ul>
  59. 59. Trends in standardization <ul><li>Standardization efforts in quality assessment and metrics </li></ul><ul><ul><li>Video Quality Experts Group (VQEG) </li></ul></ul><ul><ul><li>ITU-T SG 12 (Performance, QoS and QoE) </li></ul></ul><ul><ul><li>ITU-R WP6C (Prog. production and quality assessment) </li></ul></ul><ul><ul><li>JPEG (Advanced Image Coding - AIC) </li></ul></ul><ul><ul><li>MPEG (High performance Video Coding – HVC) </li></ul></ul><ul><ul><li>… </li></ul></ul>
  60. 60. Challenges ahead <ul><li>Some key issues in QoE need to be better addressed </li></ul><ul><ul><li>Content -dependent quality assessment methods and metrics </li></ul></ul><ul><ul><li>Context -dependent quality assessment methods and metrics </li></ul></ul><ul><ul><li>Quality assessment methods and metrics beyond AV (haptics, …) </li></ul></ul><ul><ul><li>Multi-modal quality assessment methods and metrics (AV, … ) </li></ul></ul><ul><ul><li>3D quality assessment methods and metrics (3D sound, 3D video, …) </li></ul></ul><ul><ul><li>HDR content quality assessment methods and metrics </li></ul></ul><ul><ul><li>Interaction quality metrics (closely related to usability ) </li></ul></ul><ul><ul><li>Presence/immersion quality metrics </li></ul></ul><ul><ul><li>… </li></ul></ul><ul><li>Need for Quality Certification Mechanisms of multimedia services and products </li></ul><ul><ul><li>Similar in idea to ISO 9000 series </li></ul></ul>
  61. 61. What does this all mean to you? <ul><li>Era of user-centric multimedia has already started </li></ul><ul><ul><li>It is not anymore sufficient to merely add new features and functionalities to multimedia systems </li></ul></ul><ul><ul><li>True added value in terms of impact on user’s experience of such features and functions should be demonstrated </li></ul></ul><ul><ul><li>Quality of Experience plays a central role in this new game </li></ul></ul>
  62. 62. Thank you for your attention Questions?
  63. 63. Acknowledgements <ul><li>Some of the concepts and illustrations presented in this talk are the results of collaborations with individuals with whom I have had the pleasure of working. In particular: </li></ul><ul><ul><li>Francesca de Simone (EPFL/MMSPG) </li></ul></ul><ul><ul><li>Frederic Dufaux (EPFL/MMSPG) </li></ul></ul><ul><ul><li>Lutz Goldmann (EPFL/MMSPG) </li></ul></ul><ul><ul><li>Jong-Seok Lee (EPFL/MMSPG) </li></ul></ul><ul><ul><li>Andrew Perkis (NTNU/Q2S) </li></ul></ul><ul><ul><li>Ulrich Hoffmann (NTNU/Q2S) </li></ul></ul><ul><ul><li>Fitri Rahayu (NTNU/Q2S) </li></ul></ul><ul><li>Part of the work presented here are the fruits of the research projects: </li></ul><ul><ul><li>EC funded NoE Petamedia </li></ul></ul><ul><ul><li>Swiss NSF funded NCCR IM2 </li></ul></ul>

×