Quality of Experience in emerging visual communications

1,902 views

Published on

The slides of my keynote presentation at VPQM2014, Chandler, Arizona

Published in: Education, Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,902
On SlideShare
0
From Embeds
0
Number of Embeds
16
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Challenging to encode since it has relatively high SI and TI indexesArtifacts are more visible in the upper left corner due to higher sensitivity of the human visual system in low intensity areas (Weber law)Blockiness was perceived in AVC encoded sequences while the content was smoothed out in HEVC encoded sequences, which is less annoying
  • UHD – HD conversion – bilinear subsampling
  • In this paper, we used MOS that were computed by the MPEG test coordinator on a total of 36 naive viewers coming from three different laboratories.Outlier detection was performed by the MPEG test coordinator according to the procedure adopted by the ITU Video Quality Experts Group (VQEG) for its Multimedia Project.
  • The random stereo pair is located in-between two decoded views; one view of the stereo pair is always located closer to one of the decoded views than the other view of the stereo pair.Thus, we denote them as closer and farther views rather than left and right views.The objective metrics are ranked for each objective video quality model and the ranking number is specified below each performance index value.The difference is particularly strong between SNR-based metrics (PCC <= 0.7633 and SCC <= 0.7784) and perceptual metrics (PCC >= 0.9050 and SCC >= 0.9326)PSNR (PCC <= 0.7122 and SCC <= 0.7415) has a significantly lower correlation with perceived quality compared to VIF (PCC >=0.9373 and SCC >= 0.9442)
  • Quality of Experience in emerging visual communications

    1. 1. 1 Quality of Experience in emerging visual communications Touradj Ebrahimi Touradj.Ebrahimi@epfl.ch Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    2. 2. Some old (still unanswered?) questions What is the best way to apply PSNR to color images? What is the best way to apply PSNR to video? What are the most reliable and repeatable subjective evaluation methodologies for image and video quality assessment? How to measure quality (subjective evaluations or objective metrics) of 3D image and video? How to measure quality (subjective evaluations or objective metrics) of UHD video? How to measure quality (subjective evaluations or objective metrics) of HDR image and video? How to measure quality (subjective evaluations or objective metrics of audiovisual content? … Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 2
    3. 3. Qualinet in a nutshell COST Action IC1003: – European Network on Quality of Experience in Multimedia Systems and Services Period of activity: – November 2011 to October 2014 33 countries (27+6) and 185 active researchers More information: – http://www.qualinet.eu Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 3
    4. 4. 4 QUALINET in a nutshell Certification Products & Services Multimedia Applications QUALINET International Standards ICT, Psychology& Neuroscience& Humanities, … Protocols & Methodologies & Metrics Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    5. 5. Qualinet organization Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 5
    6. 6. Organisation of Qualinet Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 6
    7. 7. A fundamental and ancient concept Aristotle classified every object of human apprehension into 10 Categories – – – – – – – – – – Substance Quantity Quality Relation Place Time Position State Action Affection Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 7
    8. 8. User experience in multimedia Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 8
    9. 9. Qualinet white paper on Quality of Experience White Paper produced by COST Action IC1003 (Qualinet): – Downloadable from http://www.qualinet.eu – Latest version: V1.2, Novi Sad, March 2013 Several definitions of quality in multimedia systems and services and other related concepts Qualinet databases - 169 individual databases - http://dbq-multimediatech.cz Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 9
    10. 10. Quality (of Experience) is like an elephant … The blind men and the elephant: Poem by John Godfrey Saxe Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 10
    11. 11. User preference Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 11
    12. 12. Context Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 12
    13. 13. User centered evaluation Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 13
    14. 14. 14 A simple model for QoE User attributes – – – – individual attributes – expectation, age, sex, personality, background… sensorial attributes – including limitations and deficiencies perceptual attributes emotional attributes System attributes QoE – technical attributes (as in QoS) Contextual attributes – – – – environmental attributes device attributes service attributes content attributes user Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne QoS context
    15. 15. User and contextual attributes Personas (user preference) – Archetypical user representing the needs, behaviors and goals of a particular group of users Scenarios (context) – Realistic usage environment Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 15
    16. 16. Quality of Experience in mobile multimedia • Evaluation of quality of experience of video streaming in mobile environment (living lab) “Gesture and Touch Controlled Video Player Interface for Mobile Devices” S. Buchinger, et al., in Proceeding of the ACM Multimedia 2010 International Conference, (2010). Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 16
    17. 17. Emotional attributes Study with 32 subjects Valence-Arousal-Liking (VAL) emotional modeling. Elicitation using 40 music clips chosen to fill the whole 2D VA space. Subjective rating using SAM (Self Assessment Manikin) Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 17
    18. 18. 18 Classification in the VA space Electroencephalography (EEG) Physiological signals: blood flow, electrodermal activity –EDA-, respiration,… (Physio.) Multimedia content analysis (MCA) Classification accuracy Valence Liking EEG 0.56 0.58 0.5 Physio. 0.61 0.53 0.54 MCA 0.61 0.62 0.63 All Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Arousal 0.65 0.62 0.63
    19. 19. 19 Evolution of content 3D TV B&W TV Color TV HD TV UHD TV ? HFR TV HDR TV Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    20. 20. UHDTV PeopleOnStreet (3840x2160@30fps) Traffic (3840x2048@30fps) Sintel2 (3840x1744@24fps) Sintel39* (3840x1744@24fps) Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 20
    21. 21. DS Methodology Double Stimulus Impairment Scale (DSIS) Variant II Test Video Reference Video Test Video Age 21 1 Imperceptible Reference Video Name 100 90 Perceptible but not annoying 80 70 Slightly annoying 60 2s 5s 2s 5s 2s 5s 6s 40 (TOT= 34 s) “Rate the level of annoyance of the visual defects that you see in stimulus B, knowing that A is the reference video.” Annoying 5s 30 20 Very annoying 2s 50 10 0 Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne
    22. 22. PeopleOnStreet PeopleOnStreet 100 90 80 70 MOS 60 50 40 30 20 10 AVC HEVC 0 4 6 8 10 12 14 bitrate [Mbps] 16 18 20 22 At similar bit rates, HEVC outperforms AVC in 4 / 5 cases Bit rate reduction – BD-PSNR: 27.5% – BD-MOS: 50.8% Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 22
    23. 23. 23 Foveated video coding of UHDTV Priority map … Localization result Compression (H.265/HEVC) Blurred image Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Gaussian pyramid of L levels
    24. 24. Example Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 24
    25. 25. Subjective evaluation experiment 7 test sequences MJF content, Tears of Steel – 10 s – Including multiple moving objects in scene – UHD and HD resolution – separate sessions – Audio-visual source localization – Visual features: differential images – Audio features: frame energy H.265/HEVC coding – HM 12.1 – Different QP – 20, 30, 33 Subjective test – Perceived quality? – Single stimulus - home like scenario – Same distance for both resolutions Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 25
    26. 26. Results: Coding Efficiency & Subjective Quality UHD C3 HD C3 41% gain 9% gain 24% gain 19% gain C6 C6 87% gain 9% gain 20% gain 5% gain Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 26
    27. 27. 3D Quality ǂ Σ2D Encode left/right images with JPEG and different QPs (0-100) Show images with decreasing quality to the subjects Determine limit of transparency for left, right and stereo image Compute PSNR of left and right images and average for stereo Find PSNR which corresponds to the QP limit for each image Average PSNRs for each image across the individual subjects Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 27
    28. 28. 3D Quality ǂ Σ2D Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 28
    29. 29. MVC assessment using PSNR as metric Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 29
    30. 30. MVC assessment by subjective evaluation Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 30
    31. 31. Left/right image [Campisi2007] • Applies common 2D image quality metrics to left and right image • Combines scores using average, main eye or visual acuity approach Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 31
    32. 32. 3D video content  2 sets of spatio-temporal resolutions (8 different contents) – Class A: 1920x1088p@25fps – Class C: 1024x768p@30fps 4 target coding bit rates 22 different codecs + 2 anchors YUV 4:2:0 uncompressed videos with 8 bits per sample Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 32
    33. 33. will stand on the screen for 1 seconds. The test subjects w 33 Evaluation methodology quality votes will be expresse Double Stimulus Impairment Scale (DSIS) evaluation 11-grade numerical categorical scale Training 1 10 9 8 7 6 5 4 3 2 1 0 Test session: 24 test pairs + 3 dummy pairs + 1 ref vs. ref pair Outlier detection Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne Exa
    34. 34. Results - Random stereo pair Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 34
    35. 35. Recent drivers behind HDR imaging HDR sensors – Backlit CMOS sensor – Binary Pixel Imager Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 35
    36. 36. Recent drivers behind HDR imaging HDR displays – Modulated LED Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 36
    37. 37. Which tone-mapping? Many subjective evaluations of tone-mapping to find the best among those proposed in literature – Not always consistent with each other What happens if we perform subjective evaluation of tone-mapping operators taking into account explicitly the influence of content: – Scenes with varying dynamic range shot at night day, with dark and bright regions And context: – Environmental parameters (ambient illumination, etc.) – Devices (type of display, etc.) – Content (type of content, etc.) Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 37
    38. 38. Subjective evaluation Five state-of-the-art tone-mapping operators – Drago – Mantiuk – Reinhard – iCam – Logarithmic One controlled environment – Eizo monitor in an ITU-R BT 500-11 compliant laboratory – Passive subjects Two uncontrolled environments – iPad Tablet and Android mobile phone – Active subjects Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 38
    39. 39. Different content Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 39
    40. 40. Evaluation protocol Paired comparison between any two tonemapping operators applied to the same image Scores: A>B, A=B, A<B 20 subjects (12 male, 8 female) 4 images with 10 paired comparisons for each Training session to obtain more stable results Reference versus reference Randomization Two short sessions to avoid visual fatigue and loss of concentration (less than 15 min each) Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 40
    41. 41. Example comparison scores Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 41
    42. 42. Example subjective scores Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 42
    43. 43. Compression comparisons Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 43
    44. 44. 44 HDR objective metrics Visibility VDP-2 SNR 38 1 36 0.9 0.8 34 0.7 32 VDP_pdet SNR (dB) 0.6 30 28 BloomingGorse2 CanadianFalls McKeesPub MtRushmore2 WillyDesk 0.5 0.4 26 0.3 BloomingGorse2 CanadianFalls McKeesPub MtRushmore2 WillyDesk 24 22 20 0.5 1 1.5 2 2.5 3 3.5 4 4.5 0.2 0.1 5 Bpp Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 0 0.5 1 1.5 2 2.5 3 Bpp 3.5 4 4.5 5
    45. 45. Drunk scientist! Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 45
    46. 46. Measuring quality of experience through user sensing Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 46
    47. 47. User sensing through wearable devices Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 47
    48. 48. Thanks for your attention Multimedia Signal Processing Group Swiss Federal Institute of Technology, Lausanne 48

    ×