Your SlideShare is downloading. ×
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Quality of experience in High Definition Television: subjective assessments and objective metrics
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Quality of experience in High Definition Television: subjective assessments and objective metrics

2,157

Published on

My Ph.D. defense presentation, on October 2nd, 2008.

My Ph.D. defense presentation, on October 2nd, 2008.

Published in: Technology, Business
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,157
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
101
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Quality of experience in High Definition Television: subjective assessments and objective metrics Stéphane Péchard October, 2nd 2008 IVC 1
  • 2. Motivations psychological technical 2
  • 3. New technologies 1 10 NEW 11 0 10 0 1 1 N EW 1110100 1 N EW 01 0 0 1 NE W 0 1 1 0 compression capture transmission restitution 5x SDTV (pixels) => new distortions 3
  • 4. Controlling quality subjective objective (MOS) (MOSp) 4
  • 5. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 5
  • 6. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 6
  • 7. What is video quality subjective assessment? getting a mean human quality evaluation observers environment methodology 7
  • 8. Subjective quality assessment how quality is globally perceived ? preference between HDTV and SDTV ? can we better understand quality judgment ? 8
  • 9. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 9
  • 10. Suitable methodology HDTV: high quality in a short range => quality measure should be precise and discriminative + important part of visual field excited => how to consider this in a methodology ? 10
  • 11. Subjective Assessment Absolute Methodology Category Rating for Video Quality European Broadcasting Union Video Quality Experts Group - random order - user-driven order - only one viewing - multiple viewing (natural?) - category scale - continuous scale Good ... - no explicit reference - explicit reference 11
  • 12. State of the art [Brotherton, 2006] both MOS (Mean Opinion Score) populations correlation on CIF (352x288): CC(MOSACR, MOSSAMVIQ) = 0.94 HDTV VGA QVGA 1080 480 240 to confirm: 320 more tests 640 1920 12
  • 13. Results visual correlation RMSDiff= field QVGA 13° 0.969 6.73 VGA 19° 0.942 9.31 HDTV 33° 0.899 14.06 ACR and SAMVIQ are equivalent up to a certain resolution 13
  • 14. Accuracy vs. Number of observers 15 confidence interval 10 SAMVIQ SAMVIQ ACR' 5 0 5 10 15 20 24 25 30 number of observers 14
  • 15. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 15
  • 16. Comparing two videos with different resolutions problem: observers can't move! H D=3H for HDTV 16
  • 17. Technical solution How? no specific protocol exists comparison HD QHD ~SDTV QHD in HD 17
  • 18. Motivation same screen for both formats QHD: 960x540 H h TVSD: 720x576 D=3H=6h 18
  • 19. Quality and preference tests A: quality tests preference tests with SAMVIQ A vs. B of SDTV qualities (good and mid-range) preference scale I prefer much more A than B +3 I prefer more A than B +2 B: quality tests I prefer a little more A than B +1 I have no preference 0 with SAMVIQ I prefer a little less A than B -1 of HDTV qualities I prefer less A than B -2 -3 I prefer much less A than B 19
  • 20. Results preference ΔQuality = 0 isopreference MOSHD - MOSSD 0 ΔQuality HD/SD Qgood: QHD may be less than QSD, benefit of the size HD/SD Qmid-range: QHD must be higher than QSD, size becomes an enemy 20
  • 21. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 21
  • 22. Classical approach ... a global distortion on an entire sequence 22
  • 23. Farias approach-2004 Proposed approach distortion-based partition content-based partition blur homogeneous ... areas blockiness blur strong textured areas from disturbance functions to global distorting system fine textured t areas Drawbacks content dependency coding system dependency from spatio-temporal distortion list exhaustivity category qualities pooling function? complex subjective assessment to global quality? 23
  • 24. spatio-temporal classification source C1 C2 C3 C4 C5 spatio-temporal segmentation (tube creation tube classification along local motion) categories masks sequence class-distorted sequences H.264 coding generation …… partly-distorted sequences usable for subjective tests 24
  • 25. Local to global? MOS(Ci): partly-distorted sequence qualities related to global MOSG: f(MOS(Ci)) = MOSG ? several relation tested: up to CC(f(MOS(Ci)), MOSg) = 0.95 YES! It's possible to relate spatio-temporal category qualities to global quality 25
  • 26. Farias approach-2004 Proposed approach distortion-based partition content-based partition blur homogeneous ... areas blockiness blur strong textured areas from disturbance functions to global distorting system fine textured areas t Drawbacks Advantages content dependency generic methodology coding system dependency simple pooling function distortion list exhaustivity real distortions pooling function? classical subjective assessment complex subjective assessment 26
  • 27. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 27
  • 28. What are objective quality metrics? reference reduced distorted reference sequence system extraction NR metric RR metric FR performance objective evaluation criteria scores (CC, RMSE, OR, difference signifiance) MOS from subjective assessments 28
  • 29. Usual approaches high level distorstions measurement models PSNR VQM [2002] low level HVS models structural models VSSIM [2004] signal perceptual approach approach 29
  • 30. Performances on HDTV 168 sequences metric CC RMSE OR VSSIM 0.790 11.27 0.55 VQM 0.898 8.09 0.40 PSNR 0.543 15.43 0.61 30
  • 31. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior knowledge) 2. comparing qualities of 2 TV services 2. generic metric based on spatio-temporal 3. towards a fine tubes quality measurement 31
  • 32. 32
  • 33. reference global motion M sequence proportions Pi model ST content parameters analysis prediction offset, slope distorted bitrate B quality sequence model quality score Q 33
  • 34. reference global motion M sequence proportions Pi model ST content parameters analysis prediction use of the spatio-temporal segmentation offset, slope 10% 20% distorted bitrate B quality 5% sequence model 60% class proportions Pi mean sequence quality score Q motion M 34
  • 35. reference global motion M sequence proportions Pi model ST content parameters analysis prediction offset, slope offset parameter: temporal complexity estimation distorted bitrate Brelated to motion Mi quality sequence model slope parameter: spatial complexity estimation related to class proportions Pi quality score Q 35
  • 36. Performances metric CC RMSE OR VSSIM 0.791 11.90 0.45 VQM 0.892 8.79 0.40 proposed 0.901 8.47 0.36 pros cons reduced reference metric (6 parameters) equal performances faster than VQM H.264-dependent 36
  • 37. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior knowledge) 2. comparing qualities of 2 TV services 2. generic metric based on spatio-temporal 3. towards a fine tubes quality measurement 37
  • 38. Interesting HVS features for this metric Visual inspection (gaze fixation) spatially localized duration (200-300 ms) smooth local motion tracking some of them have been used in part 1 38
  • 39. reference spatio-temporal distorted sequence segmentation sequence tubes features features extraction extraction features difference short-term long-term quality spatio-temporal temporal score Q pooling pooling 39
  • 40. reference spatio-temporal distorted sequence segmentation sequence tubes features features extraction a tube t extraction features difference short-term temporal quality spatio-temporal pooling score Q pooling 40
  • 41. reference spatio-temporal distorted sequence segmentation sequence tubes features features extraction spatial information feature: fSI extraction features temporal information feature: fTI difference reference distorted tube - tube short-term temporal quality spatio-temporal pooling score Q pooling 41
  • 42. reference spatio-temporal distorted sequence segmentation= sequence tubes features 5 frames 1 time-slot (200ms) features extraction extraction features difference short-term long-term quality spatio-temporal temporal score Q pooling pooling 42
  • 43. reference spatio-temporal distorted sequence segmentation sequence high level HVS properties tubes features mid-term features asymetrical extraction long-term extraction non linear temporal features quality temporal filtering difference filtering judgment short-term long-term quality spatio-temporal temporal score Q pooling pooling 43
  • 44. Training and testing 168 sequences testing training 44
  • 45. Best performances metric CC RMSE OR VSSIM 0.837 10.15 0.38 VQM 0.875 8.98 0.43 fixed tubes 0.875 9.08 0.38 motion-oriented tubes 0.898 8.30 0.31 generic metric slightly better than VQM with less features 45
  • 46. General conclusion 46
  • 47. Subjective quality assessment better knowledge of HDTV (visual) subjective quality assessment visual image size influences preference between SDTV/HDTV services generic methodology to assess fine quality => better knowledge of judgment construction 47
  • 48. Experiment effort 26 sessions (6 months) (SAMVIQ, ACR and preference) 200 observers for 600 unique sessions in 300 hours of subjective evaluation => 25,000 subjective scores more than 750 cumulative days of H.264 coding 48
  • 49. Objective quality metrics fast RR metric dedicated to H.264 systems evaluation generic metric based on motion-oriented spatio-temporal tubes both performed slightly better than VQM 49
  • 50. Future works adapting ACR to HDTV: more than 5 items? => work in progress (VQEG) considering a display model => work in progress (Tourancheau) towards a multimodal quality evaluation 50
  • 51. Q&A 51
  • 52. HDTV sequence database ref 24 -------- 7 52
  • 53. SAMVIQ ACR 100 excellent 5 80% excellent good 4 good fair 3 fair poor 2 poor bad 1 bad 0 53
  • 54. CC=0.899 RMSE=14.06 54
  • 55. large screen effect distorsions effect HDTV prefered mean preference ΔMOS <MOS MOSHD=-18 SD 0 MOSHD ΔMOS0=-8 >MOS SD SDTV prefered ΔMOS=MOSHD-MOSSD 55
  • 56. Classes five spatial activity levels smooth areas textured areas edges low high fine strong luminance textures C1 C2 C3 C4 C5 56
  • 57. Tube classification ΔV 4 spatial gradients per tube space P C4 plot in spatial space P C5(P') frontiers defined C3 to get relevant C1 C2 C4 classification ΔH 57
  • 58. DMOS and ΔMOS MOSref MOS4 ΔMOS(C4 ) ΔMOS(C MOS5 5 DMOS(Sj,Bk)= ) MOSref - MOS(Sj,Bk) MOS3 ΔMOS(C3 MOS1 ΔMOS(C1 ) global loss ) local MOS2 ΔMOS(C2 ) losses MOS(Sj,Bk) 58

×