Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Quality of experience
in High Definition Television:
   subjective assessments
    and objective metrics




 Stéphane Péc...
Motivations




psychological   technical

                            2
New technologies
             1 10 NEW
               11 0
             10 0 1 1          N EW
            1110100 1      ...
Controlling quality




subjective     objective
  (MOS)         (MOSp)
                           4
Outline
Subjective quality       Objective quality
   assessment                metrics
1. global quality        1. H.264-...
Outline
Subjective quality       Objective quality
   assessment                metrics
1. global quality        1. H.264-...
What is video quality
  subjective assessment?
getting a mean human quality evaluation




observers    environment method...
Subjective quality
       assessment
 how quality is globally perceived ?


preference between HDTV and SDTV ?


     can ...
Outline
Subjective quality       Objective quality
   assessment                metrics
1. global quality        1. H.264-...
Suitable methodology

    HDTV: high quality in a short range
   => quality measure should be precise
             and dis...
Subjective Assessment
         Absolute                            Methodology
      Category Rating                      ...
State of the art
  [Brotherton, 2006] both MOS (Mean Opinion
Score) populations correlation on CIF (352x288):
         CC(...
Results
       visual correlation   RMSDiff=
        field
QVGA    13°       0.969         6.73

VGA     19°       0.942  ...
Accuracy vs.
                           Number of observers
                      15
confidence interval




             ...
Outline
Subjective quality       Objective quality
   assessment                metrics
1. global quality        1. H.264-...
Comparing two videos
with different resolutions
     problem: observers can't move!



 H



         D=3H for HDTV
      ...
Technical solution
How? no specific protocol exists

          comparison




HD        QHD ~SDTV      QHD in HD
         ...
Motivation
    same screen for both formats

                           QHD: 960x540
H   h
                              T...
Quality and preference
             tests
 A: quality tests          preference tests
  with SAMVIQ                   A vs...
Results
preference



                                             ΔQuality =
             0              isopreference
  ...
Outline
Subjective quality       Objective quality
   assessment                metrics
1. global quality        1. H.264-...
Classical approach
                     ...




a global distortion on an entire sequence
                                ...
Farias approach-2004                  Proposed approach
 distortion-based partition           content-based partition
    ...
spatio-temporal classification
source


                                         C1     C2     C3     C4    C5
          s...
Local to global?
MOS(Ci): partly-distorted sequence qualities
related to global MOSG: f(MOS(Ci)) = MOSG ?

         severa...
Farias approach-2004                    Proposed approach
 distortion-based partition             content-based partition
...
Outline
Subjective quality       Objective quality
   assessment                metrics
1. global quality        1. H.264-...
What are objective
             quality metrics?
reference                   reduced
                           distorted
...
Usual approaches
             high level distorstions
             measurement models

 PSNR       VQM [2002]             ...
Performances on HDTV
       168 sequences

 metric CC RMSE         OR
 VSSIM 0.790 11.27     0.55
  VQM 0.898 8.09       0...
Outline
Subjective quality       Objective quality
   assessment                metrics
1. global quality        1. H.264-...
32
reference         global motion M
sequence          proportions Pi
                                 model
            ST c...
reference         global motion M
sequence          proportions Pi
                                   model
            ST...
reference         global motion M
sequence          proportions Pi
                                   model
            ST...
Performances
     metric      CC     RMSE        OR
     VSSIM      0.791   11.90      0.45
      VQM       0.892    8.79 ...
Outline
Subjective quality       Objective quality
   assessment                metrics
1. global quality        1. H.264-...
Interesting HVS features
     for this metric

  Visual inspection (gaze fixation)
           spatially localized
        ...
reference    spatio-temporal    distorted
sequence      segmentation      sequence
                    tubes
 features    ...
reference    spatio-temporal    distorted
sequence      segmentation      sequence
                    tubes
 features    ...
reference       spatio-temporal          distorted
sequence         segmentation            sequence
                     ...
reference    spatio-temporal       distorted
sequence      segmentation=        sequence
                    tubes
 featur...
reference       spatio-temporal          distorted
sequence         segmentation            sequence
               high l...
Training and testing
     168 sequences




                     testing

         training          44
Best performances
          metric           CC     RMSE    OR
          VSSIM           0.837   10.15   0.38
           V...
General conclusion




                     46
Subjective quality
         assessment
    better knowledge of HDTV (visual)
      subjective quality assessment

  visual...
Experiment effort
      26 sessions (6 months)
   (SAMVIQ, ACR and preference)

200 observers for 600 unique sessions
in 3...
Objective quality metrics
        fast RR metric dedicated
      to H.264 systems evaluation

generic metric based on moti...
Future works
adapting ACR to HDTV: more than 5 items?
       => work in progress (VQEG)

      considering a display model...
Q&A




      51
HDTV sequence database

ref
               24
--------



7
                            52
SAMVIQ                      ACR
            100
excellent         5   80%   excellent

 good             4          good

...
CC=0.899
RMSE=14.06
             54
large screen effect    distorsions effect
                             HDTV prefered
mean preference




                 ...
Classes
       five spatial activity levels




smooth areas      textured areas       edges
low      high      fine      ...
Tube classification
                          ΔV
 4 spatial gradients
      per tube                                space ...
DMOS and ΔMOS
           MOSref

                      MOS4           ΔMOS(C4
                                     )
     ...
Upcoming SlideShare
Loading in …5
×

Quality of experience in High Definition Television: subjective assessments and objective metrics

2,644 views

Published on

My Ph.D. defense presentation, on October 2nd, 2008.

Published in: Technology, Business
  • Be the first to comment

Quality of experience in High Definition Television: subjective assessments and objective metrics

  1. 1. Quality of experience in High Definition Television: subjective assessments and objective metrics Stéphane Péchard October, 2nd 2008 IVC 1
  2. 2. Motivations psychological technical 2
  3. 3. New technologies 1 10 NEW 11 0 10 0 1 1 N EW 1110100 1 N EW 01 0 0 1 NE W 0 1 1 0 compression capture transmission restitution 5x SDTV (pixels) => new distortions 3
  4. 4. Controlling quality subjective objective (MOS) (MOSp) 4
  5. 5. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 5
  6. 6. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 6
  7. 7. What is video quality subjective assessment? getting a mean human quality evaluation observers environment methodology 7
  8. 8. Subjective quality assessment how quality is globally perceived ? preference between HDTV and SDTV ? can we better understand quality judgment ? 8
  9. 9. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 9
  10. 10. Suitable methodology HDTV: high quality in a short range => quality measure should be precise and discriminative + important part of visual field excited => how to consider this in a methodology ? 10
  11. 11. Subjective Assessment Absolute Methodology Category Rating for Video Quality European Broadcasting Union Video Quality Experts Group - random order - user-driven order - only one viewing - multiple viewing (natural?) - category scale - continuous scale Good ... - no explicit reference - explicit reference 11
  12. 12. State of the art [Brotherton, 2006] both MOS (Mean Opinion Score) populations correlation on CIF (352x288): CC(MOSACR, MOSSAMVIQ) = 0.94 HDTV VGA QVGA 1080 480 240 to confirm: 320 more tests 640 1920 12
  13. 13. Results visual correlation RMSDiff= field QVGA 13° 0.969 6.73 VGA 19° 0.942 9.31 HDTV 33° 0.899 14.06 ACR and SAMVIQ are equivalent up to a certain resolution 13
  14. 14. Accuracy vs. Number of observers 15 confidence interval 10 SAMVIQ SAMVIQ ACR' 5 0 5 10 15 20 24 25 30 number of observers 14
  15. 15. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 15
  16. 16. Comparing two videos with different resolutions problem: observers can't move! H D=3H for HDTV 16
  17. 17. Technical solution How? no specific protocol exists comparison HD QHD ~SDTV QHD in HD 17
  18. 18. Motivation same screen for both formats QHD: 960x540 H h TVSD: 720x576 D=3H=6h 18
  19. 19. Quality and preference tests A: quality tests preference tests with SAMVIQ A vs. B of SDTV qualities (good and mid-range) preference scale I prefer much more A than B +3 I prefer more A than B +2 B: quality tests I prefer a little more A than B +1 I have no preference 0 with SAMVIQ I prefer a little less A than B -1 of HDTV qualities I prefer less A than B -2 -3 I prefer much less A than B 19
  20. 20. Results preference ΔQuality = 0 isopreference MOSHD - MOSSD 0 ΔQuality HD/SD Qgood: QHD may be less than QSD, benefit of the size HD/SD Qmid-range: QHD must be higher than QSD, size becomes an enemy 20
  21. 21. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 21
  22. 22. Classical approach ... a global distortion on an entire sequence 22
  23. 23. Farias approach-2004 Proposed approach distortion-based partition content-based partition blur homogeneous ... areas blockiness blur strong textured areas from disturbance functions to global distorting system fine textured t areas Drawbacks content dependency coding system dependency from spatio-temporal distortion list exhaustivity category qualities pooling function? complex subjective assessment to global quality? 23
  24. 24. spatio-temporal classification source C1 C2 C3 C4 C5 spatio-temporal segmentation (tube creation tube classification along local motion) categories masks sequence class-distorted sequences H.264 coding generation …… partly-distorted sequences usable for subjective tests 24
  25. 25. Local to global? MOS(Ci): partly-distorted sequence qualities related to global MOSG: f(MOS(Ci)) = MOSG ? several relation tested: up to CC(f(MOS(Ci)), MOSg) = 0.95 YES! It's possible to relate spatio-temporal category qualities to global quality 25
  26. 26. Farias approach-2004 Proposed approach distortion-based partition content-based partition blur homogeneous ... areas blockiness blur strong textured areas from disturbance functions to global distorting system fine textured areas t Drawbacks Advantages content dependency generic methodology coding system dependency simple pooling function distortion list exhaustivity real distortions pooling function? classical subjective assessment complex subjective assessment 26
  27. 27. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 27
  28. 28. What are objective quality metrics? reference reduced distorted reference sequence system extraction NR metric RR metric FR performance objective evaluation criteria scores (CC, RMSE, OR, difference signifiance) MOS from subjective assessments 28
  29. 29. Usual approaches high level distorstions measurement models PSNR VQM [2002] low level HVS models structural models VSSIM [2004] signal perceptual approach approach 29
  30. 30. Performances on HDTV 168 sequences metric CC RMSE OR VSSIM 0.790 11.27 0.55 VQM 0.898 8.09 0.40 PSNR 0.543 15.43 0.61 30
  31. 31. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior knowledge) 2. comparing qualities of 2 TV services 2. generic metric based on spatio-temporal 3. towards a fine tubes quality measurement 31
  32. 32. 32
  33. 33. reference global motion M sequence proportions Pi model ST content parameters analysis prediction offset, slope distorted bitrate B quality sequence model quality score Q 33
  34. 34. reference global motion M sequence proportions Pi model ST content parameters analysis prediction use of the spatio-temporal segmentation offset, slope 10% 20% distorted bitrate B quality 5% sequence model 60% class proportions Pi mean sequence quality score Q motion M 34
  35. 35. reference global motion M sequence proportions Pi model ST content parameters analysis prediction offset, slope offset parameter: temporal complexity estimation distorted bitrate Brelated to motion Mi quality sequence model slope parameter: spatial complexity estimation related to class proportions Pi quality score Q 35
  36. 36. Performances metric CC RMSE OR VSSIM 0.791 11.90 0.45 VQM 0.892 8.79 0.40 proposed 0.901 8.47 0.36 pros cons reduced reference metric (6 parameters) equal performances faster than VQM H.264-dependent 36
  37. 37. Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior knowledge) 2. comparing qualities of 2 TV services 2. generic metric based on spatio-temporal 3. towards a fine tubes quality measurement 37
  38. 38. Interesting HVS features for this metric Visual inspection (gaze fixation) spatially localized duration (200-300 ms) smooth local motion tracking some of them have been used in part 1 38
  39. 39. reference spatio-temporal distorted sequence segmentation sequence tubes features features extraction extraction features difference short-term long-term quality spatio-temporal temporal score Q pooling pooling 39
  40. 40. reference spatio-temporal distorted sequence segmentation sequence tubes features features extraction a tube t extraction features difference short-term temporal quality spatio-temporal pooling score Q pooling 40
  41. 41. reference spatio-temporal distorted sequence segmentation sequence tubes features features extraction spatial information feature: fSI extraction features temporal information feature: fTI difference reference distorted tube - tube short-term temporal quality spatio-temporal pooling score Q pooling 41
  42. 42. reference spatio-temporal distorted sequence segmentation= sequence tubes features 5 frames 1 time-slot (200ms) features extraction extraction features difference short-term long-term quality spatio-temporal temporal score Q pooling pooling 42
  43. 43. reference spatio-temporal distorted sequence segmentation sequence high level HVS properties tubes features mid-term features asymetrical extraction long-term extraction non linear temporal features quality temporal filtering difference filtering judgment short-term long-term quality spatio-temporal temporal score Q pooling pooling 43
  44. 44. Training and testing 168 sequences testing training 44
  45. 45. Best performances metric CC RMSE OR VSSIM 0.837 10.15 0.38 VQM 0.875 8.98 0.43 fixed tubes 0.875 9.08 0.38 motion-oriented tubes 0.898 8.30 0.31 generic metric slightly better than VQM with less features 45
  46. 46. General conclusion 46
  47. 47. Subjective quality assessment better knowledge of HDTV (visual) subjective quality assessment visual image size influences preference between SDTV/HDTV services generic methodology to assess fine quality => better knowledge of judgment construction 47
  48. 48. Experiment effort 26 sessions (6 months) (SAMVIQ, ACR and preference) 200 observers for 600 unique sessions in 300 hours of subjective evaluation => 25,000 subjective scores more than 750 cumulative days of H.264 coding 48
  49. 49. Objective quality metrics fast RR metric dedicated to H.264 systems evaluation generic metric based on motion-oriented spatio-temporal tubes both performed slightly better than VQM 49
  50. 50. Future works adapting ACR to HDTV: more than 5 items? => work in progress (VQEG) considering a display model => work in progress (Tourancheau) towards a multimodal quality evaluation 50
  51. 51. Q&A 51
  52. 52. HDTV sequence database ref 24 -------- 7 52
  53. 53. SAMVIQ ACR 100 excellent 5 80% excellent good 4 good fair 3 fair poor 2 poor bad 1 bad 0 53
  54. 54. CC=0.899 RMSE=14.06 54
  55. 55. large screen effect distorsions effect HDTV prefered mean preference ΔMOS <MOS MOSHD=-18 SD 0 MOSHD ΔMOS0=-8 >MOS SD SDTV prefered ΔMOS=MOSHD-MOSSD 55
  56. 56. Classes five spatial activity levels smooth areas textured areas edges low high fine strong luminance textures C1 C2 C3 C4 C5 56
  57. 57. Tube classification ΔV 4 spatial gradients per tube space P C4 plot in spatial space P C5(P') frontiers defined C3 to get relevant C1 C2 C4 classification ΔH 57
  58. 58. DMOS and ΔMOS MOSref MOS4 ΔMOS(C4 ) ΔMOS(C MOS5 5 DMOS(Sj,Bk)= ) MOSref - MOS(Sj,Bk) MOS3 ΔMOS(C3 MOS1 ΔMOS(C1 ) global loss ) local MOS2 ΔMOS(C2 ) losses MOS(Sj,Bk) 58

×