Quality of experience in High Definition Television: subjective assessments and objective metrics

Quality of experience in High Definition Television: subjective assessments and objective metrics Stéphane Péchard October, 2nd 2008 IVC 1

Motivations psychological technical 2

New technologies 1 10 NEW 11 0 10 0 1 1 N EW 1110100 1 N EW 01 0 0 1 NE W 0 1 1 0 compression capture transmission restitution 5x SDTV (pixels) => new distortions 3

Controlling quality subjective objective (MOS) (MOSp) 4

Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior 2. comparing qualities knowledge) of 2 TV services 2. generic metric based 3. towards a fine on spatio-temporal quality measurement tubes 5

What is video quality subjective assessment? getting a mean human quality evaluation observers environment methodology 7

Subjective quality assessment how quality is globally perceived ? preference between HDTV and SDTV ? can we better understand quality judgment ? 8

Suitable methodology HDTV: high quality in a short range => quality measure should be precise and discriminative + important part of visual field excited => how to consider this in a methodology ? 10

Subjective Assessment Absolute Methodology Category Rating for Video Quality European Broadcasting Union Video Quality Experts Group - random order - user-driven order - only one viewing - multiple viewing (natural?) - category scale - continuous scale Good ... - no explicit reference - explicit reference 11

State of the art [Brotherton, 2006] both MOS (Mean Opinion Score) populations correlation on CIF (352x288): CC(MOSACR, MOSSAMVIQ) = 0.94 HDTV VGA QVGA 1080 480 240 to confirm: 320 more tests 640 1920 12

Results visual correlation RMSDiff= field QVGA 13° 0.969 6.73 VGA 19° 0.942 9.31 HDTV 33° 0.899 14.06 ACR and SAMVIQ are equivalent up to a certain resolution 13

Accuracy vs. Number of observers 15 confidence interval 10 SAMVIQ SAMVIQ ACR' 5 0 5 10 15 20 24 25 30 number of observers 14

Comparing two videos with different resolutions problem: observers can't move! H D=3H for HDTV 16

Technical solution How? no specific protocol exists comparison HD QHD ~SDTV QHD in HD 17

Motivation same screen for both formats QHD: 960x540 H h TVSD: 720x576 D=3H=6h 18

Quality and preference tests A: quality tests preference tests with SAMVIQ A vs. B of SDTV qualities (good and mid-range) preference scale I prefer much more A than B +3 I prefer more A than B +2 B: quality tests I prefer a little more A than B +1 I have no preference 0 with SAMVIQ I prefer a little less A than B -1 of HDTV qualities I prefer less A than B -2 -3 I prefer much less A than B 19

Results preference ΔQuality = 0 isopreference MOSHD - MOSSD 0 ΔQuality HD/SD Qgood: QHD may be less than QSD, benefit of the size HD/SD Qmid-range: QHD must be higher than QSD, size becomes an enemy 20

Classical approach ... a global distortion on an entire sequence 22

Farias approach-2004 Proposed approach distortion-based partition content-based partition blur homogeneous ... areas blockiness blur strong textured areas from disturbance functions to global distorting system fine textured t areas Drawbacks content dependency coding system dependency from spatio-temporal distortion list exhaustivity category qualities pooling function? complex subjective assessment to global quality? 23

spatio-temporal classification source C1 C2 C3 C4 C5 spatio-temporal segmentation (tube creation tube classification along local motion) categories masks sequence class-distorted sequences H.264 coding generation …… partly-distorted sequences usable for subjective tests 24

Local to global? MOS(Ci): partly-distorted sequence qualities related to global MOSG: f(MOS(Ci)) = MOSG ? several relation tested: up to CC(f(MOS(Ci)), MOSg) = 0.95 YES! It's possible to relate spatio-temporal category qualities to global quality 25

Farias approach-2004 Proposed approach distortion-based partition content-based partition blur homogeneous ... areas blockiness blur strong textured areas from disturbance functions to global distorting system fine textured areas t Drawbacks Advantages content dependency generic methodology coding system dependency simple pooling function distortion list exhaustivity real distortions pooling function? classical subjective assessment complex subjective assessment 26

What are objective quality metrics? reference reduced distorted reference sequence system extraction NR metric RR metric FR performance objective evaluation criteria scores (CC, RMSE, OR, difference signifiance) MOS from subjective assessments 28

Usual approaches high level distorstions measurement models PSNR VQM [2002] low level HVS models structural models VSSIM [2004] signal perceptual approach approach 29

Performances on HDTV 168 sequences metric CC RMSE OR VSSIM 0.790 11.27 0.55 VQM 0.898 8.09 0.40 PSNR 0.543 15.43 0.61 30

Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior knowledge) 2. comparing qualities of 2 TV services 2. generic metric based on spatio-temporal 3. towards a fine tubes quality measurement 31

32

reference global motion M sequence proportions Pi model ST content parameters analysis prediction offset, slope distorted bitrate B quality sequence model quality score Q 33

reference global motion M sequence proportions Pi model ST content parameters analysis prediction use of the spatio-temporal segmentation offset, slope 10% 20% distorted bitrate B quality 5% sequence model 60% class proportions Pi mean sequence quality score Q motion M 34

reference global motion M sequence proportions Pi model ST content parameters analysis prediction offset, slope offset parameter: temporal complexity estimation distorted bitrate Brelated to motion Mi quality sequence model slope parameter: spatial complexity estimation related to class proportions Pi quality score Q 35

Performances metric CC RMSE OR VSSIM 0.791 11.90 0.45 VQM 0.892 8.79 0.40 proposed 0.901 8.47 0.36 pros cons reduced reference metric (6 parameters) equal performances faster than VQM H.264-dependent 36

Outline Subjective quality Objective quality assessment metrics 1. global quality 1. H.264-specific metric assessment (using prior knowledge) 2. comparing qualities of 2 TV services 2. generic metric based on spatio-temporal 3. towards a fine tubes quality measurement 37

Interesting HVS features for this metric Visual inspection (gaze fixation) spatially localized duration (200-300 ms) smooth local motion tracking some of them have been used in part 1 38

reference spatio-temporal distorted sequence segmentation sequence tubes features features extraction extraction features difference short-term long-term quality spatio-temporal temporal score Q pooling pooling 39

reference spatio-temporal distorted sequence segmentation sequence tubes features features extraction a tube t extraction features difference short-term temporal quality spatio-temporal pooling score Q pooling 40

reference spatio-temporal distorted sequence segmentation sequence tubes features features extraction spatial information feature: fSI extraction features temporal information feature: fTI difference reference distorted tube - tube short-term temporal quality spatio-temporal pooling score Q pooling 41

reference spatio-temporal distorted sequence segmentation= sequence tubes features 5 frames 1 time-slot (200ms) features extraction extraction features difference short-term long-term quality spatio-temporal temporal score Q pooling pooling 42

reference spatio-temporal distorted sequence segmentation sequence high level HVS properties tubes features mid-term features asymetrical extraction long-term extraction non linear temporal features quality temporal filtering difference filtering judgment short-term long-term quality spatio-temporal temporal score Q pooling pooling 43

Training and testing 168 sequences testing training 44

Best performances metric CC RMSE OR VSSIM 0.837 10.15 0.38 VQM 0.875 8.98 0.43 fixed tubes 0.875 9.08 0.38 motion-oriented tubes 0.898 8.30 0.31 generic metric slightly better than VQM with less features 45

General conclusion 46

Subjective quality assessment better knowledge of HDTV (visual) subjective quality assessment visual image size influences preference between SDTV/HDTV services generic methodology to assess fine quality => better knowledge of judgment construction 47

Experiment effort 26 sessions (6 months) (SAMVIQ, ACR and preference) 200 observers for 600 unique sessions in 300 hours of subjective evaluation => 25,000 subjective scores more than 750 cumulative days of H.264 coding 48

Objective quality metrics fast RR metric dedicated to H.264 systems evaluation generic metric based on motion-oriented spatio-temporal tubes both performed slightly better than VQM 49

Future works adapting ACR to HDTV: more than 5 items? => work in progress (VQEG) considering a display model => work in progress (Tourancheau) towards a multimodal quality evaluation 50

Q&A 51

HDTV sequence database ref 24 -------- 7 52

SAMVIQ ACR 100 excellent 5 80% excellent good 4 good fair 3 fair poor 2 poor bad 1 bad 0 53

CC=0.899 RMSE=14.06 54

large screen effect distorsions effect HDTV prefered mean preference ΔMOS <MOS MOSHD=-18 SD 0 MOSHD ΔMOS0=-8 >MOS SD SDTV prefered ΔMOS=MOSHD-MOSSD 55

Classes five spatial activity levels smooth areas textured areas edges low high fine strong luminance textures C1 C2 C3 C4 C5 56

Tube classification ΔV 4 spatial gradients per tube space P C4 plot in spatial space P C5(P') frontiers defined C3 to get relevant C1 C2 C4 classification ΔH 57

DMOS and ΔMOS MOSref MOS4 ΔMOS(C4 ) ΔMOS(C MOS5 5 DMOS(Sj,Bk)= ) MOSref - MOS(Sj,Bk) MOS3 ΔMOS(C3 MOS1 ΔMOS(C1 ) global loss ) local MOS2 ΔMOS(C2 ) losses MOS(Sj,Bk) 58

Quality of experience in High Definition Television: subjective assessments and objective metrics

0 comments

Notes on slide 1

3 Favorites