Subjective quality evaluation of the upcoming HEVC video compression standard

Subjective quality evaluation of the upcoming HEVC video compression standardPhilippe Hanhart, Martin Rerabek, Francesca De Simone, and Touradj Ebrahimi

Outline• Introduction• Dataset• Test methodology• Test equipment and environment• Results• Conclusion 14/8/2012 SPIE Optics and Photonic 2012 2

Next-generation video compression• Motivation – Large quantity of video material over broadcast channels, digital networks, and packaged media – Demand for increased resolution and higher quality – Evolution of video acquisition and display technologies faster than network capabilities New video coding standard with higher efficiency than H.264/AVC is needed!• Goals – Better coding efficiency at higher resolutions – Suitability to a wide variety of applications (mobile TV, home cinema, UHDTV) 14/8/2012 SPIE Optics and Photonic 2012 4

Joint Collaborative Team on Video Coding (JCT-VC)• Video coding experts from ITU-T Study Group 16 (VCEG) + ISO/IEC JTC 1/SC 29/WG 11 (MPEG) = the Joint Collaborative Team on Video Coding (JCT-VC)• The JCT-VC standardization effort is being referred to as High Efficiency Video Coding (HEVC) Test Model under HEVC Test Model (HM) Call for Proposals Consideration publicly available (January 2010) (July 2010) (January 2011)Each proponent had to: • Based on key elements • Integrates the latest• Develop and submit a binary of some of the best developments executable of the proposed codec proposals • Steady simplification and• Encode and decode test material efficiency improvement• Evaluate objective quality using • Starting point for the definition of the new • New version available at PNSR• Provide and algorithmic standard each JCT-VC meeting cycle description of the codec 14/8/2012 SPIE Optics and Photonic 2012 5

Introduction• Double the compression efficiency is expected in HEVC versus AVC• Can be reliably compared only by means of subjective tests• Compression efficiency especially for resolutions beyond HDTV• Subjective evaluations of HEVC have been so-far limited up to HDTV resolution• This paper provides results of subjective evaluation on 4K/QFHD video content – Professional high-performance 4K/QFHD LCD reference monitor 14/8/2012 SPIE Optics and Photonic 2012 6

Contents• PeopleOnStreet (3840x2160@30fps) • Traffic (3840x2048@30fps)• Sintel2 (3840x1744@24fps) • Sintel39* (3840x1744@24fps) *Only used for training 14/8/2012 SPIE Optics and Photonic 2012 8

Spatial information and temporal information indexes 30 Sintel2 Sintel39 25 PeopleOnStreet TI 20 Traffic 15 0 5 10 15 SI• Different spatio-temporal characteristics – Sintel2 and Sintel39: large TI values – Traffic: small TI index (easier to encode) 14/8/2012 SPIE Optics and Photonic 2012 9

Encoding• Codecs – AVC (JM 18.3) – HEVC (HM 6.1.1)• Configuration – Random Access – GOP size: 8 pictures – Intra Period: 1s 24 pictures @24fps and 32 @30fps• Hierarchical B-pictures – QP increase of 1 between each Temporal Level – Coding Order: 0 8 4 2 1 3 6 5 7 14/8/2012 SPIE Optics and Photonic 2012 10

Encoding• 5 different bit rates for each content and codec – Both natural and synthetic contents• Fixed QPs (no Rate Control in HM 6) – Typical QPs range for AVC: 25 - 37 – Upper bit rate limit: 20 Mbps• Bit rates selected based on expert viewing sessions – Select lower/upper bounds for each content separately – Targeting realistic bit rates – Try to cover the full quality scale *BR expressed in Mbps 14/8/2012 SPIE Optics and Photonic 2012 11

Methodology 1• Double Stimulus Impairment Scale (DSIS) Variant II 100 Imperceptible 90 Reference Test Reference Test Video Video Video Video 80 Perceptible annoying but not 70 60 annoying Slightly 50 40 2s 5s 2s 5s 2s 5s 2s 5s 6s Annoying (TOT= 34 s) 30 20 “Rate the level of annoyance of the visual defects that you see annoying Very 10 in stimulus B, knowing that A is the reference video.” 0 14/8/2012 SPIE Optics and Photonic 2012 13

Sessions• ITU BT-500 – One test session should not last more than 30 minutes – Alternate as many different contents as possible• Short sequences => evaluation task requires a lot of attention – Test sessions no longer than 15 minutes each – Each session is followed by a resting phase• Details – Never the same content in consecutive presentations – Randomization to avoid possible effect of content presentation order – Dummy sequences at beginning of 1st session to stabilize observers’ rating – Reference versus reference stimulus to check subject’s reliability 14/8/2012 SPIE Optics and Photonic 2012 14

Timing• One DSIS Variant II presentation – Theoretically: 34 s – Loading time for each video file in Media Player Classic: 4 s • Concatenate ‘A’ + ref seq + ‘B’ + test seq in a single avi file – Total time: 46 s• Sequences – 30 test sequences (2 codecs × 3 contents × 5 bit rates) – Split in 2 test sessions• Sessions – Training session: 10 min – 1st test session: 2 dummies + 1 ref vs ref + 15 stimuli = 18 x 46 s = 14 min – 2nd test session: 15 stimuli = 15 x 46 s = 11.5 min 14/8/2012 SPIE Optics and Photonic 2012 15

Test• Test planning – 2 days, 3 time slots per day – 6 subjects per slot, split in 2 groups of 3 subjects each• Subjects – 36 naive people – 30% of the observers were female – Age ranged from 20 to 61 years old• Screening – Correct visual acuity: Snellen charts – Correct color vision: Ishiara charts• Training – Oral instructions to explain the task and the meaning of each label reported on the scale – Viewing session to allow the viewer familiarizing with the assessment procedure – Training samples have quality levels representative of the labels reported on the rating scales 14/8/2012 SPIE Optics and Photonic 2012 16

Laboratory for subjective video quality assessment• Server with SSD to read and play in real time 3840x2160@30fps YUV 4:2:0 raw video (i.e., 373.25 MB/s!)• 56-inch professional high-performance 4K/QFHD LCD reference monitor Sony Trimaster SRM-L560• ITU BT.500-11 compliant test environment – 3 subjects in front of the screen – Viewing distance ≈ 3.5 x screen height 14/8/2012 SPIE Optics and Photonic 2012 18

PeopleOnStreet PeopleOnStreet 100 90 80 70 60MOS 50 40 30 20 10 AVC HEVC 0 4 6 8 10 12 14 16 18 20 22 bitrate [Mbps] • At similar bit rates, HEVC outperforms AVC in 4 / 5 cases • Bit rate reduction – BD-PSNR: 27.5% – BD-MOS: 50.8% 14/8/2012 SPIE Optics and Photonic 2012 20

Traffic Traffic 100 90 80 70 60MOS 50 40 30 20 10 AVC HEVC 0 2 4 6 8 10 12 14 16 bitrate [Mbps] • At similar bit rates, HEVC outperforms AVC in 1 / 4 cases • Bit rate reduction – BD-PSNR: 37.7% – BD-MOS: 74.0% 14/8/2012 SPIE Optics and Photonic 2012 21

Sintel2 Sintel2 100 90 80 70 60MOS 50 40 30 20 10 AVC HEVC 0 0.5 1 1.5 2 2.5 3 3.5 bitrate [Mbps] • At similar bit rates, HEVC outperforms AVC in 4 / 4 cases • Bit rate reduction – BD-PSNR: 68.0% – BD-MOS: 74.7% 14/8/2012 SPIE Optics and Photonic 2012 22

Subjective versus objective results PSNR highly correlated with perceived quality, as long as saturation limits are considered14/8/2012 SPIE Optics and Photonic 2012 23

Conclusion• First comparison of HEVC vs AVC on resolution beyond HDTV• In general, HEVC significantly outperform AVC for similar bit rates• Substantial improvement in compression performance – Natural contents: 51 to 74% bit rate reduction – Synthetic content: 75% bit rate reduction• Objective gains based on BD-PSNR are pessimistic compared to actual gains measured with subjective rating• The upcoming HEVC video compression standard seems to be one of the key elements towards a wide deployment of UHDTV 14/8/2012 SPIE Optics and Photonic 2012 25

Thank you for your attention!14/8/2012 SPIE Optics and Photonic 2012 26

http://www.scoop.it	5087
http://www.linkedin.com	218
http://pinterest.com	16
http://webcache.googleusercontent.com	10
http://translate.googleusercontent.com	6
https://twitter.com	5
http://sonar9.com	2
http://twitter.com	2
http://localhost	1

Subjective quality evaluation of the upcoming HEVC video compression standard

by Touradj Ebrahimi on Aug 15, 2012

Accessibility

Categories

Upload Details

Usage Rights

9 Embeds 5,347

Statistics

Subjective quality evaluation of the upcoming HEVC video compression standard Presentation Transcript