1. Subjective quality evaluation of the upcoming
HEVC video compression standard
Philippe Hanhart, Martin Rerabek, Francesca De Simone,
and Touradj Ebrahimi

2. Outline
• Introduction
• Dataset
• Test methodology
• Test equipment and environment
• Results
• Conclusion
14/8/2012 SPIE Optics and Photonic 2012 2

3. Outline
• Introduction
• Dataset
• Test methodology
• Test equipment and environment
• Results
• Conclusion
14/8/2012 SPIE Optics and Photonic 2012 3

4. Next-generation video compression
• Motivation
– Large quantity of video material over
broadcast channels, digital networks, and
packaged media
– Demand for increased resolution and higher
quality
– Evolution of video acquisition and display
technologies faster than network capabilities
New video coding standard with higher
efficiency than H.264/AVC is needed!
• Goals
– Better coding efficiency at higher resolutions
– Suitability to a wide variety of applications
(mobile TV, home cinema, UHDTV)
14/8/2012 SPIE Optics and Photonic 2012 4

5. Joint Collaborative Team on Video Coding (JCT-VC)
• Video coding experts from ITU-T Study Group 16 (VCEG) + ISO/IEC JTC 1/SC
29/WG 11 (MPEG) = the Joint Collaborative Team on Video Coding (JCT-VC)
• The JCT-VC standardization effort is being referred to as High Efficiency Video
Coding (HEVC)
Test Model under HEVC Test Model (HM)
Call for Proposals
Consideration publicly available
(January 2010)
(July 2010) (January 2011)
Each proponent had to: • Based on key elements • Integrates the latest
• Develop and submit a binary of some of the best developments
executable of the proposed codec proposals • Steady simplification and
• Encode and decode test material efficiency improvement
• Evaluate objective quality using
• Starting point for the
definition of the new • New version available at
PNSR
• Provide and algorithmic standard each JCT-VC meeting cycle
description of the codec
14/8/2012 SPIE Optics and Photonic 2012 5

6. Introduction
• Double the compression efficiency is
expected in HEVC versus AVC
• Can be reliably compared only by
means of subjective tests
• Compression efficiency especially for
resolutions beyond HDTV
• Subjective evaluations of HEVC have
been so-far limited up to HDTV
resolution
• This paper provides results of subjective
evaluation on 4K/QFHD video content
– Professional high-performance 4K/QFHD
LCD reference monitor
14/8/2012 SPIE Optics and Photonic 2012 6

7. Outline
• Introduction
• Dataset
• Test methodology
• Test equipment and environment
• Results
• Conclusion
14/8/2012 SPIE Optics and Photonic 2012 7

8. Contents
• PeopleOnStreet (3840x2160@30fps) • Traffic (3840x2048@30fps)
• Sintel2 (3840x1744@24fps) • Sintel39* (3840x1744@24fps)
*Only used for training
14/8/2012 SPIE Optics and Photonic 2012 8

9. Spatial information and temporal information indexes
30 Sintel2
Sintel39
25
PeopleOnStreet
TI
20
Traffic
15
0 5 10 15
SI
• Different spatio-temporal characteristics
– Sintel2 and Sintel39: large TI values
– Traffic: small TI index (easier to encode)
14/8/2012 SPIE Optics and Photonic 2012 9

10. Encoding
• Codecs
– AVC (JM 18.3)
– HEVC (HM 6.1.1)
• Configuration
– Random Access
– GOP size: 8 pictures
– Intra Period: 1s
24 pictures @24fps and 32 @30fps
• Hierarchical B-pictures
– QP increase of 1 between
each Temporal Level
– Coding Order: 0 8 4 2 1 3 6 5 7
14/8/2012 SPIE Optics and Photonic 2012 10

11. Encoding
• 5 different bit rates for each content and codec
– Both natural and synthetic contents
• Fixed QPs (no Rate Control in HM 6)
– Typical QPs range for AVC: 25 - 37
– Upper bit rate limit: 20 Mbps
• Bit rates selected based on expert viewing sessions
– Select lower/upper bounds for each content separately
– Targeting realistic bit rates
– Try to cover the full quality scale
*BR expressed in Mbps
14/8/2012 SPIE Optics and Photonic 2012 11

12. Outline
• Introduction
• Dataset
• Test methodology
• Test equipment and environment
• Results
• Conclusion
14/8/2012 SPIE Optics and Photonic 2012 12

13. Methodology
1
• Double Stimulus Impairment Scale (DSIS) Variant II 100
Imperceptible
90
Reference Test Reference Test
Video Video Video Video 80
Perceptible
annoying
but not
70
60
annoying
Slightly
50
40
2s 5s 2s 5s 2s 5s 2s 5s 6s
Annoying
(TOT= 34 s) 30
20
“Rate the level of annoyance of the visual defects that you see
annoying
Very
10
in stimulus B, knowing that A is the reference video.”
0
14/8/2012 SPIE Optics and Photonic 2012 13

14. Sessions
• ITU BT-500
– One test session should not last more than 30 minutes
– Alternate as many different contents as possible
• Short sequences => evaluation task requires a lot of attention
– Test sessions no longer than 15 minutes each
– Each session is followed by a resting phase
• Details
– Never the same content in consecutive presentations
– Randomization to avoid possible effect of content presentation order
– Dummy sequences at beginning of 1st session to stabilize observers’ rating
– Reference versus reference stimulus to check subject’s reliability
14/8/2012 SPIE Optics and Photonic 2012 14

15. Timing
• One DSIS Variant II presentation
– Theoretically: 34 s
– Loading time for each video file in Media Player Classic: 4 s
• Concatenate ‘A’ + ref seq + ‘B’ + test seq in a single avi file
– Total time: 46 s
• Sequences
– 30 test sequences (2 codecs × 3 contents × 5 bit rates)
– Split in 2 test sessions
• Sessions
– Training session: 10 min
– 1st test session: 2 dummies + 1 ref vs ref + 15 stimuli = 18 x 46 s = 14 min
– 2nd test session: 15 stimuli = 15 x 46 s = 11.5 min
14/8/2012 SPIE Optics and Photonic 2012 15

16. Test
• Test planning
– 2 days, 3 time slots per day
– 6 subjects per slot, split in 2 groups of 3 subjects each
• Subjects
– 36 naive people
– 30% of the observers were female
– Age ranged from 20 to 61 years old
• Screening
– Correct visual acuity: Snellen charts
– Correct color vision: Ishiara charts
• Training
– Oral instructions to explain the task and the meaning of each label reported on
the scale
– Viewing session to allow the viewer familiarizing with the assessment procedure
– Training samples have quality levels representative of the labels reported on the
rating scales
14/8/2012 SPIE Optics and Photonic 2012 16

17. Outline
• Introduction
• Dataset
• Test methodology
• Test equipment and environment
• Results
• Conclusion
14/8/2012 SPIE Optics and Photonic 2012 17

18. Laboratory for subjective video quality assessment
• Server with SSD to read and play in real time 3840x2160@30fps
YUV 4:2:0 raw video (i.e., 373.25 MB/s!)
• 56-inch professional high-performance 4K/QFHD LCD reference monitor
Sony Trimaster SRM-L560
• ITU BT.500-11 compliant test environment
– 3 subjects in front of the screen
– Viewing distance ≈ 3.5 x screen height
14/8/2012 SPIE Optics and Photonic 2012 18

19. Outline
• Introduction
• Dataset
• Test methodology
• Test equipment and environment
• Results
• Conclusion
14/8/2012 SPIE Optics and Photonic 2012 19

20. PeopleOnStreet
PeopleOnStreet
100
90
80
70
60
MOS
50
40
30
20
10 AVC
HEVC
0
4 6 8 10 12 14 16 18 20 22
bitrate [Mbps]
• At similar bit rates, HEVC outperforms AVC in 4 / 5 cases
• Bit rate reduction
– BD-PSNR: 27.5%
– BD-MOS: 50.8%
14/8/2012 SPIE Optics and Photonic 2012 20

21. Traffic
Traffic
100
90
80
70
60
MOS
50
40
30
20
10 AVC
HEVC
0
2 4 6 8 10 12 14 16
bitrate [Mbps]
• At similar bit rates, HEVC outperforms AVC in 1 / 4 cases
• Bit rate reduction
– BD-PSNR: 37.7%
– BD-MOS: 74.0%
14/8/2012 SPIE Optics and Photonic 2012 21

22. Sintel2
Sintel2
100
90
80
70
60
MOS
50
40
30
20
10 AVC
HEVC
0
0.5 1 1.5 2 2.5 3 3.5
bitrate [Mbps]
• At similar bit rates, HEVC outperforms AVC in 4 / 4 cases
• Bit rate reduction
– BD-PSNR: 68.0%
– BD-MOS: 74.7%
14/8/2012 SPIE Optics and Photonic 2012 22

23. Subjective versus objective results
PSNR highly correlated with perceived quality,
as long as saturation limits are considered
14/8/2012 SPIE Optics and Photonic 2012 23

24. Outline
• Introduction
• Dataset
• Test methodology
• Test equipment and environment
• Results
• Conclusion
14/8/2012 SPIE Optics and Photonic 2012 24

25. Conclusion
• First comparison of HEVC vs AVC on resolution beyond HDTV
• In general, HEVC significantly outperform AVC for similar bit rates
• Substantial improvement in compression performance
– Natural contents: 51 to 74% bit rate reduction
– Synthetic content: 75% bit rate reduction
• Objective gains based on BD-PSNR are pessimistic compared to
actual gains measured with subjective rating
• The upcoming HEVC video compression standard seems to be one
of the key elements towards a wide deployment of UHDTV
14/8/2012 SPIE Optics and Photonic 2012 25

26. Thank you for your attention!
14/8/2012 SPIE Optics and Photonic 2012 26