Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf

High Efficiency Video Coding
Coding Tools and Specification
Mathias Wien
Institut für Nachrichtentechnik
RWTH Aachen University
ICME2013
Mathias Wien (RWTH) HEVC tutorial ICME2013 1 / 165

Outline Part I
1 Introduction
HEVC – Vision and Requirements
The Joint Collaborative Team on Video Coding
Video Basics
The Hybrid Coding Scheme
Encoder Control
2 Specification and Standardization
Specification Design

Outline Part II
3 Coding Structures
Temporal Coding Structures
Spatial Coding Structures
Reference Pictures
4 High-Level Syntax
Network Abstraction Layer
Parameter Sets
Picture Order Count
Supplemental Enhancement Information
Byte Stream Format
Comparison to H.264|AVC

Outline Part III
5 Inter Prediction
Motion Compensated Prediction
6 Intra Prediction
Reference Pixels for Intra Prediction
Intra Prediction Modes
Predictive Coding of Intra Prediction Modes

Outline Part IV
7 Residual Coding
Core Transforms
Alternative 4×4 Transform
Quantization
Coded Representation of Transform Blocks
Special Modes
8 Loop Filtering
Deblocking Filter
Sample Adaptive Offset
9 Entropy Coding
Variable Length Codes
CABAC

Outline Part V
10 Profiles, Tiers, and Levels
Main Profiles
Tiers and Levels
11 Compression Performance
12 Extensions to HEVC

Part I
Introduction, Specification

Outline
1 Introduction
Video Basics
Encoder Control

The JCT-VC
Photo Jan. 23rd 2013 by Kazushi Sato

Motivation for a New Video Coding Standard
Increasing Video Resolution
High Definition (HD), Ultra High Definition (UHD), 4K×2K, 8K×4K
Mobile services in HD
Stereo video, multi-view video
Recording and Display Devices
HD and UHD resolutions available
Reaching consumer market
Transport
Video data rate grows faster than feasible network capabilities

HEVC Vision and Requirements
Compression Performance
Substantially better compression ratio than H.264|AVC High Profile
Shall not be worse than existing standards at any operating point
Formats
Large range of picture formats
Various color spaces and sampling
Large range of frame rates
Higher bit depth support
Progressive scan

HEVC Vision and Requirements
Complexity
Low complexity operating point
Lower complexity than H.264|AVC at better compression performance
High performance operating point
Higher complexity than H.264|AVC with substantial better compression
performance
Access
Low delay, random access, trick modes
Error resilience, video buffer, system layer interface
Possible support for scalability and multi-view

Development of HEVC – Time Line
Timeline
July 2005: Start of KTA in ITU-T SQ16/Q6 VCEG
July 2008: Start of HVC activity in ISO/IEC JCT1 SC29/WG11 MPEG
April 2009: Call for Evidence on HVC (MPEG doc N10553)
January 2010: Formation of the Joint Collaborative Team on Video
Coding (JCT-VC) → Joint Call for Proposals (CfP)
April 2010: 1st meeting of the JCT-VC, Evaluation of the CfP
Name: High Efficiency Video Coding (HEVC)
July 2010: Test Model under Consideration (TMuC)
October 2010: 1st HEVC Working Draft (WD-1) and Test Model (HM-1)
. . . new WD and HM every three month . . .
January 2013: 10th HEVC Working Draft and Test Model (HM-10)
Final Draft International Standard / Consent

Joint Collaborative Team on Video Coding – JCT-VC
Previous collaborations between MPEG and ITU-T
H.262|MPEG-2 Video
H.264|AVC in the Joint Video Team (last meeting Nov. 2009 in Geneva)
New collaboration between MPEG and VCEG
Joint Collaborative Team on Video Coding (JCT-VC)
Agreement on Terms of Reference Jan. 2010, N11112, VCEG-AM90 [5]
Joint Call for Proposals issues Jan. 2010, N11113, VCEG-AM91 [6]
1st meeting in Dresden, April 2010
Chairs
Gary J. Sullivan (Microsoft)
Jens-Rainer Ohm (RWTH Aachen University)
Resources

JCT-VC Method of Working
Meetings
Decisions by consensus of the group
Meeting report with all group decisions of meeting
Ad-hoc Group (AhGs)
Organize discussion on defined topic between meetings
Exist for only one meeting cycle (can be reinstalled)
AhG chair / co-chairs: responsible for synchronization, reporting
Core Experiments (CEs)
Defined from meeting to meeting
Single tools: Precise definition, well-defined test and evaluation
Verification and cross-check of reported results
Coordinator(s): responsible for conduction, reporting

Structure of a Video Sequence
Definitions
Picture: single array or set of multiple (three) arrays of samples
(pixels = picture elements). Has lines of pixels
Field picture: even and odd lines of the full resolution taken at successive
time instances → interlaced video
Top field: even lines
Bottom field: odd lines
Frame:
Pair of top and bottom field pictures for interlaced video
Full resolution picture in progressive scan

Structure of a Video Sequence
Progressive and interlaced video
(a)
frame
(b)
top field
bottom field
frame = two fields
(c) time (d)
top bot top bot top bot top bot
time

Representation of Color
Color representation: YCbCr
Luminance (Y): gray-level picture, weighted sum of the RGB components
Chrominance (Cb, Cr): delta of Luminance to single color components
Normalization according to transfer characteristics of capture/display
Luma and chroma components (Y, Cb, Cr)
Sub-sampling of chroma
luma
chroma
(a) YCbCr 4:2:0 (b) YCbCr 4:2:2
further reading: [7]

Hybrid Coding Scheme
Used in all video coding specifications of ITU-T and ISO/IEC MPEG since
H.261(first version 1988)
Based on DPCM loop
Hybrid scheme:
Prediction of the coded signal (inter, intra)
Removal of redundant information
Transform coding of the residual including quantization
Removal of irrelevant information

Hybrid Coding Scheme: Encoder
Decoder
CTB
input picture
+
−
TR+Q
TB
iTR+iQ TB
+
Intra
PB
Entropy
Cod-
ing
bitstream
Deblk. Slice
Loop
Filter
Slice
rec. picture
Inter
PB
Buffer n pics
ME
PB
CTB – Coding Tree Block
ME – Motion Estimation
PB – Prediction Block
Q – Quantization
TB – Transform Block
TR – Transform

Encoder Control
Mode decision, determination of coding parameters subject to application
constraints
Picture type
Partitioning and sub-partitioning
Inter / intra prediction mode
Motion vectors, intra prediction directions
Loop filter parameters
Decisions based on distortion metric

Distortion Measures
Current N×M block B with pixels b(n,m)
Prediction / reconstructed block P with pixels p(n,m)
Sum of Absolute Differences (SAD)
DSAD =
N−1
∑
n=0
M−1
∑
m=0
b(n,m)−p(n,m)
Sum of Squared Differences (SSD)
DSSE =
N−1
∑
n=0
M−1
∑
m=0
b(n,m)−p(n,m)
2
Peak Signal-to-Noise Ratio (PSNR)
PSNR = 10 ·log10

NM ·A2
max
DSSE

dB
Amax = 255 = 28
−1 for 8 bit video
Sequence PSNR: Average of picture PSNRs

Rate-Distortion Optimization
Minimize reconstruction error D subject to a rate constraint
Unconstrained formulation with Lagrangian parameter λ
Mopt = argmin
M
J(P ,M|λ)

with joint cost criterion
J(P ,M|λ) = D(P ,M)+λ ·R(P ,M)
P : Picture, constructed of CTBs; M: Prediction and coding modes; R: Rate

Bjøntegaard Delta Measurement
0 250 500 750 1000125015001750
35
36
37
38
39
40
41
42
43
44
Rate [kbps]
PSNR
Y
[dB]
0 250 500 750 1000125015001750
35
36
37
38
39
40
41
42
43
44
Rate [kbps]
PSNR
Y
[dB]
(a) BD-PSNR (b) BD-rate
Comparison of two rate-distortion curves with four rate points
Interpolation of curves (using log-rate)
Polynomial VCEG-M33 [9], or pice-wise cubic JCTVC-F270 [10]
Rate delta (BD-rate, %) or PSNR delta (BD-PSNR, dB) by integration
over intermediate area

Outline
1 Introduction
Video Basics
Encoder Control

Specification Fundamentals
Specification goal: Interoperability!
Scope: Decoding process with input and output
source pre-processing encoding
transmission
decoding
post-processing
display

Specification Elements: Syntax and Semantics
Syntax
Hierarchical structure
Sequence, picture, CTB, sub-block, coefficient level
Syntax tables, conditional order of syntax elements
Conforming bitstream: subset of possible combinations of syntax
structures
Semantics
’Meaning’ of values of syntax elements
Constraints and dependencies between syntax elements

Specification Elements: Decoding Process
Input
Coded video sequence (in the bitstream)
Decoding Process
Derive reconstructed video from input bitstream
Complete and exact specification of processes
→ Includes all normative operations
Does not include handling of unspecified states, occurrence of
unspecified syntax element values
Output
Reconstructed video

Specification Elements: Parsing Process
Entropy coding of syntax elements
Applicable entropy coding method from syntax table
Specification of context dependency (if applicable)
Context-based Adaptive Binary Arithmetic Coding (CABAC)
Specification of arithmetic decoding process
Context initialization
Context update and renormalization

Specification Principles: Bit-Exact Specification
Exact specification of decoding result
Unambiguous decoding result
Eases conformance testing
Approach
No floating-point decoding process operations specified
Normative operations in integer arithmetic (interpolation, filtering)
Specifically: integer transforms
→ avoid drift between different implementations

Specification Principles: Independent Parsing
Parsing independent from decoding
Parsing must be possible in the case of transmission losses
No dependency of occurrence of syntax elements based on properties of
decoded pictures
Independent slice parsing
Slices of a picture must be independent
→ separately decodable in case of slice losses
Enables parallel processing of slices
Further: no parsing dependencies in parameter sets
(e. g. on certain profile) → extensibility
SzBu12 [11]

Specification Principles: Parallelization
Respond to prevalence of multi-thread / multi-core processing
Picture level
Parallel parsing and decoding
Parallel decoding of slices
Tiles: Parallel decoding of local regions of the picture
Wavefront parallel processing: Parallelization on a row basis
Block level
Inter prediction: fetching from reference pictures
Intra prediction: in parallel to inter prediction if constraint intra prediction
used
Loop filtering operations

Specification Principles: Dynamic Range
Memory consumption
Storage size of decoded pictures, parameters and intermediate values
Bandwidth for memory access
Processing operation
Required register sizes
Bit depth of arithmetic operations
Inverse transform and quantization
Very high dynamic range possible (including 32×32 transform)
Restrictions required, by either
Normative restrictions on encoder
Sufficient head-room in decoder specification
Intermediate truncation (used in HEVC)
JCTVC-G1044, JCTVC-H0541 [12, 13]

Specification Principles: Loss Robustness
Decoder reaction on corrupted bitstream unspecified (could “catch fire. . . )
Approach
Hierarchical organization of information (sequence, picture, block level)
→ Network Abstraction Layer (NAL), Video Coding Layer (VCL)
Inclusion of hooks for resynchronization
In-band / out-of-band transmission of parameter sets
Independent parsing (see above)

Specification Principles: Timing
Concept of time
Not needed for specification of decoding process
Relation of pictures in decoding process identified by order
Needed for correct play-out, buffer control
Specification
Decoding process includes no dependency on time
Hypothetical reference decoder
Timing information for input / output buffer management
Requirements on decoding capabilities (for conformance)

Conformance
Conformance determined by the Hypothetical Reference Decoder (HRD)
Conformance scopes
Decoder conformance
Decoded pictures exactly match output of reference decoding process
Bitstream conformance
Bitstream buffer (Coded Picture Buffer, CPB) never overfilled or
unintendedly emptied
Decoded Picture Buffer (DPB) always contains all required reference
pictures as needed, and all decoded pictures needed for output

Part II
HEVC Structure, High Level Syntax

Outline
3 Coding Structures
Reference Pictures
4 High-Level Syntax
Parameter Sets
Picture Order Count
Byte Stream Format

Coding order and output order may differ
0 8
1 2 3 4 5 6 7 9 10 12
output ord.
coding ord. 0 1 2 3 4 5 6 7 8 9
(a) Hierarchical-P coding structure (coding order = output order)
0 8
1 2 3 4 5 6 7 9 10 12
output ord.
coding ord. 0 1
2
3
4 5 6
7 8 12
(b) Hierarchical-B coding structure (coding order 6= output order)

Group of Pictures / Structure of Pictures (GOP / SOP)
Closed-GOP structures: independent from each other
0 7
3 6 10
1 2 4 5 8 9
output ord.
coding ord. 0 2 3 1 5 6 4 7 9 10
(used e. g. in H.262|MPEG-2 Video)
Open-GOP structures: neighboring GOPs share at least one reference
picture (key picture)

Temporal Layers
Higher-layer pictures successively discardable
Resulting sequence decodable at lower resulting picture rate
tid = 3
tid = 2
tid = 1
tid = 0 0 8
4 12
2 6 10
1 3 5 7 9 11
(a)
tid = 2
tid = 1
tid = 0 0 6 12
3 9
1 2 4 5 7 8 10 11
(b)

Picture Types: Random Access Points
Decoding of sequence may start here
Intra-only prediction allowed
→ “Intra random access point” (IRAP)
Instantaneous Decoder Refresh (IDR)
Start of a new coded video sequence
Reset of decoding process
DPB emptied
Only reference to pictures following IDR in coding order
No reference ’across’ IDR picture

Picture Types: Random Access Point (CRA)
Clean Random Access (CRA)
Decoding process not reset if within coded video sequence
DPB left intact if within coded video sequence
Pictures after CRA in coding and output order without reference to
pictures before CRA
If used as starting point
Conformingly decodable pictures following CRA in output order
Pictures after CRA in coding order but before CRA in output order with
reference to previous pictures: to be discarded.

Picture Types: Random Access Point (BLA)
Broken Link Access (BLA)
Useful for bitstream splicing (example to follow)
Not for original use in coded video sequence
Renaming of CRA after splicing operation
Indicates removal of pictures containing broken / unavailable references

Picture Types: Leading Pictures
Precede associated RAP in output order
Follow associated RAP in coding order
Random Access Decodable Leading Picture (RADL)
Correctly decodable if decoding starts at RAP
No reference to pictures prior to RAP in coding and output order
Random Access Skipped Leading Picture (RASL)
May contain references to unavailable pictures if decoding starts at RAP
Shall not be output if associated RAP is a BLA

Picture Types: Trailing Picture
Follow the associated RAP in coding order
Follow the associated RAP in output order
Follow all leading pictures associated to the RAP in coding order

Picture Types: Leading and Trailing Pictures – Example
2 7
0 1 3 4 5 6 8 9
output ord.
coding ord. 0
1 2 3
4 5
6 7 9 8
pic. type RADL RADL IDR TRAIL TRAIL RADL RADL CRA TRAIL TRAIL
NUT 7 6 19 0 1 7 6 21 0 1
(a)
2 7
0 1 3 4 5 6 8 9 10
output ord.
coding ord. 0
1 2 3
4 5
6 7 8
9
pic. type RADL RADL IDR TRAIL TRAIL RASL RASL CRA TRAIL TRAIL
NUT 7 6 19 0 1 9 8 21 0 1
(b)
NUT: NAL unit type, NAL = Network Abstraction Layer

Picture Types: Leading and Trailing Pictures – Splicing
SEQUENCE A
2 0 1 4 3 7 5 6 9 8 12 10 11 1
output ord.
coding ord. 0 1 2 3 4 5 6 7 8 9
pic. type IDR RADL RADL TRAIL TRAIL CRA RASL RASL TRAIL TRAIL
NUT 19 7 6 1 0 21 9 8 1 0
SEQUENCE B
0 1 2 3 × × 5 4 8 6 7 10 9
output ord.
coding ord. 0 1 2 3 4 5 6 7 8 9
pic. type IDR TRAIL TRAIL BLA RASL RASL TRAIL TRAIL CRA RASL
NUT 19 1 0 16 9 8 1 0 21 9

Picture Types: Temporal Sub-Layer Access
Switch temporal layer
Temporal nesting: at any picture to higher or lower tid
General: switching of temporal layer only at tid = 0
Temporal sub-layer access: additional option
Temporal Sub-Layer Access (TSA)
Switch to any higher tid at TSA picture
No reference to higher tid by TSA picture
Stepwise Temporal Sub-Layer Access (STSA)
Switch to tid of STSA picture possible
Switch to higher temporal layers not possible

Picture Types: Temporal Sub-Layer Access – Example
tid = 3
tid = 2
tid = 1
tid = 0 0 8
4 12
2 6 10
1 3 5 7 9 11
coding ord. 0 3 2 4 1 6 5 7 8 11
pic. type IDR TRAIL TSA TRAIL TSA TRAIL TSA TRAIL CRA TRAIL
NUT 20 0 3 0 3 0 3 0 21 0
(a)
tid = 3
tid = 2
tid = 1
tid = 0 0 8
4 12
2 6 10
1 3 5 7 9 11
coding ord. 0 3 2 4 1 6 5 7 8 11
pic. type IDR TRAIL TSA TRAIL TSA TRAIL STSA TRAIL CRA TRAIL
NUT 20 1 3 1 3 1 5 0 21 8
(b)

Blocks and Units
Block: Square or rectangular area in a color component array
Unit: Collocated blocks of the (three) color components, associated
syntax elements and prediction data (e. g. motion vectors)
Picture partitioning
Coding Tree Blocks / Coding Tree Units (CTBs / CTUs)
Slices: Dependent and independent slice segments
Tiles: Modified CTB scan, re-initialization of entropy coding

Slices and Slice Segments
Slice segments
Each CTU in exactly one slice segment
Independent slice segment: Full header, independently decodable
Dependent slice segment: very short header, relies on corresponding
independent slice, inherits CABAC state
Slice types
I-slice: Intra prediction only
P-slice: Intra prediction and motion compensation with one reference
picture list
B-slice: Intra prediction and motion compensation with two reference
picture lists

Wavefront Parallel Processing (WPP)
Storage of CABAC states for synchronization
Two CTUs offset per row (availability of top-left CTU)
Entry points coded in the slice segment header
CTU CTU CTU CTU CTU CTU
0 1 2 3 4 5
CTU CTU CTU CTU
Nc Nc+1 Nc+2 Nc+3
CTU CTU
2Nc 2Nc+1
decoder1
decoder2
decoder3
slice seg.
header
CTU CTU CTU CTU CTU CTU
0 1 Nc Nc+1 2Nc 2Nc+1
··· ··· ···
ep0 ep1 ep2
Bitstream

Tiles
Change scanning order of CTBs in picture
Slices in tiles, or tiles in slices
Reset of prediction and entropy coding → parallel processing
(entry points like WPP)
Slice 1
Slice 2 Slice 3 Slice 4
Slice 5
Tile 1 Tile 2 Tile 3

Coding Tree Blocks and Coding Blocks (CBs)
Quadtree partitioning of CTB into CBs
If picture size not integer multiple of CTB size:
Automatic CTB partitioning to meet picture size (must be multiple of 8×8
pixels)
0
1 2
3 4
5 6
7
8 9
10 11
12 13
14 15
16
17 18
19 20
21
0
1 2
3 4 5 6
7 8 9 10 11
12 13 14 15
16
17 18 19 20
21
(a) CTB with CBs (b) corresponding quadtree

Prediction Blocks (PBs) and Transform Blocks (TBs)
Prediction block partitioning of a 2N×2N CB
INTER
2N×2N 2N×N N×2N N×N
2N×nU 2N×nD nL×2N nR×2N
INTRA
2N×2N
4×4
Transform block partitioning of a CB
Quadtree partitioning of CB → Residual Quad Tree (RQT)
Transform size 4×4 to 32×32
TB size 4×4 to 64×64
PB boundaries inside TBs allowed

Reference Picture Sets (RPS)
Reference Picture Set
Set of previously decoded pictures
To be used as reference for inter prediction
Identified by POC value
Picture marking
Use in current or following pictures
Unused for reference (can be removed from DPB)
Construction
Short-term before (B)
Short-term after (A)
Long-term

Reference Picture Sets
Short-term and long-term reference pictures
Signaling
Short-term reference picture sets
Pictures in the proximity of the current picture
List of usable short-term RPSs in Sequence Parameter Set (SPS)
Alternative: explicit signaling in slice segment header
Slice indicates which predefined RPS to apply
Long-term reference picture set
Signaled in SPS
Alternative: explicit signaling in slice segment header
further reading: [14]

Short-Term RPS – Example
POC
RPS
8
B0 *
B1
B2
B3
0 8
4
2 6
1 3 5 7
-1
-2
-3
-4
-5
-6
-7
-8
9
B0 A0
*
B1
10
B0 A1
A0
*
B1
3
B0 A2
A1
A0
*
4
B1 A1
A0
B0 *
5
B2 A0
B0
B1 *
6
B1 A1
B0 A0
*
7
B2 A0
B1 B0 *
0 8
4
2 6
1 3 5 7
-1
-2
-3
-4
-5
-6
-7
-8
RPS of random access configuration from the JCT-VC
common testing conditions JCTVC-K1100 [15]

Short-Term RPS – Example
POC
RPS
0 8
B0 *
B1
B2
B3
0 8
4
2 6
1 3 5 7
-1
-2
-3
-4
-5
-6
-7
-8
1 9
B0 A0
*
B1
2 10
B0 A1
A0
*
B1
3
B0 A2
A1
A0
*
4
B1 A1
A0
B0 *
5
B2 A0
B0
B1 *
6
B1 A1
B0 A0
*
7
B2 A0
B1 B0 *
0 8
4
2 6
1 3 5 7
-1
-2
-3
-4
-5
-6
-7
-8
RPS of random access configuration from the JCT-VC
common testing conditions JCTVC-K1100 [15]

Reference Picture List (RPL)
Reference picture lists constructed from available RPS
Size of RPL signaled in PPS or slice segment header
One list in P-slices (List0)
Two lists in B-slices (List0, List1)
Construction
List0
Short-term before
Short-term after
Long-term
List1
Short-term after
Short-term before
Long-term

Outline
3 Coding Structures
Reference Pictures
4 High-Level Syntax
Parameter Sets
Picture Order Count
Byte Stream Format

Network Abstraction Layer (NAL)
Encapsulation of coded video sequence for transport and storage
Video coding layer (VCL) NAL units: video data
Non-VCL NAL units: parameter sets, supplemental enhancement
information, . . .
NAL Unit Header (2 bytes)
“0” NAL unit type NUH layer id temporal id
byte 1 byte 2
NAL unit header (NUH) layer id: equal to 0 for HEVC Version 1, for use in
scalable extension (spatial layers, quality layers), multi-view extension
(view id)
temporal id (tid): temporal sub-layer identifier

NAL Unit Types
64 values available, VCL and non-VCL NAL units
New: Detailed information on picture properties conveyed in NAL unit
type (NUT) – 16 types available
Reserved NUTs with dedicated number ranges for future extensions
NUT Name NAL Unit Content Class
0 TRAIL_N CSS(∗) of a trailing picture (non-referenced) VCL
1 TRAIL_R CSS of a trailing picture (referenced) VCL
2 TSA_N CSS of a temporal sub-layer access pic. (non-ref.) VCL
3 TSA_R CSS of a TSA picture (referenced) VCL
4 STSA_N CSS of a step-wise TSA (non-referenced) VCL
5 STSA_R CSS of a STSA (referenced) VCL
6 RADL_N CSS of a random access decodable leading picture (non-referenced) VCL
7 RADL_R CSS of a RADL picture (referenced) VCL
8 RASL_N CSS of a random access skipped leading picture (non-referenced) VCL
9 RASL_R CSS of a RASL picture (referenced) VCL
.
.
.
(∗) CSS: Coded slice segment

NAL Unit Types II
NUT Name NAL Unit Content Class
(. . . )
10 RSV_VCL_N10 Reserved non-IRAP sub-layer non-referenced VCL
11 RSV_VCL_R11 Reserved non-IRAP sub-layer referenced VCL
16 BLA_W_LP CSS of a Broken Link Access (BLA) with leading pictures VCL
17 BLA_W_RADL CSS of a BLA with associated RADL pictures VCL
18 BLA_N_LP CSS of a BLA without leading pictures (LP) VCL
19 IDR_W_RADL CSS of an IDR picture with RADL VCL
20 IDR_N_LP CSS of an IDR picture without LP VCL
21 CRA_NUT CSS of a Clean Random Access picture VCL
22 RSV_IRAP_22 Reserved IRAP VCL
23 RSV_IRAP_23 Reserved IRAP VCL
24–31 RSV_VCL_24, . . . , RSV_VCL_31 Reserved non-IRAP VLC
32 VPS_NUT Video Parameter Set (VPS) non-VCL
33 SPS_NUT Sequence Parameter Set (SPS) non-VCL
34 PPS_NUT Picture Parameter Set (PPS) non-VCL
35 AUD_NUT Access Unit Delimiter (AUD) non-VCL
36 EOS_NUT End of Sequence (EOS) non-VCL
37 EOB_NUT End of Bitstream (EOB) non-VCL
38 FD_NUT Filler Data (FD) non-VCL
39 PREFIX_SEI_NUT Prefix Supplemental Enhancement Information (SEI) non-VCL
40 SUFFIX_SEI_NUT Suffix SEI non-VCL
41–47 RSV_VCL_N41, . . . , RSV_VCL_N47 Reserved non-VCL
48–63 UNSPEC48, . . . , UNSPEC63 Unspecified non-VCL

Access Units
start
AUD
(prefix)
SEI
CSS
suffix
SEI
EOS EOB
end
Access Unit (AU): Set of all NAL units associated to exactly one picture
NAL units of AU share output time of included picture
Prefix and suffix SEI messages
Decoding Units (DU)
Decoder operation on sub-picture level enabled
Decoding unit: AU or subset of AU
Enables sub-picture output before complete decoding of full picture
AUD Access unit delimiter
SEI Supplemental enhancement information
EOS End of sequence
EOB End of bitstream

Parameter Sets
Hierarchical structure
Separation of information for different hierarchy levels
Highest-priority information
In-band or out-of-band transmission
Available parameter sets
Video parameter set (VPS)
Sequence parameter set (SPS)
Picture parameter set (PPS)

Parameter Sets
VPS1
VPS2
SPS1
SPS2
SPS3
PPS1
PPS2
PPS3
PPS4
PPS5
slice headers in coded video seq. A
slice headers in coded video seq. B
slice headers in coded video seq. M
slice headers in coded video seq. N
.
.
.
.
.
.
bitstream I
bitstream II

Parameter Sets
Video Parameter Set
Introduced for handling of multi-layer bitstreams
General information, activated for all layers
Coded video sequence(s), included layers, available operation points
HEVC Version 1: not needed, copy of SPS information
Sequence Parameter Set
Profile, tier, and level
Usage of tools
Video usability information, . . .
Picture Parameter Set
May change from picture to picture (only one per picture)
Tool configuration: CABAC, quantizers, loop filters, tiles, . . .

Picture Order Count (POC)
POC-2
POC-1
POC
POC+1
Identifier of picture in the DPB
Indicates output order of the pictures, strictly increasing
POC of IDR pictures always 0: start of new coded video sequence
Used for derivation of picture distance in inter prediction, scaling of
motion vectors
Constant POC delta not necessary
VPS: indication of POC relation to picture time difference possible

Supplemental Enhancement Information (SEI)
Supplemental information not needed for the decoding process
Prefix SEI: Cannot occur after the NAL units of the coded picture
Suffix SEI: Cannot occur before the NAL units of the coded pictures
SEI Message types
Prefix: Buffering period, Picture timing, Pan-scan rectangle, Filler payload, User
data ITU-T T-35, User data unregistered, Recovery point, Scene information,
Picture snapshot, Progressive refinement start/end, Film grain characteristics,
Post filter hint, Tone mapping information, Frame packing arrangement, Display
orientation, Structure of pictures, Active parameter sets, Decoding unit
information, Temporal sub-layer zero index, Scalable nesting, Region refresh
information
Suffix: Filler payload, User data ITU-T T-35, User data unregistered,
Progressive refinement end, Post filter hint, Decoded picture hash

Byte Stream Format
start code prefix = 0x00 0x00 0x01 = 0000 0000 0000 0000 0000 0001
Specified in HEVC Annex B
Byte stream representation of the NAL unit stream, using start codes for
identification of NAL units / resynchronization
Can be used as transport format for the coded video sequence
Not required to be used if suitable transport of NAL units by other means
available

H.264|AVC picture types and temporal structures
Discrimination of only IRD / non-IDR pictures
Coded slice data partitions A, B, C
SI, SP pictures for stream switching
Concept of tid in scalable extension, Annex G SVC
H.264|AVC Picture partitioning
16×16 macroblocks, prediction partitioning subset of HEVC
4×4 and 8×8 transform blocks (+ two-stage transform for intra 16×16
blocks)
I, P, B slices
Arbitrary slice ordering
No tiles, no slice segments, no WPP

H.264|AVC reference pictures
Short-term and long-term reference pictures
Unused reference pictures
Implicit decoded picture marking (sliding window), ’FIFO’ operation
Explicit memory management control operations (MMCO)
H.264|AVC Interlaced video support
Picture adaptive frame-field coding (PAFF)
Re-arrangement of macroblocks for macroblock adaptive frame-field
coding (MBAFF)
HEVC: No interlaced video support on tool level!

H.264|AVC NAL units
One-byte H.264|AVC NAL unit header (extension with layer ids for SVC)
Much less VCL NAL unit types
SEI messages
No suffix NAL units
Parameter sets
No VPS (Scalability information SEI message for SVC)
Picture order count
Concept of frame number indication for H.264|AVC reference pictures

Part III
Inter Prediction, Intra Prediction

Outline
5 Inter Prediction
6 Intra Prediction

Inter Prediction
Decoder
CTB
input picture
+
−
TR+Q
TB
iTR+iQ TB
+
Intra
PB
Entropy
Cod-
ing
bitstream
Deblk. Slice
Loop
Filter
Slice
rec. picture
Inter
PB
Buffer n pics
ME
PB

POC-2
POC-1
POC
POC+1
Prediction from reference picture lists
Uni-prediction
P-slices only with List0, B-slices with List0 or List1
Minimum PB size 8×4 or 4×8
Bi-prediction, only in B-slices
One predictor from List0, one predictor from List1
Minimum PB size 8×8
Weighted prediction (default equal, or explicit)
Skip, merge mode, advanced motion vector prediction

Sub-Pixel Interpolation (Luma)
Interpolation of half and quarter-pixel locations
Full-pixel luma samples Ai,j used for locations a,b,c as well as d,h,n
(sub-pixel in only one direction)
Interpolated values a,b,c and d,h,n for interpolation of other locations
(sub-pixel in both directions)
Interpolation filters:
hy
a(n) = {−1,4,−10,58,17,−5,1},
h
y
b(n) = {−1,4,−11,40,40,−11,4,−1},
hy
c(n) = hy
a(−n)

Sub-Pixel Interpolation (Chroma)
Interpolation of half-, quarter-, and eights-pixel locations
Full-pixel chroma samples Bi,j used for locations ’ab’ to ’ah’ as well as ’ba’
to ’ha’ (sub-pixel in only one direction)
Interpolated values for other locations (sub-pixel in both directions)
Interpolation filters:
hc
b(n) = {−2,58,10,−2}, n = −1,...,2
hc
c(n) = {−4,54,16,−2}, n = −1,...,2
hc
d(n) = {−6,46,28,−4}, n = −1,...,2
hc
e(n) = {−4,36,36,−4}, n = −1,...,2
hc
f (n) = hc
d(1 −n)
hc
g(n) = hc
c(1 −n)
hc
h(n) = hc
b(1 −n)

Sub-Pixel Interpolation Filter Derivation
p−3 p−2 p−1 p0 p1 p2 p3 p4
pδ
δ
DCT-based interpolation filter
Interpolation filter derived from inverse DCT at shifted pixel location pδ
Forward and inverse DCT in one filter operation
Additional weighted window functions for reduction of ripple in filter
frequency response
Derivation scheme applicable for arbitrary sub-pixel locations
e. g. JCTVC-D344 [16], JCTVC-F247 [17]

Motion Data Storage Reduction
·
·
·
·
·
·
· ·
· ·
·
·
·
·
·
·
· ·
·
·
·
· ·
×
× ×
×
×
×
CTB boundaries
CB boundaries
PB boundaries
16×16 block grid
Intra coded
Sub-sampling of motion vector information in reference pictures
16×16 block grid
Motion vector of top-left block stored

Merging Motion Vectors
Merge candidate list
List with up to five different motion merge candidates (list length indicated
in the slice segment header)
Unavailable candidates are ignored
List filled to specified length → Improved loss robustness:
Available list length independent of derivation process
Additional combination of reference picture list 0 and 1 candidates for
B-slices
Merge mode granularity
PU grid size configured in the PPS
PUs below grid size share merge candidate list
Adjustable computational complexity

Merging Motion Vectors – Spatial Candidates
A0
A1
B0
B1
B2
(xP,yP)
PU
(xP,yP) (xP,yP)
(a) Merge candidates (b) 4×8 PUs
Processing order: A1,B1,B0,A0,B2
Candidates A1,B1 only if not right or bottom PU in CU with two partitions,
respectively
Maximum four spatial merge candidates
Reduced line buffer storage requirement: shifted neighbor locations for
blocks of width 4

Merging Motion Vectors – Temporal Candidates
collocated region in
reference picture
C0
C1
n −5 n −4 n −3 n −2 n −1 n
current picture
POC
td tb
Location C1 if C0 unavailable, or PU at bottom right CTU boundary
(complexity)
Applicable reference picture selected on slice basis
B-slices: Reference picture list 0 or 1 indicated by flag
Reference index of candidate always set to 0 (error resilience, complexity)
MV scaling according to POC distance
mvPU =
tb
td
·mvcol

Predictive Motion Vector Coding
A0
A1
B0
B1
B2
(xP,yP)
PU
collocated region in
reference picture
C0
C1
Advanced motion vector prediction
Reference index and motion vector difference coded!
Selection of predictor by flag (two options)
Derivation process for each reference picture list
Locations from spatial neighborhood as shown for merge mode
Candidate MVs scaled based on POC difference
Candidate A first of A0,A1, candidate B first of B0,B1,B2
Optional: Additional temporal candidate C if A or B not available

H.264|AVC motion vector representation
One motion vector predictor (median)
Temporal and spatial direct mode
H.264|AVC sub-pixel interpolation filtering
Two-stage interpolation for quarter-pixel locations: 6-tap and bi-linear
Bi-linear H.264|AVC chroma sub-pixel interpolation
Precision issue with interpolation filter for half-pixel locations

Outline
5 Inter Prediction
6 Intra Prediction

Intra Prediction
Decoder
CTB
input picture
+
−
TR+Q
TB
iTR+iQ TB
+
Intra
PB
Entropy
Cod-
ing
bitstream
Deblk. Slice
Loop
Filter
Slice
rec. picture
Inter
PB
Buffer n pics
ME
PB

Bcurr
pref
scan for available pixels
Block size scurr×scurr, set of reference pixel pref
pref extends by 2 ·scurr in horizontal and vertical direction
Scan of pref for availability along indicated directions. Unavailable
locations filled
Reference pixel filtering for PB sizes 8×8 to 32×32
(activation dependent on direction)

18
10
3
2
34
26
Planar prediction: mode 0
DC intra prediction: mode 1
Numbering from diagonal-up (2) to diagonal-down (34)
Horizontal: mode 10, vertical: mode 26

Prediction block size
PU size = CU size (2N×2N), and for smallest CU size additionally N×N
PU determines prediction direction
Applicable prediction block size derived from residual quadtree!
Table 8–2 from HEVC specification
Chroma prediction mode dependence on luma configurable

Planar and DC Prediction
y
x
a
b
c d pref
pref
vDC
(a) (b)
Planar prediction (a)
B lending between pixel values a,b,c,d
DC prediction (b)
Mean value vDC from direct neighboring pixels
Smoothing at block boundary
Mean of neighboring pixel and vDC
Omitted for chroma blocks

Horizontal and Vertical Intra Prediction
Example: Vertical intra prediction
−
Copy reference pixels of same line / column to Bcurr
Boundary smoothing: Local update
Difference between neighbor prediction pixel and corner pixel pref(−1,−1)
Scaling of update by 1/2
Boundary smoothing omitted for chroma

Angular Intra Prediction
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
=⇒
pref
p1,ref
Unified intra prediction reference
Simplified determination of horizontal / vertical reference pixels
Projection of used pixel from ’side’ reference to main reference
Interpolated prediction: Angle parameter pang
Prediction values: weighted of neighboring pixels by iFact
Pixel offset: iIdx = (y +1)·
pang
32
Weighting factor: iFact =

(y +1)·pang

mod32

Angular Prediction: Horizontal Modes
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 2
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 3
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 4
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 5

-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 6
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 7
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 8
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 9

-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 11
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 12
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 13
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 14

-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 15
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 16
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 17
-4
-3
-2
-1
0
1
2
3
4
5
6
7
8
Mode 18

Angular Prediction: Vertical Modes
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 19
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 20
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 21
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 22

-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 23
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 24
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 25
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 27

-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 28
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 29
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 30
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 31

-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 32
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 33
-4 -3 -2 -1 0 1 2 3 4 5 6 7 8
Mode 34

Bcurr
Ba
Bb
(xcurr,ycurr)
(xa,ya)
(xb,yb)
Construction of list with three most-probable modes
Candidates from neighbors a and b
Planar, DC, or Vertical (Mode 26) added to list
Indication of list entry for current mode, or
Explicit coding of intra prediction mode

H.264|AVC intra prediction modes
Nine directional prediction modes
DC and planar prediction
4×4 and 8×8 intra prediction blocks (8×8 in High profiles)
16×16 prediction block with less prediction modes
Intra prediction reference
Vertical reference pixels restricted to block size
Predictor low-pass filtering for 8×8 blocks

Part IV
Residual Coding, Loop Filters, Entropy
Coding

Outline
7 Residual Coding
Core Transforms
Quantization
Special Modes
8 Loop Filtering
Deblocking Filter
9 Entropy Coding
CABAC

Residual Coding
Decoder
CTB
input picture
+
−
TR+Q
TB
iTR+iQ TB
+
Intra
PB
Entropy
Cod-
ing
bitstream
Deblk. Slice
Loop
Filter
Slice
rec. picture
Inter
PB
Buffer n pics
ME
PB

Core Transforms
Transform block sizes 4×4, 8×8, 16×16, and 32×32
Integer approximations of the DCT-II transform matrix
’Single-norm’ design per transform block size → simple quantizer
implementation
Not all perfectly orthogonal, leakage below normalization threshold

Construction of the Transform Matrices
T4 =




t0
t1
t2
t3



 =




a0 a0 a0 a0
b0 b1 −b1 −b0
a0 −a0 −a0 a0
b1 −b0 b0 −b1





T8 =












a0 a0 a0 a0 a0 a0 a0 a0
c0 c1 c2 c3 −c3 −c2 −c1 −c0
b0 b1 −b1 −b0 −b0 −b1 b1 b0
c1 −c3 −c0 −c2 c2 c0 c3 −c1
a0 −a0 −a0 a0 a0 −a0 −a0 a0
c2 −c0 c3 c1 −c1 −c3 c0 −c2
b1 −b0 b0 −b1 −b1 b0 −b0 b1
c3 −c2 c1 −c0 c0 −c1 c2 −c3













T16 =





























a0 a0 a0 a0 a0 a0 a0 a0
d0 d1 d2 d3 d4 d5 d6 d7
c0 c1 c2 c3 −c3 −c2 −c1 −c0
d1 d4 d7 −d5 −d2 −d0 −d3 −d6
b0 b1 −b1 −b0 −b0 −b1 b1 b0
d2 d7 −d3 −d1 −d6 d4 d0 d5
c1 −c3 −c0 −c2 c2 c0 c3 −c1
d3 −d5 −d1 d7 d0 d6 −d2 −d4
a0 −a0 −a0 a0 a0 −a0 −a0 a0
d4 −d2 −d6 d0 −d7 −d1 d5 d3
c2 −c0 c3 c1 −c1 −c3 c0 −c2
d5 −d0 d4 d6 −d1 d3 d7 −d2
b1 −b0 b0 −b1 −b1 b0 −b0 b1
d6 −d3 d0 −d2 d5 d7 −d4 d1
c3 −c2 c1 −c0 c0 −c1 c2 −c3
d7 −d6 d5 −d4 d3 −d2 d1 −d0
...





























,

T32 =






























































a0 a0 a0 a0 a0 a0 a0 a0 a0 a0 a0 a0 a0 a0 a0 a0
e0 e1 e2 e3 e4 e5 e6 e7 e8 e9 e10 e11 e12 e13 e14 e15
d0 d1 d2 d3 d4 d5 d6 d7 −d7 −d6 −d5 −d4 −d3 −d2 −d1 −d0
e1 e4 e7 e10 e13 −e15 −e12 −e9 −e6 −e3 −e0 −e2 −e5 −e8 −e11 −e14
c0 c1 c2 c3 −c3 −c2 −c1 −c0 −c0 −c1 −c2 −c3 c3 c2 c1 c0
e2 e7 e12 −e14 −e9 −e4 −e0 −e5 −e10 −e15 e11 e6 e1 e3 e8 e13
d1 d4 d7 −d5 −d2 −d0 −d3 −d6 d6 d3 d0 d2 d5 −d7 −d4 −d1
e3 e10 −e14 −e7 −e0 −e6 −e13 e11 e4 e2 e9 −e15 −e8 −e1 −e5 −e12
b0 b1 −b1 −b0 −b0 −b1 b1 b0 b0 b1 −b1 −b0 −b0 −b1 b1 b0
e4 e13 −e9 −e0 −e8 e14 e5 e3 e12 −e10 −e1 −e7 e15 e6 e2 e11
d2 d7 −d3 −d1 −d6 d4 d0 d5 −d5 −d0 −d4 d6 d1 d3 −d7 −d2
e5 −e15 −e4 −e6 e14 e3 e7 −e13 −e2 −e8 e12 e1 e9 −e11 −e0 −e10
c1 −c3 −c0 −c2 c2 c0 c3 −c1 −c1 c3 c0 c2 −c2 −c0 −c3 c1
e6 −e12 −e0 −e13 e5 e7 −e11 −e1 −e14 e4 e8 −e10 −e2 −e15 e3 e9
d3 −d5 −d1 d7 d0 d6 −d2 −d4 d4 d2 −d6 −d0 −d7 d1 d5 −d3
e7 −e9 −e5 e11 e3 −e13 −e1 e15 e0 e14 −e2 −e12 e4 e10 −e6 −e8
a0 −a0 −a0 a0 a0 −a0 −a0 a0 a0 −a0 −a0 a0 a0 −a0 −a0 a0
e8 −e6 −e10 e4 e12 −e2 −e14 e0 −e15 −e1 e13 e3 −e11 −e5 e9 e7
d4 −d2 −d6 d0 −d7 −d1 d5 d3 −d3 −d5 d1 d7 −d0 d6 d2 −d4
e9 −e3 −e15 e2 −e10 −e8 e4 e14 −e1 e11 e7 −e5 −e13 e0 −e12 −e6
c2 −c0 c3 c1 −c1 −c3 c0 −c2 −c2 c0 −c3 −c1 c1 c3 −c0 c2
e10 −e0 e11 e9 −e1 e12 e8 −e2 e13 e7 −e3 e14 e6 −e4 e15 e5
d5 −d0 d4 d6 −d1 d3 d7 −d2 d2 −d7 −d3 d1 −d6 −d4 d0 −d5
e11 −e2 e6 −e15 −e7 e1 −e10 −e12 e3 −e5 e14 e8 −e0 e9 e13 −e4
b1 −b0 b0 −b1 −b1 b0 −b0 b1 b1 −b0 b0 −b1 −b1 b0 −b0 b1
e12 −e5 e1 −e8 e15 e9 −e2 e4 −e11 −e13 e6 −e0 e7 −e14 −e10 e3
d6 −d3 d0 −d2 d5 d7 −d4 d1 −d1 d4 −d7 −d5 d2 −d0 d3 −d6
e13 −e8 e3 −e1 e6 −e11 −e15 e10 −e5 e0 −e4 e9 −e14 −e12 e7 −e2
c3 −c2 c1 −c0 c0 −c1 c2 −c3 −c3 c2 −c1 c0 −c0 c1 −c2 c3
e14 −e11 e8 −e5 e2 −e0 e3 −e6 e9 −e12 e15 e13 −e10 e7 −e4 e1
d7 −d6 d5 −d4 d3 −d2 d1 −d0 d0 −d1 d2 −d3 d4 −d5 d6 −d7
e15 −e14 e13 −e12 e11 −e10 e9 −e8 e7 −e6 e5 −e4 e3 −e2 e1 −e0
...






























































.

Coefficient values
a0 b0 b1 c0 c1 c2 c3
64 83 36 89 75 50 18
d0 d1 d2 d3 d4 d5 d6 d7
90 87 80 70 57 43 25 9
e0 e1 e2 e3 e4 e5 e6 e7
90 90 88 85 82 78 73 67
e8 e9 e10 e11 e12 e13 e14 e15
61 54 46 38 31 22 13 4

Transform Approximation
Example: 8×8 transform
Non-orthogonal matrix, leakage below normalization threshold
Approximately identical norms for all bases
T8 ·TT
8 =












32768 0 0 0 0 0 0 0
0 32740 0 −50 0 50 0 0
0 0 32740 0 0 0 0 0
0 −50 0 32740 0 0 0 50
0 0 0 0 32768 0 0 0
0 50 0 0 0 32740 0 50
0 0 0 0 0 0 32740 0
0 0 0 50 0 50 0 32740













Approximation of a 4×4 DST matrix
Applied to intra 4×4 blocks (independent of intra prediction mode)
Consideration of intra prediction error properties
TA =




29 55 74 84
74 74 0 −74
84 −29 −74 55
55 −84 74 −29




TA ·TT
A =




16398 0 −15 15
0 16428 0 0
−15 0 16398 −15
15 0 −15 16398




DST = Discrete Sine Transform

Quantization
... ...
nq
x
-4 -3 -2 -1 1 2 3 4
0
∆q
Quantization and inverse quantization
N×N block B, transform TN:
Cq = round

1
∆q
·
1
kTN k2
·TN ·B ·TT
N

B̃ = round

∆q ·
1
kTN k2
·TT
N ·Cq ·TN

Integer implementation
Quantization parameter QP, transform norm 2log2(N)+12
Logarithmic design: ∆q = rs( QP%6)2QP/6
Signaling
PPS, slice, CU level, quantization groups
(configurable granularity for QP delta)

Transform Sub-Blocks (TSBs)
last significant coefficient
16
16
4
4
transform sub-block
Scan of transform sub-blocks
Last significant coefficient
Determination of transform sub-blocks with non-zero coefficients
Coded sub-block flag

Transform Sub-Block Scanning
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
0 1 2 3
4 5 6 7
8 9 10 11
12 13 14 15
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
0
1
2
3
0
2
1
3
diag.,
vert.
horiz.
(a) 4×4 diagonal (b) 4×4 horizontal (c) 4×4 vertical (d) 8×8 TSB scans
Partitioning of transform block into 4×4 transform sub-blocks (TSBs)
Scan direction selectable for 4×4 and 8×8 blocks, diagonal otherwise
Scan direction in TSB depending on (intra) prediction mode

Coded transform coefficient information in five scan passes
Significance map
Level larger than 1
Level larger than 2
Coefficient sign
Remaining absolute level
Level larger than 1 and 2 scans terminate early if too many significant
coefficients
SoJoNgJiKaClHeDu12 [19]

Sign Data Hiding
Omit coding of sign flags
Sign for first significant scan position in TSB not coded but derived
Sum of absolute levels
Sum even → positive value
Sum odd → negative value
Saves one bit per TSB

Special Modes
Transform Skip, applicable to 4×4 TUs
Omit transform step
Inverse quantization operation
Additional bit shift to compensate missing transform
PCM mode
Direct encoding of pixel levels
Lossless representation of a CU
Configurable bit depth (SPS)
Maximum PCM block size 32×32

Outline
7 Residual Coding
Core Transforms
Quantization
Special Modes
8 Loop Filtering
Deblocking Filter
9 Entropy Coding
CABAC

Loop Filtering
Decoder
CTB
input picture
+
−
TR+Q
TB
iTR+iQ TB
+
Intra
PB
Entropy
Cod-
ing
bitstream
Deblk. Slice
Loop
Filter
Slice
rec. picture
Inter
PB
Buffer n pics
ME
PB

Deblocking Filter
Design considerations
Reduction of visible blocking artifacts from block-wise processing
Computational complexity
Parallel processing
H.264|AVC deblocking filter in comparison:
Operation on 4×4 block boundary grid
Operation on macroblock basis (no parallel processing)
Higher computational complexity
[20]

Deblocking Filter Strength
vertical edges horizontal edges
(a)
CTB boundaries
PBA PBB
PBC PBD
PBE
q0,A1
p0,A1
q0,A2
p0,A2
q0,B1
p0,B1
q0,B2
p0,B2
q0,E
p0,E
q0,C
p0,C q0,D
p0,D
(b)
PBA PBB
PBC PBD
PBE
q0,A1
p0,A1
q0,A2
p0,A2
q0,B
p0,B
(i) (ii)
q0,E
q0,C q0,D
p0,C p0,D
Neighborhood determination
Filtering only on 8×8 sample grid!
q0,p0: reference pixels in current block and neighboring block
Horizontal filtering: p0,B shifted from (i) to (ii) for MV check

Deblocking Filter Operation
Boundary strength parameter bs
Neighboring blocks determined by q0,p0
Possible boundary parameter strength values: 0,1,2
Deblocking filter operation
Operation on a 4-pixel edge basis
Luma: Deblocking if bs ≥ 1
Chroma: Deblocking if bs = 2
First vertical filtering, then horizontal filtering
Independent operation on 8×8 block grid → parallel processing!

q0,0
q0,3
p0,0
p0,3
q1,0
q1,3
p1,0
p1,3
q2,0
q2,3
p2,0
p2,3
q3,0
q3,3
p3,0
p3,3
Bq
Bp
Analysis of first and last row of 4-pixel edge → complexity
Deblocking filtering condition: sum of 2nd-order derivatives β,
for p,q and rows 0,3; e. g. dpi = p |p2,i −2p1,i +p0,i |
Example: vertical edges; horizontal processing accordingly

Strong filter
-3 -2 -1 0 1 2 n
1
2
hq0
(n)
p1 p0 q0 q1 q2
pn :
−3−2−1 0 1 2
-3 -2 -1 0 1 2 n
1
hq1
(n)
p0 q0 q1 q2
pn :
−3−2−1 0 1 2
-3 -2 -1 0 1 2 n
1
3
2
hq2
(n)
p0 q0 q1 q2 q3
pn :
−3−2−1 0 1 2
(a) (b) (c)
Filtering of both pi ,qi
Additional clipping to ±2tC:
q0
i = min
h
max

(qi −2tC),q̃i

,(qi +2tC)
i

Normal filter
Edge steepness
∆0 = round

1
16
·∑
n
h∆0(n)·pn

If ∆0 10 ·tC: Clip ∆0 to ±tC
p0
0 = p0 +∆0,
q0
0 = q0 −∆0
n
-3 -1 1 2
-2 0
3
−9
9
−3
h∆0(n)
p1 p0 q0 q1
pn :
−2 −1 0 1
p1,q1: ∆0 updated by 2nd order derivative and clipped to ±1
2
tC

Sample Adaptive Offset (SAO)
New filter type in ITU-T / MPEG video coding specifications
Local processing of pixels
Depending on local neighborhood, or
Depending on pixel value (amplitude)
Operation independent of processed pixels → parallel processing
Local filter parameter adaptation
[21]

SAO: Edge Offset
Neighborhood
pc
p0 p1 pc
p1
p0
pc
p1
p0
pc
p1
p0
(a) i = 0 (b) i = 1 (c) i = 2 (d) i = 3
Offset categories: Smoothing only
p0 pc p1 p0 pc p1 p0 pc p1 p0 pc p1 p0 pc p1 p0 pc p1
(a) Offset index 1 (b) Offset index 2 (c) Offset index 3 (d) Offset index 4

SAO: Band Offset
low band transition high band
0 8 16 24 32 40 48 56 64 72 80 88 96 104 112 120 128 136 144 152 160 168 176 184 192 200 208 216 224 232 240 248
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
255
intensity value (example: bit depth 8bit)
Correction of pixel intensity values for four transition bands

Sample Adaptive Offset Signaling
Activation of SAO in SPS
SAO signaling on CTU basis
SAO merge (inherit SAO parameters from top or left neighbor)
SAO type: Edge or band offset
SAO offsets

Outline
7 Residual Coding
Core Transforms
Quantization
Special Modes
8 Loop Filtering
Deblocking Filter
9 Entropy Coding
CABAC

Entropy Coding
Decoder
CTB
input picture
+
−
TR+Q
TB
iTR+iQ TB
+
Intra
PB
Entropy
Cod-
ing
bitstream
Deblk. Slice
Loop
Filter
Slice
rec. picture
Inter
PB
Buffer n pics
ME
PB

Entropy Coding
Fixed length and variable length codes
High-level syntax
Parameter sets, slice segment header
SEI messages
Fixed-length codes, Exp-Golomb codes
Arithmetic coding
Slice level, CTUs
Context-based adaptive coding
Bypass coding (complexity, throughput)

Specification: Code Types
ae(v): context-based adaptive arithmetic entropy-coded syntax element
b(8): byte having any pattern of bit string (8 bits)
f(n): fixed-pattern bit string using n bits written with the left bit first
se(v): signed integer 0-th order Exp-Golomb-coded syntax element with
the left bit first
u(n): unsigned integer using n bits
ue(v): unsigned integer 0-th order Exp-Golomb-coded syntax element
with the left bit first.

VLCs: Exp-Golomb Codes
Exp-Golomb (EG) code of order 0
v Ceg,0(v)
0 1
1,2 0 1 x0
3,...,6 0 0 1 x1 x0
7,...,14 0 0 0 1 x2 x1 x0
.
.
.
Unsigned: see table
Signed: x0 as sign bit

CABAC
Context-based Adaptive Binary Arithmetic Coding
Binary Arithmetic Coder
syntax element
Binarizer
Context
Modeler
Adaptive
Engine
Bypass
Engine
bitstream
bin string
binary value
bin value coded bits
bin value coded bits
context update
MaScWi03 [22]

Binary Arithmetic Coding
Representation of engine state
Coding of ’1’s and ’0’s
Most probable symbol (MPS) / least probable symbol (LPS)
Representation
Probability of the LPS pLPS
Value of the MPS vMPS ∈ {0,1}
Integer arithmetic, 9 bit dynamic range
28
≤ Rs 29

CABAC State Transition
0 5 10 15 20 25 30 35 40 45 50 55 60
0.1
0.2
0.3
0.4
0.5
p̃LPSi
ipLPS
state ipLPS
state ipLPS
+1
LPS
MPS
State transitions ipLPS
= 0...62
State ipLPS
= 63 for termination
Highest value p̃LPSi = 0.5 for ipLPS
= 0
Change of vMPS if coding LPS and ipLPS
= 0
MaScWi03 [22]

Table-Based Interval Subdivision
Coding interval subdivision
Current interval Rs quantized to one of four possible values
iRs
= (Rs 6)3, iRs
∈ {0,1,2,3}
Range index iRs
and state index ipLPS
as index into table with precomputed
quantized values p̃LPSi ·Rs
Special state ipLPS
= 63: Returns RLPS = 2 for all indices for termination of
the arithmetic coding process

Binarization
v Beg3(v)
0,...,7 0 x2 x1 x0
8,...,23 1 0 x3 x2 x1 x0
24,...,55 1 1 0 x4 x3 x2 x1 x0
56,...,119 1 1 1 0 x5 x4 x3 x2 x1 x0
120,...,247 1 1 1 1 0 x6 x5 x4 x3 x2 x1 x0
.
.
.
ibin 0 1 2 3 4 5 6 ...
Representation of non-binary value by binary string
Adaptation to the symbol probabilities
Context selection per bin
EG-k binarization:
Prefix with inverted bit polarity compared to coding

Example Binarization: Coefficient Levels
v0 k = 0 v1 k = 1 v2 k = 2
0 0 0,1 0x0 0,...,3 0x1x0
1 10 2,3 10x0 4,...,7 10x1x0
2 110 4,5 110x0 8,...,11 110x1x0
3 1110 6,7 1110x0 12,...,15 1110x1x0
4,5 11110x0 8,...,11 11110x1x0 16,...,23 11110x2x1x0
6,...,9 111110x1x0 12,...,19 111110x2x1x0 24,...,39 111110x3x2x1x0
10,...,17 1111110x2x1x0 20,...,35 1111110x3x2x1x0 40,...,71 1111110x4x3x2x1x0
... ... ... ... ... ...
Coefficient levels 2 (remain abs level)
Special binarization scheme
Prefix: truncated Rice binarization
Suffix: Golomb-Rice binarization

Context Initialization
Contexts initialized
Beginning of slice
Beginning of tile
Three available sets of initialization values
I-slices
Two sets for P- and B-slices selectable by slice segment header flag

Example Context Selection: Significant Coefficient Flag
Significance flag coded for each coefficient between last significant
coefficient and (0,0)
Context selection (offset)
Slice type, TB size, separate for luma and chroma
Coefficient location in the current transform sub-block, DC
Significance of neighboring blocks:
0 1 4 5
2 3 4 5
6 6 8 8
7 7 8
2 1 0 0
1 1 0 0
0 0 0 0
0 0 0 0
2 2 2 2
1 1 1 1
0 0 0 0
0 0 0 0
2 1 0 0
2 1 0 0
2 1 0 0
2 1 0 0
2 2 2 2
2 2 2 2
2 2 2 2
2 2 2 2
(a) 4×4 TB (b) no neighb. (c) right neighb. (d) bottom neighb. (e) both neighb.

CABAC Throughput
Reduced number of bins using context adaptation: Less computational
complexity for bypass coded bins (range adaptation by bit shift)
Grouping of bypass bins
Grouping of bins with same context
Minimized context dependencies
Avoidance of parsing dependencies
SzBu12 [11]

Reduced total number of coded bins (worst case, 1.5×)
Reduced number of context coded bins (8×)
Reduced number of contexts, storage size: 153 for HEVC, 459 (289 w/o
interlaced) for H.264|AVC. 8 bit initialization values in HEVC, 16 bit in
H.264|AVC
Coefficient memory consumption: 4×4 transform sub-blocks
Cleaner structure
Grouping of bypass bins, bins with same context
Reduced parsing dependencies (without significant performance impact)
SzBu12 [11]

Part V
Profiles/Tiers/Levels, Extensions,
Performance

Outline
Main Profiles
Tiers and Levels

Profiles, Tiers, and Levels
Main Still Picture
Main
Main 10

Profiles
Main Profile
YCbCr 4:2:0 8 bit video only
CTB block sizes 16×16, 32×32, and 64×64
Either tiles or wavefront parallel processing
Minimum tile size 256×64 pixels
Maximum number of bits in the bitstream that are CABAC coded is
5*raw/3
Main 10 Profile
Additional support of bit depths of 9 and 10 bit
Main Still Picture Profile
Like Main, but only one picture in the bitstream
No timing constraints for the decoder

Tiers and Levels
Profiles
Defined tool (sub) set of specification
Tools determined from application space
Tiers and levels
Levels: Restrictions on parameters which determine decoding and
buffering capabilities (13 levels defined)
Tiers: Grouping of level limits for different application spaces (currently:
Main Tier for consumer, High Tier for professional applications)
Decoder capable of decoding Profile@Tier/Level must be able to decode
all lower levels of same and lower tier

Outline
Main Profiles
Tiers and Levels

HEVC Compression Performance
Significantly improved compression performance
Comparison of H.264|AVC and HEVC
Subjective and objective quality assessment: OhSuHeTaWi12 [3]
Complexity analysis: BoBrSuFl12 [4]
Facts
Reportedly about 49% bitrate savings for HEVC compared to H.264|AVC
for entertainment application scenario [3]
HEVC real-time encoder and decoder implementations reported and
demonstrated at the JCT-VC meetings, e. g. JCTVC-L0098 [23],
JCTVC-L0379 [24]

Subjective Assessment Results
Entertainment applications
Figure from OhSuHeTaWi12 [3]

Outline
Main Profiles
Tiers and Levels

Extensions in Development
Range extensions
Extended color formats (4 : 2 : 2, 4 : 4 : 4)
Extended bit depth
Scalable extensions
Simple, multiloop approach
No modifications on tool-level
Supports spatial, SNR scalability
Temporal scalability already supported in Main profiles
Multi-view / 3D extensions (→ tutorial by Karsten!)
Stereo / multi-view coding
Multi-view + depth

Outline
13 Resources
14 Acronyms
15 References

HEVC Resources
JCT-VC Mailing List
http://mailman.rwth-aachen.de/mailman/listinfo/jct-vc
JCT-VC Document Repository
http://phenix.it-sudparis.eu/jct/index.php
http://ftp3.itu.int/av-arch/jctvc-site/
JCT-VC HM Software Repository
SVN:
https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSoftware
Trac:
https://hevc.hhi.fraunhofer.de/trac/hevc

Outline
13 Resources
14 Acronyms
15 References

Acronyms
AHG – Ad-hoc group
AMVP – Advanced motion vector prediction
AU – Access unit
BDSNR – Bøntegaard Delta PSNR
BD-rate – Bøntegaard Delta rate
BLA – Broken link access
CABAC – Context adaptive binary arithmetic coding
CPB – Coded picture buffer
CB – Coding block
CRA – Clean random access
CTB – Coding tree block
CTU – Coding tree unit
CU – Coding unit
CE – Core experiment
DCT – Discrete cosine transform
DST – Discrete sine transform
DPB – Decoded picture buffer
DU – Decoding unit
GOP – Group of pictures
HRD – Hypothetical reference decoder

Acronyms II
HVS – Human visual system
IDR – Instantaneous decoder refresh
MPM – Most probable mode
Merge – Merge Mode (MV prediction)
NAL – Network abstraction layer
NUT – NAL unit type
QP – Quantization parameter
PB – Prediction block
POC – Picture order count
PPS – Picture parameter set
PU – Prediction unit
RADL – Random access decodable leading picture
RASL – Random access skipped leading picture
RBSP – Raw byte sequence payload
RPL – Reference picture list
RPS – Reference picture set
RQT – Residual quad-tree
SAD – Sum of absolute differences
SAO – Sample adaptive offset
SAR – Sample aspect ratio

Acronyms III
SEI – Supplemental enhancement information
SODB – String of data bits
SOP – Structure of pictures
SPS – Sequence parameter set
SSD – Sum of squared differences
STSA – Stepwise temporal sub-layer access
TB – Transform block
TSA – Temporal sub-layer access
TSB – Transform sub-block
TU – Transform unit
VCL – Video coding layer
VLC – Variable length code
VPS – Video parameter set
VUI – Video usability information
WPP – Wavefront parallel processing

Outline
13 Resources
14 Acronyms
15 References

References I
[1] High efficiency video coding. ITU-T Recommendation H.265 and ISO/IEC 23008-2 (HEVC). Apr. 2013.
[2] Gary J. Sullivan et al. “Overview of the High Efficiency Video Coding (HEVC) Standard”. In: IEEE Transactions on Circuits and
Systems for Video Technology 22.12 (Dec. 2012), pp. 1649–1668. DOI: 10.1109/TCSVT.2012.2221191.
[3] Jens-Rainer Ohm et al. “Comparison of the Coding Efficiency of Video Coding Standards—Including High Efficiency Video Coding
(HEVC)”. In: IEEE Transactions on Circuits and Systems for Video Technology 22.12 (Dec. 2012), pp. 1669–1684. DOI:
10.1109/TCSVT.2012.2221192.
[4] Frank Bossen et al. “HEVC Complexity and Implementation Analysis”. In: IEEE Transactions on Circuits and Systems for Video
Technology 22.12 (Dec. 2012), pp. 1685–1696. DOI: 10.1109/TCSVT.2012.2221255.
[5] VCEG and MPEG. Terms of Reference of the Joint Collaborative Team on Video Coding Standard Development. Doc. VCEG-AM90.
Kyoto, JP: ITU-T SG16/Q6 VCEG, Jan. 2010.
[6] VCEG and MPEG. Joint Call for Proposals on Video Compression Technology. Doc. VCEG-AM91. Kyoto, JP: ITU-T SG16/Q6 VCEG,
Jan. 2010.
[7] Charles Poynton. Digital Video and HD: Algorithms and Interfaces. Waltham, MA, USA: Morgan Kaufman Publishers, 2012.
[8] Video codec for audiovisual services at p ×64kbit/s. ITU-T Recommendation H.261. 1993. URL:
http://www.itu.int/rec/T-REC-H.261/en.
[9] Gisle Bjontegaard. Calculation of average PSNR differences between RD curves. Doc. VCEG-M33. Austin, TX, USA: ITU-T SG16/Q6
VCEG, Apr. 2001.
[10] Jing Wang, Xiang Yu, and Dake He. On BD-rate calculation. Doc. JCTVC-F270. Torino, IT, 6th meeting: Joint Collaborative Team on
Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, July 2011.
[11] Vivienne Sze and Madhukar Budagavi. “High Throughput CABAC Entropy Coding in HEVC”. In: IEEE Transactions on Circuits and
[12] Madhukar Budagavi and Louis Kreofsky. BoG report on Quantization. Doc. JCTVC-G1044. Geneva, CH, 7th meeting: Joint
Collaborative Team on Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, Nov. 2011.

References II
[13] Louis Kerofsky and Andrew Segall. Non CE4: Limiting dynamic range when using a quantization weighting matrix. Doc. JCTVC-H0541.
San Jose, CA, USA, 8th meeting: Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, Jan. 2012.
[14] Rickard Sjöberg et al. “Overview of HEVC high-level syntax and reference picture management”. In: IEEE Transactions on Circuits and
[15] Frank Bossen. Common test conditions and software reference configurations. Doc. JCTVC-K1100. Shanghai, CN, 11th meeting: Joint
Collaborative Team on Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, Oct. 2012.
[16] Elena Alshina et al. CE3: Experimental results of DCTIF by Samsung. Doc. JCTVC-D344. Daegu, KR, 4th meeting: Joint Collaborative
Team on Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, Jan. 2011.
[17] Elena Alshina and Alexander Alshin. CE3: DCT derived interpolation filter test by Samsung. Doc. JCTVC-F247. Torino, IT, 6th meeting:
Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, July 2011.
[18] Jani Lainema et al. “Intra Coding of the HEVC Standard”. In: IEEE Transactions on Circuits and Systems for Video Technology 22.12
(Dec. 2012), pp. 1792–1801. DOI: 10.1109/TCSVT.2012.2221525.
[19] Joel Sole et al. “Transform Coefficient Coding in HEVC”. In: IEEE Transactions on Circuits and Systems for Video Technology 22.12
(Dec. 2012), pp. 1765–1777. DOI: 10.1109/TCSVT.2012.2223055.
[20] Andrey Norkin et al. “HEVC Deblocking Filter”. In: IEEE Transactions on Circuits and Systems for Video Technology 22.12 (Dec. 2012),
pp. 1746–1754. DOI: 10.1109/TCSVT.2012.2223053.
[21] Chih-Ming Fu et al. “Sample Adaptive Offset in the HEVC Standard”. In: IEEE Transactions on Circuits and Systems for Video
Technology 22.12 (Dec. 2012), pp. 1755–1764. DOI: 10.1109/TCSVT.2012.2221529.
[22] Detlev Marpe, Heiko Schwarz, and Thomas Wiegand. “Context-Based Adaptive Binary Arithmetic Coding in the H.264/AVC Video
Compression Standard”. In: IEEE Transactions on Circuits and Systems for Video Technology 13.7 (July 2003), pp. 620–637.
[23] Thiow K. Tan, Yoshinori Suzuki, and Frank Bossen. On software complexity: decoding 4K60p content on a laptop. Doc. JCTVC-L0098.
Geneva, CH, 12th meeting: Joint Collaborative Team on Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, Jan. 2013.
[24] A. Minezawa et al. HEVC Real-time Hardware Encoder for HDTV signal. Doc. JCTVC-L0379. Geneva, CH, 12th meeting: Joint
Collaborative Team on Video Coding (JCT-VC) of ITU-T VCEG and ISO/IEC MPEG, Jan. 2013.

Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf

More Related Content

What's hot

Similar to Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf

Recently uploaded

Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf