-
1.
A History of Video Coding
Invited Talk,
United States Patent and Trademark Office
Iain Richardson
December 2014
Slide 1
vcodex.com
-
2.
What this talk is about
Slide 2
-
3.
Slide 3
From
To
-
4.
Video coding : the basics
Slide 4
-
5.
A video codec (1)
Slide 5
Video Source
Decompress
(Decode)
Compress
(Encode)
Video Display
Coded
video
enCOder / DECoder
-
6.
A video codec (2)
Slide 6
video
source partition transform
entropy
encode
predict
(add)
inverse
transform
entropy
decode
compressed
video
video
output
VIDEO ENCODER
VIDEO DECODER
predict
(subtract)
reconstruct
-
7.
1. Partitioning
Slide 7
frame or
picture
slice
or tile
macroblock (MB)
or
coding tree unit
(CTU)
sequence
-
8.
2. Prediction
Slide 8
Original Frame Prediction Frame
Residual
-
9.
3. Transform + Quantize
Block of samples After transform After quantization
Slide 9
-
10.
4. Entropy coding
Slide 10
1
1
0
1
1
1
1
1
0
0
0
1
1
coded components
(syntax elements)
bitstream
-
11.
source frames
........ ........
coded bitstream
decoded frames
ENCODE
DECODE
Slide 11
-
12.
Video Encoder
Form
prediction
Transform
+ quantize
Inverse
transform
+ quantize
Entropy
encoder
Current MB
Prediction MB
Residual MB
Decoded
Residual MB
Intra
Inter
....
Current frame or field
Previously coded
frames or fields
+ Coded bitstream
-
+
+
Slide 12
-
13.
Video Decoder
Coded bitstream
....
Entropy
decoder
Inverse
transform
+ quantize
Decoded
Residual MB
Current decoded
frame or field
Inter
Previously decoded
frames or fields
Prediction MB
Form
prediction
Intra
Decoded MB
+
+
Slide 13
-
14.
Ancient history
Slide 14
-
15.
Slide 15
1851
-
16.
1910
Slide 16
-
17.
Slide 17
1964
-
18.
The foundations
Slide 18
-
19.
Slide 19
1952 Variable length coding
1972 Frame differencing
1974 Discrete Cosine
Transform
1981
Motion compensated
video codec
1990
Bidirectional prediction
Variable block size
motion compensation
-
20.
Huffman variable length coding, 1952
Slide 20
D A Huffman, Minimum redundancy codes
-
21.
Frame differencing, 1972
Slide 21
US 3,679,821, Schroeder, Bell Labs
-
22.
Discrete Cosine Transform, 1974
Slide 22
The Discrete Cosine Transform
Ahmed, Natarajan and Rao
-
23.
Video codec with motion compensation, 1981
Slide 23
Displacement measurement and its application in
interframe video coding, Jain and Jain
-
24.
Bidirectional + variable block size motion
prediction, circa 1990
Slide 24
MPEG-1, LeGall, 1991 Chan et al, 1990
-
25.
By 1991, the basic codec was complete
Slide 25
MPEG-1 encoder, 1991
-
26.
Video Encoder
Form
prediction
Transform
+ quantize
Inverse
transform
+ quantize
Entropy
encoder
Current MB
Prediction MB
Residual MB
Decoded
Residual MB
Intra
Inter
....
Current frame or field
Previously coded
frames or fields
+ Coded bitstream
-
+
+
Slide 26
-
27.
The video coding standards
Slide 27
-
28.
Slide 28
Moving Picture Experts Group
- MPEG standards
- Broadcast / entertainment
Joint Video Team /
Joint Collaborative Team
Video Coding Experts Group
- H.26x standards
- Communications
-
29.
Slide 29
-
30.
Slide 30
1990
1993
1995
1996
1999
2003
2013
ITU-T JOINT ISO MPEG
MPEG-1
H.263
H.262/MPEG-2
H.264/AVC
H.265/HEVC
MPEG-4
H.261
….
-
31.
ITU-T H.261 (1990)
Slide 31
2-way video telephony
ISDN lines (p x 64kbps)
-
32.
ITU-T H.261 (1990)
Slide 32
16x16 blocks 8x8 DCT
I P P P
Forward
prediction
…011011000100…
Variable
length coding
-
33.
MPEG-1 Video (1991-93)
Slide 33
Video CD playback
-
34.
MPEG-1 Video (1991-93)
Slide 34
I B B P B B P B B P B B I
0 1 2 3 4 5 6 7 8 9 10 11 12
Intra, forward and
bidirectional prediction
Half-pixel motion vectors
-
35.
MPEG-2 Video (1995)
Slide 35
Digital TV broadcasting,
DVD
-
36.
MPEG-2 Video (1995)
Slide 36
16x16 and Interlaced video
16x8 block sizes
-
37.
H.263 (1996), MPEG-4 Part 2 (1999)
Slide 37
-
38.
H.263 (1996), MPEG-4 Part 2 (1999)
Slide 38
16x16, 8x8
blocks
Arithmetic
coding Error
resilience
Advanced
prediction Deblocking
filter
Object
coding Mesh
coding
Face
animation
Many optional
coding tools
-
39.
H.264 / Advanced Video Coding (2003)
Slide 39
-
40.
H.264 / Advanced Video Coding (2003)
Slide 40
16x16 down
to 4x4 blocks
Intra
prediction
0 1 2 3 4 5 6
Multiple reference
frames
+ context adaptive coding
(CABAC / CAVLC),
¼ pixel motion vectors,
loop filter….
-
41.
H.265 / High Efficiency Video Coding (2013)
Slide 41
-
42.
H.265 / High Efficiency Video Coding (2013)
Courtesy of Parabola Research
Slide 42
64x64 down
to 4x4 blocks Parallel processing support
+ Lots of “H.264 like” optimizations….
-
43.
HEVC analyzer screenshot
Courtesy of Parabola Research
Slide 43
-
44.
Performance
Slide 44
-
45.
Compression performance
Slide 45
MPEG-2
~2x better
H.264/AVC
~2x better
H.265/HEVC
-
46.
Performance examples
Slide 46
-
47.
Slide 47
MPEG-1 MPEG-2
H.264 HEVC
-
48.
Slide 48
Difference image (MPEG-1)
Difference image (HEVC)
-
49.
Patents and patent pools
Slide 49
-
50.
Example
Slide 50
September
2001
Patent application is filed
September
2001
Proposal document is submitted to
Joint Video Team
May 2003 H.264 is published, including the
concept of the application & proposal
2005 Patent is granted
-
51.
Slide 51
Published
Standard
Patents
Implementors
Patent
pool
-
52.
Patents and patent pools: the early
standards
Slide 52
H.261, MPEG-1, H.263 :
- no formal patent pool
MPEG-2 :
- first patent pool, managed by MPEG-LA
MPEG-4 Part 2 :
- patent pool, managed by MPEG-LA
-
53.
Patents and patent pools: H.264 / AVC
Slide 53
2001-2003 Royalty Free Baseline intention
announced by 32 patent holders
May 2003 First edition of H.264 published
Late 2003 MPEG-LA and Via Licensing
announce patent pools. Royalty
Free Baseline abandoned.
2004
onwards
Via Licensing pool “folds”. MPEG-LA
pool dominates.
-
54.
Patents and patent pools: recent
developments
Slide 54
Open source VP8 format:
MPEG-LA attempts to create patent pool,
Google and MPEG-LA reach agreement and pool
is abandoned.
HEVC:
Patent pool created by MPEG-LA (2014).
-
55.
Conclusion
Slide 55
Many of the key components of modern video codecs
originated before 1990.
In the last 25 years, developments in video coding
have been strongly influenced by standardisation.
Performance continues to improve, driven by
optimization of compression tools.
-
56.
Useful information sources
Slide 56
Low bit rate / Video Coding Experts Group
archives, 1996-present:
http://wftp3.itu.int/av-arch/video-site/
Joint Video Team archives (H.264 development):
http://wftp3.itu.int/av-arch/jvt-site/
JCT-VC archives (HEVC development):
http://wftp3.itu.int/av-arch/jctvc-site/
Video coding resources:
http://vcodex.com/
On the left is a typical block of pixel samples, 4x4 in this case. In any region of the picture, we’re going to have a range of brightnesses or levels. Each one of those numbers is important to the video image and each one takes up space in the video file.
The idea of a video coding transform is to convert a block of samples into a frequency representation, such as the example in the middle. In a video encoder, we follow the transform with quantization, which removes small or insignificant values. So instead of the 16 image samples on the left, we have a transformed and quantized block on the right. Most of the numbers are zero, which makes this transformed and quantized block very easy to store in a compressed form.
And if we do this in the right way, it’s possible to reverse the process and get a decoded image block that looks almost the same as the original.