1.
A History of Video Coding
Invited Talk,
United States Patent and Trademark Office
Iain Richardson
December 2014
Slide 1
vcodex.com

2.
What this talk is about
Slide 2

3.
Slide 3
From
To

4.
Video coding : the basics
Slide 4

5.
A video codec (1)
Slide 5
Video Source
Decompress
(Decode)
Compress
(Encode)
Video Display
Coded
video
enCOder / DECoder

6.
A video codec (2)
Slide 6
video
source partition transform
entropy
encode
predict
(add)
inverse
transform
entropy
decode
compressed
video
video
output
VIDEO ENCODER
VIDEO DECODER
predict
(subtract)
reconstruct

7.
1. Partitioning
Slide 7
frame or
picture
slice
or tile
macroblock (MB)
or
coding tree unit
(CTU)
sequence

8.
2. Prediction
Slide 8
Original Frame Prediction Frame
Residual

9.
3. Transform + Quantize
Block of samples After transform After quantization
Slide 9

10.
4. Entropy coding
Slide 10
1
1
0
1
1
1
1
1
0
0
0
1
1
coded components
(syntax elements)
bitstream

11.
source frames
........ ........
coded bitstream
decoded frames
ENCODE
DECODE
Slide 11

12.
Video Encoder
Form
prediction
Transform
+ quantize
Inverse
transform
+ quantize
Entropy
encoder
Current MB
Prediction MB
Residual MB
Decoded
Residual MB
Intra
Inter
....
Current frame or field
Previously coded
frames or fields
+ Coded bitstream
-
+
+
Slide 12

13.
Video Decoder
Coded bitstream
....
Entropy
decoder
Inverse
transform
+ quantize
Decoded
Residual MB
Current decoded
frame or field
Inter
Previously decoded
frames or fields
Prediction MB
Form
prediction
Intra
Decoded MB
+
+
Slide 13

14.
Ancient history
Slide 14

15.
Slide 15
1851

16.
1910
Slide 16

17.
Slide 17
1964

18.
The foundations
Slide 18

19.
Slide 19
1952 Variable length coding
1972 Frame differencing
1974 Discrete Cosine
Transform
1981
Motion compensated
video codec
1990
Bidirectional prediction
Variable block size
motion compensation

20.
Huffman variable length coding, 1952
Slide 20
D A Huffman, Minimum redundancy codes

21.
Frame differencing, 1972
Slide 21
US 3,679,821, Schroeder, Bell Labs

22.
Discrete Cosine Transform, 1974
Slide 22
The Discrete Cosine Transform
Ahmed, Natarajan and Rao

23.
Video codec with motion compensation, 1981
Slide 23
Displacement measurement and its application in
interframe video coding, Jain and Jain

24.
Bidirectional + variable block size motion
prediction, circa 1990
Slide 24
MPEG-1, LeGall, 1991 Chan et al, 1990

25.
By 1991, the basic codec was complete
Slide 25
MPEG-1 encoder, 1991

26.
Video Encoder
Form
prediction
Transform
+ quantize
Inverse
transform
+ quantize
Entropy
encoder
Current MB
Prediction MB
Residual MB
Decoded
Residual MB
Intra
Inter
....
Current frame or field
Previously coded
frames or fields
+ Coded bitstream
-
+
+
Slide 26

27.
The video coding standards
Slide 27

28.
Slide 28
Moving Picture Experts Group
- MPEG standards
- Broadcast / entertainment
Joint Video Team /
Joint Collaborative Team
Video Coding Experts Group
- H.26x standards
- Communications

29.
Slide 29

30.
Slide 30
1990
1993
1995
1996
1999
2003
2013
ITU-T JOINT ISO MPEG
MPEG-1
H.263
H.262/MPEG-2
H.264/AVC
H.265/HEVC
MPEG-4
H.261
….

31.
ITU-T H.261 (1990)
Slide 31
2-way video telephony
ISDN lines (p x 64kbps)

32.
ITU-T H.261 (1990)
Slide 32
16x16 blocks 8x8 DCT
I P P P
Forward
prediction
…011011000100…
Variable
length coding

33.
MPEG-1 Video (1991-93)
Slide 33
Video CD playback

34.
MPEG-1 Video (1991-93)
Slide 34
I B B P B B P B B P B B I
0 1 2 3 4 5 6 7 8 9 10 11 12
Intra, forward and
bidirectional prediction
Half-pixel motion vectors

35.
MPEG-2 Video (1995)
Slide 35
Digital TV broadcasting,
DVD

36.
MPEG-2 Video (1995)
Slide 36
16x16 and Interlaced video
16x8 block sizes

37.
H.263 (1996), MPEG-4 Part 2 (1999)
Slide 37

38.
H.263 (1996), MPEG-4 Part 2 (1999)
Slide 38
16x16, 8x8
blocks
Arithmetic
coding Error
resilience
Advanced
prediction Deblocking
filter
Object
coding Mesh
coding
Face
animation
Many optional
coding tools

39.
H.264 / Advanced Video Coding (2003)
Slide 39

40.
H.264 / Advanced Video Coding (2003)
Slide 40
16x16 down
to 4x4 blocks
Intra
prediction
0 1 2 3 4 5 6
Multiple reference
frames
+ context adaptive coding
(CABAC / CAVLC),
¼ pixel motion vectors,
loop filter….

41.
H.265 / High Efficiency Video Coding (2013)
Slide 41

42.
H.265 / High Efficiency Video Coding (2013)
Courtesy of Parabola Research
Slide 42
64x64 down
to 4x4 blocks Parallel processing support
+ Lots of “H.264 like” optimizations….

43.
HEVC analyzer screenshot
Courtesy of Parabola Research
Slide 43

44.
Performance
Slide 44

45.
Compression performance
Slide 45
MPEG-2
~2x better
H.264/AVC
~2x better
H.265/HEVC

46.
Performance examples
Slide 46

47.
Slide 47
MPEG-1 MPEG-2
H.264 HEVC

48.
Slide 48
Difference image (MPEG-1)
Difference image (HEVC)

49.
Patents and patent pools
Slide 49

50.
Example
Slide 50
September
2001
Patent application is filed
September
2001
Proposal document is submitted to
Joint Video Team
May 2003 H.264 is published, including the
concept of the application & proposal
2005 Patent is granted

51.
Slide 51
Published
Standard
Patents
Implementors
Patent
pool

52.
Patents and patent pools: the early
standards
Slide 52
H.261, MPEG-1, H.263 :
- no formal patent pool
MPEG-2 :
- first patent pool, managed by MPEG-LA
MPEG-4 Part 2 :
- patent pool, managed by MPEG-LA

53.
Patents and patent pools: H.264 / AVC
Slide 53
2001-2003 Royalty Free Baseline intention
announced by 32 patent holders
May 2003 First edition of H.264 published
Late 2003 MPEG-LA and Via Licensing
announce patent pools. Royalty
Free Baseline abandoned.
2004
onwards
Via Licensing pool “folds”. MPEG-LA
pool dominates.

54.
Patents and patent pools: recent
developments
Slide 54
Open source VP8 format:
MPEG-LA attempts to create patent pool,
Google and MPEG-LA reach agreement and pool
is abandoned.
HEVC:
Patent pool created by MPEG-LA (2014).

55.
Conclusion
Slide 55
Many of the key components of modern video codecs
originated before 1990.
In the last 25 years, developments in video coding
have been strongly influenced by standardisation.
Performance continues to improve, driven by
optimization of compression tools.

56.
Useful information sources
Slide 56
Low bit rate / Video Coding Experts Group
archives, 1996-present:
http://wftp3.itu.int/av-arch/video-site/
Joint Video Team archives (H.264 development):
http://wftp3.itu.int/av-arch/jvt-site/
JCT-VC archives (HEVC development):
http://wftp3.itu.int/av-arch/jctvc-site/
Video coding resources:
http://vcodex.com/