Video coding is an essential component of video streaming, digital TV, video chat and many other technologies. This presentation, an invited lecture to the US Patent and Trade Mark Office, describes some of the key developments in the history of video coding.
Many of the components of present-day video codecs were originally developed before 1990. From 1990 onwards, developments in video coding were closely associated with industry standards such as MPEG-2, H.264 and H.265/HEVC.
The presentation covers:
- Basic concepts of video coding
- Fundamental inventions prior to 1990
- Industry standards from 1990 to 2014
- Video coding patents and patent pools.
5. A video codec (1)
Slide 5
Video Source
Decompress
(Decode)
Compress
(Encode)
Video Display
Coded
video
enCOder / DECoder
6. A video codec (2)
Slide 6
video
source partition transform
entropy
encode
predict
(add)
inverse
transform
entropy
decode
compressed
video
video
output
VIDEO ENCODER
VIDEO DECODER
predict
(subtract)
reconstruct
7. 1. Partitioning
Slide 7
frame or
picture
slice
or tile
macroblock (MB)
or
coding tree unit
(CTU)
sequence
12. Video Encoder
Form
prediction
Transform
+ quantize
Inverse
transform
+ quantize
Entropy
encoder
Current MB
Prediction MB
Residual MB
Decoded
Residual MB
Intra
Inter
....
Current frame or field
Previously coded
frames or fields
+ Coded bitstream
-
+
+
Slide 12
13. Video Decoder
Coded bitstream
....
Entropy
decoder
Inverse
transform
+ quantize
Decoded
Residual MB
Current decoded
frame or field
Inter
Previously decoded
frames or fields
Prediction MB
Form
prediction
Intra
Decoded MB
+
+
Slide 13
23. Video codec with motion compensation, 1981
Slide 23
Displacement measurement and its application in
interframe video coding, Jain and Jain
24. Bidirectional + variable block size motion
prediction, circa 1990
Slide 24
MPEG-1, LeGall, 1991 Chan et al, 1990
25. By 1991, the basic codec was complete
Slide 25
MPEG-1 encoder, 1991
26. Video Encoder
Form
prediction
Transform
+ quantize
Inverse
transform
+ quantize
Entropy
encoder
Current MB
Prediction MB
Residual MB
Decoded
Residual MB
Intra
Inter
....
Current frame or field
Previously coded
frames or fields
+ Coded bitstream
-
+
+
Slide 26
28. Slide 28
Moving Picture Experts Group
- MPEG standards
- Broadcast / entertainment
Joint Video Team /
Joint Collaborative Team
Video Coding Experts Group
- H.26x standards
- Communications
34. MPEG-1 Video (1991-93)
Slide 34
I B B P B B P B B P B B I
0 1 2 3 4 5 6 7 8 9 10 11 12
Intra, forward and
bidirectional prediction
Half-pixel motion vectors
40. H.264 / Advanced Video Coding (2003)
Slide 40
16x16 down
to 4x4 blocks
Intra
prediction
0 1 2 3 4 5 6
Multiple reference
frames
+ context adaptive coding
(CABAC / CAVLC),
¼ pixel motion vectors,
loop filter….
41. H.265 / High Efficiency Video Coding (2013)
Slide 41
42. H.265 / High Efficiency Video Coding (2013)
Courtesy of Parabola Research
Slide 42
64x64 down
to 4x4 blocks Parallel processing support
+ Lots of “H.264 like” optimizations….
50. Example
Slide 50
September
2001
Patent application is filed
September
2001
Proposal document is submitted to
Joint Video Team
May 2003 H.264 is published, including the
concept of the application & proposal
2005 Patent is granted
52. Patents and patent pools: the early
standards
Slide 52
H.261, MPEG-1, H.263 :
- no formal patent pool
MPEG-2 :
- first patent pool, managed by MPEG-LA
MPEG-4 Part 2 :
- patent pool, managed by MPEG-LA
53. Patents and patent pools: H.264 / AVC
Slide 53
2001-2003 Royalty Free Baseline intention
announced by 32 patent holders
May 2003 First edition of H.264 published
Late 2003 MPEG-LA and Via Licensing
announce patent pools. Royalty
Free Baseline abandoned.
2004
onwards
Via Licensing pool “folds”. MPEG-LA
pool dominates.
54. Patents and patent pools: recent
developments
Slide 54
Open source VP8 format:
MPEG-LA attempts to create patent pool,
Google and MPEG-LA reach agreement and pool
is abandoned.
HEVC:
Patent pool created by MPEG-LA (2014).
55. Conclusion
Slide 55
Many of the key components of modern video codecs
originated before 1990.
In the last 25 years, developments in video coding
have been strongly influenced by standardisation.
Performance continues to improve, driven by
optimization of compression tools.
56. Useful information sources
Slide 56
Low bit rate / Video Coding Experts Group
archives, 1996-present:
http://wftp3.itu.int/av-arch/video-site/
Joint Video Team archives (H.264 development):
http://wftp3.itu.int/av-arch/jvt-site/
JCT-VC archives (HEVC development):
http://wftp3.itu.int/av-arch/jctvc-site/
Video coding resources:
http://vcodex.com/
Editor's Notes
On the left is a typical block of pixel samples, 4x4 in this case. In any region of the picture, we’re going to have a range of brightnesses or levels. Each one of those numbers is important to the video image and each one takes up space in the video file.
The idea of a video coding transform is to convert a block of samples into a frequency representation, such as the example in the middle. In a video encoder, we follow the transform with quantization, which removes small or insignificant values. So instead of the 16 image samples on the left, we have a transformed and quantized block on the right. Most of the numbers are zero, which makes this transformed and quantized block very easy to store in a compressed form.
And if we do this in the right way, it’s possible to reverse the process and get a decoded image block that looks almost the same as the original.