A short history of video coding

Iain Richardson
Iain RichardsonExpert at Vcodex / Beamshare
A History of Video Coding 
Invited Talk, 
United States Patent and Trademark Office 
Iain Richardson 
December 2014 
Slide 1 
vcodex.com
What this talk is about 
Slide 2
Slide 3 
From 
To
Video coding : the basics 
Slide 4
A video codec (1) 
Slide 5 
Video Source 
Decompress 
(Decode) 
Compress 
(Encode) 
Video Display 
Coded 
video 
enCOder / DECoder
A video codec (2) 
Slide 6 
video 
source partition transform 
entropy 
encode 
predict 
(add) 
inverse 
transform 
entropy 
decode 
compressed 
video 
video 
output 
VIDEO ENCODER 
VIDEO DECODER 
predict 
(subtract) 
reconstruct
1. Partitioning 
Slide 7 
frame or 
picture 
slice 
or tile 
macroblock (MB) 
or 
coding tree unit 
(CTU) 
sequence
2. Prediction 
Slide 8 
Original Frame Prediction Frame 
Residual
3. Transform + Quantize 
Block of samples After transform After quantization 
Slide 9
4. Entropy coding 
Slide 10 
1 
1 
0 
1 
1 
1 
1 
1 
0 
0 
0 
1 
1 
coded components 
(syntax elements) 
bitstream
source frames 
........ ........ 
coded bitstream 
decoded frames 
ENCODE 
DECODE 
Slide 11
Video Encoder 
Form 
prediction 
Transform 
+ quantize 
Inverse 
transform 
+ quantize 
Entropy 
encoder 
Current MB 
Prediction MB 
Residual MB 
Decoded 
Residual MB 
Intra 
Inter 
.... 
Current frame or field 
Previously coded 
frames or fields 
+ Coded bitstream 
- 
+ 
+ 
Slide 12
Video Decoder 
Coded bitstream 
.... 
Entropy 
decoder 
Inverse 
transform 
+ quantize 
Decoded 
Residual MB 
Current decoded 
frame or field 
Inter 
Previously decoded 
frames or fields 
Prediction MB 
Form 
prediction 
Intra 
Decoded MB 
+ 
+ 
Slide 13
Ancient history 
Slide 14
Slide 15 
1851
1910 
Slide 16
Slide 17 
1964
The foundations 
Slide 18
Slide 19 
1952 Variable length coding 
1972 Frame differencing 
1974 Discrete Cosine 
Transform 
1981 
Motion compensated 
video codec 
1990 
Bidirectional prediction 
Variable block size 
motion compensation
Huffman variable length coding, 1952 
Slide 20 
D A Huffman, Minimum redundancy codes
Frame differencing, 1972 
Slide 21 
US 3,679,821, Schroeder, Bell Labs
Discrete Cosine Transform, 1974 
Slide 22 
The Discrete Cosine Transform 
Ahmed, Natarajan and Rao
Video codec with motion compensation, 1981 
Slide 23 
Displacement measurement and its application in 
interframe video coding, Jain and Jain
Bidirectional + variable block size motion 
prediction, circa 1990 
Slide 24 
MPEG-1, LeGall, 1991 Chan et al, 1990
By 1991, the basic codec was complete 
Slide 25 
MPEG-1 encoder, 1991
Video Encoder 
Form 
prediction 
Transform 
+ quantize 
Inverse 
transform 
+ quantize 
Entropy 
encoder 
Current MB 
Prediction MB 
Residual MB 
Decoded 
Residual MB 
Intra 
Inter 
.... 
Current frame or field 
Previously coded 
frames or fields 
+ Coded bitstream 
- 
+ 
+ 
Slide 26
The video coding standards 
Slide 27
Slide 28 
Moving Picture Experts Group 
- MPEG standards 
- Broadcast / entertainment 
Joint Video Team / 
Joint Collaborative Team 
Video Coding Experts Group 
- H.26x standards 
- Communications
Slide 29
Slide 30 
1990 
1993 
1995 
1996 
1999 
2003 
2013 
ITU-T JOINT ISO MPEG 
MPEG-1 
H.263 
H.262/MPEG-2 
H.264/AVC 
H.265/HEVC 
MPEG-4 
H.261 
….
ITU-T H.261 (1990) 
Slide 31 
2-way video telephony 
ISDN lines (p x 64kbps)
ITU-T H.261 (1990) 
Slide 32 
16x16 blocks 8x8 DCT 
I P P P 
Forward 
prediction 
…011011000100… 
Variable 
length coding
MPEG-1 Video (1991-93) 
Slide 33 
Video CD playback
MPEG-1 Video (1991-93) 
Slide 34 
I B B P B B P B B P B B I 
0 1 2 3 4 5 6 7 8 9 10 11 12 
Intra, forward and 
bidirectional prediction 
Half-pixel motion vectors
MPEG-2 Video (1995) 
Slide 35 
Digital TV broadcasting, 
DVD
MPEG-2 Video (1995) 
Slide 36 
16x16 and Interlaced video 
16x8 block sizes
H.263 (1996), MPEG-4 Part 2 (1999) 
Slide 37
H.263 (1996), MPEG-4 Part 2 (1999) 
Slide 38 
16x16, 8x8 
blocks 
Arithmetic 
coding Error 
resilience 
Advanced 
prediction Deblocking 
filter 
Object 
coding Mesh 
coding 
Face 
animation 
Many optional 
coding tools
H.264 / Advanced Video Coding (2003) 
Slide 39
H.264 / Advanced Video Coding (2003) 
Slide 40 
16x16 down 
to 4x4 blocks 
Intra 
prediction 
0 1 2 3 4 5 6 
Multiple reference 
frames 
+ context adaptive coding 
(CABAC / CAVLC), 
¼ pixel motion vectors, 
loop filter….
H.265 / High Efficiency Video Coding (2013) 
Slide 41
H.265 / High Efficiency Video Coding (2013) 
Courtesy of Parabola Research 
Slide 42 
64x64 down 
to 4x4 blocks Parallel processing support 
+ Lots of “H.264 like” optimizations….
HEVC analyzer screenshot 
Courtesy of Parabola Research 
Slide 43
Performance 
Slide 44
Compression performance 
Slide 45 
MPEG-2 
~2x better 
H.264/AVC 
~2x better 
H.265/HEVC
Performance examples 
Slide 46
Slide 47 
MPEG-1 MPEG-2 
H.264 HEVC
Slide 48 
Difference image (MPEG-1) 
Difference image (HEVC)
Patents and patent pools 
Slide 49
Example 
Slide 50 
September 
2001 
Patent application is filed 
September 
2001 
Proposal document is submitted to 
Joint Video Team 
May 2003 H.264 is published, including the 
concept of the application & proposal 
2005 Patent is granted
Slide 51 
Published 
Standard 
Patents 
Implementors 
Patent 
pool
Patents and patent pools: the early 
standards 
Slide 52 
H.261, MPEG-1, H.263 : 
- no formal patent pool 
MPEG-2 : 
- first patent pool, managed by MPEG-LA 
MPEG-4 Part 2 : 
- patent pool, managed by MPEG-LA
Patents and patent pools: H.264 / AVC 
Slide 53 
2001-2003 Royalty Free Baseline intention 
announced by 32 patent holders 
May 2003 First edition of H.264 published 
Late 2003 MPEG-LA and Via Licensing 
announce patent pools. Royalty 
Free Baseline abandoned. 
2004 
onwards 
Via Licensing pool “folds”. MPEG-LA 
pool dominates.
Patents and patent pools: recent 
developments 
Slide 54 
Open source VP8 format: 
MPEG-LA attempts to create patent pool, 
Google and MPEG-LA reach agreement and pool 
is abandoned. 
HEVC: 
Patent pool created by MPEG-LA (2014).
Conclusion 
Slide 55 
Many of the key components of modern video codecs 
originated before 1990. 
In the last 25 years, developments in video coding 
have been strongly influenced by standardisation. 
Performance continues to improve, driven by 
optimization of compression tools.
Useful information sources 
Slide 56 
Low bit rate / Video Coding Experts Group 
archives, 1996-present: 
http://wftp3.itu.int/av-arch/video-site/ 
Joint Video Team archives (H.264 development): 
http://wftp3.itu.int/av-arch/jvt-site/ 
JCT-VC archives (HEVC development): 
http://wftp3.itu.int/av-arch/jctvc-site/ 
Video coding resources: 
http://vcodex.com/
1 of 56

Recommended

Iain Richardson: An Introduction to Video Compression by
Iain Richardson: An Introduction to Video CompressionIain Richardson: An Introduction to Video Compression
Iain Richardson: An Introduction to Video CompressionIain Richardson
8.4K views55 slides
H.264 vs HEVC by
H.264 vs HEVCH.264 vs HEVC
H.264 vs HEVCMarcin Walendowski
18.8K views72 slides
Introduction to HEVC by
Introduction to HEVCIntroduction to HEVC
Introduction to HEVCYoss Cohen
4.2K views23 slides
An Overview of High Efficiency Video Codec HEVC (H.265) by
An Overview of High Efficiency Video Codec HEVC (H.265)An Overview of High Efficiency Video Codec HEVC (H.265)
An Overview of High Efficiency Video Codec HEVC (H.265)Varun Ravi
7.5K views23 slides
Video coding standards ppt by
Video coding standards pptVideo coding standards ppt
Video coding standards pptLokesh Reddy Avula
4K views41 slides
H.264 video standard by
H.264 video standardH.264 video standard
H.264 video standardSajan Sahu
3.3K views16 slides

More Related Content

What's hot

Versatile Video Coding: Compression Tools for UHD and 360° Video by
Versatile Video Coding: Compression Tools for UHD and 360° VideoVersatile Video Coding: Compression Tools for UHD and 360° Video
Versatile Video Coding: Compression Tools for UHD and 360° VideoMathias Wien
2.1K views61 slides
HEVC intra coding by
HEVC intra codingHEVC intra coding
HEVC intra codingManohar Kuse
5K views35 slides
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE... by
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...Mathias Wien
3.2K views245 slides
Introduction to Video Compression Techniques - Anurag Jain by
Introduction to Video Compression Techniques - Anurag JainIntroduction to Video Compression Techniques - Anurag Jain
Introduction to Video Compression Techniques - Anurag JainVideoguy
3.8K views77 slides
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ... by
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...
Versatile Video Coding – Video Compression beyond HEVC: Coding Tools for SDR ...Förderverein Technische Fakultät
5.2K views41 slides
Compression: Images (JPEG) by
Compression: Images (JPEG)Compression: Images (JPEG)
Compression: Images (JPEG)danishrafiq
7.5K views20 slides

What's hot(20)

Versatile Video Coding: Compression Tools for UHD and 360° Video by Mathias Wien
Versatile Video Coding: Compression Tools for UHD and 360° VideoVersatile Video Coding: Compression Tools for UHD and 360° Video
Versatile Video Coding: Compression Tools for UHD and 360° Video
Mathias Wien2.1K views
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE... by Mathias Wien
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...
ICME 2016 - High Efficiency Video Coding - Coding Tools and Specification: HE...
Mathias Wien3.2K views
Introduction to Video Compression Techniques - Anurag Jain by Videoguy
Introduction to Video Compression Techniques - Anurag JainIntroduction to Video Compression Techniques - Anurag Jain
Introduction to Video Compression Techniques - Anurag Jain
Videoguy3.8K views
Compression: Images (JPEG) by danishrafiq
Compression: Images (JPEG)Compression: Images (JPEG)
Compression: Images (JPEG)
danishrafiq7.5K views
VVC tutorial at ICME 2020 together with Benjamin Bross by Mathias Wien
VVC tutorial at ICME 2020 together with Benjamin BrossVVC tutorial at ICME 2020 together with Benjamin Bross
VVC tutorial at ICME 2020 together with Benjamin Bross
Mathias Wien3.6K views
An Introduction to AV1 - The Next-Gen Royalty-Free Codec From the Alliance fo... by Bitmovin Inc
An Introduction to AV1 - The Next-Gen Royalty-Free Codec From the Alliance fo...An Introduction to AV1 - The Next-Gen Royalty-Free Codec From the Alliance fo...
An Introduction to AV1 - The Next-Gen Royalty-Free Codec From the Alliance fo...
Bitmovin Inc3.1K views
AV1: the next generation video codec by Ting-Li Chou
AV1: the next generation video codecAV1: the next generation video codec
AV1: the next generation video codec
Ting-Li Chou1.1K views
Video Compression Techniques by cnssources
Video Compression TechniquesVideo Compression Techniques
Video Compression Techniques
cnssources6.9K views
Compression: Video Compression (MPEG and others) by danishrafiq
Compression: Video Compression (MPEG and others)Compression: Video Compression (MPEG and others)
Compression: Video Compression (MPEG and others)
danishrafiq11.9K views
Video Compression Basics by Sanjiv Malik
Video Compression BasicsVideo Compression Basics
Video Compression Basics
Sanjiv Malik25.2K views
Video compression by DarkNight14
Video compressionVideo compression
Video compression
DarkNight141.3K views
MPEG-1 Part 2 Video Encoding by Christian Kehl
MPEG-1 Part 2 Video EncodingMPEG-1 Part 2 Video Encoding
MPEG-1 Part 2 Video Encoding
Christian Kehl3.8K views
Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf by ssuserc5a4dd
Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdfTutorial High Efficiency Video Coding Coding - Tools and Specification.pdf
Tutorial High Efficiency Video Coding Coding - Tools and Specification.pdf
ssuserc5a4dd80 views
An Introduction to Versatile Video Coding (VVC) for UHD, HDR and 360 Video by Dr. Mohieddin Moradi
An Introduction to  Versatile Video Coding (VVC) for UHD, HDR and 360 VideoAn Introduction to  Versatile Video Coding (VVC) for UHD, HDR and 360 Video
An Introduction to Versatile Video Coding (VVC) for UHD, HDR and 360 Video

Viewers also liked

Configurable Video Coding by
Configurable Video CodingConfigurable Video Coding
Configurable Video CodingIain Richardson
1.9K views32 slides
Book Launch: The H.264 Advanced Video Compression Standard by
Book Launch: The H.264 Advanced Video Compression StandardBook Launch: The H.264 Advanced Video Compression Standard
Book Launch: The H.264 Advanced Video Compression StandardIain Richardson
2K views19 slides
Introduction to H.264 Advanced Video Compression by
Introduction to H.264 Advanced Video CompressionIntroduction to H.264 Advanced Video Compression
Introduction to H.264 Advanced Video CompressionIain Richardson
11.6K views40 slides
Lec-03 Entropy Coding I: Hoffmann & Golomb Codes by
Lec-03 Entropy Coding I: Hoffmann & Golomb CodesLec-03 Entropy Coding I: Hoffmann & Golomb Codes
Lec-03 Entropy Coding I: Hoffmann & Golomb CodesUnited States Air Force Academy
3.4K views10 slides
Getting the most out of H.264 by
Getting the most out of H.264Getting the most out of H.264
Getting the most out of H.264Iain Richardson
2.6K views41 slides
Compressed Video Quality by
Compressed Video QualityCompressed Video Quality
Compressed Video QualityIain Richardson
864 views40 slides

Viewers also liked(6)

Similar to A short history of video coding

Video Compression Technology by
Video Compression TechnologyVideo Compression Technology
Video Compression TechnologyTong Teerayuth
807 views48 slides
Video Coding Standard by
Video Coding StandardVideo Coding Standard
Video Coding StandardVideoguy
2K views47 slides
4 imtc wiegand 131009 by
4 imtc wiegand 1310094 imtc wiegand 131009
4 imtc wiegand 131009IMTC
975 views39 slides
Emerging H.264 Standard: by
Emerging H.264 Standard:Emerging H.264 Standard:
Emerging H.264 Standard:Videoguy
488 views13 slides
Performance Analysis of Various Video Compression Techniques by
Performance Analysis of Various Video Compression TechniquesPerformance Analysis of Various Video Compression Techniques
Performance Analysis of Various Video Compression TechniquesInternational Journal of Science and Research (IJSR)
665 views4 slides
/conferences/spr2004/presentations/eubanks/eubanks_mpeg4.ppt by
/conferences/spr2004/presentations/eubanks/eubanks_mpeg4.ppt/conferences/spr2004/presentations/eubanks/eubanks_mpeg4.ppt
/conferences/spr2004/presentations/eubanks/eubanks_mpeg4.pptVideoguy
246 views11 slides

Similar to A short history of video coding(20)

Video Compression Technology by Tong Teerayuth
Video Compression TechnologyVideo Compression Technology
Video Compression Technology
Tong Teerayuth807 views
Video Coding Standard by Videoguy
Video Coding StandardVideo Coding Standard
Video Coding Standard
Videoguy2K views
4 imtc wiegand 131009 by IMTC
4 imtc wiegand 1310094 imtc wiegand 131009
4 imtc wiegand 131009
IMTC975 views
Emerging H.264 Standard: by Videoguy
Emerging H.264 Standard:Emerging H.264 Standard:
Emerging H.264 Standard:
Videoguy488 views
/conferences/spr2004/presentations/eubanks/eubanks_mpeg4.ppt by Videoguy
/conferences/spr2004/presentations/eubanks/eubanks_mpeg4.ppt/conferences/spr2004/presentations/eubanks/eubanks_mpeg4.ppt
/conferences/spr2004/presentations/eubanks/eubanks_mpeg4.ppt
Videoguy246 views
H263.ppt by Videoguy
H263.pptH263.ppt
H263.ppt
Videoguy3.1K views
Spatial Scalable Video Compression Using H.264 by IOSR Journals
Spatial Scalable Video Compression Using H.264Spatial Scalable Video Compression Using H.264
Spatial Scalable Video Compression Using H.264
IOSR Journals220 views
Emerging H.264 Standard: Overview and TMS320DM642- Based ... by Videoguy
Emerging H.264 Standard: Overview and TMS320DM642- Based ...Emerging H.264 Standard: Overview and TMS320DM642- Based ...
Emerging H.264 Standard: Overview and TMS320DM642- Based ...
Videoguy410 views
H264 video compression explained by cnssources
H264 video compression explainedH264 video compression explained
H264 video compression explained
cnssources5.4K views
h.264 video compression standard. by Videoguy
h.264 video compression standard.h.264 video compression standard.
h.264 video compression standard.
Videoguy457 views

Recently uploaded

Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates by
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesKeynote Talk: Open Source is Not Dead - Charles Schulz - Vates
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesShapeBlue
210 views15 slides
The Power of Heat Decarbonisation Plans in the Built Environment by
The Power of Heat Decarbonisation Plans in the Built EnvironmentThe Power of Heat Decarbonisation Plans in the Built Environment
The Power of Heat Decarbonisation Plans in the Built EnvironmentIES VE
69 views20 slides
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ... by
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...ShapeBlue
85 views10 slides
Ransomware is Knocking your Door_Final.pdf by
Ransomware is Knocking your Door_Final.pdfRansomware is Knocking your Door_Final.pdf
Ransomware is Knocking your Door_Final.pdfSecurity Bootcamp
90 views46 slides
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...James Anderson
156 views32 slides
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue by
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlueCloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlueShapeBlue
93 views15 slides

Recently uploaded(20)

Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates by ShapeBlue
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesKeynote Talk: Open Source is Not Dead - Charles Schulz - Vates
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates
ShapeBlue210 views
The Power of Heat Decarbonisation Plans in the Built Environment by IES VE
The Power of Heat Decarbonisation Plans in the Built EnvironmentThe Power of Heat Decarbonisation Plans in the Built Environment
The Power of Heat Decarbonisation Plans in the Built Environment
IES VE69 views
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ... by ShapeBlue
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
ShapeBlue85 views
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N... by James Anderson
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
GDG Cloud Southlake 28 Brad Taylor and Shawn Augenstein Old Problems in the N...
James Anderson156 views
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue by ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlueCloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
ShapeBlue93 views
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R... by ShapeBlue
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
Setting Up Your First CloudStack Environment with Beginners Challenges - MD R...
ShapeBlue132 views
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by Bernd Ruecker
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
Bernd Ruecker50 views
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or... by ShapeBlue
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
Zero to Cloud Hero: Crafting a Private Cloud from Scratch with XCP-ng, Xen Or...
ShapeBlue158 views
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha... by ShapeBlue
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
ShapeBlue138 views
The Role of Patterns in the Era of Large Language Models by Yunyao Li
The Role of Patterns in the Era of Large Language ModelsThe Role of Patterns in the Era of Large Language Models
The Role of Patterns in the Era of Large Language Models
Yunyao Li80 views
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit... by ShapeBlue
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
Transitioning from VMware vCloud to Apache CloudStack: A Path to Profitabilit...
ShapeBlue117 views
Business Analyst Series 2023 - Week 4 Session 7 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 7Business Analyst Series 2023 -  Week 4 Session 7
Business Analyst Series 2023 - Week 4 Session 7
DianaGray10126 views
Data Integrity for Banking and Financial Services by Precisely
Data Integrity for Banking and Financial ServicesData Integrity for Banking and Financial Services
Data Integrity for Banking and Financial Services
Precisely78 views
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ... by ShapeBlue
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
How to Re-use Old Hardware with CloudStack. Saving Money and the Environment ...
ShapeBlue123 views
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by ShapeBlue
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue181 views
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ... by ShapeBlue
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
Backup and Disaster Recovery with CloudStack and StorPool - Workshop - Venko ...
ShapeBlue144 views

A short history of video coding

  • 1. A History of Video Coding Invited Talk, United States Patent and Trademark Office Iain Richardson December 2014 Slide 1 vcodex.com
  • 2. What this talk is about Slide 2
  • 4. Video coding : the basics Slide 4
  • 5. A video codec (1) Slide 5 Video Source Decompress (Decode) Compress (Encode) Video Display Coded video enCOder / DECoder
  • 6. A video codec (2) Slide 6 video source partition transform entropy encode predict (add) inverse transform entropy decode compressed video video output VIDEO ENCODER VIDEO DECODER predict (subtract) reconstruct
  • 7. 1. Partitioning Slide 7 frame or picture slice or tile macroblock (MB) or coding tree unit (CTU) sequence
  • 8. 2. Prediction Slide 8 Original Frame Prediction Frame Residual
  • 9. 3. Transform + Quantize Block of samples After transform After quantization Slide 9
  • 10. 4. Entropy coding Slide 10 1 1 0 1 1 1 1 1 0 0 0 1 1 coded components (syntax elements) bitstream
  • 11. source frames ........ ........ coded bitstream decoded frames ENCODE DECODE Slide 11
  • 12. Video Encoder Form prediction Transform + quantize Inverse transform + quantize Entropy encoder Current MB Prediction MB Residual MB Decoded Residual MB Intra Inter .... Current frame or field Previously coded frames or fields + Coded bitstream - + + Slide 12
  • 13. Video Decoder Coded bitstream .... Entropy decoder Inverse transform + quantize Decoded Residual MB Current decoded frame or field Inter Previously decoded frames or fields Prediction MB Form prediction Intra Decoded MB + + Slide 13
  • 19. Slide 19 1952 Variable length coding 1972 Frame differencing 1974 Discrete Cosine Transform 1981 Motion compensated video codec 1990 Bidirectional prediction Variable block size motion compensation
  • 20. Huffman variable length coding, 1952 Slide 20 D A Huffman, Minimum redundancy codes
  • 21. Frame differencing, 1972 Slide 21 US 3,679,821, Schroeder, Bell Labs
  • 22. Discrete Cosine Transform, 1974 Slide 22 The Discrete Cosine Transform Ahmed, Natarajan and Rao
  • 23. Video codec with motion compensation, 1981 Slide 23 Displacement measurement and its application in interframe video coding, Jain and Jain
  • 24. Bidirectional + variable block size motion prediction, circa 1990 Slide 24 MPEG-1, LeGall, 1991 Chan et al, 1990
  • 25. By 1991, the basic codec was complete Slide 25 MPEG-1 encoder, 1991
  • 26. Video Encoder Form prediction Transform + quantize Inverse transform + quantize Entropy encoder Current MB Prediction MB Residual MB Decoded Residual MB Intra Inter .... Current frame or field Previously coded frames or fields + Coded bitstream - + + Slide 26
  • 27. The video coding standards Slide 27
  • 28. Slide 28 Moving Picture Experts Group - MPEG standards - Broadcast / entertainment Joint Video Team / Joint Collaborative Team Video Coding Experts Group - H.26x standards - Communications
  • 30. Slide 30 1990 1993 1995 1996 1999 2003 2013 ITU-T JOINT ISO MPEG MPEG-1 H.263 H.262/MPEG-2 H.264/AVC H.265/HEVC MPEG-4 H.261 ….
  • 31. ITU-T H.261 (1990) Slide 31 2-way video telephony ISDN lines (p x 64kbps)
  • 32. ITU-T H.261 (1990) Slide 32 16x16 blocks 8x8 DCT I P P P Forward prediction …011011000100… Variable length coding
  • 33. MPEG-1 Video (1991-93) Slide 33 Video CD playback
  • 34. MPEG-1 Video (1991-93) Slide 34 I B B P B B P B B P B B I 0 1 2 3 4 5 6 7 8 9 10 11 12 Intra, forward and bidirectional prediction Half-pixel motion vectors
  • 35. MPEG-2 Video (1995) Slide 35 Digital TV broadcasting, DVD
  • 36. MPEG-2 Video (1995) Slide 36 16x16 and Interlaced video 16x8 block sizes
  • 37. H.263 (1996), MPEG-4 Part 2 (1999) Slide 37
  • 38. H.263 (1996), MPEG-4 Part 2 (1999) Slide 38 16x16, 8x8 blocks Arithmetic coding Error resilience Advanced prediction Deblocking filter Object coding Mesh coding Face animation Many optional coding tools
  • 39. H.264 / Advanced Video Coding (2003) Slide 39
  • 40. H.264 / Advanced Video Coding (2003) Slide 40 16x16 down to 4x4 blocks Intra prediction 0 1 2 3 4 5 6 Multiple reference frames + context adaptive coding (CABAC / CAVLC), ¼ pixel motion vectors, loop filter….
  • 41. H.265 / High Efficiency Video Coding (2013) Slide 41
  • 42. H.265 / High Efficiency Video Coding (2013) Courtesy of Parabola Research Slide 42 64x64 down to 4x4 blocks Parallel processing support + Lots of “H.264 like” optimizations….
  • 43. HEVC analyzer screenshot Courtesy of Parabola Research Slide 43
  • 45. Compression performance Slide 45 MPEG-2 ~2x better H.264/AVC ~2x better H.265/HEVC
  • 47. Slide 47 MPEG-1 MPEG-2 H.264 HEVC
  • 48. Slide 48 Difference image (MPEG-1) Difference image (HEVC)
  • 49. Patents and patent pools Slide 49
  • 50. Example Slide 50 September 2001 Patent application is filed September 2001 Proposal document is submitted to Joint Video Team May 2003 H.264 is published, including the concept of the application & proposal 2005 Patent is granted
  • 51. Slide 51 Published Standard Patents Implementors Patent pool
  • 52. Patents and patent pools: the early standards Slide 52 H.261, MPEG-1, H.263 : - no formal patent pool MPEG-2 : - first patent pool, managed by MPEG-LA MPEG-4 Part 2 : - patent pool, managed by MPEG-LA
  • 53. Patents and patent pools: H.264 / AVC Slide 53 2001-2003 Royalty Free Baseline intention announced by 32 patent holders May 2003 First edition of H.264 published Late 2003 MPEG-LA and Via Licensing announce patent pools. Royalty Free Baseline abandoned. 2004 onwards Via Licensing pool “folds”. MPEG-LA pool dominates.
  • 54. Patents and patent pools: recent developments Slide 54 Open source VP8 format: MPEG-LA attempts to create patent pool, Google and MPEG-LA reach agreement and pool is abandoned. HEVC: Patent pool created by MPEG-LA (2014).
  • 55. Conclusion Slide 55 Many of the key components of modern video codecs originated before 1990. In the last 25 years, developments in video coding have been strongly influenced by standardisation. Performance continues to improve, driven by optimization of compression tools.
  • 56. Useful information sources Slide 56 Low bit rate / Video Coding Experts Group archives, 1996-present: http://wftp3.itu.int/av-arch/video-site/ Joint Video Team archives (H.264 development): http://wftp3.itu.int/av-arch/jvt-site/ JCT-VC archives (HEVC development): http://wftp3.itu.int/av-arch/jctvc-site/ Video coding resources: http://vcodex.com/

Editor's Notes

  1. On the left is a typical block of pixel samples, 4x4 in this case. In any region of the picture, we’re going to have a range of brightnesses or levels. Each one of those numbers is important to the video image and each one takes up space in the video file. The idea of a video coding transform is to convert a block of samples into a frequency representation, such as the example in the middle. In a video encoder, we follow the transform with quantization, which removes small or insignificant values. So instead of the 16 image samples on the left, we have a transformed and quantized block on the right. Most of the numbers are zero, which makes this transformed and quantized block very easy to store in a compressed form. And if we do this in the right way, it’s possible to reverse the process and get a decoded image block that looks almost the same as the original.