Remember all those non-FAANG entities who deployed HEVC and VP9? Yeah, me neither. AV1? Eh. VVC? lol. Are newer, more complex codecs, even useful for anyone but megcorps? Have we reache The End, like we have for audio codecs, where only telcos are interested in lowering bandwidth more, to save money? Have we reached the point where the compute/bandwidth costs will never make sense again small/medium players, who are too big for hardware encoding, too small to reap the benefits of software encoding, and too small to invest in bespoke hardware? Similarily, does anyone under the age of 50 work on codecs anymore? Have we made the barrier to entry so high that you need to spend 10 years banging your head against esoteric papers to understand everything in VVC? Are we all doomed to glue things together? Are video codecs dead? This is an open question. Let's discuss.

1. Are Video Codecs... Done?
derekb@vimeo.com / derek@videolan.org
@daemon404
Derek Buitenhuis
13 October 2022
San Francisco, USA

2. Who’s this?
1
13 October 2022
• Senior Principal Video Engineer @ Vimeo
• Open source developer (FFmpeg, FFMS2, rav1e, obuparse, etc.)
• VideoLAN non-profit board member
• Involved in codecs, containers, DSP, etc. since 2004.
• Professional Twitter Sh*tposter

3. Context / Biases
2
13 October 2022
• Vimeo is:
• Large but not FAANG-sized. Scale, but not FAANG-level cash.
• User Generated Content (Long Tail, Large Corpus, Bad Input, etc.)
• VOD, Live, OTT, etc.
• Primarily familiar with western companies.
• I am cognizant of the use of HEVC, etc. in e.g. China – but that’s not a sector I have much
connection to, or who publish cost numbers, at least in English.
• The patent situation there can be “interesting”.
• I would love to learn more and fill in gaps in my understanding about use in these
markets.

4. 3
13 October 2022
Part One:
The Practicalities of Deployment

5. First: A Confession
4
13 October 2022
• 10 years ago, I was a HEVC True Believer.
• The H.264 good times were in full swing.
• Why wouldn’t the next MPEG codec be just as great of a success as H.264?
• Surely we’d have people chomping at the bit to write “the x264 of HEVC”.
• MPEG-LA definitely wouldn’t do something silly like mess with royalties.
• VP8 (and WebM) looked to be dead on arrival.
• Spec? Who needs that? Not Google.
• Only danger was Chrome announced in 2011 that they’d remove support for H.264.
• I think they’re getting close, probably just needs a final code review.
• Boy, was I wrong.

6. It’s 2022: Why have people actually deployed HEVC?
5
13 October 2022
• Specifically non-megacorps.
• A few main reasons:
• Apple and Dolby.
• HDR:
• Many devices require HEVC for any HDR support at all, even types which are
not codec-specific (i.e. not Dolby Vision).
• Not all of us can be YouTube and get special codec support for HDR put in our
favorite OSes.
• Some cheaper devices decided not to get better H.264 chipsets, so simply required HEVC
for >1080p support (e.g. Roku).
• They worked on HEVC and their employer has patents on it.
• Notably not included here: lower bandwidth and storage costs.

7. Anecdotally: It is not saving money for those who have.
6
13 October 2022
• In addition to HEVC being just generally more intensive to encode, due to the reasons on the
previous slide, this is also when many of us made the jump to 10-bit, making it even more costly.
• We also need to remember: We are not extremely spoiled with H.264 encoding speeds and
costs 10-15 years later, and many finance departments now take this for granted – slower codecs
are a hard sell.
• While I can’t share exact numbers, I can say we definitely aren’t saving money by using HEVC.
• It’s treated as a feature, not a cost savings thing.
• Still mainly only Apple devices are consuming it.
• Slower encodes, even chunked, are worse at using spot instances.
• Even ingesting HEVC (and HDR – hooray tone mapping) is slower!
• When the iPhone started doing HDR HEVC by default, us and industry colleagues noted we
could really see the cost increases tick up starting right after launch!

8. VP{8,9}: Eh…
7
13 October 2022
• As before: VP8 is irrelevant to most of us, aside from GIF hosts.
• VP9 may have not had the patent issues of HEVC (more on that later), but saw, from my
observation, even less adoption.
• Not just a new codec, also required whole new tooling an infra for WebM.
• ‘WebM DASH’ was a gross hack that abused Matroska cues.
• libvpx was terrible… especially at rate control. Now it is merely bad.
• I am convinced Google just doesn’t get what non-Google orgs need to drive creation-side adoption.
• Either that or they don’t care, as long as it saves YouTube money.
• Was mostly only embraced by big corps like Google, and much later, Netflix.
• Netflix waited for MP4 support, I think?
• Google had to make its own hardware to make it truly economical.

9. So who do / don’t slower, better codecs work for?
8
13 October 2022
• General: Patient liability / risk tolerance is too high for HEVC for SMB.
• Small corpus, high consumption:
• Netflix
• High bandwidth websites like fashion, where video is not the main focus.
• UGC – only at megacorp scale
• The long tail only really economical with Google-level “we made out own hardware” investment.
• Off the shelf hardware encoders aren’t economical or scalable past SMB-level.
• Live:
• Unclear.
• Twitch trialed it but not widely deployed.
• Most YouTube livestreams are still H.264.
• Can be done, but either at a cost, or without sufficient bandwidth savings vs H.264.

10. The Future
9
13 October 2022
• AV1:
• YouTube. Duh.
• Hot take: Real bandwidth returns on cost are diminishing vs compute, and much like audio codecs: only
telcos the big bois care about the amount shaved off.
• Real use case – Cisco:
• Replaced H.264, not HEVC or VP9.
• Very small subset of tools.
• Could be a way forward for next gen codecs? No one seems to be targeting this.
• VVC:
• Hotter take: Nobody cares about VVC except people involved in creating it (MPEG companies).
• Patents. “Let’s just wait and see approach”.
• I’ve heard rumors some Indian vendors have deployed it to get The Smallest Files.
• I would love to know the cost and patent situation.

11. The Result
10
13 October 2022
H.264 is the JPEG of video.

12. 11
13 October 2022
Part Two:
Development and Deep Understanding

13. Back in my day…
12
13 October 2022
• When I started in video, MPEG-2, MPEG-4 ASP, and H.264 were all relevant.
• A sole person could learn every bit of these, and implement them themselves,
reasonably quickly.
• Reasonably-ish well documented. Sort of. Kind of… Well. OK.
• You could apply your newly gained knowledge to current problems.
• Tools in these codecs were less intertwined.
• Does anyone do this nowadays (besides Ronald?)
• All literature online (DIY, DSP intros, etc.) hasn’t really moved past these, even
in 2022.

14. Current Year Argument
13
13 October 2022
• Houses built on top of other houses built on top of other houses. Turtles all the way down.
• Only the Old Gods (Yuriy, Gary, etc.) remember why things were done.
• At 32, I am considered “young” in the field.
• MPEG: In theory public, but good luck finding the confusingly named ZIP file with the right doc.
• AOM: Alliance for non-open documents.
• Not only that, but the original “how did we get to X” stuff was never well documented to begin with.
• KLT->DCT? Zigzag? YCbCr (luma illuminants, etc). 33 degrees YIQ rotation? Hope you like
scanned documents.
• This sort of deeper understanding is important – if we don’t understand why then how can we
effectively apply it?
• Only going to get worse with AI.
• The coding tools are too numerous to feasibly be implemented and understood by one person
in a sane amount of time.

15. Current Year Argument + 1
14
13 October 2022
• Aside: Following WGs, it’s all very boring and iterative
• More angles, bigger / different shaped / better structured “blocks”, non-binary arithmetic coding.
• Have we reached the limits of Novel New Tools on hybrid block-based video codecs?
• This doesn’t even touch on things like RDO, quantization strategies (deadzones, trellis), etc.

16. Result
15
13 October 2022
• MASSIVE barrier to entry, let alone a deep understanding.
• A lot of mediocre multimedia engineers due to not having a background on why/how.
• A lot of repeating things from the 90s/80s/70s.
• Multi-symbol arithmetic coding… also known as “normal arithmetic coding”.
• Wavelets. Again. Still.
• A lot of smart people who are specialized in e.g. transforms, or color spaces, etc. but lack the full
picture view on how tools interact with other tools.
• None of this is taught well in uni or literature – we all learned it by smashing our heads on desks for
10 years. What is taught as “DSP” is only foundational, and poorly taught at that. Lecturers often lack
deep understanding.

17. 16
13 October 2022
So how do we fix this?

18. Economics
17
13 October 2022
• I don’t actually have a solution. If I said I did, I would be a snake oil salesman.
• We may have to make peace with less profits (omg) in order to invest in a better user experience.
• AI in encoders choosing tools? They’re all currently slower aside from a few. Jury’s out.
• Specialized video SIMD instructions (more scalable than ASICs – e.g. one per CPU QuickSync).
• YOU tell ME! I want to be wrong and dumb.
• Please don’t include a sales pitch.

19. Understanding / New Blood
18
13 October 2022
• I would do unspeakable things for a large tome of All Historical Video Info written by the Old Gods.
• Current literature is fragmented, scanned from 1970s papers, lacking, missing, or undiscoverable.
• Better and deeper uni education? Seems unlikely.
• If codecs “done”, does it matter? Will codecs become the new COBOL?
• Hey at least I can make a bomb consulting when I’m 70.
• We haven’t even solved the COBOL problem, though.

20. 19
13 October 2022
I want to hear from you after this talk!