A gentle(*) introduction to video encoding (*)for some definition of “gentle” Mark Pilgrim January 8, 2009

Video is everywhere Handhelds iPod, cell phone, PSP, Archos Internet YouTube, Daily Motion, Metacafe Television Broadcast, cable, DVD, Blu-Ray “ It’s a bird…It’s a plane…It’s…Beauregard!” PCs QuickTime, Windows Media Player, VLC

Video is everywhere But how does it get there?

Encoding process Source material (video + audio) Raw video Compressed video Demuxing separates streams Encoding compresses streams Single video file Muxing combines streams again Raw audio Compressed audio

Encoding process Source material (video + audio) Raw video Compressed video Single video file Raw audio Compressed audio Time-coded captions Hey, don’t forget about accessibility! Transcription

The Zen of video encoding

Elements of a video The video container holds it all together 4 The caption format defines when captions are displayed 3 The audio codec defines how waveforms are compressed 2 The video codec defines how the pixels are compressed 1

How to choose the right formats Pick a video container that works on your target platform 4 Pick a caption format that works on your target platform 3 Pick a audio codec that works on your target platform 2 Pick a video codec that works on your target platform 1

Constraints CPU capacity for encoding CPU capacity for decoding and playing in real time Acceptable delay between recording and delivery Acceptable quality degradation Screen dimensions Your bandwidth My bandwidth Hard limits on storage size (for physical media) Video codec compatibility Audio codec compatibility Patent licensing costs And on and on…

CPU capacity for encoding

CPU capacity for decoding and playing in real time

Acceptable delay between recording and delivery

Acceptable quality degradation

Screen dimensions

Your bandwidth

My bandwidth

Hard limits on storage size (for physical media)

Video codec compatibility

Audio codec compatibility

Patent licensing costs

And on and on…

The Zen of video encoding “ There is no right or wrong; there is only what works and what doesn’t.”

Video codecs The wool pulled over your eyes to blind you from the truth

What is a video codec? “ A video codec defines the compression algorithm used to store the array of pixels that are eventually displayed on the screen.” … and they cheat. A lot. And hope you won’t notice.

Acronym overload: video codecs Dirac Theora VC-1 WMV H.264 MPEG-4 ASP MPEG-2 MPEG-1 WTF? “ I could tell by the pixels…”

Common video codecs (*) As far as we know Source: Wikipedia No YouTube, Google Video MPEG WG H.263 HDTV broadcast Free Software hippies Blu-Ray Microsoft world domination YouTube HD, iTunes, Blu-Ray Movie pirates DVD VCD Popular usage Yes* On2, Xiph.org Theora No Microsoft VC-1 Yes* BBC Dirac No Microsoft WMV No MPEG WG MPEG-2 No MPEG WG MPEG-1 No MPEG WG H.264 No MPEG WG MPEG-4 ASP Patent-free? Inventor Codec name

Video encoders Source: DigitalContentProducer.com, doom9.org, Wikipedia Adobe Premiere ffmpeg H.263 none none Windows Media Encoder, Compressor, Squeeze Windows Media Encoder, Expression Encoder Adobe Media Encoder, Apple Compressor, On2 Flix Pro, Sorenson Squeeze, Telestream DivX, 3ivX Compressor, Sonic, TMPGEnc QuickTime Non-OSS tools oggenc, Thusnelda Theora none VC-1 dirac-research, Schroedinger Dirac none WMV ffmpeg, mpeg2enc MPEG-2 ffmpeg, MP1E MPEG-1 x264 H.264 ffmpeg, Xvid MPEG-4 ASP OSS tools Codec name

Audio codecs The hand is quicker than the ear

What is an audio codec? “ An audio codec defines the compression algorithm used to store the waveforms that are eventually played through your speakers.” … and they cheat, too.

Acronym overload: audio codecs DTS AC3 Vorbis WMA AAC MP3 MP2 AMR Seriously, WTF? “ Can you hear me now?”

Common audio codecs (*) As far as we know Source: Wikipedia No iPod, iTunes Store videos MPEG WG AAC Blu-Ray DVD Free Software hippies Microsoft world domination Music VCD, DVD, digital TV Ringtones, VoIP Popular usage No DTS Inc. DTS No Dolby AC3 Yes* Xiph.org Vorbis No MPEG WG MP2 No 3GPP AMR No Microsoft WMA No MPEG WG MP3 Patent-free? Owner Codec name

Audio encoders Sources: HydrogenAudio.org, Wikipedia iTunes, Nero FAAC AAC DTS, Inc. Dolby none Windows Media Encoder iTunes Audio Transcoder, Easy CD-DA Extractor 3GPP reference encoder Non-OSS tools none DTS ffmpeg, Aften AC3 ffmpeg, aoTuV Vorbis TwoLAME MP2 ffmpeg, RetroCode AMR ffmpeg WMA LAME MP3 OSS tools Codec name

Captioning formats Just shoot me

What is a captioning format? “ A captioning format defines when, where, and how captions for the hearing impaired are displayed on screen while a video is playing.” … Some formats are reused for subtitles, translations, & commentary.

Acronym overload: caption formats DVD MP4TT SMIL SAMI ASS (really) SRT Teletext Line 21 Baby Jesus cries “ No, I can’t hear you now. What else ya got?”

Overview of caption formats Source: Wikipedia DVD Stored as images DVD-Forum DVD Analog TV (Europe) custom BBC et. al. Teletext Analog TV (US + Canada) pseudo-ASCII Electronic Industries Alliance Line 21 UTF-8 (multiple) (multiple) UTF-8 windows-1252 Encoding Windows Media Player MSDN page SAMI MP4 files, iTunes Store QuickTime (SMIL 1.0 only) AVI files, “fansubbing” AVI files, “fansubbing” Popular usage ISO 14966-17 MPEG-4 Timed Text A doom9.org forum post from 2004 SubRip W3C Recommendation SMIL Some Word document floating around the net Advanced SubStation Alpha Spec Caption format

Container formats … those should be simple, right?

What is a video container? “ A container format defines how video tracks, audio tracks, captions, and metadata are stored together in a single file.” … like a ZIP archive

Video metadata Technical metadata How many video tracks? How many audio tracks? How many caption tracks? Where is everything within the file? (track IDs, byte ranges) Human-readable metadata Video title Genre Release date MPAA rating Cover art

Technical metadata

How many video tracks?

How many audio tracks?

How many caption tracks?

Where is everything within the file? (track IDs, byte ranges)

Human-readable metadata

Video title

Genre

Release date

MPAA rating

Cover art

Metadata matters Without human-readable metadata, the iTunes Store would look like this: Source: list of top movies from RottenTomatoes.com $9.99 The.Chronicles.Of.Narnia.Prince.Caspian.2008.SciFi.Fantasy.720x480.02h29m.m4v $9.99 Smart.People.2008.Comedy.720x480.01h35m.m4v $9.99 Iron.Man.2008.Action.Adventure.720x480.02h42m.m4v $9.99 Kung.Fu.Panda.2008.Animation.Comedy.720x480.01h50m.m4v $9.99 WALL-E.2008.Animation.Action.Adventure.720x480.01h38m.m4v $9.99 Step.Brothers.2008.Comedy.720x480.01h45m.m4v $9.99 Wanted.2008.Action.Adventure.720x480.01h50m.m4v $9.99 Hancock.2008.Action.Adventure.720x480.01h42m.m4v

Acronym overload: container formats EVO VOB ASF OGG FLV MP4 MKV AVI LOL One ring to rule them all, and in the darkness bind them.

Overview of container formats (*) Seriously, some guys in Russia. They contributed it to the public domain, so technically you own it. Source: Wikipedia ffmpeg, MP4Box, mp4creator MPEG-LA .MP4, .M4V MPEG-4 ffmpeg Apple .MOV QuickTime ffmpeg, oggmux Xiph.org .OGG Ogg ffmpeg (in progress) Microsoft .ASF, .WMV Advanced Systems Format ffmpeg, AVIMux-GUI Microsoft .AVI Audio/Video Interchange ffmpeg, mkvtoolnix Some guys in Russia* .MKV Matroska ffmpeg Adobe .FLV Flash Video OSS tools Owner Extension Container Name

Putting it all together Where’s the kaboom? There was supposed to be an Earth-shattering kaboom!

Common combinations Source: Wikipedia SAMI WMA WMV, VC-1 ASF Windows Kate Vorbis Theora OGG Linux QTTextTrack AAC H.264 MOV Mac/QuickTime SRT, ASS MP3 MPEG-4 ASP AVI DVD pirates Blu-Ray subtitles (SUP) AC3, DTS H.264, VC-1 EVOB Blu-Ray SRT, ASS AC3, DTS H.264 MKV Blu-Ray pirates MP4 MP4 MP4 FLV VOB Container Custom, if any MP3 H.263 Flash 8 Line 21, DVD subtitles AC3 MPEG-2 DVD AAC+AC3 AAC AAC Audio MPEG-4 Timed Text H.264 iPod/iPhone MPEG-4 Timed Text Flash Timed Text Captions H.264 Flash 9+ H.264 AppleTV Video Platform/Audience

Thank You! Q&A