Moving Pictures Implementing Video on Flickr Cal Henderson

Hello

Hello

Flickr Large scale kitten sharing website Started 2003, launched 2004 5 years old this december Almost 3 billion photos

Large scale kitten sharing website

Started 2003, launched 2004

5 years old this december

Almost 3 billion photos

Enter: Video Video was added this year Launched April 2008 Many hundreds of thousands of videos uploaded Many millions of playbacks

Video was added this year

Launched April 2008

Many hundreds of thousands of videos uploaded

Many millions of playbacks

“ Video? That’s just like photos!” -Me, before Flickr Video

“ Video? That’s just like photos!”

-Me, before Flickr Video

12 4 Steps 4 main tasks Uploading Transcoding Storage Serving & Playback

4 main tasks

Uploading

Transcoding

Storage

Serving & Playback

1. Upload

1.

Upload

Simple upload Web forms Just like any other file <form action=&quot;/uploadify/&quot; method=&quot;post&quot; enctype=&quot;multipart/form-data&quot;> <input type=&quot;file&quot; name=&quot;fred&quot; /> <input type=&quot;submit&quot; value=&quot;Go!&quot; /> </form> But files are large / huge

Web forms

Just like any other file

<form action=&quot;/uploadify/&quot; method=&quot;post&quot; enctype=&quot;multipart/form-data&quot;>

<input type=&quot;file&quot; name=&quot;fred&quot; /> <input type=&quot;submit&quot; value=&quot;Go!&quot; />

</form>

But files are large / huge

Issues Two components for uploading: Sending from the client Receiving on the server Both of these present problems

Two components for uploading:

Sending from the client

Receiving on the server

Both of these present problems

Sending from the client Multiple options Simple form Flash Desktop app

Multiple options

Simple form

Flash

Desktop app

Simple Forms Pros Very easy to implement Works on every browser out of the box Cons Upload progress is harder ‘Slow’ Select a single file at once

Pros

Very easy to implement

Works on every browser out of the box

Cons

Upload progress is harder

‘Slow’

Select a single file at once

Flash Pros Upload progress is easy ‘Fast’ Multi select of files Cons Harder to implement Flash isn’t quite ubiquitous

Pros

Upload progress is easy

‘Fast’

Multi select of files

Cons

Harder to implement

Flash isn’t quite ubiquitous

Desktop App Pros Upload progress is easy ‘Very fast’ Multi select of files Drag and drop Cons Hard to develop Hard to deploy (relative to the web)

Pros

Upload progress is easy

‘Very fast’

Multi select of files

Drag and drop

Cons

Hard to develop

Hard to deploy (relative to the web)

Making Progress Upload progress Not impossible with plain forms Just need to be able to query the upload progress via AJAX Multiple machines The VIP issue Enter Perlbal

Upload progress

Not impossible with plain forms

Just need to be able to query the upload progress via AJAX

Multiple machines

The VIP issue

Enter Perlbal

Making Progress Browser Web 1 Web 2 1. Browser starts upload 2. Web server broadcasts progress via UDP 3. Browser queries progress via AJX call Load balancer

Receiving on the server File uploads are slow Much slower than serving pages Apache processes are heavy Waste of resources Use a poll based server (like jetty)

File uploads are slow

Much slower than serving pages

Apache processes are heavy

Waste of resources

Use a poll based server (like jetty)

Receiving on the server Or, use a buffering layer Perlbal is great for this Or a lightweight Apache E.g. w/ mod_proxy Browser Buffer Server Slow Fast

Or, use a buffering layer

Perlbal is great for this

Or a lightweight Apache

E.g. w/ mod_proxy

But wait It’s not just the first step that’s slow Moving files around between servers is slow Do it out of band Asynchronous jobs are in order anyway Do it!

It’s not just the first step that’s slow

Moving files around between servers is slow

Do it out of band

Asynchronous jobs are in order anyway

Do it!

2. Transcode

2.

Transcode

Transcode? Why transcode at all? Input comes from many sources Point & shoots DV Cams Mobile devices Video editing software All in different formats

Why transcode at all?

Input comes from many sources

Point & shoots

DV Cams

Mobile devices

Video editing software

All in different formats

So many formats But very few of these formats can be played back cross platform Without special software or hardware Formats are designed to do one thing well They don’t always manage even that Transcoding puts all videos on an equal footing

But very few of these formats can be played back cross platform

Without special software or hardware

Formats are designed to do one thing well

They don’t always manage even that

Transcoding puts all videos on an equal footing

Video file basics Most file types are really just containers MOV, FLV, AVI The data inside can be in multiple formats We call these codecs (encoder + decoder) Files contains multiple ‘streams’ Both audio and video

Most file types are really just containers

MOV, FLV, AVI

The data inside can be in multiple formats

We call these codecs (encoder + decoder)

Files contains multiple ‘streams’

Both audio and video

Interleave Audio and video are often interleaved Hence AVI A video file looks like this: Headers Video chunk Audio chunk Video chunk Audio chunk Etc

Audio and video are often interleaved

Hence AVI

A video file looks like this:

Headers

Video chunk

Audio chunk

Video chunk

Audio chunk

Etc

Compress Raw video files are huge A bitmap for every frame Rarely used, even in post production At 30 fps, that gets crazy pretty quickly We don’t need to store every frame Static backgrounds don’t change (much) between frames

Raw video files are huge

A bitmap for every frame

Rarely used, even in post production

At 30 fps, that gets crazy pretty quickly

We don’t need to store every frame

Static backgrounds don’t change (much) between frames

Compresssss Intraframe Treat each frame as a picture Compress it (just like JPEG) DCT (Discrete Cosine Transform) Interframe Store the differences between frames Treat the pixels as a 3D array to be compressed

Intraframe

Treat each frame as a picture

Compress it (just like JPEG)

DCT (Discrete Cosine Transform)

Interframe

Store the differences between frames

Treat the pixels as a 3D array to be compressed

The IPB Three frame types: I, P & B Intra coded pictures A full raw frame Predicted pictures Based on a single reference frame Bi-predictive pictures Based on two or more reference frames

Three frame types: I, P & B

Intra coded pictures

A full raw frame

Predicted pictures

Based on a single reference frame

Bi-predictive pictures

Based on two or more reference frames

IPBIPBIPBIPB Reference frames may be I, P or B P & B frames may contain a mix of image data and motion vector displacements I frames require the most bits Then P frames Then B frames

Reference frames may be I, P or B

P & B frames may contain a mix of image data and motion vector displacements

I frames require the most bits

Then P frames

Then B frames

Bad terminology We should really say picture (Not frame) Because of interlacing Really, we encode fields not frames Picture is the general term And H.264 contains ‘slices’ Sub-regions of the field ‘ Macroblocks’ & ‘Artifacts’

We should really say picture

(Not frame)

Because of interlacing

Really, we encode fields not frames

Picture is the general term

And H.264 contains ‘slices’

Sub-regions of the field

‘ Macroblocks’ & ‘Artifacts’

I-Frames Also called Key Frames Allow easy random seeking Twice a second for Digital TV & DVDs More widely spaced online

Also called Key Frames

Allow easy random seeking

Twice a second for Digital TV & DVDs

More widely spaced online

Seekable

Seekable

Oh, and audio too We can worry less about this Older problem, well solved MP3 is pretty good Who cares how it works? Syncing is the only issue Presentation Time Stamps (PTS) and Decode Time Stamps (DTS) in MPEG-2

We can worry less about this

Older problem, well solved

MP3 is pretty good

Who cares how it works?

Syncing is the only issue

Presentation Time Stamps (PTS) and Decode Time Stamps (DTS) in MPEG-2

Flash! Woah-oh! The big question: Flash?

The big question:

Flash?

Non-flash sites QuickTime Windows Media This is gradually disappearing Flash player is ubiquitous Compression is good enough Interactive too But no 3D/VR as with QuickTime :(

QuickTime

Windows Media

This is gradually disappearing

Flash player is ubiquitous

Compression is good enough

Interactive too

But no 3D/VR as with QuickTime :(

The Flash Player Flash Player 6 March 2002 Video: Sorenson Spark (H.263) Audio: MP3 Or ADPCM / Uncompressed Or Nellymoser Asao

Flash Player 6

March 2002

Video: Sorenson Spark (H.263)

Audio: MP3

Or ADPCM / Uncompressed

Or Nellymoser Asao

Second Generation Flash Player 7 August 2005 Video: On2 TrueMotion VP6 Audio: MP3

Flash Player 7

August 2005

Video: On2 TrueMotion VP6

Audio: MP3

The hot shit Flash Player 9 (update 3) December 2007 Video: H.264 (MPEG-4 Part 10) w/ container formats from MPEG-4 Part 14 Audio: AAC (MPEG-4 Part 3) Plus 3GPP Timed Text (MPEG-4 Part 17)

Flash Player 9 (update 3)

December 2007

Video: H.264 (MPEG-4 Part 10)

w/ container formats from MPEG-4 Part 14

Audio: AAC (MPEG-4 Part 3)

Plus 3GPP Timed Text (MPEG-4 Part 17)

TrueMotion VP6 Proprietary Reasonable compression Created by On2 Patented Probably illegal for GPL code YouTube uses it for lower quality and old streams

Proprietary

Reasonable compression

Created by On2

Patented

Probably illegal for GPL code

YouTube uses it for lower quality and old streams

H.264 Not proprietary Good compression MPEG Standard Open, but patented Patent licenses from the MPEG LA Unclear how this applies to (L)GPL code But probably badly YouTube using it for higher quality streams iPhones and AppleTV

Not proprietary

Good compression

MPEG Standard

Open, but patented

Patent licenses from the MPEG LA

Unclear how this applies to (L)GPL code

But probably badly

YouTube using it for higher quality streams

iPhones and AppleTV

Software Open source transcode tools FFmepg libavcodec for VP6 Probably illegal – dubious Also pretty shoddy Can only decode H.264

Open source transcode tools

FFmepg

libavcodec for VP6

Probably illegal – dubious

Also pretty shoddy

Can only decode H.264

More software MEncoder libmpcodecs uses libavcodec VLC libvlc uses libavcodec So basically the same Different muxing, same codecs

MEncoder

libmpcodecs uses libavcodec

VLC

libvlc uses libavcodec

So basically the same

Different muxing, same codecs

Free H.264? Unfortunately, not really x264 is the only usable one It’s pretty good MEncoder can use it Still limited in options at this point Again, dubiously legal

Unfortunately, not really

x264 is the only usable one

It’s pretty good

MEncoder can use it

Still limited in options at this point

Again, dubiously legal

Non-free tools Flash encoder Not automatable On2 FlixEngine Creators of VP6 Windows or Linux Some support for H.264 Rhozet Carbon Coder The new hot shit Good H.264 support Windows

Flash encoder

Not automatable

On2 FlixEngine

Creators of VP6

Windows or Linux

Some support for H.264

Rhozet Carbon Coder

The new hot shit

Good H.264 support

Windows

Choices Video codec Resolution Bitrate (VBR, CBR) Keyframes Audio codec Channels Bit depth Sampling rate

Video codec

Resolution

Bitrate (VBR, CBR)

Keyframes

Audio codec

Channels

Bit depth

Sampling rate

Doing it at scale Not really a problem Very easily parallelizable Amazon EC2 is awesome here Exactly what it was design for Grow/shrink as needed But, per-CPU software licensing

Not really a problem

Very easily parallelizable

Amazon EC2 is awesome here

Exactly what it was design for

Grow/shrink as needed

But, per-CPU software licensing

3. Store

3.

Store

Easy! Really, just like photos But with bigger files Same disk layout as any other serving But the serving part is slower

Really, just like photos

But with bigger files

Same disk layout as any other serving

But the serving part is slower

But.. Remember the files are huge Operations take time More likely to fail halfway through Checksums are your friend Do it all asynchronously

Remember the files are huge

Operations take time

More likely to fail halfway through

Checksums are your friend

Do it all asynchronously

4. Serve & Playback

4.

Serve & Playback

The choice Streaming vs Progressive

Streaming

vs

Progressive

Streaming Pros Easily seekable Live feeds Cons Special server software Slower to start Firewall troubles

Pros

Easily seekable

Live feeds

Cons

Special server software

Slower to start

Firewall troubles

Progressive download Pros Just use a web server Play offline Firewall/proxy friendly Cons Harder to seek ahead (but not impossible)

Pros

Just use a web server

Play offline

Firewall/proxy friendly

Cons

Harder to seek ahead (but not impossible)

Streaming tech Non flash stuff We’ll ignore that You’re using flash, right? RTMP Real-time Messaging Protocol Proprietary (thanks Adobe!)

Non flash stuff

We’ll ignore that

You’re using flash, right?

RTMP

Real-time Messaging Protocol

Proprietary (thanks Adobe!)

RTMP RTMP - Raw TCP socket stuffs RTMPT – RTMP tunneled over HHTP For firewalls, etc Flash Media Server previously Flash Communication Server Wowza Pro $1000/server

RTMP - Raw TCP socket stuffs

RTMPT – RTMP tunneled over HHTP

For firewalls, etc

Flash Media Server

previously Flash Communication Server

Wowza Pro

$1000/server

Open source It’s not all bad Red5 Java implementation of RTMP server Mostly feature complete Beta quality, but usable in production Facebook

It’s not all bad

Red5

Java implementation of RTMP server

Mostly feature complete

Beta quality, but usable in production

Facebook

Progressive Used by the majority of large sites Very simple! Seekable with server support

Used by the majority of large sites

Very simple!

Seekable with server support

Seeking Serve the FLV starting at a different point Just add a simple FLV preamble before seeking into the file Simple to do in PHP, Perl, etc mod_flvx for Apache mod_secdownload for lighttpd

Serve the FLV starting at a different point

Just add a simple FLV preamble before seeking into the file

Simple to do in PHP, Perl, etc

mod_flvx for Apache

mod_secdownload for lighttpd

5. Other considerations

5.

Other considerations

Review Videos are slow Expensive to review Review grids Doesn’t cover audio

Videos are slow

Expensive to review

Review grids

Doesn’t cover audio

Not enough? Social tools are useful here

Social tools are useful here

Summary

Summary

Summing up Flash makes sense For uploading too H.264 is probably your best bet today Transcoding software still costs money Unless you’re willing to take on the risk Progressive download is basically awesome

Flash makes sense

For uploading too

H.264 is probably your best bet today

Transcoding software still costs money

Unless you’re willing to take on the risk

Progressive download is basically awesome

The end!

Awesome! These slides are available online: iamcal.com/talks/

These slides are available online:

iamcal.com/talks/