Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Moving Pictures - Web 2.0 Expo NYC


Published on

Published in: Technology, News & Politics
  • Be the first to comment

Moving Pictures - Web 2.0 Expo NYC

  1. 1. Moving Pictures Implementing Video on Flickr Cal Henderson
  2. 2. <ul><li>Hello </li></ul>
  3. 3. Flickr <ul><li>Large scale kitten sharing website </li></ul><ul><li>Started 2003, launched 2004 </li></ul><ul><ul><li>5 years old this december </li></ul></ul><ul><li>Almost 3 billion photos </li></ul>
  4. 4. Enter: Video <ul><li>Video was added this year </li></ul><ul><ul><li>Launched April 2008 </li></ul></ul><ul><li>Many hundreds of thousands of videos uploaded </li></ul><ul><li>Many millions of playbacks </li></ul>
  5. 10. <ul><li>“ Video? That’s just like photos!” </li></ul><ul><li>-Me, before Flickr Video </li></ul>
  6. 11. 12 4 Steps <ul><li>4 main tasks </li></ul><ul><ul><li>Uploading </li></ul></ul><ul><ul><li>Transcoding </li></ul></ul><ul><ul><li>Storage </li></ul></ul><ul><ul><li>Serving & Playback </li></ul></ul>
  7. 12. <ul><li>1. </li></ul><ul><li>Upload </li></ul>
  8. 13. Simple upload <ul><li>Web forms </li></ul><ul><ul><li>Just like any other file </li></ul></ul><ul><ul><li><form action=&quot;/uploadify/&quot; method=&quot;post&quot; enctype=&quot;multipart/form-data&quot;> </li></ul></ul><ul><ul><li><input type=&quot;file&quot; name=&quot;fred&quot; /> <input type=&quot;submit&quot; value=&quot;Go!&quot; /> </li></ul></ul><ul><ul><li></form> </li></ul></ul><ul><li>But files are large / huge </li></ul>
  9. 14. Issues <ul><li>Two components for uploading: </li></ul><ul><ul><li>Sending from the client </li></ul></ul><ul><ul><li>Receiving on the server </li></ul></ul><ul><li>Both of these present problems </li></ul>
  10. 15. Sending from the client <ul><li>Multiple options </li></ul><ul><ul><li>Simple form </li></ul></ul><ul><ul><li>Flash </li></ul></ul><ul><ul><li>Desktop app </li></ul></ul>
  11. 16. Simple Forms <ul><li>Pros </li></ul><ul><ul><li>Very easy to implement </li></ul></ul><ul><ul><li>Works on every browser out of the box </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Upload progress is harder </li></ul></ul><ul><ul><li>‘Slow’ </li></ul></ul><ul><ul><li>Select a single file at once </li></ul></ul>
  12. 17. Flash <ul><li>Pros </li></ul><ul><ul><li>Upload progress is easy </li></ul></ul><ul><ul><li>‘Fast’ </li></ul></ul><ul><ul><li>Multi select of files </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Harder to implement </li></ul></ul><ul><ul><li>Flash isn’t quite ubiquitous </li></ul></ul>
  13. 18. Desktop App <ul><li>Pros </li></ul><ul><ul><li>Upload progress is easy </li></ul></ul><ul><ul><li>‘Very fast’ </li></ul></ul><ul><ul><li>Multi select of files </li></ul></ul><ul><ul><li>Drag and drop </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Hard to develop </li></ul></ul><ul><ul><li>Hard to deploy (relative to the web) </li></ul></ul>
  14. 19. Making Progress <ul><li>Upload progress </li></ul><ul><ul><li>Not impossible with plain forms </li></ul></ul><ul><ul><li>Just need to be able to query the upload progress via AJAX </li></ul></ul><ul><li>Multiple machines </li></ul><ul><ul><li>The VIP issue </li></ul></ul><ul><ul><li>Enter Perlbal </li></ul></ul>
  15. 20. Making Progress Browser Web 1 Web 2 1. Browser starts upload 2. Web server broadcasts progress via UDP 3. Browser queries progress via AJX call Load balancer
  16. 21. Receiving on the server <ul><li>File uploads are slow </li></ul><ul><ul><li>Much slower than serving pages </li></ul></ul><ul><li>Apache processes are heavy </li></ul><ul><ul><li>Waste of resources </li></ul></ul><ul><li>Use a poll based server (like jetty) </li></ul>
  17. 22. Receiving on the server <ul><li>Or, use a buffering layer </li></ul><ul><ul><li>Perlbal is great for this </li></ul></ul><ul><li>Or a lightweight Apache </li></ul><ul><ul><li>E.g. w/ mod_proxy </li></ul></ul>Browser Buffer Server Slow Fast
  18. 23. But wait <ul><li>It’s not just the first step that’s slow </li></ul><ul><li>Moving files around between servers is slow </li></ul><ul><ul><li>Do it out of band </li></ul></ul><ul><li>Asynchronous jobs are in order anyway </li></ul><ul><ul><li>Do it! </li></ul></ul>
  19. 24. <ul><li>2. </li></ul><ul><li>Transcode </li></ul>
  20. 25. Transcode? <ul><li>Why transcode at all? </li></ul><ul><li>Input comes from many sources </li></ul><ul><ul><li>Point & shoots </li></ul></ul><ul><ul><li>DV Cams </li></ul></ul><ul><ul><li>Mobile devices </li></ul></ul><ul><ul><li>Video editing software </li></ul></ul><ul><li>All in different formats </li></ul>
  21. 26. So many formats <ul><li>But very few of these formats can be played back cross platform </li></ul><ul><ul><li>Without special software or hardware </li></ul></ul><ul><li>Formats are designed to do one thing well </li></ul><ul><ul><li>They don’t always manage even that </li></ul></ul><ul><li>Transcoding puts all videos on an equal footing </li></ul>
  22. 27. Video file basics <ul><li>Most file types are really just containers </li></ul><ul><ul><li>MOV, FLV, AVI </li></ul></ul><ul><li>The data inside can be in multiple formats </li></ul><ul><ul><li>We call these codecs (encoder + decoder) </li></ul></ul><ul><li>Files contains multiple ‘streams’ </li></ul><ul><ul><li>Both audio and video </li></ul></ul>
  23. 28. Interleave <ul><li>Audio and video are often interleaved </li></ul><ul><ul><li>Hence AVI </li></ul></ul><ul><li>A video file looks like this: </li></ul><ul><ul><li>Headers </li></ul></ul><ul><ul><li>Video chunk </li></ul></ul><ul><ul><li>Audio chunk </li></ul></ul><ul><ul><li>Video chunk </li></ul></ul><ul><ul><li>Audio chunk </li></ul></ul><ul><ul><li>Etc </li></ul></ul>
  24. 29. Compress <ul><li>Raw video files are huge </li></ul><ul><ul><li>A bitmap for every frame </li></ul></ul><ul><ul><li>Rarely used, even in post production </li></ul></ul><ul><li>At 30 fps, that gets crazy pretty quickly </li></ul><ul><li>We don’t need to store every frame </li></ul><ul><ul><li>Static backgrounds don’t change (much) between frames </li></ul></ul>
  25. 30. Compresssss <ul><li>Intraframe </li></ul><ul><ul><li>Treat each frame as a picture </li></ul></ul><ul><ul><li>Compress it (just like JPEG) </li></ul></ul><ul><ul><li>DCT (Discrete Cosine Transform) </li></ul></ul><ul><li>Interframe </li></ul><ul><ul><li>Store the differences between frames </li></ul></ul><ul><ul><li>Treat the pixels as a 3D array to be compressed </li></ul></ul>
  26. 31. The IPB <ul><li>Three frame types: I, P & B </li></ul><ul><li>Intra coded pictures </li></ul><ul><ul><li>A full raw frame </li></ul></ul><ul><li>Predicted pictures </li></ul><ul><ul><li>Based on a single reference frame </li></ul></ul><ul><li>Bi-predictive pictures </li></ul><ul><ul><li>Based on two or more reference frames </li></ul></ul>
  27. 32. IPBIPBIPBIPB <ul><li>Reference frames may be I, P or B </li></ul><ul><li>P & B frames may contain a mix of image data and motion vector displacements </li></ul><ul><li>I frames require the most bits </li></ul><ul><ul><li>Then P frames </li></ul></ul><ul><ul><li>Then B frames </li></ul></ul>
  28. 33. Bad terminology <ul><li>We should really say picture </li></ul><ul><ul><li>(Not frame) </li></ul></ul><ul><ul><li>Because of interlacing </li></ul></ul><ul><li>Really, we encode fields not frames </li></ul><ul><ul><li>Picture is the general term </li></ul></ul><ul><li>And H.264 contains ‘slices’ </li></ul><ul><ul><li>Sub-regions of the field </li></ul></ul><ul><ul><li>‘ Macroblocks’ & ‘Artifacts’ </li></ul></ul>
  29. 34. I-Frames <ul><li>Also called Key Frames </li></ul><ul><li>Allow easy random seeking </li></ul><ul><li>Twice a second for Digital TV & DVDs </li></ul><ul><li>More widely spaced online </li></ul>
  30. 35. Seekable
  31. 36. Seekable
  32. 37. Oh, and audio too <ul><li>We can worry less about this </li></ul><ul><ul><li>Older problem, well solved </li></ul></ul><ul><li>MP3 is pretty good </li></ul><ul><ul><li>Who cares how it works? </li></ul></ul><ul><li>Syncing is the only issue </li></ul><ul><ul><li>Presentation Time Stamps (PTS) and Decode Time Stamps (DTS) in MPEG-2 </li></ul></ul>
  33. 38. Flash! Woah-oh! <ul><li>The big question: </li></ul><ul><li>Flash? </li></ul>
  34. 39. Non-flash sites <ul><li>QuickTime </li></ul><ul><li>Windows Media </li></ul><ul><li>This is gradually disappearing </li></ul><ul><li>Flash player is ubiquitous </li></ul><ul><li>Compression is good enough </li></ul><ul><li>Interactive too </li></ul><ul><li>But no 3D/VR as with QuickTime :( </li></ul>
  35. 40. The Flash Player <ul><li>Flash Player 6 </li></ul><ul><ul><li>March 2002 </li></ul></ul><ul><li>Video: Sorenson Spark (H.263) </li></ul><ul><li>Audio: MP3 </li></ul><ul><ul><li>Or ADPCM / Uncompressed </li></ul></ul><ul><ul><li>Or Nellymoser Asao </li></ul></ul>
  36. 41. Second Generation <ul><li>Flash Player 7 </li></ul><ul><ul><li>August 2005 </li></ul></ul><ul><li>Video: On2 TrueMotion VP6 </li></ul><ul><li>Audio: MP3 </li></ul>
  37. 42. The hot shit <ul><li>Flash Player 9 (update 3) </li></ul><ul><ul><li>December 2007 </li></ul></ul><ul><li>Video: H.264 (MPEG-4 Part 10) </li></ul><ul><ul><li>w/ container formats from MPEG-4 Part 14 </li></ul></ul><ul><li>Audio: AAC (MPEG-4 Part 3) </li></ul><ul><li>Plus 3GPP Timed Text (MPEG-4 Part 17) </li></ul>
  38. 43. TrueMotion VP6 <ul><li>Proprietary </li></ul><ul><li>Reasonable compression </li></ul><ul><li>Created by On2 </li></ul><ul><ul><li>Patented </li></ul></ul><ul><ul><li>Probably illegal for GPL code </li></ul></ul><ul><li>YouTube uses it for lower quality and old streams </li></ul>
  39. 44. H.264 <ul><li>Not proprietary </li></ul><ul><li>Good compression </li></ul><ul><li>MPEG Standard </li></ul><ul><ul><li>Open, but patented </li></ul></ul><ul><ul><li>Patent licenses from the MPEG LA </li></ul></ul><ul><ul><li>Unclear how this applies to (L)GPL code </li></ul></ul><ul><ul><li>But probably badly </li></ul></ul><ul><li>YouTube using it for higher quality streams </li></ul><ul><li>iPhones and AppleTV </li></ul>
  40. 45. Software <ul><li>Open source transcode tools </li></ul><ul><li>FFmepg </li></ul><ul><ul><li>libavcodec for VP6 </li></ul></ul><ul><ul><li>Probably illegal – dubious </li></ul></ul><ul><ul><li>Also pretty shoddy </li></ul></ul><ul><ul><li>Can only decode H.264 </li></ul></ul>
  41. 46. More software <ul><li>MEncoder </li></ul><ul><ul><li>libmpcodecs uses libavcodec </li></ul></ul><ul><li>VLC </li></ul><ul><ul><li>libvlc uses libavcodec </li></ul></ul><ul><li>So basically the same </li></ul><ul><ul><li>Different muxing, same codecs </li></ul></ul>
  42. 47. Free H.264? <ul><li>Unfortunately, not really </li></ul><ul><li>x264 is the only usable one </li></ul><ul><ul><li>It’s pretty good </li></ul></ul><ul><ul><li>MEncoder can use it </li></ul></ul><ul><ul><li>Still limited in options at this point </li></ul></ul><ul><ul><li>Again, dubiously legal </li></ul></ul>
  43. 48. Non-free tools <ul><li>Flash encoder </li></ul><ul><ul><li>Not automatable </li></ul></ul><ul><li>On2 FlixEngine </li></ul><ul><ul><li>Creators of VP6 </li></ul></ul><ul><ul><li>Windows or Linux </li></ul></ul><ul><ul><li>Some support for H.264 </li></ul></ul><ul><li>Rhozet Carbon Coder </li></ul><ul><ul><li>The new hot shit </li></ul></ul><ul><ul><li>Good H.264 support </li></ul></ul><ul><ul><li>Windows </li></ul></ul>
  44. 49. Choices <ul><li>Video codec </li></ul><ul><li>Resolution </li></ul><ul><li>Bitrate (VBR, CBR) </li></ul><ul><li>Keyframes </li></ul><ul><li>Audio codec </li></ul><ul><ul><li>Channels </li></ul></ul><ul><ul><li>Bit depth </li></ul></ul><ul><ul><li>Sampling rate </li></ul></ul>
  45. 50. Doing it at scale <ul><li>Not really a problem </li></ul><ul><li>Very easily parallelizable </li></ul><ul><li>Amazon EC2 is awesome here </li></ul><ul><ul><li>Exactly what it was design for </li></ul></ul><ul><ul><li>Grow/shrink as needed </li></ul></ul><ul><ul><li>But, per-CPU software licensing </li></ul></ul>
  46. 51. <ul><li>3. </li></ul><ul><li>Store </li></ul>
  47. 52. Easy! <ul><li>Really, just like photos </li></ul><ul><ul><li>But with bigger files </li></ul></ul><ul><li>Same disk layout as any other serving </li></ul><ul><li>But the serving part is slower </li></ul>
  48. 53. But.. <ul><li>Remember the files are huge </li></ul><ul><li>Operations take time </li></ul><ul><ul><li>More likely to fail halfway through </li></ul></ul><ul><ul><li>Checksums are your friend </li></ul></ul><ul><li>Do it all asynchronously </li></ul>
  49. 54. <ul><li>4. </li></ul><ul><li>Serve & Playback </li></ul>
  50. 55. The choice <ul><li>Streaming </li></ul><ul><li>vs </li></ul><ul><li>Progressive </li></ul>
  51. 56. Streaming <ul><li>Pros </li></ul><ul><ul><li>Easily seekable </li></ul></ul><ul><ul><li>Live feeds </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Special server software </li></ul></ul><ul><ul><li>Slower to start </li></ul></ul><ul><ul><li>Firewall troubles </li></ul></ul>
  52. 57. Progressive download <ul><li>Pros </li></ul><ul><ul><li>Just use a web server </li></ul></ul><ul><ul><li>Play offline </li></ul></ul><ul><ul><li>Firewall/proxy friendly </li></ul></ul><ul><li>Cons </li></ul><ul><ul><li>Harder to seek ahead (but not impossible) </li></ul></ul>
  53. 58. Streaming tech <ul><li>Non flash stuff </li></ul><ul><ul><li>We’ll ignore that </li></ul></ul><ul><ul><li>You’re using flash, right? </li></ul></ul><ul><li>RTMP </li></ul><ul><ul><li>Real-time Messaging Protocol </li></ul></ul><ul><ul><li>Proprietary (thanks Adobe!) </li></ul></ul>
  54. 59. RTMP <ul><li>RTMP - Raw TCP socket stuffs </li></ul><ul><li>RTMPT – RTMP tunneled over HHTP </li></ul><ul><ul><li>For firewalls, etc </li></ul></ul><ul><li>Flash Media Server </li></ul><ul><ul><li>previously Flash Communication Server </li></ul></ul><ul><li>Wowza Pro </li></ul><ul><ul><li>$1000/server </li></ul></ul>
  55. 60. Open source <ul><li>It’s not all bad </li></ul><ul><li>Red5 </li></ul><ul><ul><li>Java implementation of RTMP server </li></ul></ul><ul><ul><li>Mostly feature complete </li></ul></ul><ul><ul><li>Beta quality, but usable in production </li></ul></ul><ul><ul><li>Facebook </li></ul></ul>
  56. 61. Progressive <ul><li>Used by the majority of large sites </li></ul><ul><li>Very simple! </li></ul><ul><li>Seekable with server support </li></ul>
  57. 62. Seeking <ul><li>Serve the FLV starting at a different point </li></ul><ul><li>Just add a simple FLV preamble before seeking into the file </li></ul><ul><li>Simple to do in PHP, Perl, etc </li></ul><ul><li>mod_flvx for Apache </li></ul><ul><li>mod_secdownload for lighttpd </li></ul>
  58. 63. <ul><li>5. </li></ul><ul><li>Other considerations </li></ul>
  59. 64. Review <ul><li>Videos are slow </li></ul><ul><li>Expensive to review </li></ul><ul><li>Review grids </li></ul><ul><ul><li>Doesn’t cover audio </li></ul></ul>
  60. 66. Not enough? <ul><li>Social tools are useful here </li></ul>
  61. 67. <ul><li>Summary </li></ul>
  62. 68. Summing up <ul><li>Flash makes sense </li></ul><ul><ul><li>For uploading too </li></ul></ul><ul><li>H.264 is probably your best bet today </li></ul><ul><li>Transcoding software still costs money </li></ul><ul><ul><li>Unless you’re willing to take on the risk </li></ul></ul><ul><li>Progressive download is basically awesome </li></ul>
  63. 69. The end!
  64. 70. Awesome! <ul><li>These slides are available online: </li></ul><ul><li> </li></ul>