WKSP07: Media Processing Workflow


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

WKSP07: Media Processing Workflow

  1. 1. Media Processing Workflow<br />Alex Zambelli<br />Media Technology Evangelist<br />Developer & Platform Evangelism<br />alexzam@microsoft.com<br />
  2. 2. Workshop Overview<br />Silverlight media capabilities<br />Smooth Streaming 101<br />15 min break<br />Live Smooth Streaming<br />Workflow and architecture<br />15 min break<br />Encoding for Smooth Streaming<br />
  3. 3. A Brave New World<br />Encoding 320x240 video on a laptop is “so 2002”<br />HTTP adaptive streaming has entirely changed the game in the past 2 years<br />The future for Silverlight is all Smooth Streaming, all HD<br />Video quality is better than ever, but at the price of complexity and bandwidth<br />Don’t worry, it only gets easier from here<br />
  4. 4. Silverlight Media Formats<br />Windows Media<br />ASF file format (*.asf, *.wmv, *.wma)<br />VC-1 (WMV9), WMV8, WMV7 video codecs<br />WMA Pro, WMA Standard, MP3 audio codecs<br />MPEG-4<br />MP4 file format (*.mp4, *.3gp, *.mov)<br />H.264 video codec (Baseline, Main, High profiles)<br />AAC audio codec (AAC-LC profile)<br />
  5. 5. Silverlight Media Extensibility<br />Smooth Streaming<br />VC-1 and H.264 video<br />WMA Pro, WMA Standard and AAC-LC audio<br />Supported via Smooth Streaming Media Element extension<br />MediaStreamSource lets developers add support for 3rd party codecs, formats and delivery methods<br />
  6. 6. Silverlight Media Protection<br />Silverlight DRM, powered by PlayReady<br />PlayReady AES-256 encryption<br />Windows Media DRM encryption<br />Silverlight 3<br />VC-1 and WMA encryption only<br />Live DRM (Direct License Acquisition) only<br />Silverlight 4<br />H.264 and AAC encryption<br />Offline DRM<br />
  7. 7. Silverlight Media Delivery<br />Windows Media Services streaming<br />Windows Server 2008 and 2003<br />Unicast (WMS HTTP)<br />Multicast<br />HTTP progressive download<br />Any HTTP 1.0/1.1 web server<br />IIS Smooth Streaming<br />Windows Server 2008 (IIS7)<br />
  8. 8. Silverlight Media Framework<br />No need to reinvent the wheel every time a Silverlight video player is needed<br />Silverlight Media Framework:http://smf.codeplex.com<br />All essential playback features plus advanced Smooth Streaming features<br />DVR, Fast Forward, Rewind, Replay, Slow Motion<br />Markers, captions, dynamic ad insertion<br />Rich analytics and logging<br />
  9. 9. Media Delivery Methods<br />Unicast<br />Traditional Streaming<br />Progressive Download<br />HTTP-based Adaptive Streaming<br />Multicast<br />IP Multicast<br />
  10. 10. Unicast<br />Many one-to-one streams<br />Requires more bandwidth per user<br />May require more servers<br />For private and public networks<br />Origin Server<br />
  11. 11. Multicast<br />A single one-to-many stream<br />Uses bandwidth of only one stream<br />Requires multicast-enabled networks<br />Typically requires fewer servers<br />Origin Server<br />
  12. 12. Movie<br />Unicast Methods<br />HTTP Adaptive Streaming<br />Video @ 01:04?<br />Video @ 01:06?<br />Video @ 01:08?<br />Traditional Streaming<br />Play<br />Seek<br />Pause<br />Progressive Download<br />Movie<br />
  13. 13. Internet Streaming Challenge<br />Traditional streaming designed for efficient media delivery<br />Works well in small networks but on the Internet suffers from serious drawbacks:<br />Doesn’t scale to today’s Internet audience<br />Doesn’t take into account today’s Internet structure and organization<br />Designed for network conditions less volatile than Internet traffic<br />
  14. 14. HTTP Adaptive Streaming<br />Hybrid media delivery method <br />Acts like streaming but is in fact a series of short progressive downloads<br />Video and audio delivered as series of small files over HTTP <br />Built for the Web<br />Leverages existing HTTP caches<br />Scales exceptionally well to meet high demand<br />Resilient to network unpredictability<br />Client can seamlessly switch video quality and bit rate based on perceived network bandwidth and CPU resources<br />Applicable to both on-demand and live delivery<br />
  15. 15. Windows Server Media Delivery<br />IIS Media Services<br /><ul><li> Traditional Streaming
  16. 16. Unicast
  17. 17. WMS RTSP
  18. 18. WMS HTTP
  19. 19. Multicast
  20. 20. WMS Multicast
  21. 21. Progressive Download
  22. 22. Bit Rate Throttling
  23. 23. HTTP Adaptive Streaming
  24. 24. Smooth Streaming</li></li></ul><li>Smooth Streaming<br />Smooth Streaming<br />Microsoft implementation of HTTP-based adaptive streaming<br />Best of both worlds<br />Responsive and efficient like streaming<br />Cheap and scalable like progressive download<br />Superior user experience<br />No buffering, no stutter<br />Instant start, instant seek<br />Seamless bit rate switching based on bandwidth and CPU<br />
  25. 25. Multi Bit Rate Revisited<br />00:27.24<br />03:47.16<br />05:05.04<br />07:33.10<br />2.4 Mbps<br />1.5 Mbps<br />700 kbps<br />300 kbps<br />
  26. 26. Adapting Bit Rate in Real-Time<br />300K @ 00:00?<br />700K @ 00:02?<br />2.4M @ 00:04?<br />1.5M @ 00:06?<br />2.4M @ 00:08?<br />300K (start quickly)<br />00:00<br />00:02<br />00:04<br />00:06<br />00:08<br />700K (good network)<br />2.4M (great network)<br />2.4M<br />1.5M (glitch)<br />2.4M (play on…)<br />…<br />Bit Rate Heuristics<br />300K<br />
  27. 27. Smooth Streaming Design<br />Smooth Streaming File Format based on MP4 (ISO Base Media File Format)<br />Video is encoded and stored on disk as one contiguous MP4 file<br />One file per bit rate<br />Each video Group of Pictures (GOP) is stored in a separate Movie Fragment box<br />This allows accurate fragmentation at key frames<br />Contiguous file is virtually split up into chunks when responding to a client request<br />
  28. 28. Smooth Streaming File Format<br />File Type (ftyp)<br />
  29. 29. On-Demand Presentations<br />MP4 files containing video/audio/data fragments<br />New file extensions instead of *.mp4<br />*.ismvfor files containing video and audio tracks, or only video tracks<br />*.ismafor files containing only audio tracks<br />On-demand server manifest file<br />File extension: *.ism<br />Describes the relationships between media tracks, bitrates and files on disk<br />Uses SMIL 2.0 XML format<br />Client manifest file<br />File extension: *.ismc<br />Describes available streams, codecs, bitrates, resolutions, metadata<br />Uses XML format<br />
  30. 30. On Demand Server Manifest<br />
  31. 31. On Demand Client Manifest<br />
  32. 32. Client-Server Communication<br />All requests to the server are formed as RESTful URLs<br />Client always first requests the client manifest:http://iis.net/video.ism/Manifest<br />After parsing the manifest the client begins making requests for fragments by indicating bit rates and offset times it learned from the manifest:http://iis.net/video.ism/QualityLevels(350000)/Fragments(video=40040000)http://iis.net/video.ism/QualityLevels(48000)/Fragments(audio=62293333)<br />Every valid HTTP request returns a file of MIME type “video/mp4”<br />
  33. 33. Serving Fragments On Demand<br />IIS7 Smooth Streaming handler interprets the RESTful URL request<br />It maps the quality level to a physical ISMV file based on the server manifest (*.ism) and calculates the fragment location within the contiguous ISMV file based on the time offset<br />Because the wire format (fragment) is just a subset of the disk format (MP4), the extraction process is simple<br />No re-muxing necessary<br />Dynamic stream switching logic is fully implemented in Silverlight application code – no server-side detection<br />
  34. 34. Demo<br />DEMO:<br />Setting up On Demand publishing points<br />Using Fiddler to monitor Smooth Streaming traffic<br />
  35. 35. Live Smooth Streaming<br />Live publishing point identified by *.isml extension<br />Client-server communication very similar to on-demand<br />Server continually appends MP4 file with new fragments from encoder and makes them available to client<br />This allows DVR-like seeking within the existing archive<br />
  36. 36. Encoder-Server Communication<br />Encoder pushes fragmented MP4 stream to server in body of a long-running HTTP POST requesthttp://iis.net/live.isml/stream(720p)<br />Encoder inserts Live Server Manifest Box into start of MP4 stream<br />Contains a SMIL-formatted Live Server Manifest very similar to On-Demand Manifest<br />Describes all tracks in that encoder’s stream<br />
  37. 37. Encoder-Server Communication<br />Each sent fragment contains a box with absolute time and duration<br />Timestamps are typically derived from the input source Linear Time Code (LTC)<br />If multiple encoders are used to encode the same video source, it’s essential that a time code generator is used to keep their LTC in sync<br />
  38. 38. Serving Live Fragments<br />Server parses the encoder manifests and starts collecting MP4 fragments<br />Server builds a cumulative runtime index in memory for all incoming fragments<br />Server stores fragments into a temporary DVR ISMV or a permanent ISMV archive<br />When the pub point is shut down, server generates ‘mfra’ index box for ISMV archive<br />
  39. 39. Live Client Workflow<br />Client requests the latest manifesthttp://iis.net/live.isml/Manifest<br />Server adds up all the encoder manifests and the runtime fragment index and sends the latest version to the client<br />Every fragment contains information about the next fragment<br />No need to repeatedly download manifest<br />Client builds its own cumulative runtime index based on the lookahead info in the fragments<br />
  40. 40. Demo<br />DEMO:<br />Basic Live Smooth Streaming workflow<br />
  41. 41. Live Smooth Publishing Points<br />ISML files are SMIL 2.0 formatted XML configuration files<br />Live publishing points can be created via the IIS Manager UI or by manually creating ISML files on server<br />Both push and pull publishing models are supported<br />Live publishing points can also push to and pull from other live publishing points<br />
  42. 42. Live Publishing Point – IIS Mgr<br />
  43. 43. Live Publishing Point - ISML<br />
  44. 44. ISML Syntax<br />sourceType<br />Push: source streams will be pushed to pub point<br />Pull: pub point will pull streams from listed URLs<br />publishing<br />Streams: pub point can serve streams to other live pub points<br />Fragments: clients can request fragments<br />Archives: archiving is enabled<br />lookaheadChunks: number of fragments server will buffer before making them available to clients<br />manifestWindowLength: length of DVR moving window<br />archivePath: local path where archives will be written<br />
  45. 45. Ingest Server<br />Origin Server:<br />Realtime Stream<br />Ingests and Origins<br />Origin Server:<br />Delayed Stream<br />Origin Server:<br />Delayed Stream<br />Encoder<br />Origin Server:<br />Realtime Stream<br />8 Mbps<br />Origin Server:<br />VOD Archive<br />Origin Server:<br />VOD Archive<br />
  46. 46. DEMO<br />DEMO:<br />Advanced Live Smooth Streaming workflow<br />
  47. 47. Network Design Considerations<br />Bandwidth, bandwidth, bandwidth<br />Encoder egress bit rate =<br />(All video and audio bit rates together)<br />x<br />(Number of ingest publishing points)<br />Leave enough headroom for bit rate spikes and bursts<br />Play it: leave 50%-100% headroom<br />
  48. 48. Building Smooth Streaming Origins<br />Network<br />Make sure NICs can handle the cumulative ingress and egress bandwidth<br />CPU and memory<br />Follow typical server configuration guidelines<br />Disk<br />Live Smooth Streaming can require high read/write rates<br />10,000 RPM disks are a good idea<br />
  49. 49. Redundancy and Failover<br />Encoders<br />Active-active redundant encoder configuration not supported<br />Encoder hot swap is possible as long as encoder doesn’t signal end-of-stream to server<br />Configurable time-out periods can help survive temporary loss of network connectivity <br />Some encoders support pushing to multiple publishing points concurrently, while others do pub point roll over in case of failure<br />
  50. 50. Redundancy and Failover<br />Servers<br />IIS is designed to discard duplicate fragments<br />This allows ingest and origin servers to be set in active-active configuration<br />Ingest server:<br />Primary<br />Encoder<br />Origin server<br />Ingest server:<br />Secondary<br />
  51. 51. Redundancy and Failover<br />Origins<br />CDNs can dynamically change DNS to redirect traffic to secondary origin when primary fails<br />Ingest server:<br />Primary<br />Origin server:<br />Primary<br />DNS<br />Encoder<br />Ingest server:<br />Secondary<br />Origin server:<br />Secondary<br />
  52. 52. Video Encoding<br />
  53. 53. Smooth Streaming Encoders<br />Many vendors to choose from:<br />Anystream (Grab Networks)<br />Digital Rapids<br />Envivio<br />Inlet Technologies<br />Microsoft<br />Rhozet (Harmonic)<br />Telestream<br />VBrick<br />ViewCast<br />Winnov<br />
  54. 54. H.264 or VC-1?<br />Customers should choose whichever codec best fits their encoding workflow and meets their quality requirements<br />However, keep in mind:<br />H.264 decoding is typically more CPU intensive than VC-1 decoding for the same resolution and frame rate<br />In Silverlight 3: H.264 decoding requires about 15-25% more CPU time than VC-1 decoding with similar content properties<br />Good rule of thumb (for now):<br />For HD video, use VC-1 to reach largest audience<br />For SD video and smaller, use H.264 if quality gains are noticeable<br />
  55. 55. Encoding for Smooth Streaming<br />Use VC-1 Advanced Profile or H.264 Main/High Profile<br />2 second key frame distance works very well<br />Must be closed GOP<br />Client heuristics work best with fixed length (non-adaptive) GOP<br />Set video buffer size to 2x-3x key frame distance<br />Use WMA Professional audio codec for best quality at low bitrates<br />Comparable to HE-AAC – excellent stereo quality even at 48 kbps<br />Client heuristics are currently tuned for CBR content, so if using VBR set the peak bit rate within 15% of average bit rate for best playback<br />
  56. 56. Multi Bit Rate Encoding Revisited<br />00:27.24<br />03:47.16<br />05:05.04<br />07:33.10<br />2.5 Mbps<br />1.5 Mbps<br />750 kbps<br />350 kbps<br />
  57. 57. Variable Resolution<br />Why do we vary encoded resolution together with bit rate?<br />Encoding the same resolution at inappropriately low bit rates introduces objectionable compression side effects into the video: blockiness, twirling details, color smearing, etc.<br />By lowering the resolution proportionally to the bit rate we maintain a consistent level of compression quality in exchange for giving up some visual detail<br />It’s a compromise: between two evils, customers prefer a blurry picture over a blocky picture<br />
  58. 58. VC-1: 640x360 at 1250 kbps<br />
  59. 59. VC-1: 640x360 at 300 kbps<br />
  60. 60. VC-1: 288x160 at 300 kbps<br />
  61. 61. How Many Bit Rates Do I Need?<br />You can use as few as you’d like – even just a single bit rate<br />Upper bound is typically set by the encoder<br />Anything more than 12 video bit rates is probably overkill<br />4 video bit rates for SD and 6 bit rates for 720p HD are good starting points<br />Silverlight client currently only supports 1 audio bit rate<br />
  62. 62. Choosing the Quality Levels<br />Smooth Streaming Multi Bit Rate Calculatorhttp://alexzambelli.com/WMV/MBRCalc.html<br />
  63. 63. Choosing the Quality Levels<br />Recommended minimum quality level:320x176 at 350 kbps<br />Never set the minimum quality level below 288x160 or 300 kbps<br />Small video = Bad full screen experience<br />Compression is most efficient when width and height are multiples of 16<br />One of the resolutions should match the player video window size<br />
  64. 64. NBC Sunday Night Football<br />Video:<br />VC-1 Advanced Profile<br />Audio:<br />WMA 10 Professional<br />48 kbps 48 kHz 16-bit stereo<br />
  65. 65. Configuring Live Encoders<br />Live encoders have finite CPU resources<br />Higher resolution = More CPU cycles<br />Overloading a live video encoder will cause poor image quality and dropped frames<br />Most live encoders are bound by number of CPU cores<br />
  66. 66. Configuring Live Encoders<br />Good rule of thumb for live VC-1 encoding:For each resolution you’d like to use, count the number of CPU cores needed per this table:<br />If total number of needed cores exceeds the number of CPU cores in your encoder, reduce the number of quality levels until they fit<br />
  67. 67. Distributed Live Encoding<br />Some live encoders (Inlet Spinnaker) support distributing bit rates of the same video across multiple encoders<br />Muse use LTC time code generator for sync<br />Always attach audio stream to the video stream with lowest bit rate<br />Make sure encoder load is distributed evenly between encoders<br />Balance based on pixel count<br />
  68. 68. CBS March Madness<br />
  69. 69. Live Encoding Tips & Tricks<br />Most live Smooth Streaming encoders can’t handle above 720p<br />If source is 720p50 or 720p59.94, don’t forget to set frame rate to half<br />If source is 1080i, it’s faster to do single field resizing (bobbing) instead of full frame deinterlacing<br />If source is 480i or 576i, use full frame deinterlacing, preferably motion adaptive<br />Super Sampling scaling gives best quality<br />Denoising filters can improve compressed quality<br />
  70. 70. Expression Encoder 3: Smooth VC-1<br />Required Settings:<br />Profile: VC-1 Advanced Profile<br />Adaptive GOP: Disabled<br />Closed GOP: Enabled<br />Output Mode: Elementary Stream Sequence Header<br />Create separate file per stream: Enabled<br />Recommended Settings:<br />Resize Mode: Stretch<br />Adaptive Dead Zone: Conservative<br />In-Loop: Enabled<br />Overlap: Enabled<br />B-Frame Number: 1<br />Search Range: Adaptive<br />
  71. 71. Expression Encoder 3: Smooth H.264<br />Required Settings:<br />Profile: H.264 Main or Baseline<br />Create separate file per stream: Enabled<br />Recommended Settings:<br />Profile: H.264 Main Profile<br />Resize Mode: Stretch<br />Number of Reference Frames: 4<br />B-Frame Number: 2<br />Search Range: 256<br />Entropy Mode: CABAC<br />In-Loop Filter: Enabled<br />
  72. 72. DEMO<br />DEMO:<br />Configuring Inlet Spinnaker HD<br />
  73. 73. The More You Know…<br />Related MIX10 Sessions:<br />Microsoft Silverlight "Media" : Moving at 60fps<br />Tuesday 11:30 am<br />Introducing the Silverlight Rough Cut Editor<br />Tuesday 1:30 pm<br />Smooth Streaming Live in HD: From Camera to Screen<br />Tuesday 3:00 pm<br />Smooth Streaming Live in HD: 2010 Olympic Winter Games<br />Tuesday 4:30 pm<br />
  74. 74. That’s All!<br />Please fill out the evaluation forms!<br />Enjoy MIX!<br />
  75. 75.
  76. 76. © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.<br />The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.<br />
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.