WKSP07: Media Processing Workflow
Upcoming SlideShare
Loading in...5

WKSP07: Media Processing Workflow






Total Views
Views on SlideShare
Embed Views



2 Embeds 9

http://www.slideshare.net 8
http://webcache.googleusercontent.com 1



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

WKSP07: Media Processing Workflow WKSP07: Media Processing Workflow Presentation Transcript

  • Media Processing Workflow
    Alex Zambelli
    Media Technology Evangelist
    Developer & Platform Evangelism
  • Workshop Overview
    Silverlight media capabilities
    Smooth Streaming 101
    15 min break
    Live Smooth Streaming
    Workflow and architecture
    15 min break
    Encoding for Smooth Streaming
  • A Brave New World
    Encoding 320x240 video on a laptop is “so 2002”
    HTTP adaptive streaming has entirely changed the game in the past 2 years
    The future for Silverlight is all Smooth Streaming, all HD
    Video quality is better than ever, but at the price of complexity and bandwidth
    Don’t worry, it only gets easier from here
  • Silverlight Media Formats
    Windows Media
    ASF file format (*.asf, *.wmv, *.wma)
    VC-1 (WMV9), WMV8, WMV7 video codecs
    WMA Pro, WMA Standard, MP3 audio codecs
    MP4 file format (*.mp4, *.3gp, *.mov)
    H.264 video codec (Baseline, Main, High profiles)
    AAC audio codec (AAC-LC profile)
  • Silverlight Media Extensibility
    Smooth Streaming
    VC-1 and H.264 video
    WMA Pro, WMA Standard and AAC-LC audio
    Supported via Smooth Streaming Media Element extension
    MediaStreamSource lets developers add support for 3rd party codecs, formats and delivery methods
  • Silverlight Media Protection
    Silverlight DRM, powered by PlayReady
    PlayReady AES-256 encryption
    Windows Media DRM encryption
    Silverlight 3
    VC-1 and WMA encryption only
    Live DRM (Direct License Acquisition) only
    Silverlight 4
    H.264 and AAC encryption
    Offline DRM
  • Silverlight Media Delivery
    Windows Media Services streaming
    Windows Server 2008 and 2003
    Unicast (WMS HTTP)
    HTTP progressive download
    Any HTTP 1.0/1.1 web server
    IIS Smooth Streaming
    Windows Server 2008 (IIS7)
  • Silverlight Media Framework
    No need to reinvent the wheel every time a Silverlight video player is needed
    Silverlight Media Framework:http://smf.codeplex.com
    All essential playback features plus advanced Smooth Streaming features
    DVR, Fast Forward, Rewind, Replay, Slow Motion
    Markers, captions, dynamic ad insertion
    Rich analytics and logging
  • Media Delivery Methods
    Traditional Streaming
    Progressive Download
    HTTP-based Adaptive Streaming
    IP Multicast
  • Unicast
    Many one-to-one streams
    Requires more bandwidth per user
    May require more servers
    For private and public networks
    Origin Server
  • Multicast
    A single one-to-many stream
    Uses bandwidth of only one stream
    Requires multicast-enabled networks
    Typically requires fewer servers
    Origin Server
  • Movie
    Unicast Methods
    HTTP Adaptive Streaming
    Video @ 01:04?
    Video @ 01:06?
    Video @ 01:08?
    Traditional Streaming
    Progressive Download
  • Internet Streaming Challenge
    Traditional streaming designed for efficient media delivery
    Works well in small networks but on the Internet suffers from serious drawbacks:
    Doesn’t scale to today’s Internet audience
    Doesn’t take into account today’s Internet structure and organization
    Designed for network conditions less volatile than Internet traffic
  • HTTP Adaptive Streaming
    Hybrid media delivery method
    Acts like streaming but is in fact a series of short progressive downloads
    Video and audio delivered as series of small files over HTTP
    Built for the Web
    Leverages existing HTTP caches
    Scales exceptionally well to meet high demand
    Resilient to network unpredictability
    Client can seamlessly switch video quality and bit rate based on perceived network bandwidth and CPU resources
    Applicable to both on-demand and live delivery
  • Windows Server Media Delivery
    IIS Media Services
    • Traditional Streaming
    • Unicast
    • WMS RTSP
    • WMS HTTP
    • Multicast
    • WMS Multicast
    • Progressive Download
    • Bit Rate Throttling
    • HTTP Adaptive Streaming
    • Smooth Streaming
  • Smooth Streaming
    Smooth Streaming
    Microsoft implementation of HTTP-based adaptive streaming
    Best of both worlds
    Responsive and efficient like streaming
    Cheap and scalable like progressive download
    Superior user experience
    No buffering, no stutter
    Instant start, instant seek
    Seamless bit rate switching based on bandwidth and CPU
  • Multi Bit Rate Revisited
    2.4 Mbps
    1.5 Mbps
    700 kbps
    300 kbps
  • Adapting Bit Rate in Real-Time
    300K @ 00:00?
    700K @ 00:02?
    2.4M @ 00:04?
    1.5M @ 00:06?
    2.4M @ 00:08?
    300K (start quickly)
    700K (good network)
    2.4M (great network)
    1.5M (glitch)
    2.4M (play on…)

    Bit Rate Heuristics
  • Smooth Streaming Design
    Smooth Streaming File Format based on MP4 (ISO Base Media File Format)
    Video is encoded and stored on disk as one contiguous MP4 file
    One file per bit rate
    Each video Group of Pictures (GOP) is stored in a separate Movie Fragment box
    This allows accurate fragmentation at key frames
    Contiguous file is virtually split up into chunks when responding to a client request
  • Smooth Streaming File Format
    File Type (ftyp)
  • On-Demand Presentations
    MP4 files containing video/audio/data fragments
    New file extensions instead of *.mp4
    *.ismvfor files containing video and audio tracks, or only video tracks
    *.ismafor files containing only audio tracks
    On-demand server manifest file
    File extension: *.ism
    Describes the relationships between media tracks, bitrates and files on disk
    Uses SMIL 2.0 XML format
    Client manifest file
    File extension: *.ismc
    Describes available streams, codecs, bitrates, resolutions, metadata
    Uses XML format
  • On Demand Server Manifest
  • On Demand Client Manifest
  • Client-Server Communication
    All requests to the server are formed as RESTful URLs
    Client always first requests the client manifest:http://iis.net/video.ism/Manifest
    After parsing the manifest the client begins making requests for fragments by indicating bit rates and offset times it learned from the manifest:http://iis.net/video.ism/QualityLevels(350000)/Fragments(video=40040000)http://iis.net/video.ism/QualityLevels(48000)/Fragments(audio=62293333)
    Every valid HTTP request returns a file of MIME type “video/mp4”
  • Serving Fragments On Demand
    IIS7 Smooth Streaming handler interprets the RESTful URL request
    It maps the quality level to a physical ISMV file based on the server manifest (*.ism) and calculates the fragment location within the contiguous ISMV file based on the time offset
    Because the wire format (fragment) is just a subset of the disk format (MP4), the extraction process is simple
    No re-muxing necessary
    Dynamic stream switching logic is fully implemented in Silverlight application code – no server-side detection
  • Demo
    Setting up On Demand publishing points
    Using Fiddler to monitor Smooth Streaming traffic
  • Live Smooth Streaming
    Live publishing point identified by *.isml extension
    Client-server communication very similar to on-demand
    Server continually appends MP4 file with new fragments from encoder and makes them available to client
    This allows DVR-like seeking within the existing archive
  • Encoder-Server Communication
    Encoder pushes fragmented MP4 stream to server in body of a long-running HTTP POST requesthttp://iis.net/live.isml/stream(720p)
    Encoder inserts Live Server Manifest Box into start of MP4 stream
    Contains a SMIL-formatted Live Server Manifest very similar to On-Demand Manifest
    Describes all tracks in that encoder’s stream
  • Encoder-Server Communication
    Each sent fragment contains a box with absolute time and duration
    Timestamps are typically derived from the input source Linear Time Code (LTC)
    If multiple encoders are used to encode the same video source, it’s essential that a time code generator is used to keep their LTC in sync
  • Serving Live Fragments
    Server parses the encoder manifests and starts collecting MP4 fragments
    Server builds a cumulative runtime index in memory for all incoming fragments
    Server stores fragments into a temporary DVR ISMV or a permanent ISMV archive
    When the pub point is shut down, server generates ‘mfra’ index box for ISMV archive
  • Live Client Workflow
    Client requests the latest manifesthttp://iis.net/live.isml/Manifest
    Server adds up all the encoder manifests and the runtime fragment index and sends the latest version to the client
    Every fragment contains information about the next fragment
    No need to repeatedly download manifest
    Client builds its own cumulative runtime index based on the lookahead info in the fragments
  • Demo
    Basic Live Smooth Streaming workflow
  • Live Smooth Publishing Points
    ISML files are SMIL 2.0 formatted XML configuration files
    Live publishing points can be created via the IIS Manager UI or by manually creating ISML files on server
    Both push and pull publishing models are supported
    Live publishing points can also push to and pull from other live publishing points
  • Live Publishing Point – IIS Mgr
  • Live Publishing Point - ISML
  • ISML Syntax
    Push: source streams will be pushed to pub point
    Pull: pub point will pull streams from listed URLs
    Streams: pub point can serve streams to other live pub points
    Fragments: clients can request fragments
    Archives: archiving is enabled
    lookaheadChunks: number of fragments server will buffer before making them available to clients
    manifestWindowLength: length of DVR moving window
    archivePath: local path where archives will be written
  • Ingest Server
    Origin Server:
    Realtime Stream
    Ingests and Origins
    Origin Server:
    Delayed Stream
    Origin Server:
    Delayed Stream
    Origin Server:
    Realtime Stream
    8 Mbps
    Origin Server:
    VOD Archive
    Origin Server:
    VOD Archive
  • DEMO
    Advanced Live Smooth Streaming workflow
  • Network Design Considerations
    Bandwidth, bandwidth, bandwidth
    Encoder egress bit rate =
    (All video and audio bit rates together)
    (Number of ingest publishing points)
    Leave enough headroom for bit rate spikes and bursts
    Play it: leave 50%-100% headroom
  • Building Smooth Streaming Origins
    Make sure NICs can handle the cumulative ingress and egress bandwidth
    CPU and memory
    Follow typical server configuration guidelines
    Live Smooth Streaming can require high read/write rates
    10,000 RPM disks are a good idea
  • Redundancy and Failover
    Active-active redundant encoder configuration not supported
    Encoder hot swap is possible as long as encoder doesn’t signal end-of-stream to server
    Configurable time-out periods can help survive temporary loss of network connectivity
    Some encoders support pushing to multiple publishing points concurrently, while others do pub point roll over in case of failure
  • Redundancy and Failover
    IIS is designed to discard duplicate fragments
    This allows ingest and origin servers to be set in active-active configuration
    Ingest server:
    Origin server
    Ingest server:
  • Redundancy and Failover
    CDNs can dynamically change DNS to redirect traffic to secondary origin when primary fails
    Ingest server:
    Origin server:
    Ingest server:
    Origin server:
  • Video Encoding
  • Smooth Streaming Encoders
    Many vendors to choose from:
    Anystream (Grab Networks)
    Digital Rapids
    Inlet Technologies
    Rhozet (Harmonic)
  • H.264 or VC-1?
    Customers should choose whichever codec best fits their encoding workflow and meets their quality requirements
    However, keep in mind:
    H.264 decoding is typically more CPU intensive than VC-1 decoding for the same resolution and frame rate
    In Silverlight 3: H.264 decoding requires about 15-25% more CPU time than VC-1 decoding with similar content properties
    Good rule of thumb (for now):
    For HD video, use VC-1 to reach largest audience
    For SD video and smaller, use H.264 if quality gains are noticeable
  • Encoding for Smooth Streaming
    Use VC-1 Advanced Profile or H.264 Main/High Profile
    2 second key frame distance works very well
    Must be closed GOP
    Client heuristics work best with fixed length (non-adaptive) GOP
    Set video buffer size to 2x-3x key frame distance
    Use WMA Professional audio codec for best quality at low bitrates
    Comparable to HE-AAC – excellent stereo quality even at 48 kbps
    Client heuristics are currently tuned for CBR content, so if using VBR set the peak bit rate within 15% of average bit rate for best playback
  • Multi Bit Rate Encoding Revisited
    2.5 Mbps
    1.5 Mbps
    750 kbps
    350 kbps
  • Variable Resolution
    Why do we vary encoded resolution together with bit rate?
    Encoding the same resolution at inappropriately low bit rates introduces objectionable compression side effects into the video: blockiness, twirling details, color smearing, etc.
    By lowering the resolution proportionally to the bit rate we maintain a consistent level of compression quality in exchange for giving up some visual detail
    It’s a compromise: between two evils, customers prefer a blurry picture over a blocky picture
  • VC-1: 640x360 at 1250 kbps
  • VC-1: 640x360 at 300 kbps
  • VC-1: 288x160 at 300 kbps
  • How Many Bit Rates Do I Need?
    You can use as few as you’d like – even just a single bit rate
    Upper bound is typically set by the encoder
    Anything more than 12 video bit rates is probably overkill
    4 video bit rates for SD and 6 bit rates for 720p HD are good starting points
    Silverlight client currently only supports 1 audio bit rate
  • Choosing the Quality Levels
    Smooth Streaming Multi Bit Rate Calculatorhttp://alexzambelli.com/WMV/MBRCalc.html
  • Choosing the Quality Levels
    Recommended minimum quality level:320x176 at 350 kbps
    Never set the minimum quality level below 288x160 or 300 kbps
    Small video = Bad full screen experience
    Compression is most efficient when width and height are multiples of 16
    One of the resolutions should match the player video window size
  • NBC Sunday Night Football
    VC-1 Advanced Profile
    WMA 10 Professional
    48 kbps 48 kHz 16-bit stereo
  • Configuring Live Encoders
    Live encoders have finite CPU resources
    Higher resolution = More CPU cycles
    Overloading a live video encoder will cause poor image quality and dropped frames
    Most live encoders are bound by number of CPU cores
  • Configuring Live Encoders
    Good rule of thumb for live VC-1 encoding:For each resolution you’d like to use, count the number of CPU cores needed per this table:
    If total number of needed cores exceeds the number of CPU cores in your encoder, reduce the number of quality levels until they fit
  • Distributed Live Encoding
    Some live encoders (Inlet Spinnaker) support distributing bit rates of the same video across multiple encoders
    Muse use LTC time code generator for sync
    Always attach audio stream to the video stream with lowest bit rate
    Make sure encoder load is distributed evenly between encoders
    Balance based on pixel count
  • CBS March Madness
  • Live Encoding Tips & Tricks
    Most live Smooth Streaming encoders can’t handle above 720p
    If source is 720p50 or 720p59.94, don’t forget to set frame rate to half
    If source is 1080i, it’s faster to do single field resizing (bobbing) instead of full frame deinterlacing
    If source is 480i or 576i, use full frame deinterlacing, preferably motion adaptive
    Super Sampling scaling gives best quality
    Denoising filters can improve compressed quality
  • Expression Encoder 3: Smooth VC-1
    Required Settings:
    Profile: VC-1 Advanced Profile
    Adaptive GOP: Disabled
    Closed GOP: Enabled
    Output Mode: Elementary Stream Sequence Header
    Create separate file per stream: Enabled
    Recommended Settings:
    Resize Mode: Stretch
    Adaptive Dead Zone: Conservative
    In-Loop: Enabled
    Overlap: Enabled
    B-Frame Number: 1
    Search Range: Adaptive
  • Expression Encoder 3: Smooth H.264
    Required Settings:
    Profile: H.264 Main or Baseline
    Create separate file per stream: Enabled
    Recommended Settings:
    Profile: H.264 Main Profile
    Resize Mode: Stretch
    Number of Reference Frames: 4
    B-Frame Number: 2
    Search Range: 256
    Entropy Mode: CABAC
    In-Loop Filter: Enabled
  • DEMO
    Configuring Inlet Spinnaker HD
  • The More You Know…
    Related MIX10 Sessions:
    Microsoft Silverlight "Media" : Moving at 60fps
    Tuesday 11:30 am
    Introducing the Silverlight Rough Cut Editor
    Tuesday 1:30 pm
    Smooth Streaming Live in HD: From Camera to Screen
    Tuesday 3:00 pm
    Smooth Streaming Live in HD: 2010 Olympic Winter Games
    Tuesday 4:30 pm
  • That’s All!
    Please fill out the evaluation forms!
    Enjoy MIX!
  • © 2010 Microsoft Corporation. All rights reserved. Microsoft, Windows, Windows Vista and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries.
    The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.