1.
Building a Dash-264 Player Jeff Tapper / Michael Labriola Digital Primates jtapper@digitalprimates.net mlabriola@digitalprimates.net Twitter: @jtapper / @mlabriola

2.
Jeff Tapper• Senior Consultant at Digital Primates – Building next generation client applications• Built video applications for clients including …• Developing Internet applications for 17 years• Author of 12 books on Internet technologies

3.
Who am I?• Michael Labriola• Senior Consultant at Digital Primates – Building next generation client applications• Built video applications for clients including …well lots…• Developing Internet applications for 17 years• Author of 5 books on Internet technologies

4.
Agenda• Video and the Internet today• Understanding HTTP Streaming• What is DASH• Flash as a video platform• The inner workings of DASH• The challenge• Getting video bytes info flash• Introducing the Digital Primates DASH-264 Player• Questions

5.
Video is dominating the Internet• Desktop: Video makes up 50% of traffic at peak periods – notably 30% from Netflix and 11% from YouTube• Mobile: Video traffic is growing exponentially Fixed Internet Mobile Internet

6.
Encode each segment HTTP Adaptive Streaming 1 at multiple bitrates Split the video into 2 small segments Client splices together 5 and plays back Make each segment 0010101000 01010 0101010100 01110 0111010001 1010101 3 addressable via a HTTP-URL 0010101000010 10 0101010100011 10 0111010001101 0101 0010101000 0101010100 0111010001 01010 01110 1010101 0010101000 0101010100 0111010001 01010 01110 1010101 Client makes decisionMedia Capture & Encoding 4 on which segment to download Media Origin Servers HTTP Cache Servers Client Devices

7.
HTTP Streaming Landscape• Apple’s HTTP Live Streaming (HLS)• Microsoft’s Smooth Streaming• Adobe’s HTTP Dynamic Streaming (HDS)• And many more…

8.
The challenge• Most agree that HTTP Streaming is the most efficient choice• Different devices support different streaming protocols• No one standard is currently supported ubiquitously• Results in media being served in several different formats to support the broadest range of devices

9.
What is MPEG-DASH DASH – Dynamic Adaptive Streaming via HTTP International open standard, developed and published by ISO Addresses both simple and advanced use cases Enables highest-quality multiscreen distribution and efficient dynamic adaptive switching Enables reuse of existing content, devices and infrastructure Attempts to unify to a single standard for HTTP Streaming

10.
DASH and codecs• The DASH specification is codec agnostic• Any existing or future codec can work with DASH• DASH manifest describes which codec is used – Different codecs store the actual video data differently

11.
DASH264• H.264 is dominant format today• Many vendors and service providers are committed to supporting/enabling DASH264• Provides support for today’s requirements such as DRM• H.264 is backed by rigorous testing and conformance

12.
DASH Industry Forum• Addressing the dramatic growth of broadband video by recommending a universal delivery format that provides end users with the best possible media experience by dynamically adapting to changing network conditions.

13.
DASH Industry Forum• Objectives: – promote and catalyze market adoption of MPEG-DASH – publish interoperability and deployment guidelines – facilitate interoperability tests – collaborate with standard bodies and industry consortia in aligning ongoing DASH standards development and the use of common profiles across industry organizations• Currently 46 members• Visit http://dashif.org for more information

14.
Members

15.
Building a Desktop DASH player• There are many technologies available for playing video in modern web browsers – Flash – SilverLight – HTML5 – Etc.• We focused on Flash for our first DASH player – Largest percentage of users can support it

16.
Building Video Players in Flash• Flash Player includes a native video player – flash.media.Video• Video accepts a source from a Camera or a NetStream – flash.media.Camera – flash.net.NetStream• Netstream supports: – Streaming FLV file – Manually appending FLV Bytes into its buffer

17.
Flash Video (continued)• NetStream and Video classes can be manipulated manually• Open Source frameworks are available for easing the process – Adobe’s Open Source Media Framework (OSMF) – Open Video Player (OVP)• We have chosen to build on OSMF

18.
OSMF• Extensible framework with plugin architecture• Native support for DRM• Contains classes to work with HTTP Streaming technologies – Only HDS is supported natively – Extensible architecture makes it possible to support new technologies

19.
How to DASH in Flash• Extract Video Bytes from DASH segments• Wrap bytes in format understood by flash (FLV)• Append FLV Bytes into Netstream

20.
Understanding DASH structure• Three types of files – Manifest (.mpd) • XML file describing the segments – Initialization file • Contains headers needed to decode bytes in segments – Segment Files • Contains playable media • Includes: – 0…many video tracks – 0…many audio tracks

21.
DASH Manifest• Manifest contains: – Program Info (including title) – 1 or more periods • Periods contain 1 adaptation set per video stream and • Periods contain 1 adaptation set per audio stream • Adaptation Sets contain: – Content Composition nodes (for each video or audio track) – 1 or more Representation node » Each representation describes a single bitrate » Representations contain data on finding the actual segments » Different ways a representation can describe segments

22.
Describing Representations• SegmentBase – Describes a stream with only a single Segment per bitrate – Can be used for Byte Range Requests• SegmentList – A SegmentList will contain a specific list of each SegmentURL (individual HTTP packet with media data) – Can be used for Byte Range Requests• SegmentTemplate – Defines a known url for the fragment with wildcards resolved at runtime to request a segments (see bbb.mpd) – Alternatively, can specify a list of segments based on duration

23.
SegmentList<Representation id="h264bl_hd" mimeType="video/mp4"codecs="avc1.42c01f" width="1280" height="720" startWithSAP="1"bandwidth="514864"> <SegmentList timescale="1000" duration="10000"> <Initialization sourceURL="mp4-main-multi-h264bl_hd-.mp4"/> <SegmentURL media="mp4-main-multi-h264bl_hd-1.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-2.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-3.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-4.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-5.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-6.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-7.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-8.m4s"/>

24.
SegmentTemplate fixed segment duration<AdaptationSet> <ContentComponent id="1" contentType="video"/> <SegmentTemplateinitialization="BigBuckBunny_720p_1800kbps_44khz_track1_dash.mp4"/> <Representation id="1" mimeType="video/mp4“ codecs="avc1.64001f" width="1280" height="720“ startWithSAP="1" bandwidth="1809954"> <SegmentTemplate timescale="1000" duration="13809" media="bbb_seg_BigBuckBunny_720p_1800kbps_44khz_track1$Number$.m4s" startNumber="1"/> </Representation></AdaptationSet>

25.
SegmentTemplate variable segment duration<AdaptationSet group="2" mimeType="video/mp4" par="16:9“ minBandwidth="475000“ maxBandwidth="6589000" minWidth="176" maxWidth="1680" minHeight="99" maxHeight="944“ segmentAlignment="true“ startWithSAP="1"> <SegmentTemplate timescale="1000" initialization="dash/ateam-video=$Bandwidth$.dash" media="dash/ateam-video=$Bandwidth$-$Time$.dash"> <SegmentTimeline> <S t="0" d="4171" /> <S d="2503" /> <S d="2961" /> <S d="2461" /> <S d="2127" r="2" />…

26.
Understanding MP4 – H.264 structure• Each segment is binary, with a series of boxes contained within• First 8 bytes of every box defines – Box type – Number of bytes for the box.

27.
Initialization Segment• FTYP – file type and compatibility• FREE – free space• MOOV – container of all metadata – MVHD – movie header – 1.* TRAK – container for individual stream data • TKHD – track header • MDIA – container for all media info – Lots of child boxes which contain more detailed stream info

28.
Content Segments• STYP - segment type• SIDX - segment index• MOOF -movie fragment – MFHD - movie fragment header – TRAF - track fragment • TFHD - track fragment header • TFDT - track fragment decode time • TRUN - track fragment run• MDAT - media data container (actual media data) – Contains one or more samples (smallest described piece of content)

29.
Pulling it all together• Using OSMF, we wrote a DASH specific IndexHandler and FileHandler• IndexHandler responsible for deciding url of next http request• FileHandler responsible for parsing data from segments and returning FLV Bytes

30.
DashIndexHandler• When OSMF determines a new fragment needs to be loaded, it calls the DashIndexHandler.• When playing, getNextFile is called• When seeking, getFileForTime is called• Both return an HTTPStreamRequest, which OSMF uses to request the proper segment

31.
DashFileHandler• When the requested file is returned, it is handed to the DashFileHandler for processing – initializeProcessing – gets decoder – processFileSegment – asks decoder to process data• We are using a Decoder called MP4Decoder• MP4Decoder returns FLVBytes for the data within the MDAT• processFileSegment returns ByteArray (FLVBytes) to OSMF, for adding to the NetStream buffer

32.
MP4Decoder• processData() – iterates through all boxes in segment – Calls getNextBox()• getNextBox knows how to process each specific box type• When processing MDAT, individual samples are wrapped with FLVBytes• FLVBytes wrap specific binary header around data, as required by NetStream.appendBytes()

33.
Digital Primates DASH Player• Currently our DASH player is not commercially available• We use it as the basis for DASH players we build for our clients• Initially built as a learning exercise to better understand DASH

34.
Current DP Dash Player• We currently support SegmentLists, SegmentTemplates and SegmentBase in manifest – Byte Range Requests only currently supported via Akamai• We support Dynamic Adaptive Streaming for VOD Streams• We have tested with streams up to 1080p• We have tested with streams from several different encoders

35.
DP Dash Player plans• 4Q12 • 1Q13 – Basic Seek Fixes – Initial JavaScript Player – Improved multi-channel – Support for DRM audio support – Support for Enhanced seeking – Support for Live • 2Q13 Streams – Investigate HEVC (h.265) – Continue to test and fix streams from more – Initial Android Player encoding partners • 3Q13 – Initial iOS Player

36.
Resources• GPAC – http://gpac.wp.mines-telecom.fr – Provides baseline test streams – Provides baseline player• MP4Parser – http://code.google.com/p/mp4parser/ – Open Source java project – Allows for display of contents within boxes• DASH Industry Forum – http://www.dashif.org

37.
Questions?