Slides from Streaming Media West Presentation, "How to Create A DASH-264 player for Adobe Flash"

1. Building a Dash-264 Player
Jeff Tapper / Michael Labriola
Digital Primates
jtapper@digitalprimates.net
mlabriola@digitalprimates.net
Twitter: @jtapper / @mlabriola

2. Jeff Tapper
• Senior Consultant at Digital Primates
– Building next generation client applications
• Built video applications for clients including
…
• Developing Internet applications for 17 years
• Author of 12 books on Internet technologies

3. Who am I?
• Michael Labriola
• Senior Consultant at Digital Primates
– Building next generation client applications
• Built video applications for clients including
…well lots…
• Developing Internet applications for 17 years
• Author of 5 books on Internet technologies

4. Agenda
• Video and the Internet today
• Understanding HTTP Streaming
• What is DASH
• Flash as a video platform
• The inner workings of DASH
• The challenge
• Getting video bytes info flash
• Introducing the Digital Primates DASH-264 Player
• Questions

5. Video is dominating the Internet
• Desktop: Video makes up 50% of traffic at peak periods
– notably 30% from Netflix and 11% from YouTube
• Mobile: Video traffic is growing exponentially
Fixed Internet Mobile Internet

6. Encode each segment
HTTP Adaptive Streaming
1 at multiple bitrates
Split the video into
2 small segments
Client splices together
5 and plays back
Make each segment
0010101000
01010
0101010100
01110
0111010001
1010101 3 addressable via a HTTP-URL 0010101000010
10
0101010100011
10
0111010001101
0101
0010101000 0101010100 0111010001
01010 01110 1010101
0010101000 0101010100 0111010001
01010 01110 1010101
Client makes decision
Media Capture & Encoding 4 on which segment to
download
Media Origin Servers HTTP Cache Servers Client Devices

7. HTTP Streaming Landscape
• Apple’s HTTP Live Streaming (HLS)
• Microsoft’s Smooth Streaming
• Adobe’s HTTP Dynamic Streaming (HDS)
• And many more…

8. The challenge
• Most agree that HTTP Streaming is the most
efficient choice
• Different devices support different streaming
protocols
• No one standard is currently supported
ubiquitously
• Results in media being served in several
different formats to support the broadest
range of devices

9. What is MPEG-DASH
 DASH – Dynamic Adaptive Streaming via HTTP
 International open standard, developed and
published by ISO
 Addresses both simple and advanced use cases
 Enables highest-quality multiscreen distribution
and efficient dynamic adaptive switching
 Enables reuse of existing content, devices and
infrastructure
 Attempts to unify to a single standard for HTTP
Streaming

10. DASH and codecs
• The DASH specification is codec agnostic
• Any existing or future codec can work with
DASH
• DASH manifest describes which codec is used
– Different codecs store the actual video data
differently

11. DASH264
• H.264 is dominant format today
• Many vendors and service providers are
committed to supporting/enabling DASH264
• Provides support for today’s requirements
such as DRM
• H.264 is backed by rigorous testing and
conformance

12. DASH Industry Forum
• Addressing the dramatic growth of broadband
video by recommending a universal delivery
format that provides end users with the best
possible media experience by dynamically
adapting to changing network conditions.

13. DASH Industry Forum
• Objectives:
– promote and catalyze market adoption of MPEG-DASH
– publish interoperability and deployment guidelines
– facilitate interoperability tests
– collaborate with standard bodies and industry
consortia in aligning ongoing DASH standards
development and the use of common profiles across
industry organizations
• Currently 46 members
• Visit http://dashif.org for more information

14. Members

15. Building a Desktop DASH player
• There are many technologies available for
playing video in modern web browsers
– Flash
– SilverLight
– HTML5
– Etc.
• We focused on Flash for our first DASH player
– Largest percentage of users can support it

16. Building Video Players in Flash
• Flash Player includes a native video player
– flash.media.Video
• Video accepts a source from a Camera or a
NetStream
– flash.media.Camera
– flash.net.NetStream
• Netstream supports:
– Streaming FLV file
– Manually appending FLV Bytes into its buffer

17. Flash Video (continued)
• NetStream and Video classes can be
manipulated manually
• Open Source frameworks are available for
easing the process
– Adobe’s Open Source Media Framework (OSMF)
– Open Video Player (OVP)
• We have chosen to build on OSMF

18. OSMF
• Extensible framework with plugin architecture
• Native support for DRM
• Contains classes to work with HTTP Streaming
technologies
– Only HDS is supported natively
– Extensible architecture makes it possible to
support new technologies

19. How to DASH in Flash
• Extract Video Bytes from DASH segments
• Wrap bytes in format understood by flash
(FLV)
• Append FLV Bytes into Netstream

20. Understanding DASH structure
• Three types of files
– Manifest (.mpd)
• XML file describing the segments
– Initialization file
• Contains headers needed to decode bytes in segments
– Segment Files
• Contains playable media
• Includes:
– 0…many video tracks
– 0…many audio tracks

21. DASH Manifest
• Manifest contains:
– Program Info (including title)
– 1 or more periods
• Periods contain 1 adaptation set per video stream and
• Periods contain 1 adaptation set per audio stream
• Adaptation Sets contain:
– Content Composition nodes (for each video or audio track)
– 1 or more Representation node
» Each representation describes a single bitrate
» Representations contain data on finding the actual segments
» Different ways a representation can describe segments

22. Describing Representations
• SegmentBase
– Describes a stream with only a single Segment per bitrate
– Can be used for Byte Range Requests
• SegmentList
– A SegmentList will contain a specific list of each
SegmentURL (individual HTTP packet with media data)
– Can be used for Byte Range Requests
• SegmentTemplate
– Defines a known url for the fragment with wildcards
resolved at runtime to request a segments (see bbb.mpd)
– Alternatively, can specify a list of segments based on
duration

23. SegmentList
<Representation id="h264bl_hd" mimeType="video/mp4"
codecs="avc1.42c01f" width="1280" height="720" startWithSAP="1"
bandwidth="514864">
<SegmentList timescale="1000" duration="10000">
<Initialization sourceURL="mp4-main-multi-h264bl_hd-.mp4"/>
<SegmentURL media="mp4-main-multi-h264bl_hd-1.m4s"/>
<SegmentURL media="mp4-main-multi-h264bl_hd-2.m4s"/>
<SegmentURL media="mp4-main-multi-h264bl_hd-3.m4s"/>
<SegmentURL media="mp4-main-multi-h264bl_hd-4.m4s"/>
<SegmentURL media="mp4-main-multi-h264bl_hd-5.m4s"/>
<SegmentURL media="mp4-main-multi-h264bl_hd-6.m4s"/>
<SegmentURL media="mp4-main-multi-h264bl_hd-7.m4s"/>
<SegmentURL media="mp4-main-multi-h264bl_hd-8.m4s"/>

24. SegmentTemplate fixed segment
duration
<AdaptationSet>
<ContentComponent id="1" contentType="video"/>
<SegmentTemplate
initialization="BigBuckBunny_720p_1800kbps_44khz_track1_dash.mp4"/>
<Representation id="1"
mimeType="video/mp4“ codecs="avc1.64001f"
width="1280" height="720“
startWithSAP="1" bandwidth="1809954">
<SegmentTemplate timescale="1000" duration="13809"
media="bbb_seg_BigBuckBunny_720p_1800kbps_44khz_track1$Number$.m4s"
startNumber="1"/>
</Representation>
</AdaptationSet>

25. SegmentTemplate variable segment
duration
<AdaptationSet group="2" mimeType="video/mp4" par="16:9“ minBandwidth="475000“
maxBandwidth="6589000" minWidth="176" maxWidth="1680"
minHeight="99" maxHeight="944“ segmentAlignment="true“
startWithSAP="1">
<SegmentTemplate timescale="1000"
initialization="dash/ateam-video=$Bandwidth$.dash"
media="dash/ateam-video=$Bandwidth$-$Time$.dash">
<SegmentTimeline>
<S t="0" d="4171" />
<S d="2503" />
<S d="2961" />
<S d="2461" />
<S d="2127" r="2" />
…

26. Understanding MP4 – H.264 structure
• Each segment is binary, with a series of boxes
contained within
• First 8 bytes of every box defines
– Box type
– Number of bytes for the box.

27. Initialization Segment
• FTYP – file type and compatibility
• FREE – free space
• MOOV – container of all metadata
– MVHD – movie header
– 1.* TRAK – container for individual stream data
• TKHD – track header
• MDIA – container for all media info
– Lots of child boxes which contain more detailed stream info

28. Content Segments
• STYP - segment type
• SIDX - segment index
• MOOF -movie fragment
– MFHD - movie fragment header
– TRAF - track fragment
• TFHD - track fragment header
• TFDT - track fragment decode time
• TRUN - track fragment run
• MDAT - media data container (actual media data)
– Contains one or more samples (smallest described
piece of content)

29. Pulling it all together
• Using OSMF, we wrote a DASH specific
IndexHandler and FileHandler
• IndexHandler responsible for deciding url of
next http request
• FileHandler responsible for parsing data from
segments and returning FLV Bytes

30. DashIndexHandler
• When OSMF determines a new fragment
needs to be loaded, it calls the
DashIndexHandler.
• When playing, getNextFile is called
• When seeking, getFileForTime is called
• Both return an HTTPStreamRequest, which
OSMF uses to request the proper segment

31. DashFileHandler
• When the requested file is returned, it is handed
to the DashFileHandler for processing
– initializeProcessing – gets decoder
– processFileSegment – asks decoder to process data
• We are using a Decoder called MP4Decoder
• MP4Decoder returns FLVBytes for the data within
the MDAT
• processFileSegment returns ByteArray (FLVBytes)
to OSMF, for adding to the NetStream buffer

32. MP4Decoder
• processData() – iterates through all boxes in
segment
– Calls getNextBox()
• getNextBox knows how to process each
specific box type
• When processing MDAT, individual samples
are wrapped with FLVBytes
• FLVBytes wrap specific binary header around
data, as required by NetStream.appendBytes()

33. Digital Primates DASH Player
• Currently our DASH player is not commercially
available
• We use it as the basis for DASH players we
build for our clients
• Initially built as a learning exercise to better
understand DASH

34. Current DP Dash Player
• We currently support SegmentLists,
SegmentTemplates and SegmentBase in manifest
– Byte Range Requests only currently supported via
Akamai
• We support Dynamic Adaptive Streaming for VOD
Streams
• We have tested with streams up to 1080p
• We have tested with streams from several
different encoders

35. DP Dash Player plans
• 4Q12 • 1Q13
– Basic Seek Fixes – Initial JavaScript Player
– Improved multi-channel – Support for DRM
audio support – Support for Enhanced
seeking
– Support for Live
• 2Q13
Streams
– Investigate HEVC (h.265)
– Continue to test and fix
streams from more – Initial Android Player
encoding partners • 3Q13
– Initial iOS Player

36. Resources
• GPAC
– http://gpac.wp.mines-telecom.fr
– Provides baseline test streams
– Provides baseline player
• MP4Parser
– http://code.google.com/p/mp4parser/
– Open Source java project
– Allows for display of contents within boxes
• DASH Industry Forum
– http://www.dashif.org

37. Questions?