Update on DASH-IF live media ingest protocol, CMAF ingest, HLS ingest, DASH ingest, encoder synchronization, live OTT streaming, content delivery

1. DASH-IF Live Ingest
Protocol
Status update of the DASH-IF Live
Media Ingest protocol
DASH-Industry Forum 9th of October
2020, special session
Rufael Mekuria (PhD), Jamie Fletcher

2. Streaming Architecture: ingest in HTTP territory
Encoder Origin CDNPackager
DRM Client
Content Encoding Content Preparation
Content Delivery
CPIX
Key Exchange
HLS/
DASH
HLS/DASH
MPD
proxy
MPEG-2 TS
SDI
Smpte ST 2110
mpd
enhancement
at the edge
profiles
DASH/HLS
* Based on DASH-IF architecture for low latency
Ingest interface 1 Ingest interface 2
Content distribution

3. Why a new protocol for live ingest ?
RTMP/Smooth becoming deprecated (no HEVC, VVC, HDR, CMAF, MPEG-H)
Fmp4/CMAF becoming the dominant media segment format for OTT
-> CMAF ingest makes sense
Push Based DASH and HLS are common but under specified -> spec needed
Timed metadata & content splice information -> enable JITP & ad Insertion
Fault tolerance, low latency and redundancy for large scale streaming
(requires encoder synchronization)
Industry Convergence, document current best practices

4. Requirements DASH-IF Live ingest
https://github.com/unifiedstreaming/fmp4-ingest/blob/master/archive/requirements.md
General Requirements (G)
•fMP4 ingest shall use clock synchronization between streams, preferably based on UTC timestamps form the original (e.g. SDI) signal
•fMP4 ingest shall be based on MPEG technologies and container formats
•fMP4 ingest shall support low latency media source ingest
•fMP4 shall support redundant robust workflows with multiple sources and multiple processing entities/publishing points
•fMP4 ingest shall be supported by an openly available reference implementation
// added by me to make it more clear that live ingest needs to support active media processing and not only pass through
•fMP4 ingest shall support further media processing in the cloud or network for value added presentations
•fMP4 ingest shall support re-encryption, content stitching, live to VoD conversion, graphics overlay processing based on accurate timing information downstream
•fMP4 ingest shall support re-generation of the manifest at the publishing point or source node to support different streaming protocols and personalized presentations.
•fMP4 ingest shall decouple the delivery between the live encoder and the processing node from the processing node and the client
•fMP4 ingest shall support ingesting to content delivery networks operating in pass through mode
•fMP4 ingest shall support ingesting to active media processing origins that perform active media processing
Requirements for source media content (S)
•fMP4 ingest specification shall support ingest of MPEG-H and MPEG-4 media including HEVC, AVC, AAC, MPEG-H
•fMP4 ingest specification shall support ingest of media in a media format following the fragmented MPEG-4 format containing moof and mdat boxes
•fMP4 ingest should support encrypted media content based on commonly deployed schemes such as specified in MPEG common encryption
•fMP4 shall support ingest of multiple tracks to accomodate for different languages, audio etc.
Requirements on ingest for timed meta data and auxiliary information (M)
•fMP4 ingest shall support timed meta data such as based on ID3 and SCTE-35, DASH e messages
•fMP4 ingest shall support ingest of captioning information
•fMP4 ingest shall support ingest information suitable to different streaming protocols such as MPEG DASH, Apple HLS and Microsoft Smooth
Requirements on ingest media transport protocol (P)
•fMP4 ingest shall support push based transmission of live stream events
•fMP4 ingest shall support reconnection procedure in case of a disconnect
•fMP4 ingest shall support secure connections
•fMP4 ingest shall support a method for authentication of the contributor
Requirements on media ingest deployments and workflows (W)
•fMP4 ingest spec shall specify failover and restart procedures to gracefully restart in case of failovers
•fMP4 ingest spec document shall specify best practices for media source redundancy and service redundancy (i.e. continuation of the live event after failure of media source or service node)
•fMP4 ingest specification shall support graceful teardown of ingest of live stream events

5. PULL versus PUSH in streaming
Origin
Server
HTTP POST a.mpd
HTTP POST rep1/rep1.init
HTTP POST rep1/seg1.mp4
HTTP GET rep1/rep1.init
DASH Ingest
Pull Based
HTTP GET a.mpd
(Live) ABR
encoder
Smart
Server
HTTP POST moov
HTTP POST moof + mdat
HTTP POST moof + mdat
.....
Fmp4 Ingest
Just in
Time
Packager
HTTP GET rep1/seg1.mp4
HTTP GET rep1/rep1.init
HTTP GET a.mpd
HTTP GET rep1/seg1.mp4
Push Based
(Live) ABR
encoder
DASH
Client
DASH
Client
Get the url right

6. Difficulties with PUSH based (HTTP POST + DASH)
- Under-specified
- How to handle upstream errors in (different origin/BaseURL must have the same
content!)
- Certain values like TimeShiftBuffer, availability times etc. may only be known at the origin
server
- Posting manifest changes to multiple encoders, avoiding race conditions, like different
origins having different manifests confusing a player
- When to delete the segment from the origin
- Detecting disconnects of the encoder (in case of short running post)
- See DASH as ingest format dash as ingest format issue
- So further specification is needed

7. Common Media Application Format (CMAF)
- https://www.iso.org/standard/71975.html
- Fragmented mpeg-4 based specification supported by Apple, Microsoft
- First edition published early 2018
- Potential sucessor of semi-proprietary ism[vat] formats developed for smooth streaming
- Potential to replace usage of MPEG-2 TS in HTTP Live Streaming
- Format was selected as the main intermediate for media in MPEG Network Based Media
Processing
- Supported by DASH and HLS clients
- Future proof ingest format avoid dependency on specific companies, codecs etc.
- Combination with HTTP POST would be a future proof media ingest protocol/format
- Media profiles developed through CTA and MPEG, allowing quick adoption of new codecs
through media profiles usage

8. DASH-IF Live Media Ingest Specification
(Nov 2018)
Supporting companies (initial):

9. Published in March 2020 by DASH-IF

10. Two interfaces defined for live media ingest
1. Interface 1: CMAF Ingest -> leverage the new CMAF specification
2. Interface 2: DASH/HLS Ingest -> document existing practice, not
excluding using the new CMAF specification

11. General shared ingest requirements
HTTPS, TLS client certificates, Basic authentication, Digest Auth
Harmonized error response codes over HTTP with the other ingest
interface for DASH/HLS
Typically the source resends the CMAF header and last fragment in case of
failure
The protocol is kept simple as a design requirement, however many aspects
that SHOULD be supported are included

12. DASH Ingest (interface 2)
Uniqueness by adding timestamps in the names
File extensions and mime types for media objects
source identification via a HTTP header
Upload order requirement
Relative path requirement
Source optionally also deletes content from origin (HTTP DELETE)
Short running post, URL of segments map to manifest
DASH and HLS modes specified

13. CMAF Ingest 1: basics
Origin
Packager
ABR
Encoder
HTTP POST Video1.cmfv
HTTP POST Video2.cmfv
HTTP POST audio1.cmfa
HTTP POST subt1.cmft
HTTP POST meta.cmfm
DASH/HLS/MSS
Long running HTTP POST request per track
or multiple short running POST request per track
Fragment boundary trigggers receiver, not POST request, at least one CMAF fragment per POST request
http://pub_point/channel1

14. CMAF Ingest 2: track requirements
CMAF tracks: video, audio, timed text, subtitle, timed metadata
One TCP connection per track, short running fixed length POST or
HTTP POST with chunked transfer encoding (no fixed length)
Continuity + increasing fragment timestamp and/or sequence number
Fmp4/CMAF fragment boundaries for retransmission and identification
New optional boxes introduced in CMAF and DASH (styp/emsg/prft)
Support for Low Latency and Ad insertion based workflows
Can support encoder synchronization and redundancy

15. CMAF ingest 3: Switching set signaling
Implicit (mandatory)
CMAF Switching set constraints (implicit and mandatory)
Defined by CMAF specification, identical boxes + content can be used to identify tracks in
switching sets
Explicit (recommended/optional)
Encoder generates switching ids for each unique switching set
Embed switching set id in post_uri by Switching(sw_id) keyword
Embed switching set id in kind box in udta
schemeIdUir urn:dashif:ingest:switchingset_id value = sw_idnd level
Optionally be a manifest (e.g. using DASH CMAF profile)

16. Redundancy and hot failover (I1)
Live Source 1 Live Source 2
Origin 1 Origin 2
CMAF ingestCMAF ingest
SDISDI
Clock
Sync.
Live Source 3#
new
1. Redundant ingest/cross post
2. Procedures for
starting new instance
(send new init segment)
3. Option to join in sync without
Explicit communication
DASH/HLS/SMOOTH DASH/HLS/SMOOTH
Contribution source
SDI
Encoder Synchronization
Fixed segment duration and fixed anchor
Segment Boundaries tfdt: Anchor + K * seg_dur
Anchor can be based on epoch or timestamps from
Input to encoder
Any encoder joining can start fragment boundary at
K *seg_dur + anchor as close to current time
All fragment boundaries are K*seg_dur + anchor generating
A shared CMAF/ISOBMFF timeline
Anchor
Fixed_segment
duration

17. Splice conditioning
SCTE-35 urn:scte:scte35:2013:bin embedded in event message track Presentation time corresponds to SCTE-35 splice point
CMAF content conditioning for splice point: insert IDR frame or fragment boundary (splice conditioned encoding or
packaging)
Document in DASH-IF ad insertion too

18. Reference Software interface 1
Live ingest of CMAF files using CMAF ingest profile C++ reference
https://github.com/unifiedstreaming/fmp4-ingest/tree/master/ingest-tools
fmp4Ingest (ingest cmaf files using CMAF ingest)
Tools for converting timed metadata track (still in progress)
File Based ingest, includes wvtt tracks, scte-35 tracks etc.
ACM publication (MMSys): https://dl.acm.org/doi/abs/10.1145/3339825.3394933
Demo with Unified Origin and Docker Compose
https://github.com/unifiedstreaming/cmaf-ingest-demo (cmaf ingest)
https://github.com/RufaelDev/live-demo/tree/cmaf_ingest (ffmpeg)

19. Interface 1 & @ ffmpeg Try it yourself !
• Both DASH ingest and CMAF ingest are in FFMPEG (work by fflabs), ffmpeg
can ingest CMAF and/or DASH to your origin server!
• Demonstration by fflabs: https://gitlab.com/fflabs (includes ffmpeg, origin
and command line shell script) and nginx/node.js
• Docker based demo CMAF ingest (interface 1) ffmpeg & Apache :
• https://github.com/unifiedstreaming/live-demo-cmaf
• Docker based demo CMAF ingest (interface 1) + multiple encoders:
• https://github.com/unifiedstreaming/live-demo-cmaf/tree/encoder_sync
• Docker based demo DASH/HLS ingest (interface 2) ffmpeg & Apache :
• https://github.com/unifiedstreaming/live-demo-cmaf/tree/dash_hls

20. What it looks like

21. Interface 1 & encoder sync demo
• Two encoders posting CMAF fragments (interface 1)
• Fixed segment duration, fixed anchor (e.g. epoch, UTC time)
• Encoder triggers segment boundary at each t = anchor + K * seg_dur with t being
close to current time and K a positive integer
• Use BaseMediaDecodeTime resulting in a shared ISOBMFF/CMAF timeline
• One encoder can fail and can then rejoin later using the same algorithm, resending
initialization segments and potentially resending some older segments (1-2), at anchor
+ N * seg_dur where N is a positive integer and close to now
Play this demo in your own environment:
• synched encoders
• Container id of encoder A is displayed
• When encoder A fails (is killed), the segment of the redundant encoder is used
• The container of encoder B is displayed
• Encoder can A rejoins, when redundant encoder fails is killed the new id will be
displayed

22. Acknowledgements
Special thanks to all contributors in the DASH-IF

23. Open Issues Interface 1
By alligning to CMAF, missing features from CMAF are also
missing, some examples:
adding or removing a representation
Sequencing or presentations
Handling explicit gaps/sparsity (missing segments, sparse
segments), for example because of loss of input
But when these are addressed in CMAF it will be easy to
update the specification accordingly

24. Open Issues Interface 2
Having a good mapping between the manifest and media formats
URL generation recommendations (structure)
Redundancy/sync and race conditions
Include CMAF support
Recommending Using DASH CMAF profile might be a way forward

25. Thank you