1_MWS2018_Tutorial1_Pham_Internet Delivered Media.pdf

Stefan Pham, Daniel Silhavy
May 15, 2018
INTERNET DELIVERED MEDIA – TRENDS & BEST PRACTICES
TUTORIAL 1 – 7TH FOKUS MEDIA WEB SYMPOSIUM

2
1. Foundations of Adaptive Streaming
2. OTT Strategy
3. Advanced Web Media Streaming
AGENDA

3
− DASH – Dynamic Adaptive Streaming over HTTP for live and on demand video; MPEG standard
− SAND – Server and network assisted DASH to enable interoperable metrics and client coordination
− HLS – HTTP Live Streaming for live and on demand video by Apple
− CMAF – Common Media Application Format for HLS and DASH
− CENC – Common Encryption for many DRM & delivery channels
− MSE – Media Source Extension to trick-function HTML5 video-objects via JavaScript (control AV
media streams)
− EME – Encrypted Media Extension to play back DRM-protected media in standard browsers w/o
the use of proprietary plug-ins
− CDM – Content Decryption Module - addition to the browser that provides functionality for one or
more Key Systems
− CPIX – Content Protection Information Exchange Format - standardizes the way entities involved
in the content creation workflow exchange protection information
− VMAF – Video Multi-Method Assessment Fusion- perceptual video quality assessment algorithm
by Netflix
DELIVERING MEDIA: TECH TO UNDERSTAND

Where we are:
− Very popular commercial OTT streaming services, e.g. Netflix, Amazon Video, DAZN, Spotify, …
What we work with and where we start:
− Different device platforms & interoperability – how to reach them all? Best practices?
− Other factors:
− Monetization (DAI),
− Regulatory (DRM, watermarking),
− Quality (UHD, encoding),
− Analytics (streaming metrics),
− UX (low latency, 360°)
4
INTERNET DELIVERED MEDIA - MISSION ACCOMPLISHED?

6
Streaming Formats
Encoding
Web Media APIs
Standards Update
1. FOUNDATIONS OF INTERNET DELIVERED MEDIA

MPEG-DASH
− Standard for streaming multimedia over the Internet
− Dynamic Adaptive Streaming over HTTP (DASH) - ISO/IEC 23009
− Part 1: Media presentation description and segment formats
− Part 2: Conformance and reference software
− Part 3: Implementation guidelines
− Part 4: Segment encryption and authentication
− Part 5: Server and network assisted DASH (SAND)
− Part 6: DASH with Server Push and Web Sockets
− Extensions for common DRM-interoperable encryption and encoding (CENC)
− DASH-IF Interoperability Guidelines
− Different profiles: DVB, HbbTV 1.5/2.0, ATSC 3.0 etc.

MPEG-DASH FILE FORMAT
8
• ISO base media file format (ISOBMFF) or MPEG2-TS
• Codec-agnostic
Box model in ISOBMFF (init + media segment):
File
Type
('ftyp')
Movie Metadata ('moov')
Movie
Header
('mvhd')
Track
('trak')
Movie Extends
('mvex')
Movie
Extends
Header
('mehd')
Track
Extends
('trex')
Movie
Fragment
('moof')
Movie
Fragm.
Header
(’mfhd’)
Track
Fragme
nt (’traf’)
Media
Data
('mdat')

MPEG-DASH - SCOPE
HTTP Server DASH Client
Control Heuristics
Media
Player
HTTP Client
Segment
Parser
MPD Parser
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
MPD
MPD
MPD
MPD
HTTP 1.1
MPD Delivery
DASH Spec covers the red blocks.
[Note: Client’s behavior is only defined to the extend needed for compliancy with MPD
and segment formats.]

− MPEG-DASH 3rd edition reached FDIS (to be published as ISO/IEC 23009-1:2018)
− New Features
− Media presentation description (MPD) can be daisy chained to simplify
implementation of pre-roll ads in cases of targeted dynamic advertising for live
linear services
− support for pre-selection in order to signal suitable combinations of audio elements
that are offered in different adaptation sets
− Availability Time Synchronisation, Spatial Relationship Description (SRD),
generalized URL parameters, Authentication, MPD linking, Callback Event, Period
Continuity
DASH 3RD EDITION

− Draft: IETF RFC 8216
− Mainly used for iPhone, iPad, Mac, AppleTV
− New Features:
− HEVC support
− CENC (‘cbcs’) and f-MP4 / CMAF support
− IMSC1 subtitles
− FairPlay / offline playback
− PHP-like playlist variables
− Synchronized playback of multiple streams (via AVFoundation)
APPLE HTTP LIVE STREAMING (HLS)

COMMON ENCRYPTION (CENC)
CENC
Header
Media File
Body
Key Generator
Audio Video Other
Scrambler
Content Asset
Management
System
Encrypted media data uses the Advanced
Encryption Standard specified by AES using
128-bit keys in Counter mode (AES-CTR) or
Block cipher mode (AES-CBC)
Advanced Web Technologies | Web & Media I | lecture winter term 2015/2016

− Common encryption means the same short fragment of video can be decrypted and
decoded by devices using different DRMs.
− Improved encoding, network cache (origin vs. edge server) and CDN efficiencies
− ISO/IEC 23007-1 (3rd edition) – Common encryption in ISO based media file format files
− Protection schemes: ’cenc’, ’cbc1’, ‘cens’, ‘cbcs’
− ‘cens’ and ‘cbcs’ are pattern encryption schemes
COMMON ENCRYPTION (CENC)

14
© Fraunhofer FOKUS
CENC – MULTI-DRM FOR DASH
‘pssh’ Common Encrypted Media Data
Audio Video Other
DRM
Info B
DRM
Info C
DRM
Info A
DRM
Info D
DRM Client A
DRM
Server A
Content Asset
Management
System
License
DRM Client B
DRM
Server B
License

PROTECTION SYSTEM HEADER BOX
File
Type
('ftyp')
Movie ('moov')
Movie
Header
('mvhd')
Protectio
n System
Specific
Header
‘pssh’ [1]
Protectio
n System
Specific
Header
‘pssh’ [2]
Track
('trak')
Movie Extends
('mvex')
Movie
Extends
Header
('mehd')
Track
Extends
('trex')
Media Segment with multiple ‘pssh’ boxes

16
− ISOBMFF-based adaptive media packaging format for HLS and DASH
− CMAF Chunks for low latency encode, delivery and playback
− Defines encoding of segmented media for delivery and decoding on device
− Published in 01/2018 as ISO/IEC 23000-19
− Does not specify manifest, player or delivery protocol
− Does not mandate encryption scheme for CENC
− Apple „Using CMAF content with HLS“ available
− DASH-IF work on CMAF mapping started
COMMON MEDIA APPLICATION FORMAT (CMAF)

DASH MANIFEST MODEL
MPD
Period 1
Period 2
AdaptationSet 1
AdaptationSet 1
AdaptationSet 2
Representation 1 : Segment 1, Segment 2, Segment 3 …
Representation 2 : Segment 1, Segment 2, Segment 3 …

− Role of the Manifest is divided between the HLS Master Playlist and Media Playlists it
references
HLS MANIFEST MODEL
Master Playlist
english.m3u8
french.m3u8
video.m3u8
uhd-video.m3u8
Media Playlist
Segment1
Segment2
Segment3
…
Media Playlist
Segment1
Segment2
Segment3
…
Media Playlist
Segment1
Segment2
Segment3
…
Media Playlist
Segment1
Segment2
Segment3
…

19
Streaming Formats
Encoding
Web Media APIs
Standards update
1. FOUNDATIONS OF INTERNET-DELIVERED MEDIA

20
WHY DO WE ENCODE? - THE NUMBERS
Compression
320x240@0.235Mbit/s
No compression
w𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 ∗ 𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉 ∗ 𝒇𝒇𝒇𝒇𝒇𝒇 ∗ 𝒃𝒃𝒃𝒃𝒃𝒃𝒃𝒃 𝒑𝒑𝒑𝒑𝒑𝒑 𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑
20x16@0.228Mbit/s
320x240p@55.3Mbit/s

21
DIFFERENT CODECS ON THE MARKET - TIMELINE
Source: https://github.com/leandromoreira/digital_video_introduction

22
DIFFERENT CODECS ON THE MARKET - MARKETSHARE
Which codecs are you using currently? Which video codecs are you planning
to use in the next 12 months?
Source: Video Developer Report 2017 - Bitmovin

23
„Our testing shows AV1 surpasses its stated goal of 30% better compression than VP9, and achieves gains of 50.3%,
46.2% and 34.0%, compared to x264 main profile, x264 high profile and libvpx-vp9, respectively. The new codec requires
longer encoding times vs. current alternatives, however, due to increased complexity.“ – Facebook
„ AV1 producing the same quality as x264 at 55% of the data rate, with x265 running in three-pass
and two-pass Placebo mode at 67% and 69% the data rate respectively... AV1 encoder has extremely low speed—2500-
3000 times lower than competitors. X265 Placebo presets (2 and 3 passes) have 10-15 times lower speed than the
competitors“ – Jan Ozer on Moscow State University results
„ AV1 video quality is pretty much on par with HEVC. The tests found that AV1 performs better than HEVC on the PSNR
and VMAF metrics, but not on subjective testing. Looking at the overall codec efficiency, there is not much difference
compared with HEVC“ – Harmonic, HHI, B<>Com
CODEC DISCUSSIONS: HEVC VS. AV1

24
• Peak Signal-to-Noise Ratio (PSNR)
 Based on mean squared error, calculates differences between
original and compressed frame in decibels
 No direct correlation between score and perceived subjective quality
 Should be between 35-45 dB
• Video Multimethod Assessment Fusion (VMAF)
 Perceptual video quality assessment algorithm developed by Netflix
• Structural similarity Index (SSIM)
 Measures the perceptual difference between two similar images.
VIDEO QUALITY METRICS

25
Streaming Formats
Encoding
Web Media APIs
Standards update

− Today browsers are more than just HTML interpreter and render.
− Common platform across devices
− NPAPI deprecation: Plugins (Flash, Silverlight, ...)
− Some HTML5 Features
− HTML5 <video> and <audio> + extensions
− Text tracks
− Canvas (2d shapes and bitmap images)
− SVG
− Advanced CSS3 features
− Storage/offline applications
− Websockets
− Hardware acceleration (WebGL…)
− …
HTML5
Advanced Web Technologies | Web & Media I | lecture winter term 2016/2017 26
*www.apple.com
*www.google.com
*www.amazon.com
*wwwsony.com

27
© Fraunhofer FOKUS
TYPES OF BROWSER-BASED PLAYBACK
− Type 1: HTML5 <video>
<video id="video" controls width=1280 height=720 src=”video.m3u8"></video>
− Type 2: HTML5 <video> + ABR API
− Type 3: HTML5 <video> +
Media Source Extension (MSE) +
Encrypted Media Extensions (EME)
Source: http://blogs.windows.com/msedgedev/2015/07/02/moving-to-html5-premium-media/

28
MSE SUPPORT IN WEB BROWSERS
http://caniuse.com/#search=media%20source

29
− Current version: Recommendation 18 September 2017
− JS interface between DRM License Server and CDM
− Minor differences across browsers / recent Chrome changes:
− Setting MediaKeySystemMediaCapability.contentType (mimeType+ codecs)
mandatory
− Setting robustness is recommended
− VMP enforced, setServerCertificate to encrypt messages to the license server
− Secure origin and transport / mixed content  require https
W3C ENCRYPTED MEDIA EXTENSIONS

30
EME SUPPORT IN WEB BROWSERS
http://caniuse.com/#search=media%20source

31
− Media Capabilities API
− Standardization of media decoding capabilities APIs
− Replacing isTypeSupported, canPlayType APIs
− Display capabilities: color gamut, luminance
− Media Playback Quality: total frames, corrupted frames, dropped
frames etc.
− Picture-in-Picture (PiP) to create a floating video window
− EME Extensions: HDCP Policy Check, encryption scheme
− Shape Detection API
− …
W3C WEB INCUBATOR COMMUNITY GROUP (WICG)
Source: https://github.com/WICG

32
− Status: CG Draft
− Initiated by CTA WAVE
− ”Web APIs for media web apps that are supported across all four of the
most widely used user agent code bases”
− “encourage manufacturers to develop products that support the APIs in
the most recent version of Web Media API Snapshot”
− Targeting Smart TVs, game machines, set-top boxes
− Some Examples:
− MSE/EME
− Canvas, Fullscreen API
− Web Workers, Service Workers (not required), Indexed DB
W3C WEB MEDIA API SNAPSHOT 2017
Source: https://w3c.github.io/webmediaapi/

33
− In May 2015 IETF standardized HTTP/2
− RFC7540 and 7541
− Main goal to reduce transport overhead
− combine requests (multiplex)
− better data compression that include header information
− server-push (PUSH_PROMISE)
− binary encoding
HTTP/2
index.html
style1.css
style1.css
script1.js
Script2.js
scriptx.js
icon1.png
icon2.png
iconx.png
font.woff2 picture1.jpg
picture1.jpg
picturex.jpg
…
…
…
…
With HTTP/1.1 one HTTP and thus TCP request must be made for each ressource

34
HTTP/2 SUPPORT IN WEB BROWSERS

35
− The web has been largely built around the so-called request/response paradigm of HTTP
− A client loads up a web page and then nothing happens until the user clicks onto the next page
− Around 2005, AJAX started to make the web feel more dynamic but still, all HTTP communication
was steered by the client, which required user interaction or periodic polling to load new data from the
server.
− Technologies that enable the server to send data to the client in the very moment when it knows that new
data is available have been around for quite some time
− They go by names such as "Push" or "Comet“
− One of the most common hacks to create the illusion of a server initiated connection is called long
polling.
− the client opens an HTTP connection to the server which keeps it open until sending response
− Whenever the server actually has new data it sends the response
− However, all of these work-arounds share one problem
− They carry the overhead of HTTP, which doesn't make them well suited for low latency applications
TUB ODS VL "Advanced Web Technologies“: Web Basics
SSE AND WEB SOCKETS

36
− API for opening an HTTP connection for receiving push notifications from a server in the form of DOM
events
− Server Sent Events (SSE) are commonly used to send message updates or continuous data streams to
a browser client
− e.g., Facebook/Twitter updates, stock price updates, news feeds, sport results where no local client
request is the origin of the event
− This is also possible using XHR but the web page would have to ask periodically if any updates were
available.
− Simple message protocol, two types of updates
− “data: This is some updated data message”
− „event: updateGameResult data: 3:5“
− To enable servers to push data to Web pages over HTTP the EventSource Object is used
SERVER SENT EVENTS
var source = new EventSource(“updates.php");
source.onmessage = function(event) {
document.getElementById("result").innerHTML +=
event.data + "<br>";
};
var source = new EventSource(“updates.php");
source.addEventListener(„updateGameResult“, function(event) {
document.getElementById("result").innerHTML += event.data
+ "<br>";
});

37
DYNAMIC WEB: SERVER SENT EVENTS
Load Page Server Sent Events Stream
Using Web Site
Request
Response
create EventSource
Data
Data
HTTP
Event Event
Server
Client close
…

38
− The W3C Web Socket API enables Web pages to use the WebSocket protocol for two-
way communication with a remote host
− with a persistent connection between the client and the server
− where both parties can start sending data at any time
− Servers must be prepared to handle many open socket connections at once
− e.g, node.js, erlang, Jetty
− Use WebSocket whenever a truly low latency, near realtime connection between the
client and the server is needed, e.g., Multiplayer online games or Chat applications and
all where SSE are used
WEB SOCKETS

39
− After a connection to the web socket server is established (when an open event is fired) data can be
send using the send('your message') method on the connection object
− supported messages types
− text messages as string (typically JSON)
− binary messages as Blob or ArrayBuffer
WEB SOCKETS
Basic WebSocket usage
var connection = new WebSocket('ws://html5rocks.websocket.org/echo');
connection.onopen = function () {
connection.send('Ping');
};
connection.onerror = function (error) {
console.log('WebSocket Error ' + error);
};
connection.onmessage = function (e) {
console.log('Server: ' + e.data);
};
WebSockets with JSON
var msg = {
type: "message",
text: document.getElementById("text").value,
id: clientID,
date: Date.now()
};
connection.send(JSON.stringify(msg));
connection.onmessage = function (event) {
processNewMessageHelper(JSON.parse(event.data));
}

40
WEB SOCKETS
Load Page Web Socket Server
Request
Response
create WebSocket
connection
Data
Data
HTTP/s
Event Event
Server
Client
Data
WebSocket Protocoll ws/wss
close
Using Web Site
Event

41
Why?
− No software downloads and no Plug-Ins
− No installations like Skype, WebEx or GotoMeeting
− Solves incompatibilities of browsers for real time communication
− Real Peer-2-Peer Connection between Browsers
− Direct exchange of audio, video, and data between Browsers
For what?
− Communication in generell (Telcos)
− (Existing) Video/Web conference system providers
− Peer 2 Peer Data exchange
− Training Providers, Customer Support
WEB RTC

42
DYNAMIC WEB: WEB RTC
What we need
Image: http://www.html5rocks.com/en/tutorials/webrtc/infrastructure/

43
What we most likely have

44
Using STUN servers to get the public
IP to be communicated via signaling
and trying to create a UDP session
between the clients

45
If a direct Peer-2-Peer connection
cannot be established a TURN media
stream relay server can be used.

46
Streaming Formats
Encoding
Web Media APIs
Standards update

47
WEB APPLICATION VIDEO ECOSYSTEM (CTA WAVE)
Content Specification
• Based on MPEG Common
Media Application Format
(CMAF)
• Compatible with DASH
and HLS.
Testable requirements
• covering most common
playback interoperability
issues.
Reference application
framework
• Based on HTML5
• Provides functional
guidelines for playback
interoperability.
Content HTML5 Reference
Platform
Device Playback
Capabilities
WAVE Test Suite

48
2018 WAVE OUTPUT PLAN
Sep’18 Oct’18
Apr’18 May’1
8
Jun’18 Dec’18
Jul’19 Aug’18 Nov’18
Web Media
Application
Dev
Guidelines
Web Media User Agent Integration Specification
(ongoing)
Test Tools & Test Cases
Content
Specification
Releasing
concurrent
with 2018
NAB Show
Device Playback Requirements Specification Test Tools & Test Cases
Web Media
API
Snapshot
2017
Test Tools & Test Cases

49
DASH-IF 2018
DASH-IF IOP
4.1
V4.0 +
Test database
49
V
DASH-AVC/264 2.0
DASH-IF IOP
3.3
• Live services
• Ad-insertion
• AC-4
• MPEG-H Audio
• HTTPS support
• Key Rotation
• Cross Adaptation
Set Switching
• Callback events
• Period continuity
CPIX V1.0
live Services
for AVC/264
2.0
Conformance
3.3
Test Suite 3.0
UHD/HDR/
Dynamic
Metadata
dash.js v1.3 dash.js v2.0
ATSC 3.0 DASH
Profile V1
•Broadcast TV
profile
•Next Gen
Audio
•Temp scalable
HEVC
•App based
xlink
dash.js v2.5
CPIX V2.0
Content Anno-
tation &
Selection
VP9
CMAF & DASH
Conformance
3.0
Test Suite 3.0
Token Access Control V1.0
Robust linear
Live w lower
latency w DVB
DASH-IF IOP
4.0
V3.3 +
• UHD/HDR
• Alignment w
23009-1 Amd 3 &
4
DRM Support
Improvement
DASH SAND White
Paper
Ad insertion
improvements
SAND Modes
Conformance
4.0
Test Suite 4.0
Dolby Vision
Segment Variants
for Watermarking
Thumbnail
navigation
DASH Metric Position
Paper
dash.js 2.6
W3 Clear Key
Last segment
signaling
DASH-IF IOP V5
ATSC 3.0 DASH
Profile V1.1
V1 +
• HDR
• DASH for NRT
content
Position Paper on uncommon
enc.
Multi-dependent
stream - ESPEX
dash.js 2.6.3
DASH-IF IOP
4.2
• Bugfixes
• Audio
alignments
CPIX
V2.1
USAC IOP
ETSI SPEC

− Different profiles:
− HbbTV 1.5: MPEG DASH ISOBMFF Live Profile
(urn:hbbtv:dash:profile:isofflive:2012)
− HbbTV 2.0.x: DVB DASH profile (urn:dvb:dash:profile:dvb-dash:2014)
− Regional specifications (TNT, HbbTV Forum Nederland, AEDETI etc.) mandate
Marlin Simple Secure Streaming (MS3) or Microsoft PlayReady
− Playback:
− “Type 1 player“
− DRM license acquisition via oipfDRMAgent API or EME (HbbTV 2.0.1)
− What’s new?
− Work on HbbTV/DASH profile validator based on DASH-IF validator
− Reference App for DASH/DRM playback
HBBTV & DASH

52
Cross-platform Deployment
Web, Native or Hybrid Apps?
Best Practices
Multi-DRM backend with CPIX
2. OTT STRATEGY

54
CODEC SUPPORT IN DESKTOP & MOBILE BROWSERS
Browser Platform AVC/H.264 HEVC/H.265 VP9 AV1
Chrome Win Yes No Yes (Yes)
OSX Yes No Yes (Yes)
Android Yes No Yes No
Firefox Win Yes No Yes (Yes)
OSX Yes No Yes (Yes)
Safari >= OSX High Sierra Yes Yes No No
< OSX High Sierra Yes No No No
iOS >= 11 Yes Yes No No
iOS < 11 Yes No No No
Edge Win 10 Yes Yes Yes No

55
© Fraunhofer FOKUS
DRM SUPPORT IN DESKTOP WEB BROWSER
Desktop
Browser
Platform EME/CDM Security Level Flash Player/
Primetime
NPAPI/
Silverlight 5
Chrome Win Widevine SW (Yes) No
OSX (Yes) No
Linux (Yes) No
Firefox Win Widevine SW (Yes) No
OSX (Yes) No
Linux (Yes) No
Safari > macOS 10.10.3
(Yosemite)
Fairplay undefined (Yes) Yes
< macOS 10.10.3 (Fairplay)** - Yes Yes
IE/Edge < Win 7 No - Yes Yes
Win 10 PlayReady SL3000 * (Yes) No
* depends on chipset
** only Netflix

56
© Fraunhofer FOKUS
DRM SUPPORT ON MOBILE
Mobile
Platform
HTML5 DRM Security Level
iOS only EME
HLS (‘cbcs’) via Type 1
playback
FairPlay undefined
Android MSE/EME Widevine SW-HW*
* Depends on chipset

57
© Fraunhofer FOKUS
DRM SUPPORT IN TV/GAME CONSOLES
Platform MSE/EME
(DASH)
DRM
HbbTV 2.0 (Yes) Marlin/ PlayReady
Samsung
(SDK 1.4)
Yes PlayReady, Widevine,
Verimatrix;
DASH, HLS, Smooth
FireTV Yes Widevine, PlayReady
native ExoPlayer Amazon
Port
AppleTV No HLS + Fairplay
Chromecast Yes Dash + PlayReady/
Widevine;
Smooth + PlayReady
HLS + ‘cbcs’
Platform Web Browser
(MSE/EME)
Native DRM
XBox Yes (as Win10
Hosted App)
PlayReady
PS4 No PlayReady(?),
Marlin
RDK Yes
(OCDM)
PlayReady, Widevine
AndroidTV For Google devices – see Android mobile
and Cast, difference only in Security
Levels

58
DASH and HLS
 With CMAF segments HLS and DASH manifests will reference the
same file format
 ‘cenc’ will be needed for legacy devices. Updates to PlayReady and
Widevine will enable ‘cbcs’ support
COMMON FILE FORMAT AND ENCRYPTION MODES
CMAF +
'cbcs'
Desktop -
OS
Consoles:
PS, XBox
TV:
Smart TV
Alliance,
Android TV,
HbbTV, RDK
Mobile:
WM, iOS,
Android
HDMI
Dongles:
Chromecast/
FireTV
3rd party
STBs
Streaming File Format Encryption DRM
DASH ISOBMFF
(CMAF)
‘cenc’ PlayReady PK<4.0,
Widevine
DASH ISOBMFF
(CMAF)
‘cbcs’ PlayReady PK>4.0,
>Android N,
Chromecast
Streaming File Format Encryption DRM
HLS MPEG2TS Sample-AES (AES-
CBC)
FairPlay
HLS ISOBMFF
(CMAF)
‘cbcs’ FairPlay

59
Best Practices
2. OTT STRATEGY

60
BASIC ARCHITECTURE: WEB APP VS. NATIVE
Hardware
Operating System
Application APIs
Browser
HTML JS
CSS SVG
e.g., Internet Explorer, Firefox,
Chrome, Safari
e.g., Windows, MacOS, Linux,
Android, iOS
e.g., iPhone, GalaxySII, PC, TV
Native Applications

61
BASIC ARCHITECTURE: PACKAGED WEB APP
Hardware
Operating System
Application APIs
Packaged Web App
The core functionality of a Browser, is
de-coupled into UI and Runtime to
share its functionalities.
• e.g, webview
The JavaScript API is extended to
provide access to device features.
Web Runtime
HTML JS
CSS SVG
Native
Applications
Script
Extension

62
WEB VS. NATIVE: AN OVERVIEW
 Pro Web
 Reuse of existing skills
 web skills are easier to find and less costly
than native app development skills
 Easy to use with a big user community to ask
 lots of documentation
 hooking into the underlying OS through
technologies known to web developers
 Create an app for multiple platforms at the
same time
 Rapid prototyping using Web Browsers
 Reuse of code and design assets
 Con Web
 Potential compatibility issues with different CSS,
JS, and DOM implementations
 input and output latency may be higher than
with native apps
 low latency makes using smartphones fun
 without attention HTML based apps run the
risk of feeling jerky
 User interface could easily feel inappropriate for
the specific target platform
 Poor performance possible, particularly where
functionality that makes lots of calls to the OS is
used
Web In general

63
 Pro Browser
 No need for application installation
 Web Applications are always up-2-date
 Strong developer community and a fast
adoption curve, reuse of existing skills
 Available on all common operating systems
and devices
 Con Browser
 Only access to a few devices features as
specified by W3C
 Applications does not feel like first class
applications (native)
 The same application as native app can
always provide better performance, more
features, nicer UI, better UX…
Browser usage in general

64
 Pro Packaged
 A packaged web application typically
 is Installable and transportable
 is offline usage capable by design
 and has access to advanced device features (if
implemented natively)
 Packaged web applications can be published
in platform specific application stores which
gains visibility.
 No nice way of searching for web apps
• Con Packaged
• Each installation package is targeting a specific
platform architecture. To support multiple
devices/platforms multiple packages are
needed (but tool chains may solve this issue).
• Native parts need to be re-implemented for
each platform
• The user need to perform application updates
• The application needs to consider that the
user does not always have the latest version
Packaged Apps in general

− New way to build Web Apps by leveraging
− Web & Service Workers API, push
notifications
− Offline capability (e.g. Indexed DB),
− Web App manifest etc.
− “first class app citizens” / Store integration
− Examples
− Microsoft PWA
− Google PWA
− Cross-plattform Desktop: Electron
PROGRESSIVE WEB APPS

66
− When executing scripts in an HTML page, the page becomes unresponsive until the script is
finished
− single threaded, event based
− A web worker is a JavaScript that runs in the background
− independently of other scripts
− without affecting the performance of the page
You can continue to do whatever you want: clicking, selecting things, etc., while the web
worker runs in the background
− A Web Worker script is defined in a separate JavaScript file
− Creating a Web Worker is simple, the code in worker.js is executed directly after creating the
Worker
− As “extension” service workers are currently in development at W3C which allow scripts to
react on events without the web page even being active (e.g., Web Push)
WEB WORKERS
new Worker("worker.js");

67
− web workers are linked to their creator in a 1:1 relationship
− Since web workers are independently from the web applications execution context
communication between the Worker and the main applications is done using messaging
− Web workers do not have access to the following JavaScript objects
− The window object
− The document object
− The parent object
WEB WORKERS
Worker Script
self.addEventListener('message',
function(e) {
console.log(‚Main said: ', e.data);
self.postMessage(e.data);
});
Main Script
var worker = new
Worker(‚worker.js');
worker.addEventListener('message',
function(e) {
console.log('Worker said: ', e.data);
});
worker.postMessage('Hello World');

68
− service worker can be used for
− push notifications
− background sync (offline usage)
− intercept and handle network requests, including
programmatically managing a cache of responses
− A service worker has a lifecycle that is completely
separate from your web page
− either the service worker is terminated
to save memory
− or it handles fetch and message events
that occur when a network request or
message is made from your page
− service worker must be provided using
a HTTPS connection
SERVICE WORKER

69
− The Push API enables sending of a push message
to a webapp via a push service
− An application server can send a push message at
any time, even when a WebApp or user agent is inactive
− Push messages are delivered to a Service Worker
that runs in the origin of the WebApp
− use the provided information to update
the local state or display a notification to the user
SERVICE WORKER: WEB PUSH

• Open-Source framework for creating web-based desktop apps
• MIT License
• Combines frontend and backend Javascript code to single package
• Electron uses Chromium for frontend and Node.js for backend
• „full featured browser and full operating system resource access, e.g. access to filesystem/devices“
• Desktop apps can be built using web technologies / reuse existing website
• Supported platforms macOS 10.9 and later, Windows 7 and later, Ubuntu 12.04 and later
• Packaging/installation/updates via Electron framework
• App can be launched via URL (e.g. electron-app://) from web site
• To optimize performance, parts of the web app can be packaged (e.g. logo, splash screen, libraries
etc.)
© Fraunhofer FOKUS
ELECTRON FOR DESKTOP

• Chrome is based on Chromium => Media Playback in Electron is comparable to Chrome
• MSE/EME is supported => dash.js, shaka-player etc. can be used
• Widevine CDM can be added
• Chromecast API can be integrated
• Modern replacement for out-of-browser Silverlight
• Possibility to add offline playback capability via filesystem
• Standard HTML5 APIs to detect online/offline mode
• Media is stored in filesystem or HTML5 storage
• Widevine persistent licenses needed
© Fraunhofer FOKUS
ELECTRON MEDIA PLAYBACK

Pros Cons
Modern desktop app to replace out-of-
browser Silverlight
Big installation package (browser is
built-in), heavy on resources (memory,
CPU)
Reuse of existing HTML5 web app Users need to download and install a
package to start the app
Same media playback behavior as in
Chrome
No platform DRM security (same
security level as in Chrome)
Not subject to Chrome browser
updates – updates are controlled by
app provider
Security updates are only integrated
into Electron, after Chromium has
been fixed (appr. 2 week delay)
© Fraunhofer FOKUS
ELECTRON PROS/CONS

73
Best Practices
2. OTT STRATEGY

74
− Open Source
− dash.js, shaka-player, hls.js, exoplayer etc.
− MSS and HLS client-side conversion to
DASH/ISOBMFF + CENC
− Hasplayer (MSS/HLSDASH)
− hls.js (HLSDASH)
− dash.js (MSSDASH)
− Shaka-player (HLSDASH)
− Customized versions for specific target platforms
(e.g. hls.js-light, exoplayer-amazon-port)
− All players are production-ready and used in many
commercial projects
WIDE AVAILABILITY OF OPEN SOURCE DASH/HLS PLAYERS

75
BEST PRACTICES: MANIFEST FILES
− Large manifests can lead to long parsing times and an therefore increased startup time

76
− Improve parsing performance and therefore improve startup time by
− Decreasing Segment and Manifest size
 (DASH MPD) Use SegmentTemplate instead of SegmentTimeline or
SegmentList
 Remove „old“ segments from the manifest according to the DVR window
 Reduce segment length/size (for client-side conversion)
− Player improvements
 Check for latest updates of the libraries (e.g. for XML, box parsing)
 Move computational heavy tasks to separate threads (e.g. WebWorker)
 Check for type 1 players to utilize native parsing
 Limit manifest updates for the player
BEST PRACTICES: SEGMENT + MANIFEST FILES

77
BEST PRACTICES: SEGMENT + MANIFEST FILES
− Recommendations:
 Use validators to check the conformance segments and manifests
 DASH-IF Conformance Validator
 Apple Media Stream Validator Tool
 (MSE) Make sure that the timeline is continuous to prevent playback
stalling
 Gzip to compress large Manifest files and improve HTTP latency

78
− (Widevine) Different KIDs for each track
− (Widevine, HDCP) depends on OS support; use
EME MediaKeyStatus enumeration
− (EME,PlayReady) HW security can be queried:
com.microsoft.playready.hardware
− (EME,Fairplay) Different key system versions
exist: com.apple.fps.x_0
− (EME) Specify robustness level, codecs, when
calling EME
− (HTTPS) Browsers enforce no mixed content. All
resources (e.g. license servers) need to be
HTTPS
BEST PRACTICES: DRM + PLAYBACK

79
App/Player Best Practices
2. OTT STRATEGY

80
MULTI-DRM DRIVERS
8
0
• NPAPI deprecation
• Studio requirements
HW DRM
SW DRM

EXAMPLE :NETFLIX SYSTEM REQUIREMENTS FOR HTML5 PLAYER
https://help.netflix.com/en/node/23742

82
MAPPING EME ROBUSTNESS LEVELS
W3C EME Level PR Policy * FP Policy * WV Policy
1 SL2000 N/A SW_SECURE_CRYPTO (L3)
2 SL2000 N/A SW_SECURE_DECODE (L3)
3 SL2000 N/A HW_SECURE_CRYPTO (L2)
4 SL2000 N/A HW_SECURE_DECODE (L1)
5 SL3000 N/A HW_SECURE_ALL (L1)
* Not specified

83
− Components of such an ecosystem are:
 Packager/Segmenter
 Encryptor
 MPD creator
 DRM server
 How will these components exchange
sensitive data for content creation (encryption
key and DRM-specific information)
© Fraunhofer FOKUS
MULTI-DRM BACKEND
©Matthias
Heyde
/
Fraunhofer
FOKUS

84
© Fraunhofer FOKUS
− Aims to standardize the way entities involved in the content creation workflow exchange
protection information
− Entities depend on existing headend,
− for example: Packager, Encryptor, DRM License server, MPD generator etc.
− How to secure Content Key (CK) delivery between entities
− CPIX document contains keys and DRM policy information
− Latest published version 2.0
− Compatible with DASH and HLS
DASH-IF CPIX
CONTENT PROTECTION INFORMATION EXCHANGE FORMAT
©Matthias
Heyde
/
Fraunhofer
FOKUS
©Fotograf
/
Bildagentur

85
© Fraunhofer FOKUS
CPIX INTERFACES

86
© Fraunhofer FOKUS
− Communication flow during the creation of DASH-protected content with PlayReady
CPIX EXAMPLES

87
© Fraunhofer FOKUS
− Communication flow during the creation of DASH-protected content with Widevine
CPIX EXAMPLES

88
© Fraunhofer FOKUS
− Example of a CPIX file
CPIX EXAMPLES

89
DEMO: CREATION AND PLAYBACK OF MPEG-DASH CONTENT WITH DRM
• DASH+CENC (PR, WV, ClearKey) content created with CPIX
• Host DASH .mpd and segments on a HTTP server
• Playback in dash.js (using MSE/EME) with keysystem selection
• Add HTTP URL to the .mpd in the DASH Player
512kB
1024kB
2048kB
1280 x 720
1024kB
2048kB
4096kB
1920 x 1080

91
Watermarking
Per-title Encoding
OTT and HbbTV Ad-Insertion
Streaming Metric Analytics using SAND
Low-latency Streaming
360° Media Streaming
3. ADVANCED STREAMING

© Fraunhofer FOKUS
WATERMARKING
Watermarking embeds identifiable information (watermark identifier) into video signal in a way that
(a) is imperceptible to a human viewer,
(b) allows decoding of the embedded information (thus e.g. making it possible to achieving
identification of each viewing session), and
(c) is robust enough to survive multiple iterations of transcoding, image processing, and camcording,
while maintaining its veracity.

© Fraunhofer FOKUS
ONE STEP WATERMARKING
• One Step Watermarking [1] :
• Embed a single watermark id into a single complete encoded asset
• During encode on the server or during playback in the secure video pipeline
• Client side approach: Requires tight integration with both hardware and DRM vendors
[1]https://ultrahdforum.org/wp-content/uploads/Ultra-HD-Forum-Guidelines-v1.4-final-for-
release.pdf

© Fraunhofer FOKUS
TWO STEP WATERMARKING
• Two Step Watermarking [1] [2] :
• Step 1: Preprocessing
• Transformation of a
single raw video
input into two raw
video outputs.
• Can be done prior
to encoding or post
encoding
[1]https://ultrahdforum.org/wp-content/uploads/Ultra-HD-Forum-Guidelines-v1.4-
final-for-release.pdf
[2] Integrating Forensic Watermarking Into Adaptive Streaming Workflow

© Fraunhofer FOKUS
TWO STEP WATERMARKING
• Two Step Watermarking [1] [2] :
• Step 2: Embedding
• Generation of a unique session-
specific segment sequence
• Example: ABBBAAB
• Can be done with manifest
manipulation or with variant
decision at the CDN edge
[1]https://ultrahdforum.org/wp-content/uploads/Ultra-HD-Forum-Guidelines-v1.4-final-for-
release.pdf
[2] Integrating Forensic Watermarking Into Adaptive Streaming Workflow

© Fraunhofer FOKUS
EXAMPLE TWO-STEP WATERMARKING ARCHITECTURE
Giladi/comcast
Unique ID
Encoder Origin
CD
N
Manifest
Generator
Player
License
Server
A
B
A/B segments
(multiple
bitrates)
Unique
A/B
Playlist
Detection &
Monitoring
System
Watermark
preprocessor
A
B
Unique
sequence of
A/B segments
A
B
A/B segments
(high quality mezz)
ABBAABAB
A
Embedder
Vendor Proprietary
Preprocessing
(part of ingest workflow)
Embedding
(real-time, on GET request)

© Fraunhofer FOKUS
WATERMARKING IN STANDARDIZATION
• Ultra HD Forum: Phase A Guidelines
• DASH-IF
• ABR-specific two-step watermarking with unique A/B variants sequence (based on UHD Forum)
• Manifest manipulator creates a unique manifest and A/B variants per session
• Alternative: Edge-side approach with proxy or redirect
• DASH-IF work to focus on efficient MPD/segments aspects
• ISO/IEC 23001-12 defines carriage of variant samples for ISO-BMFF files to store only
differing frames => storage overhead lower than 5%
• SVA not published
• HbbTV
• Application Discovery over Broadband (ADB) specification to use watermarking for AIT signalling
• ATSC watermarking [1] in consideration
[1] https://www.atsc.org/wp-content/uploads/2016/09/A335-2016-Video-Watermark-Emission-2.pdf

98
Watermarking
Per-title Encoding

99
ffmpeg -i input.mp4 -c:v libx264 -b:v 4000k -maxrate 4000k -bufsize 2000k out/cbr-4k.mp4
RATE CONTROL MODES – CONSTANT BITRATE (CBR)

101
ffmpeg -y -i input.mp4 -c:v libx264 -b:v 4000k -pass 1 -f mp4 NUL &&
ffmpeg -i input.mp4 -c:v libx264 -b:v 4000k -maxrate 6000k -bufsize 4000k -pass 2 out/cvbr-
4k.mp4
RATE CONTROL MODES – CONSTRAINED VARIABLE BITRATE (CVBR)

102
ffmpeg -i input.mp4 -c:v libx264 -crf 23 out/crf-23.mp4
RATE CONTROL MODES – CONSTANT RATE FACTOR

103
𝑀𝑀𝑀𝑀𝑀𝑀 =
∑𝑀𝑀,𝑁𝑁[𝐼𝐼1 𝑚𝑚, 𝑛𝑛 − 𝐼𝐼2 𝑚𝑚, 𝑛𝑛 ]2
𝑀𝑀 ∗ 𝑁𝑁
𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃 = 10𝑙𝑙𝑙𝑙𝑙𝑙10(
𝑀𝑀𝑀𝑀𝑀𝑀𝐼𝐼
2
𝑀𝑀𝑀𝑀𝑀𝑀
)
VIDEO QUALITY METRICS - PSNR IN DETAIL
CRF35: 32.888db
CRF25: 38.929db
CRF18: 43.491db

104
PER-TITLE ENCODING - DIFFERENT TYPES OF CONTENT

105
PER-TITLE ENCODING – FUNDAMENTALS
Source: https://medium.com/netflix-techblog/per-title-encode-optimization-7e99442b62a2

106
PER-TITLE ENCODING – HOW TO DO IT
Complexity
Analysis
• Perform test
encoding using
CRF, CBR or CQ
encoding with
different values
Convex Hull
• Select bitrate-
resolution
pairs that are
close to the
convex hull
Standard
encoding
• Perform the
standard CBR
/ Two-pass
constrained
VBR
encoding
1.Optional:
PSNR
analysis
• Find corrupt
frames or
scenes

107
PER-TITLE ENCODING - STEP 1: TEST ENCODES
Bitrate 1080p 720p 540p
4000 44.24
3400 43.65 42.80
2800 42.89 42.35
2500 42.42 42.07 41.39
1900 41.16 41.27 40.84
1300 39.20 39.91 39.87
1000 37.70 38.75 39.00
700 37.01 37.54
500 35.97
Source: Video Encoding By The Numbers – Jan Ozer

108
PER-TITLE ENCODING - STEP 1: TEST ENCODES

109
PER-TITLE ENCODING - STEP 2: CONVEX HULL

110
ffmpeg -y -i input.mp4 -vf scale=1280:720 -c:v libx264 -b:v [targetbitrate] -pass 1 -f mp4
NUL &&
ffmpeg -i input.mp4 -vf scale=1280:720 -c:v libx264 -b:v [targetbitrate] -bufsize
[targetbitrate] –maxrate [targetbitrate] -pass 2 out/cvbr.mp4
PER-TITLE ENCODING - STEP 3: CBR/CVBR ENCODING

111
PER-TITLE ENCODING – RESULTS: LOW COMPLEXITY CONTENT
Bitrate (kbps) Resolution
235 320x240
375 384x288
560 512x384
750 512x385
1050 640x480
1750 720x480
2350 1280x720
3000 1280x720
4300 1920x1080
5800 1920x1080 Simple animated content

114
PER-TITLE ENCODING - RESULTS: HIGH COMPLEXITY CONTENT
Bitrate (kbps) Resolution
235 320x240
375 384x288
560 512x384
750 512x385
1050 640x480
1750 720x480
2350 1280x720
3000 1280x720
4300 1920x1080
5800 1920x1080 Complex content

115
− Different types of content require different bitrates
− Simple content has more spatial and temporal redundancy than complex content and
therefor requires a lower bitrate to achieve a certain quality
− Per-title encoding has the potential to:
 Reduce storage
 Reduce bandwidth costs
 Improve the perceptual quality of a video
− Requires additional test encodings and complexity analysis
PER-TITLE ENCODING : SUMMARY

116
- Estimate complexity factor using only 1-2 test encodes and adjust the the static encoding
ladder accordingly
- Per-chunk encoding to tackle different complexities within a single content
- Analyze lowest PSNR to identify problematic frames/scenes
- Use advanced quality metrics like VMAF
- Capped/Constrained CRF encoding for quality based adaptive media streaming
- Bitmovin approach: Send PSNR in manifest to further improve client algorithm
PER-TITLE ENCODING: OUTLOOK & IMPROVEMENTS

117
Watermarking
Per-title Encoding

118
• Server-based Architecture
 MPD centric
 Multi-Period
 Uses Xlink, DASH-specific events for additional periods
 MPD Chaining as a new alternative
 Single-Period
 Ads and content are transcoded and packaged into a continuous
stream
 Uses just-in-time packaging for DAI per user
• App-based Architecture
 Requires additional application
 Uses DASH user-defined events
AD INSERTION ARCHITECTURES

119
• Messages signaled to the DASH client
 MPD Update information
 Splice information
 Metadata
• Appear inline (MPD level) or inband (Segment level) as ‘emsg‘
boxes
• Important attributes
• schemeIdUri
• value
• presentationTime
DASH EVENTS

120
scheme_id_uri value Description
urn:mpeg:dash:event:2012 1 MPD Validity Expiration event : Triggers MPD
reload
urn:mpeg:dash:event:2012 2 MPD Patch event: Includes XML data to update
the MPD
urn:mpeg:dash:event:2012 3 MPD Update event: Includes a complete new
MPD
DASH SPECIFIC EVENTS

121
Postroll Ad Insertion: MPD before the reload
© Fraunhofer FOKUS
<!-– Content period !-->
<Period start="PT0S" id="1">
<AdaptationSet mimeType="audio/mp4"></AdaptationSet>
<AdaptationSet mimeType=„video/mp4"></AdaptationSet>
</Period>
EXAMPLE: MPD VALIDITY EXPIRATION

122
Postroll Ad Insertion: MPD after the reload (multi-period MPD)
© Fraunhofer FOKUS
<!-– Content period !-->
<Period duration="PT20S" start="PT0S" id="1">
</Period>
<!– Ad period !-->
<Period duration="PT30S" start="PT20S" id=„2">
</Period>
EXAMPLE: MPD VALIDITY EXPIRATION

123
• Not directly processed by a DASH client
• DASH has defined how SCTE 35 cue messages may be carried
in DASH Media Segments and DASH Media Presentation
Description (MPD)
• Handed to Ad Management Module
• Example SCTE35 events
• schemeIdUri: „urn:scte:scte35:2014:xml+bin“
• Value: „1001“
CUSTOM EVENTS

124
• Remote Period Elements
• Example: <Period
xlink:href="http://dash.fokus.fraunhofer.de:8080/mws17/period_timescapes"
xlink:actuate="onLoad"/>
• Requires two attributes in the element
 xlink:href: Defines URL to remote content
 xlink:actuate : Defines resolution time
o onLoad: Directly after the manifest is parsed
o onRequest: At the time the element is accessed
• Enables ad targeting by adding user specific parameters to the xlink:href
attribute
XLINK

125
XLINK RESOLUTION
Source: DASH-IF IOP guidelines

127
DEMO: AD-INSERTION
− Server- and app-
based ad insertion
− Web interface for
stream scheduling
− Manifest manipulation:
− Multi-period
− Xlink
− Inline Event
− Live conversion

128
DAI WORKFLOW LEVERAGING FAMIUM DAI COMPONENTS

129
1. Stream scheduler
− Content provider schedules playlist with content and ad assets and requests
sitched manifest from the Ad. Agg. Service
2. Ad Agg. Service
− Ad. Agg. Service contacts Ad Server and requests target ads
− URLs to the manifest files are handed to the Ad Stitcher
3. Ad Stitcher
− Manifests are parsed and a multiperiod MPD is created
− If necessary Xlink elements are included in the MPD
4. Client Player
− Client plays multiperiod MPD and resolves Xlink elements.
SERVER-SIDE AD-INSERTION
Manifest Manipulation

130
1. Stream scheduler
− Content provider schedules playlist with content and ad assets and requests
sitched manifest from the Ad. Agg. Service
2. Ad Agg. Service
− URLs to the manifest files are handed to the Ad Stitcher
3. Ad Stitcher
− Manifests are parsed and a MPD with a single period and DASH custom
events is created
4. Client Player
− Client plays MPD with a single period. An external application processes the
DASH events and plays the ad in a second video elemen
CLIENT-SIDE AD-INSERTION
External application on the client

131
− Server-side logging of HTTP GET requests
− Dispatch IAB VAST tracking events to the client
− Example: EventStream@schemeIdUri = http://dashif.org/identifiers/vast30
− DASH Callback events to issue HTTP GET request to a provided URL
AD TRACKING

− Scenario: OTT/Broadband Ad-Insertion
− DVB CM-TA Group working on commercial requirements for targeted advertising, not
exclusive to HbbTV
− Different options possible (mainly HbbTV 2.0.x)
− Ad Signalling:
− MPEG DASH events (MPD or Inband) are mapped to TextTrack objects; contains
ad metadata
− Xlink
− Ad Playback:
− Support for Multiple HTML5 media elements enables pre-fetching in the second
<video> element
− Multiple Periods
DASH+AD-INSERTION IN HBBTV

133
DYNAMIC SPOT REPLACEMENT HBBTV 2.0
linear broadcast ad break
broadcast
IP
ads
/
HbbTV

135
Server-based Ad Insertion
App-based Ad
Insertion
Manifest manipulation
(multiple periods)
Individual packaging
(single period)
Additional application on
the client
Ad Request Server Server Client
Tracking complexity
High High Low
Content generation
complexity
Medium High Low
Ad blocking risk Medium Low High
Seamless Switching Nearly frame-accurate * Frame-accurate Nearly frame-accurate *
Player compatibility Medium High Low
AD INSERTION – DIFFERENT APPROACHES
* Depends on player implementation

136
Watermarking
Per-title Encoding

137
© Fraunhofer FOKUS
MOTIVATION FOR STREAMING ANALYTICS
©Matthias
Heyde
/
Fraunhofer
FOKUS
− Viewers start abandon streams that take more than 2 seconds to startup
− 1% of rebuffering causes people to watch 5% less of a video
− Viewers who experience video failure are less likely to revisit a web page
− 40% of customers are unlikely to give a streaming service a chance on a second
device after being confronted with and inferior stream
Stream quality influences user behaviour
“Video Stream Quality Impacts Viewer Behavior: Inferring Causality
Using Quasi-Experimental Designs”, IEEE/ACM Transactions on
Networking, Krishnan and Sitaraman, Akamai

138
© Fraunhofer FOKUS
MPEG-DASH PART 5 – SAND
©Matthias
Heyde
/
Fraunhofer
FOKUS
− Published as ISO/IEC 23009-5:2017 Server and network assisted DASH (SAND)
− DASH (SAND) introduces messages between DASH clients and network elements or
between various network elements for the purpose to improve efficiency of streaming
sessions
− DASH-IF SAND integration was in community review until April 15th, 2018
− Modes defining subsets of SAND messages and mandatory SAND protocols to use for specific deployment
environments
− E.g. Home Gateway, CDN Edge, Network Assistance
− Security guidelines for SAND messages delivery
− Procedures on DANE discovery for SAND
− Standardizing Streaming Metrics
− ISO/IEC 23009-1 AnnexD AMD.5 adds DeviceInformationList
− DASH-IF position paper on proposed QoE Media Metrics
− Streaming Video Alliance
− CTA R4WG20

139
© Fraunhofer FOKUS
MPEG-DASH PART 5 – SAND
©Matthias
Heyde
/
Fraunhofer
FOKUS
General: Bidirectional Messaging System
− DANE: DASH Aware Network Element
− Status messages
− PER: Parameters Enhancing Reception
− PED: Parameters Enhancing Delivery

140
© Fraunhofer FOKUS
OPTIMIZE STREAMING
©Matthias
Heyde
/
Fraunhofer
FOKUS
− DASH is client centric
− Media servers have no or limited knowledge of the streaming process
− How to measure QoE and identify potential playback issues on certain devices
 SAND Metric Reporting
− Clients have no information about network and server circumstances
− ABR algorithms may perform badly due to missing information (e.g. other clients)
 SAND SharedResourceAllocation
SAND Metrics Reporting and SharedResourceAllocation

141
© Fraunhofer FOKUS
SAND MESSAGES
©Matthias
Heyde
/
Fraunhofer
FOKUS
Metrics and Status Messages
Metrics
Messages
Status
Messages

143
• SAND specifies a reporting scheme for metrics
• Reporting is enabled via MPD
<Metrics
metrics="BufferLevel,HttpList,RepSwitchList,PlayList">
<Reporting schemeIdUri="urn:mpeg:dash:sand:channel:2016"
value="http://example.url"/>
</Metrics>
MPEG-DASH PART 5 – SAND - REPORTING

145
© Fraunhofer FOKUS
SAND SHARED RESOURCE ALLOCATION USE CASE
©Matthias
Heyde
/
Fraunhofer
FOKUS
• Problem:
• on-off patterns cause oscillating clients
 bad QoE
• Use cases: WiFi at sports events, in
planes, trains and home networks
• Solution:
• Coordinate clients via server (e.g. cloud
or home gateway)
• Clients signal intent to share resources
(optional: “weight”)
• Server assigns bitrate (e.g. equally
shared, priviliged, max. # clients etc.)

148
© Fraunhofer FOKUS
SAND DOWNLOADS
©Matthias
Heyde
/
Fraunhofer
FOKUS
− SANDLibrary: https://www.fokus.fraunhofer.de/go/sand
− Support for Metric Reporting
− Supports status and PER messages for SharedResourceAllocation
− Can be attached to any DASH player
− Dash.js sample available
− SANDServer: https://www.fokus.fraunhofer.de/go/sand
− Node.js-based
− Persists Metrics in a database
− Manages SharedResourceAllocation messaging
− Can be used for server-side ABR algorithms
SANDLibrary and SANDServer

149
DEMO: STREAMING METRICS REPORTING AND ANALYTICS USING SAND

150
DEMO: PLAYER COORDINATION USING SAND

151
Watermarking
Per-title Encoding

153
WHAT CAUSES LATENCY?
Source: CMAF Low Latency Streaming – Will Law 11.17

154
− End-to-End (E2E) latency or hand-waving latency = time between video or audio being
captured at the source and when it is shown on the screen
− Time to first frame (TTFF): The time it takes for content to initially load
− Low latency results in small buffers  Tradeoff between robustness and playing close to
the live edge
END-TO-END LATENCY

155
TTFF VS. E2E - WOWZA REPORT 2017
Source: https://www.wowza.com/blog/spoiler-alert-low-latency-crucial-for-sports-apps

156
LATENCY FOR DIFFERENT PROTOCOLS
Source: CMAF Low Latency Streaming – Will Law 11.17

157
− With HTTP/1.1 one HTTP and thus TCP request must be
made for each resource
 HTTP/2 reduces transport overhead: multiplexing requests,
better data compression, server-push, binary encoding
 Server Sent Events (SSE) to send updates or continuous
data stream to the app
 Web Sockets for two-way communication with a remote
host
 Web RTC for peer 2 peer communication via data channel
 QUIC for multicast delivery
TRANSPORT-LEVEL OPTIMIZATION

158
CMAF CHUNKS FOR LOW-LATENCY STREAMING
Source: ISO/IEC 23000-19 Common Media Application Format for Segmented Media

159
STANDARD CMAF FRAGMENT VS LL CMAF FRAGMENT
Famium Packager: Low latency CMAF fragment
Famium Packager: Standard CMAF fragment

161
CMAF CHUNKS FOR LOW LATENCY STREAMING

162
LOW-LATENCY : SAMPLE DASH MANIFEST

163
- @availabilityTimeOffset (ATO) : “Provides the time how much earlier segments are
available compared to their computed availability start time (AST)“
− @availabilityTimeComplete: „Specifies if all Segments of all associated Representation
are complete at the adjusted availability start time. If the value is set to false, then it may
be inferred by the client that the segment is available at its announced location prior
being complete.“
LOW-LATENCY: MPD PARAMETERS
𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 = 𝐴𝐴𝐴𝐴𝐴𝐴 − 𝐴𝐴𝐴𝐴𝐴𝐴
𝐴𝐴𝐴𝐴𝐴𝐴 = 𝑑𝑑𝑠𝑠 − 𝑑𝑑𝑐𝑐 = 7 𝑠𝑠𝑠𝑠𝑠𝑠
Source: Chunked DASH in JavaScript – Robert Alnesjö

165
− Low latency is key for live events (traditonal sport and eSport)
− Activity in DVB and DASH-IF
− DVB CM-AVC Report in Low-Latency Live Service with DASH
− DASH-IF IOP Low Latency DASH
− CMAF offers possibility for low-latency streaming with DASH and HLS
− Modified manifest required
− Modified player required  dash.js 2.6.8 has support for low-latency streaming using the
fetch API.
− Client/packager snychronization must be accurate to avoid early/late requests
− Sophisticated ABR algorithm required to compensate for download idle times
SUMMARY

167
Watermarking
Per-title Encoding

168
360° Video Playout - Fraunhofer FOKUS
COMMERCIAL HMDS VS UHD TV
Source: Harmonic and Viaccess-Orca, VR360 – the operator gateway into Virtual Reality

169
DEMO: 360° 4K FOV ON TV
360° / 16.384 pixel
180°
/
x
8.192
pixel
4K / UHD

170
High quality
360°
streaming
Live 360°
streaming
Cross-
device
Scalable UX
CHALLENGES OF HIGH QUALITY 360° STREAMING

171
Streaming Approaches
− CSP1: Apply adaptive streaming to the whole source 360° video:
Same as for streaming classical videos using DASH (Different
bitrates/representations available for the source 360° video).
− CSP2: Apply adaptive streaming based on selected FOV: Many
versions of the source 360° video are available. In each version, a part of
the video which corresponds to a FOV is available in higher quality and
the rest in lower quality.
− CSP3: Apply adaptive streaming based on video tiles: 360° video is
segmented in tiles with different representations. HEVC/VP9 for best
performance.
360° CLIENT SIDE PROCESSING (CSP)

172
360° CLIENT SIDE PROCESSING
Stitching
360° Video
Storage
360°
Processing
360°
Camera
Server Client
Network
Stream 360°
Video!

173
Playback (Browser)
360° CLIENT SIDE PROCESSING WITH WEB TECH
Request 360°
Segments
• XMLHttpRequest
• Fetch API+Streaming
• WebSocket API
Play 360° Segments
• Video Element
• MSE API
• (EME API)
360° Transformation
& FOV Display
• WebGL API
• Canvas API *
• WebVR API (for HMD)
Capture User Inputs
• Orientation Events
• Touch Events
• Key Events
• Mouse Events
DASH 360°
Segments
360°
Frames
FOV
Frames
User
Inputs
* Canvas API not compatible with DRM-protected media

174
Streaming Approaches
− SSP1: 360° Server Side Live-rendering: the source 360° video is
rendered in the cloud and only the the requested FOV is sent to the
client. The Server needs to handle each client in a separate rendering-
session.
− SSP2: 360° Server Side Pre-rendering: the source 360° video will be
pre-rendered for a different variations of FOVs the output FOV videos will
be stored in a static storage server. DASH is applied to FOVs videos in
the same way as for classical videos.
360° SERVER SIDE PROCESSING (SSP)

175
360° SERVER SIDE PROCESSING (SSP)
Stitching
360° Video
Storage
360°
Processing
360°
Camera
Server Client
Network
Stream FOV
Video!

176
How it works?
360° SERVER SIDE PRE-RENDERING
1s 2s 3s 4s time
15°
30°
45°
60°
75°
90°
angle
Static
Video for
FOV 15°
Static
Video for
FOV 90°
Transition
Video from
FOV 15° at
0.666 s
DASH
Adaptation Set
DASH
Adaptation Set
DASH
Adaptation Sets

177
Playback (Browser)
360° SERVER SIDE PROCESSING
Request FOV
Segments
• XMLHttpRequest
• Fetch API+Streaming
• WebSocket API?
Play Segments
• Video Element
• MSE API
• (EME API)
Capture User Inputs
• Orientation Events
• Touch Events
• Key Events
• Mouse Events
User
Inputs
DASH FOV
Segments
FOV
Frames

178
360° Server Side Pre-rendering (Fraunhofer Cloud Playout) – Live Streaming
Architecture
360° SERVER SIDE PROCESSING
360°
Camera
…
360° Live
Stream
FOV Live
Renderer 1
FOV Live
Renderer 2
FOV Live
Renderer N
FOV Video
Storage
…
CDN
2
1
3
4
5
Player
FOV Video
Storage
…

179
CSP1 (full
adaptive)
CSP2 (FoV
adaptive)
CSP3 (tiles)
SSP1 (cloud
browser)
SSP2 (pre-
rendered)
360° Processing Client Client Client Server Server
Storage Low High Medium Low High
Bandwidth
High Medium Medium Low Low
Motion-to-
photon latency
Low Low Low Medium High
360° SOLUTIONS COMPARISON

180
360° VIDEO STREAMING - DEMO

181
• Streaming Formats
• Encoding
• Web Media APIs
• Standards update
2. OTT Strategy
SUMMARY

182
• Cross-platform Deployment
• Web, Native or Hybrid Apps?
• Best Practices
• Multi-DRM backend with CPIX
2. OTT Strategy
SUMMARY

183
2. OTT Strategy
• Watermarking
• Per-title Encoding
• OTT and HbbTV Ad-Insertion
• Streaming Metric Analytics using SAND
• Low-latency Streaming
• 360° Media Streaming
SUMMARY

184
Fraunhofer FOKUS
Kaiserin-Augusta-Allee 31
10589 Berlin, Germany
www.fokus.fraunhofer.de
Stefan Pham
Project Manager R&D
stefan.pham@fokus.fraunhofer.de
Phone +49 (0)30 3463-7103
© Fraunhofer FOKUS
CONTACT
THANKS FOR YOUR
ATTENTION
Daniel Silhavy
Project Manager R&D
daniel.silhavy@fokus.fraunhofer.de
Phone +49 (0)30 3463-7680

1_MWS2018_Tutorial1_Pham_Internet Delivered Media.pdf

More Related Content

Similar to 1_MWS2018_Tutorial1_Pham_Internet Delivered Media.pdf

More from AliIssa53

Recently uploaded

1_MWS2018_Tutorial1_Pham_Internet Delivered Media.pdf