1. Stefan Pham, Daniel Silhavy
May 15, 2018
INTERNET DELIVERED MEDIA – TRENDS & BEST PRACTICES
TUTORIAL 1 – 7TH FOKUS MEDIA WEB SYMPOSIUM
2. 2
1. Foundations of Adaptive Streaming
2. OTT Strategy
3. Advanced Web Media Streaming
AGENDA
3. 3
− DASH – Dynamic Adaptive Streaming over HTTP for live and on demand video; MPEG standard
− SAND – Server and network assisted DASH to enable interoperable metrics and client coordination
− HLS – HTTP Live Streaming for live and on demand video by Apple
− CMAF – Common Media Application Format for HLS and DASH
− CENC – Common Encryption for many DRM & delivery channels
− MSE – Media Source Extension to trick-function HTML5 video-objects via JavaScript (control AV
media streams)
− EME – Encrypted Media Extension to play back DRM-protected media in standard browsers w/o
the use of proprietary plug-ins
− CDM – Content Decryption Module - addition to the browser that provides functionality for one or
more Key Systems
− CPIX – Content Protection Information Exchange Format - standardizes the way entities involved
in the content creation workflow exchange protection information
− VMAF – Video Multi-Method Assessment Fusion- perceptual video quality assessment algorithm
by Netflix
DELIVERING MEDIA: TECH TO UNDERSTAND
4. Where we are:
− Very popular commercial OTT streaming services, e.g. Netflix, Amazon Video, DAZN, Spotify, …
What we work with and where we start:
− Different device platforms & interoperability – how to reach them all? Best practices?
− Other factors:
− Monetization (DAI),
− Regulatory (DRM, watermarking),
− Quality (UHD, encoding),
− Analytics (streaming metrics),
− UX (low latency, 360°)
4
INTERNET DELIVERED MEDIA - MISSION ACCOMPLISHED?
7. MPEG-DASH
− Standard for streaming multimedia over the Internet
− Dynamic Adaptive Streaming over HTTP (DASH) - ISO/IEC 23009
− Part 1: Media presentation description and segment formats
− Part 2: Conformance and reference software
− Part 3: Implementation guidelines
− Part 4: Segment encryption and authentication
− Part 5: Server and network assisted DASH (SAND)
− Part 6: DASH with Server Push and Web Sockets
− Extensions for common DRM-interoperable encryption and encoding (CENC)
− DASH-IF Interoperability Guidelines
− Different profiles: DVB, HbbTV 1.5/2.0, ATSC 3.0 etc.
8. MPEG-DASH FILE FORMAT
8
• ISO base media file format (ISOBMFF) or MPEG2-TS
• Codec-agnostic
Box model in ISOBMFF (init + media segment):
File
Type
('ftyp')
Movie Metadata ('moov')
Movie
Header
('mvhd')
Track
('trak')
Movie Extends
('mvex')
Movie
Extends
Header
('mehd')
Track
Extends
('trex')
Movie
Fragment
('moof')
Movie
Fragm.
Header
(’mfhd’)
Track
Fragme
nt (’traf’)
Media
Data
('mdat')
9. MPEG-DASH - SCOPE
HTTP Server DASH Client
Control Heuristics
Media
Player
HTTP Client
Segment
Parser
MPD Parser
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
Segment
MPD
MPD
MPD
MPD
HTTP 1.1
MPD Delivery
DASH Spec covers the red blocks.
[Note: Client’s behavior is only defined to the extend needed for compliancy with MPD
and segment formats.]
10. − MPEG-DASH 3rd edition reached FDIS (to be published as ISO/IEC 23009-1:2018)
− New Features
− Media presentation description (MPD) can be daisy chained to simplify
implementation of pre-roll ads in cases of targeted dynamic advertising for live
linear services
− support for pre-selection in order to signal suitable combinations of audio elements
that are offered in different adaptation sets
− Availability Time Synchronisation, Spatial Relationship Description (SRD),
generalized URL parameters, Authentication, MPD linking, Callback Event, Period
Continuity
DASH 3RD EDITION
11. − Draft: IETF RFC 8216
− Mainly used for iPhone, iPad, Mac, AppleTV
− New Features:
− HEVC support
− CENC (‘cbcs’) and f-MP4 / CMAF support
− IMSC1 subtitles
− FairPlay / offline playback
− PHP-like playlist variables
− Synchronized playback of multiple streams (via AVFoundation)
APPLE HTTP LIVE STREAMING (HLS)
12. COMMON ENCRYPTION (CENC)
CENC
Header
Media File
Body
Key Generator
Audio Video Other
Scrambler
Content Asset
Management
System
Encrypted media data uses the Advanced
Encryption Standard specified by AES using
128-bit keys in Counter mode (AES-CTR) or
Block cipher mode (AES-CBC)
Advanced Web Technologies | Web & Media I | lecture winter term 2015/2016
13. − Common encryption means the same short fragment of video can be decrypted and
decoded by devices using different DRMs.
− Improved encoding, network cache (origin vs. edge server) and CDN efficiencies
− ISO/IEC 23007-1 (3rd edition) – Common encryption in ISO based media file format files
− Protection schemes: ’cenc’, ’cbc1’, ‘cens’, ‘cbcs’
− ‘cens’ and ‘cbcs’ are pattern encryption schemes
COMMON ENCRYPTION (CENC)
15. PROTECTION SYSTEM HEADER BOX
File
Type
('ftyp')
Movie ('moov')
Movie
Header
('mvhd')
Protectio
n System
Specific
Header
‘pssh’ [1]
Protectio
n System
Specific
Header
‘pssh’ [2]
Track
('trak')
Movie Extends
('mvex')
Movie
Extends
Header
('mehd')
Track
Extends
('trex')
Media Segment with multiple ‘pssh’ boxes
16. 16
− ISOBMFF-based adaptive media packaging format for HLS and DASH
− CMAF Chunks for low latency encode, delivery and playback
− Defines encoding of segmented media for delivery and decoding on device
− Published in 01/2018 as ISO/IEC 23000-19
− Does not specify manifest, player or delivery protocol
− Does not mandate encryption scheme for CENC
− Apple „Using CMAF content with HLS“ available
− DASH-IF work on CMAF mapping started
COMMON MEDIA APPLICATION FORMAT (CMAF)
18. − Role of the Manifest is divided between the HLS Master Playlist and Media Playlists it
references
HLS MANIFEST MODEL
Master Playlist
english.m3u8
french.m3u8
video.m3u8
uhd-video.m3u8
Media Playlist
Segment1
Segment2
Segment3
…
Media Playlist
Segment1
Segment2
Segment3
…
Media Playlist
Segment1
Segment2
Segment3
…
Media Playlist
Segment1
Segment2
Segment3
…
20. 20
WHY DO WE ENCODE? - THE NUMBERS
Compression
320x240@0.235Mbit/s
No compression
w𝒊𝒊𝒊𝒊𝒊𝒊𝒊𝒊 ∗ 𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉𝒉 ∗ 𝒇𝒇𝒇𝒇𝒇𝒇 ∗ 𝒃𝒃𝒃𝒃𝒃𝒃𝒃𝒃 𝒑𝒑𝒑𝒑𝒑𝒑 𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑𝒑
20x16@0.228Mbit/s
320x240p@55.3Mbit/s
21. 21
DIFFERENT CODECS ON THE MARKET - TIMELINE
Source: https://github.com/leandromoreira/digital_video_introduction
22. 22
DIFFERENT CODECS ON THE MARKET - MARKETSHARE
Which codecs are you using currently? Which video codecs are you planning
to use in the next 12 months?
Source: Video Developer Report 2017 - Bitmovin
23. 23
„Our testing shows AV1 surpasses its stated goal of 30% better compression than VP9, and achieves gains of 50.3%,
46.2% and 34.0%, compared to x264 main profile, x264 high profile and libvpx-vp9, respectively. The new codec requires
longer encoding times vs. current alternatives, however, due to increased complexity.“ – Facebook
„ AV1 producing the same quality as x264 at 55% of the data rate, with x265 running in three-pass
and two-pass Placebo mode at 67% and 69% the data rate respectively... AV1 encoder has extremely low speed—2500-
3000 times lower than competitors. X265 Placebo presets (2 and 3 passes) have 10-15 times lower speed than the
competitors“ – Jan Ozer on Moscow State University results
„ AV1 video quality is pretty much on par with HEVC. The tests found that AV1 performs better than HEVC on the PSNR
and VMAF metrics, but not on subjective testing. Looking at the overall codec efficiency, there is not much difference
compared with HEVC“ – Harmonic, HHI, B<>Com
CODEC DISCUSSIONS: HEVC VS. AV1
24. 24
• Peak Signal-to-Noise Ratio (PSNR)
Based on mean squared error, calculates differences between
original and compressed frame in decibels
No direct correlation between score and perceived subjective quality
Should be between 35-45 dB
• Video Multimethod Assessment Fusion (VMAF)
Perceptual video quality assessment algorithm developed by Netflix
• Structural similarity Index (SSIM)
Measures the perceptual difference between two similar images.
VIDEO QUALITY METRICS
26. − Today browsers are more than just HTML interpreter and render.
− Common platform across devices
− NPAPI deprecation: Plugins (Flash, Silverlight, ...)
− Some HTML5 Features
− HTML5 <video> and <audio> + extensions
− Text tracks
− Canvas (2d shapes and bitmap images)
− SVG
− Advanced CSS3 features
− Storage/offline applications
− Websockets
− Hardware acceleration (WebGL…)
− …
HTML5
Advanced Web Technologies | Web & Media I | lecture winter term 2016/2017 26
*www.apple.com
*www.google.com
*www.amazon.com
*wwwsony.com
28. 28
MSE SUPPORT IN WEB BROWSERS
http://caniuse.com/#search=media%20source
29. 29
− Current version: Recommendation 18 September 2017
− JS interface between DRM License Server and CDM
− Minor differences across browsers / recent Chrome changes:
− Setting MediaKeySystemMediaCapability.contentType (mimeType+ codecs)
mandatory
− Setting robustness is recommended
− VMP enforced, setServerCertificate to encrypt messages to the license server
− Secure origin and transport / mixed content require https
W3C ENCRYPTED MEDIA EXTENSIONS
30. 30
EME SUPPORT IN WEB BROWSERS
http://caniuse.com/#search=media%20source
31. 31
− Media Capabilities API
− Standardization of media decoding capabilities APIs
− Replacing isTypeSupported, canPlayType APIs
− Display capabilities: color gamut, luminance
− Media Playback Quality: total frames, corrupted frames, dropped
frames etc.
− Picture-in-Picture (PiP) to create a floating video window
− EME Extensions: HDCP Policy Check, encryption scheme
− Shape Detection API
− …
W3C WEB INCUBATOR COMMUNITY GROUP (WICG)
Source: https://github.com/WICG
32. 32
− Status: CG Draft
− Initiated by CTA WAVE
− ”Web APIs for media web apps that are supported across all four of the
most widely used user agent code bases”
− “encourage manufacturers to develop products that support the APIs in
the most recent version of Web Media API Snapshot”
− Targeting Smart TVs, game machines, set-top boxes
− Some Examples:
− MSE/EME
− Canvas, Fullscreen API
− Web Workers, Service Workers (not required), Indexed DB
W3C WEB MEDIA API SNAPSHOT 2017
Source: https://w3c.github.io/webmediaapi/
33. 33
− In May 2015 IETF standardized HTTP/2
− RFC7540 and 7541
− Main goal to reduce transport overhead
− combine requests (multiplex)
− better data compression that include header information
− server-push (PUSH_PROMISE)
− binary encoding
HTTP/2
index.html
style1.css
style1.css
script1.js
Script2.js
scriptx.js
icon1.png
icon2.png
iconx.png
font.woff2 picture1.jpg
picture1.jpg
picturex.jpg
…
…
…
…
With HTTP/1.1 one HTTP and thus TCP request must be made for each ressource
35. 35
− The web has been largely built around the so-called request/response paradigm of HTTP
− A client loads up a web page and then nothing happens until the user clicks onto the next page
− Around 2005, AJAX started to make the web feel more dynamic but still, all HTTP communication
was steered by the client, which required user interaction or periodic polling to load new data from the
server.
− Technologies that enable the server to send data to the client in the very moment when it knows that new
data is available have been around for quite some time
− They go by names such as "Push" or "Comet“
− One of the most common hacks to create the illusion of a server initiated connection is called long
polling.
− the client opens an HTTP connection to the server which keeps it open until sending response
− Whenever the server actually has new data it sends the response
− However, all of these work-arounds share one problem
− They carry the overhead of HTTP, which doesn't make them well suited for low latency applications
TUB ODS VL "Advanced Web Technologies“: Web Basics
SSE AND WEB SOCKETS
36. 36
− API for opening an HTTP connection for receiving push notifications from a server in the form of DOM
events
− Server Sent Events (SSE) are commonly used to send message updates or continuous data streams to
a browser client
− e.g., Facebook/Twitter updates, stock price updates, news feeds, sport results where no local client
request is the origin of the event
− This is also possible using XHR but the web page would have to ask periodically if any updates were
available.
− Simple message protocol, two types of updates
− “data: This is some updated data message”
− „event: updateGameResult data: 3:5“
− To enable servers to push data to Web pages over HTTP the EventSource Object is used
TUB ODS VL "Advanced Web Technologies“: Web Basics
SERVER SENT EVENTS
var source = new EventSource(“updates.php");
source.onmessage = function(event) {
document.getElementById("result").innerHTML +=
event.data + "<br>";
};
var source = new EventSource(“updates.php");
source.addEventListener(„updateGameResult“, function(event) {
document.getElementById("result").innerHTML += event.data
+ "<br>";
});
37. 37
TUB ODS VL "Advanced Web Technologies“: Web Basics
DYNAMIC WEB: SERVER SENT EVENTS
Load Page Server Sent Events Stream
Using Web Site
Request
Response
create EventSource
Data
Data
HTTP
Event Event
Server
Client close
…
38. 38
− The W3C Web Socket API enables Web pages to use the WebSocket protocol for two-
way communication with a remote host
− with a persistent connection between the client and the server
− where both parties can start sending data at any time
− Servers must be prepared to handle many open socket connections at once
− e.g, node.js, erlang, Jetty
− Use WebSocket whenever a truly low latency, near realtime connection between the
client and the server is needed, e.g., Multiplayer online games or Chat applications and
all where SSE are used
TUB ODS VL "Advanced Web Technologies“: Web Basics
WEB SOCKETS
39. 39
− After a connection to the web socket server is established (when an open event is fired) data can be
send using the send('your message') method on the connection object
− supported messages types
− text messages as string (typically JSON)
− binary messages as Blob or ArrayBuffer
TUB ODS VL "Advanced Web Technologies“: Web Basics
WEB SOCKETS
Basic WebSocket usage
var connection = new WebSocket('ws://html5rocks.websocket.org/echo');
connection.onopen = function () {
connection.send('Ping');
};
connection.onerror = function (error) {
console.log('WebSocket Error ' + error);
};
connection.onmessage = function (e) {
console.log('Server: ' + e.data);
};
WebSockets with JSON
var msg = {
type: "message",
text: document.getElementById("text").value,
id: clientID,
date: Date.now()
};
connection.send(JSON.stringify(msg));
connection.onmessage = function (event) {
processNewMessageHelper(JSON.parse(event.data));
}
40. 40
TUB ODS VL "Advanced Web Technologies“: Web Basics
WEB SOCKETS
Load Page Web Socket Server
Request
Response
create WebSocket
connection
Data
Data
HTTP/s
Event Event
Server
Client
Data
WebSocket Protocoll ws/wss
close
Using Web Site
Event
41. 41
Why?
− No software downloads and no Plug-Ins
− No installations like Skype, WebEx or GotoMeeting
− Solves incompatibilities of browsers for real time communication
− Real Peer-2-Peer Connection between Browsers
− Direct exchange of audio, video, and data between Browsers
For what?
− Communication in generell (Telcos)
− (Existing) Video/Web conference system providers
− Peer 2 Peer Data exchange
− Training Providers, Customer Support
TUB ODS VL "Advanced Web Technologies“: Web Basics
WEB RTC
42. 42
TUB ODS VL "Advanced Web Technologies“: Web Basics
DYNAMIC WEB: WEB RTC
What we need
Image: http://www.html5rocks.com/en/tutorials/webrtc/infrastructure/
43. 43
TUB ODS VL "Advanced Web Technologies“: Web Basics
DYNAMIC WEB: WEB RTC
What we most likely have
Image: http://www.html5rocks.com/en/tutorials/webrtc/infrastructure/
44. 44
TUB ODS VL "Advanced Web Technologies“: Web Basics
DYNAMIC WEB: WEB RTC
Using STUN servers to get the public
IP to be communicated via signaling
and trying to create a UDP session
between the clients
Image: http://www.html5rocks.com/en/tutorials/webrtc/infrastructure/
45. 45
TUB ODS VL "Advanced Web Technologies“: Web Basics
DYNAMIC WEB: WEB RTC
If a direct Peer-2-Peer connection
cannot be established a TURN media
stream relay server can be used.
Image: http://www.html5rocks.com/en/tutorials/webrtc/infrastructure/
47. 47
WEB APPLICATION VIDEO ECOSYSTEM (CTA WAVE)
Content Specification
• Based on MPEG Common
Media Application Format
(CMAF)
• Compatible with DASH
and HLS.
Testable requirements
• covering most common
playback interoperability
issues.
Reference application
framework
• Based on HTML5
• Provides functional
guidelines for playback
interoperability.
Content HTML5 Reference
Platform
Device Playback
Capabilities
WAVE Test Suite
48. 48
2018 WAVE OUTPUT PLAN
Sep’18 Oct’18
Apr’18 May’1
8
Jun’18 Dec’18
Jul’19 Aug’18 Nov’18
Web Media
Application
Dev
Guidelines
Web Media User Agent Integration Specification
(ongoing)
Test Tools & Test Cases
Content
Specification
Releasing
concurrent
with 2018
NAB Show
Device Playback Requirements Specification Test Tools & Test Cases
Web Media
API
Snapshot
2017
Test Tools & Test Cases
49. 49
DASH-IF 2018
DASH-IF IOP
4.1
V4.0 +
Test database
49
V
DASH-AVC/264 2.0
DASH-IF IOP
3.3
• Live services
• Ad-insertion
• AC-4
• MPEG-H Audio
• HTTPS support
• Key Rotation
• Cross Adaptation
Set Switching
• Callback events
• Period continuity
CPIX V1.0
live Services
for AVC/264
2.0
Conformance
3.3
Test Suite 3.0
UHD/HDR/
Dynamic
Metadata
dash.js v1.3 dash.js v2.0
ATSC 3.0 DASH
Profile V1
•Broadcast TV
profile
•Next Gen
Audio
•Temp scalable
HEVC
•App based
xlink
dash.js v2.5
CPIX V2.0
Content Anno-
tation &
Selection
VP9
CMAF & DASH
Conformance
3.0
Test Suite 3.0
Token Access Control V1.0
Robust linear
Live w lower
latency w DVB
DASH-IF IOP
4.0
V3.3 +
• UHD/HDR
• Alignment w
23009-1 Amd 3 &
4
DRM Support
Improvement
DASH SAND White
Paper
Ad insertion
improvements
SAND Modes
Conformance
4.0
Test Suite 4.0
Dolby Vision
Segment Variants
for Watermarking
Thumbnail
navigation
DASH Metric Position
Paper
dash.js 2.6
W3 Clear Key
Last segment
signaling
DASH-IF IOP V5
ATSC 3.0 DASH
Profile V1.1
V1 +
• HDR
• DASH for NRT
content
Position Paper on uncommon
enc.
Multi-dependent
stream - ESPEX
dash.js 2.6.3
DASH-IF IOP
4.2
• Bugfixes
• Audio
alignments
CPIX
V2.1
USAC IOP
ETSI SPEC
50. − Different profiles:
− HbbTV 1.5: MPEG DASH ISOBMFF Live Profile
(urn:hbbtv:dash:profile:isofflive:2012)
− HbbTV 2.0.x: DVB DASH profile (urn:dvb:dash:profile:dvb-dash:2014)
− Regional specifications (TNT, HbbTV Forum Nederland, AEDETI etc.) mandate
Marlin Simple Secure Streaming (MS3) or Microsoft PlayReady
− Playback:
− “Type 1 player“
− DRM license acquisition via oipfDRMAgent API or EME (HbbTV 2.0.1)
− What’s new?
− Work on HbbTV/DASH profile validator based on DASH-IF validator
− Reference App for DASH/DRM playback
HBBTV & DASH
54. 54
CODEC SUPPORT IN DESKTOP & MOBILE BROWSERS
Browser Platform AVC/H.264 HEVC/H.265 VP9 AV1
Chrome Win Yes No Yes (Yes)
OSX Yes No Yes (Yes)
Android Yes No Yes No
Firefox Win Yes No Yes (Yes)
OSX Yes No Yes (Yes)
Safari >= OSX High Sierra Yes Yes No No
< OSX High Sierra Yes No No No
iOS >= 11 Yes Yes No No
iOS < 11 Yes No No No
Edge Win 10 Yes Yes Yes No
58. 58
DASH and HLS
With CMAF segments HLS and DASH manifests will reference the
same file format
‘cenc’ will be needed for legacy devices. Updates to PlayReady and
Widevine will enable ‘cbcs’ support
COMMON FILE FORMAT AND ENCRYPTION MODES
CMAF +
'cbcs'
Desktop -
OS
Consoles:
PS, XBox
TV:
Smart TV
Alliance,
Android TV,
HbbTV, RDK
Mobile:
WM, iOS,
Android
HDMI
Dongles:
Chromecast/
FireTV
3rd party
STBs
Streaming File Format Encryption DRM
DASH ISOBMFF
(CMAF)
‘cenc’ PlayReady PK<4.0,
Widevine
DASH ISOBMFF
(CMAF)
‘cbcs’ PlayReady PK>4.0,
>Android N,
Chromecast
Streaming File Format Encryption DRM
HLS MPEG2TS Sample-AES (AES-
CBC)
FairPlay
HLS ISOBMFF
(CMAF)
‘cbcs’ FairPlay
60. 60
BASIC ARCHITECTURE: WEB APP VS. NATIVE
Hardware
Operating System
Application APIs
Browser
HTML JS
CSS SVG
e.g., Internet Explorer, Firefox,
Chrome, Safari
e.g., Windows, MacOS, Linux,
Android, iOS
e.g., iPhone, GalaxySII, PC, TV
Native Applications
61. 61
BASIC ARCHITECTURE: PACKAGED WEB APP
Hardware
Operating System
Application APIs
Packaged Web App
The core functionality of a Browser, is
de-coupled into UI and Runtime to
share its functionalities.
• e.g, webview
The JavaScript API is extended to
provide access to device features.
Web Runtime
HTML JS
CSS SVG
Native
Applications
Script
Extension
62. 62
WEB VS. NATIVE: AN OVERVIEW
Pro Web
Reuse of existing skills
web skills are easier to find and less costly
than native app development skills
Easy to use with a big user community to ask
lots of documentation
hooking into the underlying OS through
technologies known to web developers
Create an app for multiple platforms at the
same time
Rapid prototyping using Web Browsers
Reuse of code and design assets
Con Web
Potential compatibility issues with different CSS,
JS, and DOM implementations
input and output latency may be higher than
with native apps
low latency makes using smartphones fun
without attention HTML based apps run the
risk of feeling jerky
User interface could easily feel inappropriate for
the specific target platform
Poor performance possible, particularly where
functionality that makes lots of calls to the OS is
used
Web In general
63. 63
WEB VS. NATIVE: AN OVERVIEW
Pro Browser
No need for application installation
Web Applications are always up-2-date
Strong developer community and a fast
adoption curve, reuse of existing skills
Available on all common operating systems
and devices
Con Browser
Only access to a few devices features as
specified by W3C
Applications does not feel like first class
applications (native)
The same application as native app can
always provide better performance, more
features, nicer UI, better UX…
Browser usage in general
64. 64
WEB VS. NATIVE: AN OVERVIEW
Pro Packaged
A packaged web application typically
is Installable and transportable
is offline usage capable by design
and has access to advanced device features (if
implemented natively)
Packaged web applications can be published
in platform specific application stores which
gains visibility.
No nice way of searching for web apps
• Con Packaged
• Each installation package is targeting a specific
platform architecture. To support multiple
devices/platforms multiple packages are
needed (but tool chains may solve this issue).
• Native parts need to be re-implemented for
each platform
• The user need to perform application updates
• The application needs to consider that the
user does not always have the latest version
Packaged Apps in general
65. − New way to build Web Apps by leveraging
− Web & Service Workers API, push
notifications
− Offline capability (e.g. Indexed DB),
− Web App manifest etc.
− “first class app citizens” / Store integration
− Examples
− Microsoft PWA
− Google PWA
− Cross-plattform Desktop: Electron
PROGRESSIVE WEB APPS
66. 66
− When executing scripts in an HTML page, the page becomes unresponsive until the script is
finished
− single threaded, event based
− A web worker is a JavaScript that runs in the background
− independently of other scripts
− without affecting the performance of the page
You can continue to do whatever you want: clicking, selecting things, etc., while the web
worker runs in the background
− A Web Worker script is defined in a separate JavaScript file
− Creating a Web Worker is simple, the code in worker.js is executed directly after creating the
Worker
− As “extension” service workers are currently in development at W3C which allow scripts to
react on events without the web page even being active (e.g., Web Push)
WEB WORKERS
new Worker("worker.js");
67. 67
− web workers are linked to their creator in a 1:1 relationship
− Since web workers are independently from the web applications execution context
communication between the Worker and the main applications is done using messaging
− Web workers do not have access to the following JavaScript objects
− The window object
− The document object
− The parent object
WEB WORKERS
Worker Script
self.addEventListener('message',
function(e) {
console.log(‚Main said: ', e.data);
self.postMessage(e.data);
});
Main Script
var worker = new
Worker(‚worker.js');
worker.addEventListener('message',
function(e) {
console.log('Worker said: ', e.data);
});
worker.postMessage('Hello World');
68. 68
− service worker can be used for
− push notifications
− background sync (offline usage)
− intercept and handle network requests, including
programmatically managing a cache of responses
− A service worker has a lifecycle that is completely
separate from your web page
− either the service worker is terminated
to save memory
− or it handles fetch and message events
that occur when a network request or
message is made from your page
− service worker must be provided using
a HTTPS connection
SERVICE WORKER
69. 69
− The Push API enables sending of a push message
to a webapp via a push service
− An application server can send a push message at
any time, even when a WebApp or user agent is inactive
− Push messages are delivered to a Service Worker
that runs in the origin of the WebApp
− use the provided information to update
the local state or display a notification to the user
SERVICE WORKER: WEB PUSH
74. 74
− Open Source
− dash.js, shaka-player, hls.js, exoplayer etc.
− MSS and HLS client-side conversion to
DASH/ISOBMFF + CENC
− Hasplayer (MSS/HLSDASH)
− hls.js (HLSDASH)
− dash.js (MSSDASH)
− Shaka-player (HLSDASH)
− Customized versions for specific target platforms
(e.g. hls.js-light, exoplayer-amazon-port)
− All players are production-ready and used in many
commercial projects
WIDE AVAILABILITY OF OPEN SOURCE DASH/HLS PLAYERS
75. 75
BEST PRACTICES: MANIFEST FILES
− Large manifests can lead to long parsing times and an therefore increased startup time
76. 76
− Improve parsing performance and therefore improve startup time by
− Decreasing Segment and Manifest size
(DASH MPD) Use SegmentTemplate instead of SegmentTimeline or
SegmentList
Remove „old“ segments from the manifest according to the DVR window
Reduce segment length/size (for client-side conversion)
− Player improvements
Check for latest updates of the libraries (e.g. for XML, box parsing)
Move computational heavy tasks to separate threads (e.g. WebWorker)
Check for type 1 players to utilize native parsing
Limit manifest updates for the player
BEST PRACTICES: SEGMENT + MANIFEST FILES
77. 77
BEST PRACTICES: SEGMENT + MANIFEST FILES
− Recommendations:
Use validators to check the conformance segments and manifests
DASH-IF Conformance Validator
Apple Media Stream Validator Tool
(MSE) Make sure that the timeline is continuous to prevent playback
stalling
Gzip to compress large Manifest files and improve HTTP latency
78. 78
− (Widevine) Different KIDs for each track
− (Widevine, HDCP) depends on OS support; use
EME MediaKeyStatus enumeration
− (EME,PlayReady) HW security can be queried:
com.microsoft.playready.hardware
− (EME,Fairplay) Different key system versions
exist: com.apple.fps.x_0
− (EME) Specify robustness level, codecs, when
calling EME
− (HTTPS) Browsers enforce no mixed content. All
resources (e.g. license servers) need to be
HTTPS
BEST PRACTICES: DRM + PLAYBACK
89. 89
DEMO: CREATION AND PLAYBACK OF MPEG-DASH CONTENT WITH DRM
• DASH+CENC (PR, WV, ClearKey) content created with CPIX
• Host DASH .mpd and segments on a HTTP server
• Playback in dash.js (using MSE/EME) with keysystem selection
• Add HTTP URL to the .mpd in the DASH Player
512kB
1024kB
2048kB
1280 x 720
1024kB
2048kB
4096kB
1920 x 1080
105. 106
PER-TITLE ENCODING – HOW TO DO IT
Complexity
Analysis
• Perform test
encoding using
CRF, CBR or CQ
encoding with
different values
Convex Hull
• Select bitrate-
resolution
pairs that are
close to the
convex hull
Standard
encoding
• Perform the
standard CBR
/ Two-pass
constrained
VBR
encoding
1.Optional:
PSNR
analysis
• Find corrupt
frames or
scenes
112. 115
− Different types of content require different bitrates
− Simple content has more spatial and temporal redundancy than complex content and
therefor requires a lower bitrate to achieve a certain quality
− Per-title encoding has the potential to:
Reduce storage
Reduce bandwidth costs
Improve the perceptual quality of a video
− Requires additional test encodings and complexity analysis
PER-TITLE ENCODING : SUMMARY
113. 116
- Estimate complexity factor using only 1-2 test encodes and adjust the the static encoding
ladder accordingly
- Per-chunk encoding to tackle different complexities within a single content
- Analyze lowest PSNR to identify problematic frames/scenes
- Use advanced quality metrics like VMAF
- Capped/Constrained CRF encoding for quality based adaptive media streaming
- Bitmovin approach: Send PSNR in manifest to further improve client algorithm
PER-TITLE ENCODING: OUTLOOK & IMPROVEMENTS
115. 118
• Server-based Architecture
MPD centric
Multi-Period
Uses Xlink, DASH-specific events for additional periods
MPD Chaining as a new alternative
Single-Period
Ads and content are transcoded and packaged into a continuous
stream
Uses just-in-time packaging for DAI per user
• App-based Architecture
Requires additional application
Uses DASH user-defined events
AD INSERTION ARCHITECTURES
116. 119
• Messages signaled to the DASH client
MPD Update information
Splice information
Metadata
• Appear inline (MPD level) or inband (Segment level) as ‘emsg‘
boxes
• Important attributes
• schemeIdUri
• value
• presentationTime
DASH EVENTS
117. 120
scheme_id_uri value Description
urn:mpeg:dash:event:2012 1 MPD Validity Expiration event : Triggers MPD
reload
urn:mpeg:dash:event:2012 2 MPD Patch event: Includes XML data to update
the MPD
urn:mpeg:dash:event:2012 3 MPD Update event: Includes a complete new
MPD
DASH SPECIFIC EVENTS
120. 123
• Not directly processed by a DASH client
• DASH has defined how SCTE 35 cue messages may be carried
in DASH Media Segments and DASH Media Presentation
Description (MPD)
• Handed to Ad Management Module
• Example SCTE35 events
• schemeIdUri: „urn:scte:scte35:2014:xml+bin“
• Value: „1001“
CUSTOM EVENTS
121. 124
• Remote Period Elements
• Example: <Period
xlink:href="http://dash.fokus.fraunhofer.de:8080/mws17/period_timescapes"
xlink:actuate="onLoad"/>
• Requires two attributes in the element
xlink:href: Defines URL to remote content
xlink:actuate : Defines resolution time
o onLoad: Directly after the manifest is parsed
o onRequest: At the time the element is accessed
• Enables ad targeting by adding user specific parameters to the xlink:href
attribute
XLINK
124. 127
DEMO: AD-INSERTION
− Server- and app-
based ad insertion
− Web interface for
stream scheduling
− Manifest manipulation:
− Multi-period
− Xlink
− Inline Event
− Live conversion
126. 129
1. Stream scheduler
− Content provider schedules playlist with content and ad assets and requests
sitched manifest from the Ad. Agg. Service
2. Ad Agg. Service
− Ad. Agg. Service contacts Ad Server and requests target ads
− URLs to the manifest files are handed to the Ad Stitcher
3. Ad Stitcher
− Manifests are parsed and a multiperiod MPD is created
− If necessary Xlink elements are included in the MPD
4. Client Player
− Client plays multiperiod MPD and resolves Xlink elements.
SERVER-SIDE AD-INSERTION
Manifest Manipulation
127. 130
1. Stream scheduler
− Content provider schedules playlist with content and ad assets and requests
sitched manifest from the Ad. Agg. Service
2. Ad Agg. Service
− URLs to the manifest files are handed to the Ad Stitcher
3. Ad Stitcher
− Manifests are parsed and a MPD with a single period and DASH custom
events is created
4. Client Player
− Client plays MPD with a single period. An external application processes the
DASH events and plays the ad in a second video elemen
CLIENT-SIDE AD-INSERTION
External application on the client
128. 131
− Server-side logging of HTTP GET requests
− Dispatch IAB VAST tracking events to the client
− Example: EventStream@schemeIdUri = http://dashif.org/identifiers/vast30
− DASH Callback events to issue HTTP GET request to a provided URL
AD TRACKING
129. − Scenario: OTT/Broadband Ad-Insertion
− DVB CM-TA Group working on commercial requirements for targeted advertising, not
exclusive to HbbTV
− Different options possible (mainly HbbTV 2.0.x)
− Ad Signalling:
− MPEG DASH events (MPD or Inband) are mapped to TextTrack objects; contains
ad metadata
− Xlink
− Ad Playback:
− Support for Multiple HTML5 media elements enables pre-fetching in the second
<video> element
− Multiple Periods
DASH+AD-INSERTION IN HBBTV
132. 135
Server-based Ad Insertion
App-based Ad
Insertion
Manifest manipulation
(multiple periods)
Individual packaging
(single period)
Additional application on
the client
Ad Request Server Server Client
Tracking complexity
High High Low
Content generation
complexity
Medium High Low
Ad blocking risk Medium Low High
Seamless Switching Nearly frame-accurate * Frame-accurate Nearly frame-accurate *
Player compatibility Medium High Low
AD INSERTION – DIFFERENT APPROACHES
* Depends on player implementation
151. 154
− End-to-End (E2E) latency or hand-waving latency = time between video or audio being
captured at the source and when it is shown on the screen
− Time to first frame (TTFF): The time it takes for content to initially load
− Low latency results in small buffers Tradeoff between robustness and playing close to
the live edge
END-TO-END LATENCY
154. 157
− With HTTP/1.1 one HTTP and thus TCP request must be
made for each resource
HTTP/2 reduces transport overhead: multiplexing requests,
better data compression, server-push, binary encoding
Server Sent Events (SSE) to send updates or continuous
data stream to the app
Web Sockets for two-way communication with a remote
host
Web RTC for peer 2 peer communication via data channel
QUIC for multicast delivery
TRANSPORT-LEVEL OPTIMIZATION
155. 158
CMAF CHUNKS FOR LOW-LATENCY STREAMING
Source: ISO/IEC 23000-19 Common Media Application Format for Segmented Media
156. 159
STANDARD CMAF FRAGMENT VS LL CMAF FRAGMENT
Famium Packager: Low latency CMAF fragment
Famium Packager: Standard CMAF fragment
159. 163
- @availabilityTimeOffset (ATO) : “Provides the time how much earlier segments are
available compared to their computed availability start time (AST)“
− @availabilityTimeComplete: „Specifies if all Segments of all associated Representation
are complete at the adjusted availability start time. If the value is set to false, then it may
be inferred by the client that the segment is available at its announced location prior
being complete.“
LOW-LATENCY: MPD PARAMETERS
𝐴𝐴𝐴𝐴𝐴𝐴𝐴𝐴 = 𝐴𝐴𝐴𝐴𝐴𝐴 − 𝐴𝐴𝐴𝐴𝐴𝐴
𝐴𝐴𝐴𝐴𝐴𝐴 = 𝑑𝑑𝑠𝑠 − 𝑑𝑑𝑐𝑐 = 7 𝑠𝑠𝑠𝑠𝑠𝑠
Source: Chunked DASH in JavaScript – Robert Alnesjö
161. 165
− Low latency is key for live events (traditonal sport and eSport)
− Activity in DVB and DASH-IF
− DVB CM-AVC Report in Low-Latency Live Service with DASH
− DASH-IF IOP Low Latency DASH
− CMAF offers possibility for low-latency streaming with DASH and HLS
− Modified manifest required
− Modified player required dash.js 2.6.8 has support for low-latency streaming using the
fetch API.
− Client/packager snychronization must be accurate to avoid early/late requests
− Sophisticated ABR algorithm required to compensate for download idle times
SUMMARY
164. 168
360° Video Playout - Fraunhofer FOKUS
COMMERCIAL HMDS VS UHD TV
Source: Harmonic and Viaccess-Orca, VR360 – the operator gateway into Virtual Reality
165. 169
DEMO: 360° 4K FOV ON TV
360° / 16.384 pixel
180°
/
x
8.192
pixel
4K / UHD
166. 170
360° Video Playout - Fraunhofer FOKUS
High quality
360°
streaming
Live 360°
streaming
Cross-
device
Scalable UX
CHALLENGES OF HIGH QUALITY 360° STREAMING
167. 171
Streaming Approaches
360° Video Playout - Fraunhofer FOKUS
− CSP1: Apply adaptive streaming to the whole source 360° video:
Same as for streaming classical videos using DASH (Different
bitrates/representations available for the source 360° video).
− CSP2: Apply adaptive streaming based on selected FOV: Many
versions of the source 360° video are available. In each version, a part of
the video which corresponds to a FOV is available in higher quality and
the rest in lower quality.
− CSP3: Apply adaptive streaming based on video tiles: 360° video is
segmented in tiles with different representations. HEVC/VP9 for best
performance.
360° CLIENT SIDE PROCESSING (CSP)
168. 172
360° Video Playout - Fraunhofer FOKUS
360° CLIENT SIDE PROCESSING
Stitching
360° Video
Storage
360°
Processing
360°
Camera
Server Client
Network
Stream 360°
Video!
169. 173
Playback (Browser)
360° Video Playout - Fraunhofer FOKUS
360° CLIENT SIDE PROCESSING WITH WEB TECH
Request 360°
Segments
• XMLHttpRequest
• Fetch API+Streaming
• WebSocket API
Play 360° Segments
• Video Element
• MSE API
• (EME API)
360° Transformation
& FOV Display
• WebGL API
• Canvas API *
• WebVR API (for HMD)
Capture User Inputs
• Orientation Events
• Touch Events
• Key Events
• Mouse Events
DASH 360°
Segments
360°
Frames
FOV
Frames
User
Inputs
* Canvas API not compatible with DRM-protected media
170. 174
Streaming Approaches
360° Video Playout - Fraunhofer FOKUS
− SSP1: 360° Server Side Live-rendering: the source 360° video is
rendered in the cloud and only the the requested FOV is sent to the
client. The Server needs to handle each client in a separate rendering-
session.
− SSP2: 360° Server Side Pre-rendering: the source 360° video will be
pre-rendered for a different variations of FOVs the output FOV videos will
be stored in a static storage server. DASH is applied to FOVs videos in
the same way as for classical videos.
360° SERVER SIDE PROCESSING (SSP)
171. 175
360° Video Playout - Fraunhofer FOKUS
360° SERVER SIDE PROCESSING (SSP)
Stitching
360° Video
Storage
360°
Processing
360°
Camera
Server Client
Network
Stream FOV
Video!
172. 176
How it works?
360° Video Playout - Fraunhofer FOKUS
360° SERVER SIDE PRE-RENDERING
1s 2s 3s 4s time
15°
30°
45°
60°
75°
90°
angle
Static
Video for
FOV 15°
Static
Video for
FOV 90°
Transition
Video from
FOV 15° at
0.666 s
DASH
Adaptation Set
DASH
Adaptation Set
DASH
Adaptation Sets
173. 177
Playback (Browser)
360° Video Playout - Fraunhofer FOKUS
360° SERVER SIDE PROCESSING
Request FOV
Segments
• XMLHttpRequest
• Fetch API+Streaming
• WebSocket API?
Play Segments
• Video Element
• MSE API
• (EME API)
Capture User Inputs
• Orientation Events
• Touch Events
• Key Events
• Mouse Events
User
Inputs
DASH FOV
Segments
FOV
Frames
174. 178
360° Server Side Pre-rendering (Fraunhofer Cloud Playout) – Live Streaming
Architecture
360° Video Playout - Fraunhofer FOKUS
360° SERVER SIDE PROCESSING
360°
Camera
…
360° Live
Stream
FOV Live
Renderer 1
FOV Live
Renderer 2
FOV Live
Renderer N
FOV Video
Storage
…
CDN
2
1
3
4
5
Player
FOV Video
Storage
…
175. 179
CSP1 (full
adaptive)
CSP2 (FoV
adaptive)
CSP3 (tiles)
SSP1 (cloud
browser)
SSP2 (pre-
rendered)
360° Processing Client Client Client Server Server
Storage Low High Medium Low High
Bandwidth
High Medium Medium Low Low
Motion-to-
photon latency
Low Low Low Medium High
360° SOLUTIONS COMPARISON
360° Video Playout - Fraunhofer FOKUS
177. 181
• Streaming Formats
• Encoding
• Web Media APIs
• Standards update
1. Foundations of Adaptive Streaming
2. OTT Strategy
3. Advanced Web Media Streaming
SUMMARY
178. 182
1. Foundations of Adaptive Streaming
• Cross-platform Deployment
• Web, Native or Hybrid Apps?
• Best Practices
• Multi-DRM backend with CPIX
2. OTT Strategy
3. Advanced Web Media Streaming
SUMMARY
179. 183
1. Foundations of Adaptive Streaming
2. OTT Strategy
• Watermarking
• Per-title Encoding
• OTT and HbbTV Ad-Insertion
• Streaming Metric Analytics using SAND
• Low-latency Streaming
• 360° Media Streaming
3. Advanced Web Media Streaming
SUMMARY