Streaming Media West 2017 - HTML5 Workshop

THE INS AND OUTS OF HTML5
VIDEO WORKFLOWS
Streaming Media West – Streaming Media University
Wednesday, November 1, 2017
9:00 am to 12:00 pm

● Leaders in distributed content delivery systems to accompany growth in OTT traffic
● Experts in HTML5 and mobile video technologies through integrations into open-
source & proprietary players
Streamroot: Who are we?

I. HTML5 basics
A. APIs: the basics
B. Support
II. The HTML5 workflow
A. Encoding
B. Formats
C. Network protocols
D. DRM
E. SSAI
F. HTTPs
III. Choosing a web player
A. Modern player architecture
B. Considerations & what to look for
C. A few open-source and proprietary options
→ EXAMPLES: hls.js
Debugging, subtitles, encryption, ads, Streamroot, ABR
What we’ll be talking about today.
© 2017 Streamroot - All rights reservedFlawless quality. Global reach. FIxed cost.

IV. Choosing mobile & STB
Environments
Building blocks
Your player options
Native vs Webview
IV. Going from POC to production
A. What could go wrong
B. Monitoring / iterating: setting up an AB testing workflow
C. What happens after I do a POC? Workflow, fallbacks, etc.
V. Making it work for you
D. Optimizing: my player’s ABR for my specific use case
What we’ll be talking about today.

I. HTML5 Basics
APIs
1. Media Source Extensions
• Makes HTTP adaptive streaming
possible in HTML5
• Use JavaScript to build streams
and inject data into the video
tag’s buffer
Source: W3C
specs

I. HTML5 Basics
APIs
2. Encrypted Media Extensions
• Allows for DRMs in HTML5
• Provides a way to interact with
content protection systems
• Plugin-free
• Common Encryption (CENC):
standardized key and encryption
methods - multiple DRMs for same
file
Source: W3C
specs

Source:
Bitmovin
1. HTML5 Basics
Support

II. The HTML5 Workflow

Encoding
H.264 → HEVC
- Diverging & multiplying patent pools
- Fees mainly for device producers
but also content licenses
- But… Apple announced support for
HEVC (June)
HEVC Patent pools
- MPEG LA
- HEVC Advanced
- VELOS Media
- Technicolor (standalone)
Mobile $0.50 per device
4K TV $1.50 per device
Other $1 per device
Title by title $0.025 per title
Subscription $0.005-0.025 per user per month
Media $0.0025 per disc/title
HEVC Advance pricing structure

Encoding
VP9 → AV1
- VP9 can now be used with MP4
- AV1: feature freeze in October; hard freeze for
end of 2017
Either way:
- Encode to MP4 and then repackage as
necessary for target platforms
Source: Encoding.com
2017 Global Media Format
Report

Formats
Source: Encoding.com 2017
Global Media Format Report
72%21%7%
Containers:
- less and less linked to codecs
- WebM losing traction
- fMP4/ISOBMFF can now be used everywhere (MSE, CMAF, and even with VP9 and
AV1)
HTTP Streaming formats:

Formats: CMAF
Common Media Application Format:
- ISOBMFF + CENC + H.264/HEVC/etc. + Dash OR HLS manifest
- Apple supports fragmented MP4 in latest HLS version since iOS 10
PROs
Play DASH and HLS from single set of source files!
- Cut storage costs in half, efficient caching
- fMP4 involves less overhead than mpeg-ts
- No more need to transmux
- Single media engine for HLS and DASH
WARNING
- Legacy devices
- DRMs
- Implementation
not ready

Formats: CMAF
In September 2017, Microsoft announced CMAF & CBC support for Playready 4.0

Formats: A note on low latency
CMAF
- segment chunking combined with HTTP/2 Push mechanism for sub-2-second latency
Websockets
- Proprietary protocols (WOWZ) + SRT / UDP
L-HLS (Periscope)
- Modified HLS format
WebRTC
- Formats restrictions, scalability
Low latency (<3 seconds) is hard and expensive, don’t expect it to be magically solved.

DRM
Switch from token authorization to HTML5 DRM
- Microsoft: PlayReady -> Dash/fMP4 -> CTR, CBC
- Chrome, FF: Widevine -> Dash/fMP4 -> CTR, CBC
- iOS: FairPlay -> HLS/TS/fMP4 -> CBC
Keep in mind Flash fallbacks if necessary.

DRM
For complete compatibility lists, including mobile and STB: https://www.buydrm.com/keyos-platform-
compatibility-matrix

Advertising
You’ll need to rewrite the client-side Ad selection in HTML5
Good news: Flash ads are dying too!
On Google Ad-exchange:
- No video ads can be added in Flash since April 2017
- No video ads will be served in Flash starting July 2017
But because of legacy, still may need some Flash fallbacks...

Server-side ad insertion (SSAI)
A new approach on the market: SSAI.
PROs
- Not blocked by ad-blockers
- A smoother experience
CONs
- caching challenges
- not all players are ready, still need player integrations
- interactivity and measurement challenges

Server-side ad insertion (SSAI)
How to cache custom
manifest & ad segments?
SCTE 35 triggers support?
Ad tracking?

Network Protocols / HTTPS
HTTP2
- Improvement on HTTP 1.1
- Adds request multiplexing
- Adds server push
- Still over TCP
QUIC
- Transports HTTP2 over UDP instead
of TCP
- A lot of improvements in RTT,
bandwidth speed, packet recovery...
- Needs both server-side and client-
side implementation
TCP BBR
- Improves the TCP congestion control algorithm
- Improves RTT and available bandwidth by 5 to
50%
- requires server-side implementation only!
HTTPS
- mandatory with HTML5 players!
- Changing to HTTPs can be expensive and long,
depending on your CDN.

III. Web Player Architecture

III. Player architecture
Modern player architecture
Before choosing:
1. Understand the specificities of your use case: type of streaming, user habits,
target platforms, etc.
1. Know your feature requirements: ads, paywall, UX, channels, playlists,
analytics, etc.
More flexible architecture with separation between UX and media engine logic!

Skin
Ads
Playlists
DRM
Manager
Decoder /
Renderer
Authentication
Content decryption
module
Media Engine(s)
UI
Media Engine(s)
Playback
& DRM
Social sharing

1. Skin - the graphic design
of your player
2. UI logic - features defining
all interaction with the user
on top of video playback
Lots of plugins available, video.js for
example
User Interface

Sample UI workflow with authentication, channels and pre-roll ad
User Interface

Easily customizable, with many plugins available for use or inspiration.
Add features as plugins/modules to the core UI base.
Create a unified user experience across browsers, even if the media engine behind it
may vary from device to device.
Check out tools such as React native, Haxe
User Interface

Media Engine

Decoder & DRM manager

1. General criteria
- device, format and codec dependencies
III. What to look for
2. Custom features
- DRMs
- DVR subtitles
- QoS, etc.
3. Perfs & Quality
- startup time
- ABR
- error strategies
4. Street cred, robustness, ease of
use
Light-touch dev:
- Simplicity and stability of the media engine
- Assess extensibility & ease of incorporating
special features
- Media engine customization
- Events exposed
- Debugging
Digging deeper:
- Robust yet flexible core design
- Tests and testing coverage
- community & support

PROs
Supported by DASH-IF
Pushed & maintained by Akamai + tier-1s (BBC)
Highest visibility, big community
Lots of features and use-cases handled
Widely used in production
Huge test suite + online test page
CONs
A little bit of technical debt and complexity
Non-trivial API and customization config
H264/AAC support only
III. What’s out there?

PROs
Built by an entire at Google: solid & smart
Performances and robustness, quick to improve
Simple to get started, good tutorials
Only one supporting WebM, VP8, VP9, open audio codecs
Good support on github and Google groups
CONs
Google-centric
Today lacks some features for large broadcasters
Stricter PR & features policy
No ES6 support
Fewer OVP and open-source all-in-one integrations

PROs
Created by Dailymotion, built from scratch by author of Flashls
Enormous traction and visibility
Clear architecture design, easily extendable
Good robustness and debug demo, responsive support
Widely used in prod by all-in-one players + tier-1s
CONs
Some restrictions from the transmuxing & HLS: no DRMs today,
only AES128

PROs
Seamless Flash fallback with a MediaSource polyfill
De facto solution for HLS with VideoJS
Large community of users (Brightcove + Videojs)
In production with Brightcove with a wide range of customers
CONs
Videojs plugin: not usable without videojs
Lack of public debug tools or pages
Learning curve on providers & tech behind it

But there are also a lot of off-the-shelf options.

IV. Mobile & Other Platforms

Different environments:
- Mobile web (Safari on iOS, Chrome on Android)
- Native Apps
Different APIs, different access levels… often more restrictive than web!
IV. Mobile and other platforms

Building blocks: iOS
Base player: AV Player
No low level-APIs
Almost no fragmentation
HLS
Fairplay
Browsers: no MSE on Safari, but native HLS support

Building blocks: Android
Base Player: MediaPlayer
Low level APIs:
- MediaCodec (like MSE)
- MediaDRM (like EME)
Extended open-source framework: ExoPlayer
DASH, HLS, mp4, Smooth...
Widewine
Browsers: as on Desktop, supports MSE!

Building blocks: Other devices
TVos: like iOS
Android TV: like Android + Playready
STBs:
- Android TV based
- Chromium/Opera Middle-ware based
- Proprietary frameworks (legacy)
SmartTVs:
- big fragmentation (Android, Tizen, proprietary…)
- Not following standards very well

What players to choose?
Build yourself:
- AV Player on iOS
- Exoplayer on Android
Off-the-shelf:
- all built on AV Player & ExoPlayer
- unified UX, UI & config
- adding some bonus features
(simpler DRMs, offline storage,
etc)

HTML5 approach?
iOS:
- no MSE in Safari, nor in Webview
Android:
- can use MSE-based players directly with a WebView!
PROs
Same code and logic on Desktop & Mobile
Comparable performance
Easier UX & UI: HTML & CSS
CONs
Less options than with ExoPlayer
Need to know JS & HTML5

DRMs
iOS:
- Fairplay, only on the latest versions
Android:
- Widewine
So if you are using DRMs, you will need at least Widewine, Fairplay & Playready
to have a good device support.

V. The Path to Production!

What could go wrong?
- - Encoding
- - DRMs
- - HTTPS
Making a smooth transition.
- - HTML5 player rollout
- - Playback & QoS Metrics
- - AB Testing!
I did a POC. Now what?
- - Moving the UX to HTML5
- - Build the media engine extendable architecture (providers)
- - (optional) choose a Fallback alternative

Client AB testing methods:
1) Dynamic Config Injection
2) Different builds with a reverse proxy
AB testing pipeline: General setup

How much time does it take?
The build vs. buy tradeoff.
- Do I have the internal resources to develop,
integrate, customize and maintain?
- Are the features I need already available in
off-the-shelf and/or open-source solutions?
Consider time to market.

VI. Making it work for you

1. Maximize efficiency – stream at the highest bitrate possible
2. Minimize rebuffering – avoid underrun and playback stalls
3. Encourage stability – switch only when necessary
(4. Promote fairness across network bottlenecks)
VI. Introduction to ABR
Design Goals

1. Screen & Player Size
Resolution should never be larger than the actual size of the video player
2. Dropped frame rate
Downgrade when too many dropped frames per second
3. Startup time
Fetch the lowest quality first whenever the buffer is empty
VI. ABR Building Blocks
Constraints and parameters

Available bandwidth estimation → maximize bitrate
Estimate the available bandwidth based on last segment
Available bandwidth = size of chunk / time taken to download
Basic bandwidth-based algorithm:
Estimate bandwidth
from last segment
stats
Choose first bitrate <
estimated bandwidth
Schedule
download
Network
conditions
Estimation Bitrate selection
Bandwidth estimation

More complex bandwidth-based algorithm:
Quantizing: quantize the smoothed bandwidth to a discrete bitrate
Take into account
historical values, not
just the last segment
adjusted bandwidth
Schedule
download
Network
conditions
Estimation Quantizing
Average or adjust
range of values
obtained
Smoothing
Apply coefficient to
adjust bandwidth
estimate
Bitrate selection
Bandwidth estimation

Buffer fill size → Minimize rebuffering ratio
Buffer ratio = buffering time / (buffering time + playback time)
Abandon strategy
Buffer control mechanism

VI. Buffer based algorithms
Pitfalls of bandwidth-based estimations:
• Not resilient to sudden network fluctuations
• Often leads to bitrate oscillations
• Biased by HTTP/TCP calls on the same device/network
→ What if we only used the buffer size?

VI. Buffer-based algorithms: concept
IF Output Rate > Input Rate
THEN Switch Down
IF Input Rate > K * Output Rate
THEN Switch Up
IF BufferSize > Buffer Threshold
THEN Stay at Max Rate

VI. Buffer-based algorithms: pros and cons
Advantages of buffer-based algorithms:
• Fewer estimation issues
• Particularly effective when throughput is highly variable
BUT:
• Too conservative during ramp-up phase
• Issues when you have different chunk sizes (VBR)
• Issues when you don’t have enough buffer!
• → HYBRID Buffer-level based algorithms, with bandwidth measurement used only when
needed (at start-up)

VI. Buffer-based algorithms: pros and cons
→ HYBRID Buffer-level based algorithms, with bandwidth measurement used
only when needed
For VOD:
On startup, while BufferSize<Bmax,
Use bandwidth-based algo to ramp up
more aggressively.
For Live and short-content VOD:
In the case Playlist length << Bmax,
Always use the Bandwidth-based
approach

https://github.com/Dash-Industry-Forum/dash.js/blob/development/src/streaming/rules/abr/BolaRule.js
Uses Utility Theory:
Maximize Vn + y Sn
Where:
Vn is the bitrate utility
Sn is the playback Smoothness
y is the tradeoff weight parameter
VI. EXAMPLE
Dash.js BOLA - another approach

VI. And it works!
Improving hls.js’s ABR logic
1. Start taking historical values.
2. Use EWMA to smooth.
Take into account
historical values, not
just the last segment
adjusted bandwidth
Schedule
download
Network
condition
s
Estimation Quantizing
Average or adjust
range of values
obtained
Smoothing
Apply coefficient to
adjust bandwidth
estimate
Bitrate selection

https://github.com/dailymotion/hls.js/blob/master/src/controller/abr-controller.js
EWMA: Shaka Player:
VI. And it works!

native flashls hls.js,s=9,f=3hls.js,s=0,f=0hls.js,s=6,f=1
Rebuffering, percentile rank
VI. And it works!

Streaming type: VOD / Live ?
Devices: Desktop? Mobile? STBs?
Service: Premium? Ad based?
With data, personalize even more:
- Per country
- Per ISP
- Per network connection type
- Per viewer habits!
VI. Going further
Understand your use case

GET IN TOUCH!
Nikolay Rodionov, Co-Founder and CPO
nikolay@streamroot.io
Erica Beavers, Head of Partnerships
erica@streamroot.io

Streaming Media West 2017 - HTML5 Workshop

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Streaming Media West 2017 - HTML5 Workshop

Similar to Streaming Media West 2017 - HTML5 Workshop (20)

More from Erica Beavers

More from Erica Beavers (6)

Recently uploaded

Recently uploaded (20)

Streaming Media West 2017 - HTML5 Workshop

Editor's Notes