SlideShare a Scribd company logo

Adaptive Streaming of Traditional and Omnidirectional Media

This tutorial consists of three main parts. In the first part, we provide a detailed overview of the HTML5 standard and show how it can be used for adaptive streaming deployments. In particular, we focus on the HTML5 video, media extensions, and multi-bitrate encoding, encapsulation and encryption workflows, and survey well-established streaming solutions. Furthermore, we present experiences from the existing deployments and the relevant de jure and de facto standards (DASH, HLS, CMAF) in this space. In the second part, we focus on omnidirectional (360) media from creation to consumption. We survey means for the acquisition, projection, coding and packaging of omnidirectional media as well as delivery, decoding and rendering methods. Emerging standards and industry practices are covered as well. The last part presents some of the current research trends, open issues that need further exploration and investigation, and various efforts that are underway in the streaming industry.

1 of 105
Adaptive Streaming of Traditional and Omnidirectional Media
Ali C. Begen, Comcast/OzU
Christian Timmerer, AAU/Bitmovin
IEEE ICME Tutorial – Hong Kong, July 2017
Upon Attending This Tutorial, You Will Know About
• Principles of HTTP adaptive streaming for the Web/HTML5
• Principles of omnidirectional (360°) media delivery
• Content generation/distribution/consumption workflows for traditional and
omnidirectional media
• Standards and emerging technologies in the adaptive streaming space
• Current and future research on traditional and omnidirectional media delivery
This tutorial is public, however, some material might be copyrighted
Use proper citation when using content from this tutorial
(Thanks to T. Stockhammer, J. Simmons, K. Hughes, C. Concolato, S. Pham, W. Law
and many others for helping with the material)
IEEE ICME Tutorial - July 2017 2
Ali C. Begen
• Electrical engineering degree from Bilkent
University (2001)
• Ph.D. degree from Georgia Tech (2006)
– Video delivery and multimedia communications
• Research, development and standards at
Cisco (2007-2015)
– IPTV, content delivery, software clients
– Transport and distribution over IP networks
– Enterprise video
• Consulting at Networked Media since 2016
• Assistant professor at OzU and IEEE
Distinguished Lecturer since 2016
Christian Timmerer
• Associate Professor at the Institute of
Information Technology (ITEC), Multimedia
Communication Group (MMC), Alpen-Adria-
Universität Klagenfurt, Austria
• Co-founder and CIO of Bitmovin
• Research interests
– Immersive multimedia communication
– Streaming, adaptation, and
– Quality of experience (QoE)
• Blog: http://blog.timmerer.com; @timse7
Presenters Today
IEEE ICME Tutorial - July 2017 3
ACM MMSys 2018 CfP is Out!
IEEE ICME Tutorial - July 2017 4
• City: Amsterdam
• Dates: June 12-15, 2018
• Co-located with
– NOSSDAV
– MoVid
– MMVE
• http://www.mmsys2018.org/
Agenda
• Part I: Over-the-top (OTT) video and HTTP adaptive streaming
– HTML5 standard and media extensions
– HTTP adaptive streaming building blocks
– Multi-bitrate encoding, encapsulation and encryption workflows
– De jure and de facto standards (DASH, HLS, CMAF)
– Experiences from the existing deployments
• Part II: Omnidirectional (360°) media
– Acquisition, projection, coding and packaging
– Delivery, decoding and rendering
– Emerging standards and industry practices
– Current research trends, open issues, efforts that are underway in the streaming industry
IEEE ICME Tutorial - July 2017 5
IPTV vs. IP (Over-the-Top) Video
First Things First
IPTV IP Video
Best-effort delivery
Quality not guaranteed
Mostly on demand
Paid or ad-based service
Managed delivery
Emphasis on quality
Mostly linear TV
Always a paid service
IEEE ICME Tutorial - July 2017 6
Part I: OTT Video and HTTP Adaptive Streaming
8
Internet (IP aka OTT) Video Essentials
• Reach all connected devicesReach
• Enable live and on-demand delivery to the mass marketScale
• Provide TV-like or even better and richer viewer experienceQuality of Experience
• Enable revenue generation thru paid content, subscriptions or
targeted advertising
Business
• Satisfy regulations for captioning, ratings, parental control and
advertising
Regulatory
IEEE ICME Tutorial - July 2017
9
Creating Revenue – Attracting Eye Balls
• High-end content
– Hollywood movies, TV shows
– Sports
• Excellent quality
– HD/3D/UHD audiovisual presentation w/o artifacts such as pixelization and rebuffering
– Fast startup, fast zapping and low glass-to-glass delay
• Usability
– Navigation, content discovery, battery consumption, trick modes, social network integration
• Service flexibility
– Linear TV
– Time-shifted and on-demand services
• Reach
– Any device, any time
IEEE ICME Tutorial - July 2017
One Request, One Response
Progressive Download
HTTP Request
HTTP Response
IEEE ICME Tutorial - July 2017 10
What is Streaming?
Streaming is transmission of a continuous content from a server
to a client and its simultaneous consumption by the client
Two Main Characteristics
1. Client consumption rate may be limited by real-time constraints as opposed
to just bandwidth availability
2. Server transmission rate (loosely or tightly) matches to client consumption
rate
IEEE ICME Tutorial - July 2017 11
Stalls, Slow Start-Up, Plug-In and DRM Issues
IEEE ICME Tutorial - July 2017 12
Common Annoyances in Streaming
• Unsupported/wrong
– protocol
– plug-in
– codec
– format
– DRM
• Slow start-up
• Poor quality, quality variation
• Frequent freezes/glitches
• Lack of seeking
IEEE ICME Tutorial - July 2017 13
Video Delivery over HTTP
• Enables
playback while
still downloading
• Server sends the
file as fast as
possible
Progressive
Download
• Enables seeking
via media indexing
• Server paces
transmission based
on encoding rate
Pseudo
Streaming
• Content is divided
into short-duration
chunks
• Enables live
streaming and ad
insertion
Chunked
Streaming
• Multiple versions
of the content are
created
• Enables to adapt
to network and
device conditions
Adaptive
Streaming
Adaptive Streaming over HTTP
14
Decoding and
Presentation
Streaming Client
Media Buffer
Content Ingest
(Live or Pre-captured)
Multi-rate
Encoder
Packager Origin (HTTP)
Server
…
…
…
…
ServerStorage
HTTP GET
Request
Response
IEEE ICME Tutorial - July 2017
Adapt Video to Web Rather than Changing the Web
15
Adaptive Streaming over HTTP
• Imitation of streaming via short downloads
– Downloads small chunks to minimize bandwidth waste
– Enables to monitor consumption and track the streaming clients
• Adaptation to dynamic conditions and device capabilities
– Adapts to dynamic conditions in the Internet and home network
– Adapts to display resolution, CPU and memory resources of the streaming client
à Facilitates “any device, anywhere, anytime” paradigm
• Improved quality of experience (not always improved average quality)
– Enables faster start-up and seeking, and quicker buffer fills
– Reduces skips, freezes and stutters
• Use of HTTP
– Well-understood naming/addressing approach, and authentication/authorization infrastructure
– Provides easy traversal for all kinds of middleboxes (e.g., NATs, firewalls)
– Enables cloud access, leverages the existing and cheap HTTP caching infrastructure
IEEE ICME Tutorial - July 2017
Example Representations
Encoding
Bitrate
Resolution
Rep. #1 3.45 Mbps 1280 x 720
Rep. #2 2.2 Mbps 960 x 540
Rep. #3 1.4 Mbps 960 x 540
Rep. #4 900 Kbps 512 x 288
Rep. #5 600 Kbps 512 x 288
Rep. #6 400 Kbps 340 x 192
Rep. #7 200 Kbps 340 x 192
Source: Vertigo MIX10, Alex Zambelli’s Streaming Media Blog, Akamai
Vancouver 2010 Sochi 2014
Encoding
Bitrate
Resolution
Rep. #1 3.45 Mbps 1280 x 720
Rep. #2 1.95 Mbps 848 x 480
Rep. #3 1.25 Mbps 640 x 360
Rep. #4 900 Kbps 512 x 288
Rep. #5 600 Kbps 400 x 224
Rep. #6 400 Kbps 312 x 176
Encoding
Bitrate
Resolution
Rep. #1 3.45 Mbps 1280 x 720
Rep. #2 2.2 Mbps 1024 x 576
Rep. #3 1.4 Mbps 768 x 432
Rep. #4 950 Kbps 640 x 360
Rep. #5 600 Kbps 512 x 288
Rep. #6 400 Kbps 384 x 216
Rep. #7 250 Kbps 384 x 216
Rep. #8 150 Kbps 256 x 144
FIFA 2014
IEEE ICME Tutorial - July 2017 16
List of Accessible Segments and Their Timings
An Example Manifest Format
MPD
Period id = 1
start = 0 s
Period id = 3
start = 300 s
Period id = 4
start = 850 s
Period id = 2
start = 100 s
Adaptation Set 0
subtitle turkish
Adaptation Set 2
audio english
Adaptation Set 1
BaseURL=http://abr.rocks.com/
Representation 2
Rate = 1 Mbps
Representation 4
Rate = 3 Mbps
Representation 1
Rate = 500 Kbps
Representation 3
Rate = 2 Mbps
Resolution = 720p
Segment Info
Duration = 10 s
Template:
3/$Number$.mp4
Segment Access
Initialization Segment
http://abr.rocks.com/3/0.mp4
Media Segment 1
start = 0 s
http://abr.rocks.com/3/1.mp4
Media Segment 2
start = 10 s
http://abr.rocks.com/3/2.mp4
Adaptation Set 3
audio italian
Adaptation Set 1
video
Period id = 2
start = 100 s
Representation 3
Rate = 2 Mbps
Selection of
components/tracks
Well-defined
media format
Selection of
representations
Splicing of arbitrary
content like ads
Chunks with addresses
and timing
17IEEE ICME Tutorial - July 2017
Smart (aka Selfish and Greedy) Clients
18
Client manages
- Manifest(s)
- HTTP transport
- TCP connection(s)
Client monitors/measures
- Playout buffer
- Download times and throughput
- Local resources (CPU, memory, screen, etc.)
- Dropped frames
Client performs adaptation
Request
IEEE ICME Tutorial - July 2017
Response
End-to-End Workflow for OTT
Production Preparation and Staging Distribution Consumption
News
Gathering
Sport Events
Premium
Content
Studio
Multi-bitrate
Encoding
Encapsulation
Protection
Origin Servers
VoD
Content &
Manifests
Live
Content &
Manifests
CDN
IEEE ICME Tutorial - July 2017 19
IEEE ICME Tutorial - July 2017 20
IEEE ICME Tutorial - July 2017 21
• HTML5 is a set of technologies that
allows more powerful Web sites and
applications
– Better semantics
– Better connectivity
– Offline and storage
– Multimedia
– 2D/3D graphics and effects
– Performance and integration
– Device access
– Styling
• Most interesting new elements
– Semantic elements: <header>, <footer>,
<article> and <section>
– Graphic elements: <svg> and <canvas>
– Multimedia elements: <audio> and <video>
A Common Platform across Devices
HTML5
Source: Mozilla MDN, w3schools.com
• These apps can
– leverage native APIs and offline capability
– provide cross-device compatibility (iOS,
Windows, Android, etc.)
– be packaged and published to app stores
• For streaming:
– To a user, it looks like a regular native app
they have downloaded
– The app is actually a container and loads
loads dynamically from the web server
– Server-side changes propagate
automatically to installed apps
Hosted and Progressive Web Apps
IEEE ICME Tutorial - July 2017 22
Source: Google PWA, Microsoft HWA
IEEE ICME Tutorial - July 2017 23
Types of Browser-Based Playback
Source: CTA WAVE
Type 1: Minimum control architectural model for HTML5 support of adaptive
streaming where manifest and heuristics are managed by the user agent
Type 2: Adaptation control architectural model for HTML5 support of
adaptive streaming providing script manageable features
Type 3: Full media control architectural model for HTML5 support of adaptive
streaming allowing script to explicitly send the media segments (MSE + EME)
IEEE ICME Tutorial - July 2017 24
MSE Support in Web Browsers
Source: http://caniuse.com/#search=mse
IEEE ICME Tutorial - July 2017 25
Content Protection – Famous Quotes
• “Digital files cannot be made uncopiable, any more than water can be made not wet”
– Bruce Schneier (cryptographer), May 2001
• “We have Ph.D.'s here that know the stuff cold, and we don't believe it's possible to
protect digital content”
– Steve Jobs, December 2003
• “If we’re still talking about DRM in five years, please take me out and shoot me”
– eMusic CEO David Pakman, February 2007
• “This is unethical”
– Ian Hickson, HTML5 editor, upon learning of the Netflix-Google-Microsoft EME proposal -
February 2012
IEEE ICME Tutorial - July 2017 26
W3C Encrypted
Media Extensions
Source: https://www.w3.org/TR/encrypted-media/
IEEE ICME Tutorial - July 2017 27
EME Support in Web Browsers
Source: http://caniuse.com/#search=eme
Web Video Ecosystem
Encoding
Encryption
Rights Expression
Web Video App Framework
Decoding
Decryption
Rights Management
The fundamental Digital Rights Management
problem derives from a lack of interoperability
which prevents mobility of experience
The solution is to combine interoperable
commercial Web video content with a cross-
platform Web video app framework
IEEE ICME Tutorial - July 2017 28
Source: John Simmons
Contrary to a common misconception, with EME DRM
functionality is not in the HTML/JS app
There is no DRM in HTML5 with EME, and ECP
requires that this be the case
Digital Rights Management
The HTML/JS app selects the DRM and controls key
exchange between DRM client and server
HTML/JavaScript Application
Browser extends HTML5 media element to allow
JavaScript handled key acquisition
Browser
A CDM exposes a key system to JavaScript; it is
transparent whether the CDM is in the browser Content Decryption Module
The Web, EME and Enhanced Content Protection (ECP)
IEEE ICME Tutorial - July 2017 29
Source: John Simmons
IEEE ICME Tutorial - July 2017
DRM Support on Desktop Browsers
30
Browser OS EME/CDM Flash Player NPAPI/ Silverlight 5
Chrome Win Widevine (Yes) No
OS X (Yes) No
Linux (Yes) No
Firefox Win Widevine Yes No
OS X Yes No
Linux Yes No
Safari > OS X Yosemite (Fairplay) Yes Yes
< OS X Yosemite No Yes Yes
IE/Edge < Win 7 No Yes Yes
Win 10 PlayReady Yes No
DRM Support on Mobile Platforms
Platform Browser Native App / DRM
iOS No MSE/EME yet
HLS (AES-128 CBC) via <video>
Native SDK and WebView: FairPlay
Android MSE/EME Native + Android SDK (MediaDRM APIs) - (Widevine
+ OMA v2)
ExoPlayer
Native + WebView with Widevine
Windows 10 MSE/EME WebViews/Hosted Web Apps with PlayReady
IEEE ICME Tutorial - July 2017 31
Ad Insertion Methods
• Server-based
– MPD is constructed or updated based on the ads
– For multi-period MPDs, we can use
• XLink
• MPD chaining
– For single-period MPDs
• Ads are spliced into a continuous stream
• Splicing can be offline or just-in-time
• App-based
– Requires additional support in the application
– Uses DASH-specified user-defined events
IEEE ICME Tutorial - July 2017 32
Ad Insertion – XLink Resolution
IEEE ICME Tutorial - July 2017 33
IEEE ICME Tutorial - July 2017 34
Streaming over HTTP – The Promise
• Leverage tried-and-true Web infrastructure for scaling
– Video is just ordinary Web content!
• Leverage tried-and-true TCP
– Congestion avoidance
– Reliability
– No special QoS for video
It should all “just work” J
35
Does It Just Work?
• When streaming clients compete with other traffic, mostly yes
• When streaming clients compete with each other, we begin to see problems:
– The clients’ adaptation behaviors interact with each other
– The competing clients form an “accidental” distributed control-feedback system
• Unexpected behaviors will result in places like
– Multiple screens within a household
– ISP access and aggregation links
– Small cells in stadiums and malls
IEEE ICME Tutorial - July 2017
A Single Microsoft Smooth Streaming Client under a Controlled Environment
36
Demystifying a Streaming Client
0
1
2
3
4
5
0 50 100 150 200 250 300 350 400 450 500
Bitrate(Mbps)
Time (s)
Available Bandwidth Requests Chunk Tput Average Tput
Reading: “An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP,” ACM MMSys 2011
Buffer-filling State
Back-to-back requests
(Not pipelined)
Steady State
Periodic requests
IEEE ICME Tutorial - July 2017
10 (Commercial) Streaming Clients Sharing a 10 Mbps Link
IEEE ICME Tutorial - July 2017 37
Typical Example
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500
RequestedBitrate(Kbps)
Time (s)
Client1 Client2 Client3
38
Viewer Experience Statistics
Selfishness Hurts Everyone
Source: Conviva Viewer Experience Report, 2015
IEEE ICME Tutorial - July 2017
Inner and Outer Control Loops
39
HTTP Server
Manifest
Media
HTTP
Origin
Module
TCP Sender
Streaming Client
Manifest
Resource
Monitors
Streaming
Application
TCP ReceiverData / ACK
Request
Response
There could be multiple TCPs destined to the same or different servers
IEEE ICME Tutorial - July 2017
IEEE ICME Tutorial - July 2017 40
Tradeoffs in Adaptive Streaming
Overall quality
Quality stability
Proximity to live edge
Stalls
Zapping/seeking time
Two Competing Clients
41
Understanding the Root Cause
• Depending on the timing of the ON periods:
– Unfairness, underutilization and/or instability may occur
– Clients may grossly overestimate their fair share of the available bandwidth
Clients cannot figure out how much bandwidth to use until they use too much
Reading: “What happens when HTTP adaptive streaming players compete for bandwidth?,” ACM NOSSDAV 2012
IEEE ICME Tutorial - July 2017
IEEE ICME Tutorial - July 2017 42
How to Solve the Issues?
Solution
Approaches
Fix the clients
Enable a control
plane
Get support from
the network
IEEE ICME Tutorial - July 2017 43
One Slide on Software Defined Networking (SDN)
Control and data planes are decoupled, network intelligence and state are logically
centralized, and the underlying network infrastructure is abstracted from the apps
IEEE ICME Tutorial - July 2017 44
SDN-Based Bitrate Adaptation
Reading: “SDNDASH: improving QoE of HTTP adaptive streaming using software defined networking,” ACM MM 2016
Control Points in the Ecosystem
• I want to make sure that my content is protected and looks awesome on any device
• I want to make sure that my ads are viewed, trackable and measurable
• I want to make sure that my servers are properly used and latency is low
• I want to control the QoE of all my customers, differentiate my own services, make $$$
from OTT services
• I want to make sure that my device/app provides the best possible video quality
• I want the best quality for minimal $
ConsumerContent
Provider
ISP
CDN
ProviderAdvertiser
Device/
App
45IEEE ICME Tutorial - July 2017
IEEE ICME Tutorial - July 2017 46
DASH Part 5: Server and Network Assisted DASH (SAND)
A Control Plane Approach
DANE: DASH-assisting network element
PER: Parameters for enhancing reception
PED: Parameters for enhancing delivery
Media
PER Messages
Metrics and Status Messages
PED Messages
DANE (Origin) DANE Regular
Network Element
DASH Client
DANE (Analytics)
Dead, Surviving and New-Born Technologies
• Move Adaptive Stream (Long gone)
– http://www.movenetworks.com
• Microsoft Smooth Streaming (Legacy)
– http://www.iis.net/expand/SmoothStreaming
• Adobe Flash (Almost dead)
– http://www.adobe.com/products/flashplayer.html
• Adobe HTTP Dynamic Streaming (Legacy)
– http://www.adobe.com/products/httpdynamicstreaming
• Apple HTTP Live Streaming (The elephant in the room)
– http://tools.ietf.org/html/draft-pantos-http-live-streaming (Soon to be an RFC)
• MPEG DASH and CMAF (The standards)
– http://mpeg.chiariglione.org/standards/mpeg-dash
– http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format
IEEE ICME Tutorial - July 2017 47
• Fragmented architectures
– Advertising, DRM, metadata, blackouts, etc.
• Investing in more hardware and
software
– Increased CapEx and OpEx
• Lack of consistent analytics
• Preparing and delivering each asset in
several incompatible formats
– Higher storage and transport costs
• Confusion due to the lack of skills to
troubleshoot problems
• Lack of common experience across
devices for the same service
– Tricks, captions, subtitles, ads, etc.
What does Fragmentation Mean?
Higher Costs
Less Scalability
Smaller Reach
Frustration
Skepticism
Slow Adoption
IEEE ICME Tutorial - July 2017 48
49
DASH intends to be to
the Internet world …
what MPEG2-TS has been to
the broadcast world
IEEE ICME Tutorial - July 2017
50
DASH intendsed to be to
the Internet world …
what MPEG2-TS has been to
the broadcast world
IEEE ICME Tutorial - July 2017
What is HAPPENING in MPEG ?
DASH Design Principles
• DASH is not
– system, protocol, presentation, codec, interactivity, DRM, client specification
• DASH is an enabler
– It provides formats to enable efficient and high-quality delivery of streaming services over the Internet
– System definition is left to other organizations
• Design choices
– Enable reuse of existing technologies (containers, codecs, DRM, etc.)
– Enable deployment on top of CDNs
– Enable live and on-demand experiences
– Move intelligence from network to client, enable client differentiation
– Provide simple interoperability points (profiles)
IEEE ICME Tutorial - July 2017 52
53
Shown in Red
Scope of MPEG DASH
HTTP Server DASH Client
Control Engine
MediaEngines
HTTP ClientHTTP 1.1
Segment
ParserMPD Transport
MPD
Parser
MPD
MPD
MPD
...
...
...
IEEE ICME Tutorial - July 2017
IEEE ICME Tutorial - July 2017 54
DASH (ISO/IEC 23009) Parts
• 23009-1: Media Presentation Description and Segment Formats
– Amd. 1: NTP sync, extended profiles
– Amd. 2: SRD, URL parameter insertion, role extensions
– Amd. 3: External MPD link, period continuity, generalized HTTP header extensions/queries
– Amd. 4: TV profile, MPD chaining/resetting, data URLs in MPD, switching across adaptation sets
– à 3rd edition (FDIS), to be published soon
• 23009-2: Conformance and Reference Software
– 2nd edition (FDIS), to be published in 2017
– Amd. 1: SAND conformance rules
• 23009-3: Implementation Guidelines (Informative)
– 2nd edition Amd. 1: Multi-track content guidelines and rules, fix on dependent representations
• 23009-4: Segment Encryption and Authentication
– 2nd edition is in progress
IEEE ICME Tutorial - July 2017 55
DASH (ISO/IEC 23009) Parts
• 23009-5: Server and Network Assisted DASH (SAND)
– 1st edition, published in Feb. 2017
• 23009-6: DASH over Full Duplex HTTP-based Protocols (FDH)
– 1st edition (FDIS), to be published in 2017
• 23009-7: Delivery of CMAF Contents with DASH (Informative)
– WD available, PDTR to be published after the July meeting
IEEE ICME Tutorial - July 2017 56
Part 1 – 3rd Edition
• TV profile
– Broadcast TV content can be distributed over broadcast and/or broadband networks using DASH
– Segments do not need to be a switching access point
• MPD chaining
– Multiple MPDs can be chained to each other for easier ad insertion
• Data URLs in MPD
– Initialization segments can be included in the MPD (only in the TV profile)
• Switching across adaptation sets
– A capable client can switch between AVC and HEVC tracks
IEEE ICME Tutorial - July 2017 57
Part 6 – DASH over Full Duplex HTTP-Based Protocols
• Part 6 describes how a client can signal various types of “push” behaviors to a server in
HTTP/2 and for WebSockets
• This is primarily targeted for low-latency live streaming
• Similar activities
– DASH over LTE broadcast (3GPP SA4)
– DVB ABR multicast
– ATSC 3.0 hybrid delivery
IEEE ICME Tutorial - July 2017 58
Ongoing Work as of MPEG 119 (July 2017)
• Core Experiments
– High quality VR delivery with DASH (DASH-VR)
• Technologies under Consideration
– Support for Controlled Playback in DASH
– Playback control
– Owner defined content identifiers within MPDs
– Usage of HEVC tile tracks in DASH
– Relation between presentation time and EPT
– Service-level service protection using segment encryption
IEEE ICME Tutorial - July 2017 59
Common Media Application Format (CMAF)
• Media delivery has three main components:
– Media format
– Manifest
– Delivery
• CMAF defines the media format only (Fragments, headers, segments, chunks, tracks)
– CMAF does not specify a manifest format
– CMAF does not specify a delivery method, either
• CMAF uses ISO-BMFF and common encryption (CENC)
– CENC means the media fragments can be decrypted/decoded by devices using different DRMs
(aka multi-DRM support)
– CMAF does not mandate CTR or CBC mode (which makes CMAF useful only for unencrypted
content for now)
• MPEG technologies (DASH and MMT) may be used for delivering CMAF content
IEEE ICME Tutorial - July 2017 60
Common Media Application Format (CMAF)
• CMAF defines media profiles for
– Video
– Audio
– Subtitle
• CMAF defines presentation profiles by selection a media profile from each category
• Current status
– iOS 10 supports CMAF with HLS manifests
– PDAM1 (w16821)
• Discussions on SHVC support
• AAC 7.1 support
– Conformance and interop is WIP with DASH-IF
– Non-MPEG codecs are under discussion
IEEE ICME Tutorial - July 2017 61
CMAF Development Timeline
Feb. 2016
Working Draft
June 2016
Committee
Draft
Nov. 2016
Draft
International
Standard
Jan. 2017
Study of Draft
International
Standard
May 2017
Final Draft
International
Standard
July 2017
Approved
International
Standard
Ballot Ballot Ballot
IEEE ICME Tutorial - July 2017 62
CMAF ISO-BMFF Media Objects
Encoder
Encryption
Packaging
CMAF
Header
CMAF
Fragment
CMAF
Fragment
CMAF
Chunk
CMAF
Chunk
CMAF
Chunk
CMAF
Fragment
R
A
P
R
A
P
R
A
P
R
A
P
CMAF
Fragment
CMAF
Segment
CMAF
Segment
CMAF Track File
• Manifests typically provide URLs to
– CMAF track files
– CMAF header + CMAF segments
• single/multiple fragment(s)
– CMAF header + CMAF chunk
CMAF Chunks
CMAF
Header
CMAF Content Consumption
IEEE ICME Tutorial - July 2017 63
Un-
packaging
MSE
CMAF
Track File
CMAF Fragments
Switching can only happen seamlessly
at CMAF Fragment boundaries
CMAF
Segments
CMAF
Header
CMAF
Header
• Content format issues
– Each asset is copied to multiple media
formats
• different video codecs
• different audio codecs
• different (regional) frame rates
– Cost to content creators and distributors
– Inefficiencies in CDNs and higher storage
costs
• Platform issues
– Lack of consistent app behavior across
platforms
– Varying video features, APIs and semantics
across platforms
• Playback issues
– Codec incompatibility
– Partial profile support
– Switching bitrate glitches
– Audio discontinuities
– Ad splicing problems
– Long-term playback instability
– Request protocol deficiencies
– Memory problems, CPU weaknesses
– Scaling (display) issues
– Variable HDR support
– Unknown capabilities
Commercial OTT Issues
64IEEE ICME Tutorial - July 2017
CDN
delivery
ismc
Smooth
mpd
DASH
m3u8
HLS
Source
Encoder
f4m
HDS
m3u8
m3u8
mpd
mpd
ismc
ismc
f4m
f4m
mpd
CMAF
m3u8
mpd
m3u8
mpd
m3u8
mpd
m3u8
mpd
m3u8
mpd
m3u8
More efficient
edge caching
Storage and
packaging costs
reduced
Multi-Platform OTT Workflow with CMAF
IEEE ICME Tutorial - July 2017 65
CTA Web Application Video Ecosystem (WAVE)
Content specification based
upcoming CMAF, compatible
with DASH and HLS
Testable requirements
covering the most
common device playback
interoperability issues
Reference application
framework based on
HTML5 providing
functional guidelines for
playback interop
Content Specification
HTML5 Reference
Platform
Device Playback
Requirements
Test Suite
IEEE ICME Tutorial - July 2017 66
Web Media API Community Group
• Developers want to deploy their content on a range of devices and platforms, e.g., TVs,
set-tops and mobile devices
• To ensure a smooth user experience across devices, these user agents need to
support a minimum set of Web technologies
• This Community Group plans to specify such a set of Web technologies and
additionally plans to provide guidance for developers and implementers
• Web Media APIs 2017
– Details the Web APIs that should be included in device implementations in 2017
– https://www.w3.org/community/webmediaapi/
• Web Media Application Developer Guidelines
– Companion guide to outline best practices for implementing Web media apps
– https://w3c.github.io/webmediaguidelines/
IEEE ICME Tutorial - July 2017 67
Many Definitions Do Exist – It is Like an Elephant
What is Quality?
IEEE ICME Tutorial - July 2017 68
IEEE ICME Tutorial - July 2017 69
Segments Have Different Complexities
Bitrate
Quality Video
Segment #1
Equal Bitrate Allocation
among Segments
Consistent
Quality Video
Segment #2
70IEEE ICME Tutorial - July 2017
Guidelines Limited Bitrate Variability to (Mostly) 10% So Far
Adaptation Feature Does Not Deliver Consistent Quality
Easy
Moderate
Easy
Easy
Easy
Difficult
Difficult
Difficult
Moderate
Moderate
Moderate
Moderate
…
S
Time (s)
Segment Size
0 2 4 6 8 10 12 14 16 18 20 22 24
Segment Quality
QCBR
Small variation in
encoding bitrate
Large variation
in quality
If there is something worse than having to watch a video at a
lousy quality, it is to watch that video with varying quality
71IEEE ICME Tutorial - July 2017
What if We Encode in a More Subtle Fashion?
Easy
Moderate
Easy
Easy
Easy
Difficult
Difficult
Difficult
Moderate
Moderate
Moderate
Moderate
…
Time (s)
Segment Size
0 2 4 6 8 10 12 14 16 18 20 22 24
Segment Quality
QVBR
While we spend the same total amount of bits, we not only
increase average quality but also reduce quality variation
Large variation in
encoding bitrate
Low variation in
quality
S
HLS authoring spec for ATV allows 2x capping rate for VoD. For linear content, variability is limited to 10-25% range.
72IEEE ICME Tutorial - July 2017
Content Aware
Encoding
Content Aware
Streaming
Generating VBR-encoded segments is easy,
but streaming them is not!
IEEE ICME Tutorial - July 2017 73
Multiple Representations Naturally Enable “Cherry-Picking”
74
2.5 Mbps
Network
HTTP Server Streaming Client
…
…
…
…
…
3 4
3 4
3 4
3 4
4 Mbps
3 Mbps
2 Mbps
1 Mbps
5
5
5
5
2
2
2
2
2 135 4
Time (s)
SegmentSize
…
S
SegmentQuality
QVBR
0 2 4 6 8 10
IEEE ICME Tutorial - July 2017
Reading: “Streaming video over HTTP with consistent quality,” ACM MMSys 2014
75
Dimensions – In-Stream vs. Between-Streams
• Same principle applies to both:
– In-stream Case: Temporal bit shifting between segments
– Between-streams Case: Bit shifting between streams sharing a bottleneck link
Bitrate
Quality Video
Segment 1
Video
Segment 2
CBR
CQ
Bitrate
Quality Stream 1
(News)
Stream 2
(Sports)
Equal Bandwidth
Sharing
CQ
IEEE ICME Tutorial - July 2017
Deployment Challenges
• Challenge 1: Development of quality metrics and temporal pooling models
– A common metric that is suitable for a variety of content types
– A temporal pooling model that will reliably work for different viewer profiles, devices and networks
• Different viewers have different sensitivity levels to glitches for different content types, and they are also
forgiving in different time scales
– A young viewer (likely to have longer-term memory) watching sports on a big screen vs. an elder viewer (likely to have
short-term memory) watching news on a smaller screen
• Challenge 2: Integration into popular streaming client implementations
– Many ecosystems are closed or proprietary, and one may not have access to the client algorithm
to make the necessary changes (e.g., Apple HLS in iOS)
– Standards bodies and industry consortiums may lead the way to develop certain guidelines
76IEEE ICME Tutorial - July 2017
Reading: “Quality-aware HTTP adaptive streaming,” IBC 2015
Deployment Challenges
• Challenge 3: Development of metadata standards
– Computing the quality metric for each segment in each representation for each content is a
tedious task, which is the easiest to deal with at the encoder or packager
– Packing the metric values and conveying this information to all the clients in a timely and scalable
manner is an equally important task
– The timed metadata spec in MPEG (ISO/IEC 23001-10) is a good candidate for this task
77IEEE ICME Tutorial - July 2017
Deployment Challenges
• Challenge 4: Expansion to multi-client scenarios
– We need controlled unfairness (which is fairness in quality not bitrate) among clients adaptively
streaming the same or a different content over a network sharing resources (e.g., access
network)
• Easy scenario: One adult watching sports on a big screen vs. one adult watching a food show on a tablet
• More complex scenario: One adult watching sports on a phone vs. three adults watching news on a big screen
– The optimization across a number of streaming clients has to be done based on the utilities of the
streamed videos, which depend on factors such as:
• Spatial pooling model
• Content types
• Content features
• Rendering devices
• Audience profiles and sizes
– SAND can help deploy controlled unfairness that we need in quality-aware streaming in multi-
client scenarios
78IEEE ICME Tutorial - July 2017
79
Need for Analytics
Service
Provider:
“Your video
or CDN
provider
must be
slow”
Video/CDN
Provider:
“Your home
network
must be
slow”
Consumer:
“The device
or the app is
slow”
Device/App
Vendor:
“It must be
the OS”
OS Vendor:
“Your
Internet
connection
must be
bad”
IEEE ICME Tutorial - July 2017
IEEE ICME Tutorial - July 2017 80
Four Major Areas of Further Exploration
• How to choose bitrate/resolution pairs to make up/downshifts least visible
• How to pick the segment durations
• How to achieve content aware streaming and not just content aware
encoding
Content Preparation
• What information could the network provide to streaming clients
• How to achieve controlled unfairness
• MPTCP or QUIC better than TCP? What about multicast and HTTP/2?
Distribution and
Delivery
• How to model streaming dynamics for different genres
• How to model the impact of faster zapping and trick modes on the QoE
• Understanding the impact of QoE on viewer engagement
QoE Modeling and
Client Design
• Understanding the interaction of adaptive streaming with caching in CDNs
• Extracting actions based on real-time analytics
• Fixing issues faster and remotely
Analytics, Fault
Isolation, Diagnostics
Part II: Omnidirectional (360°) Media
MPEG’s Definition
What is Virtual Reality (VR)
VR is a rendered environment (visual and acoustic, pre-
dominantly real-world) providing an immersive experience to
a user who can interact with it in a seemingly real or physical
way using special electronic equipment
IEEE ICME Tutorial - July 2017 82
IEEE ICME Tutorial - July 2017 83
Virtual Reality (VR) Puts Us in a Virtual World
Source: Phil Chou
IEEE ICME Tutorial - July 2017 84
Augmented Reality (AR) Puts Virtual Objects in Our World
Source: Phil Chou
Delivering High-Quality VR in Economic Scale is Extremely Challenging
The VR Challenge
30K x 24K x 36 x 60 x 2 / 600
~5.2 Gbps
Resolution
Bit depth Frame rate Stereoscopic
Compression gain
IEEE ICME Tutorial - July 2017 85
Ultimate Level of Immersion
Interactions
So intuitive that they
become second nature
Sounds
So accurate that
they are true to life
Visuals
So vibrant that they are eventually
indistinguishable from the real world
IEEE ICME Tutorial - July 2017 86
Visual
quality
Sound
quality
Intuitive
interactions
Immersive VR Has Extreme Requirements
Immersion
High resolution audio
Up to human hearing capabilities
3D audio
Realistic 3D, positional, surround
audio that is accurate to the real world
Precise motion tracking
Accurate on-device motion tracking
Minimal latency
Minimized system latency
to remove perceptible lag
Natural user interfaces
Seamlessly interact with VR using
natural movements, free from wires
Extreme pixel
quantity and quality
Screen is very close to the eyes
Spherical view
Look anywhere with a
full 360° spherical view
Stereoscopic display
Humans see in 3D
IEEE ICME Tutorial - July 2017 87
Source: Thomas Stockhammer
Need for Higher Resolutions on Mobile Phones?
IEEE ICME Tutorial - July 2017 88
Source: Thomas Stockhammer
The Human Eye can Only See High Resolution Where the Fovea is Focused
Foveated Rendering Reduces Pixel Processing
High resolution
Low resolution
The GPU renders a small rectangle at a
high resolution and the rest of the FoV at a
lower resolution
IEEE ICME Tutorial - July 2017 89
Generate Video
• Simultaneously capture video with
multiple cameras from different views to
generate 360° spherical video
• Use twice the cameras for stereoscopic
video
• Undistort, stitch together and map the
discrete images to a equirectangular or
cubemap format
• Encode video
Playback Video
• Decode video
• Apply an equirectangular or cubemap
UV projection
• Determine the pose and show
appropriate view of 360° spherical video
Generating and Consuming 360° Spherical Video
Discrete unstitched
camera images for 360°
spherical view
Equirectangular
image
Cubemap image
Left eye VR
headset view
IEEE ICME Tutorial - July 2017 90
IEEE ICME Tutorial - July 2017 91
Adaptive Streaming of VR Content
• Required resolution of a panoramic video for achieving 4K resolution for viewport with
120° field of view (FoV)
• 3-DoF
– “In which direction I look,” look around the
virtual world from a fixed point
– Detect rotational movement
• 6-DoF
– “Where I am and in which direction I look,”
move freely in the virtual world and look
around corners
– Detect rotational movement and
translational movement
Precise Motion Tracking of Head Movements and Gaze
Z
X
Z
Y
Pitch
Yaw
Roll
IEEE ICME Tutorial - July 2017 92
Lag Prevents Immersion and Causes Discomfort
Minimizing Motion-to Photon Latency is Important
Low latency Noticeable latency
IEEE ICME Tutorial - July 2017 93
MPEG-I
• New MPEG project
– ISO/IEC 23090: Coded Representation of Immersive Media
• Five parts are currently in progress:
– Technical report: use cases, requirements and architectures
– Omnidirectional media format (OMAF)
– New and immersive video coding
– New and immersive audio coding
– Point cloud coding
• Two more parts are considered
– VR metadata
– VR metrics
IEEE ICME Tutorial - July 2017 94
OMAF
• Scope: 360° video and associated audio, 3-DoF only
• Focus on the media format that enables VR video production, delivery and
consumption
– Projection and region-wise packing that are used to generate 2D video from the sphere signal
– File format encapsulation and metadata signaling
– DASH encapsulation and metadata signaling
– Video/audio codecs and profiles/brands
– Informative examples of viewport-dependent approaches
IEEE ICME Tutorial - July 2017 95
DASH/OMAF System Diagram
Multi-
camera
capture
Stitch Project
HEVC
encode
OMAF
composing
DASH server
DASH
client
network
HEVC
Decode Renderer
Head mounted
device (HMD)
DASH
encapsulation
Head and eye tracking telemetry
Request for
tiles within
FoV window
Projection mapping metadata
Projection mapping metadata
authoring
CDN
VR player
IEEE ICME Tutorial - July 2017 96
Adaptive Streaming of VR Content
IEEE ICME Tutorial - July 2017 97
Projection
format:
Equirectangular
projection
Coding format: AVC
File format: ISOBMFF,
M2TS
Delivery format: DASH,
HLS
Web clients:
HTML5, MSE,
WebVR
2018 20202017 2019 2021
Internet Video
Coding
IoMT
Media Orchestration
CMAF
CDVA (Video Analysis)
Audio Wave
Field Coding
AR/VR Audio
extensions
New codec
with Lightfield
New & Immersive
Video Coding
Point Cloud
Compression
HDR TR HDR TR 2
Omnidirectional
Media Format OMAF v2
Genome Compression
Network-Distributed
Video Coding
Coding
2022
Systems
and Tools
Immersive
Media AF
Augmented
Media AF
Cross-platform
media distribution
VR360, on-demand
and live (3-DoF)
Immersive media
with 6-DoF
Combining natural
and synthetic content
MPEG’s Five-Year Roadmap
IEEE ICME Tutorial - July 2017 98
Tile-Based Streaming
IEEE ICME Tutorial - July 2017 99
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Encoding &
Packaging
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Delivery
MPEG-HEVC/H.265
Tiles in ISOBMFF
Adaptive
Player
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile-based streaming of VR/360°
content with MPEG-DASH SRD
…
…
…
…
Adaptive Streaming
using MPEG-DASH
SRD
Head Mounted Displays
Browsers, Smart (Mobile) Devices
(Stereo) 2D, (Stereo) 3D
IEEE ICME Tutorial - July 2017 100
Tile-Based Streaming Options
IEEE ICME Tutorial - July 2017 101
ODV Quality Metrics: S-PSNR
IEEE ICME Tutorial - July 2017 102
ODV Quality Metrics: V-PSNR
Thank You
“Television! Teacher, mother, secret lover”
– Homer Simpson
IEEE ICME Tutorial - July 2017 103
IEEE ICME Tutorial - July 2017 104
Source Code for Client Implementations
• DASH Industry Forum
– http://dashif.org/software/
• Open Source Implementations/Frameworks
– http://dash.itec.aau.at/
– http://gpac.wp.mines-telecom.fr/
– https://github.com/google/shaka-player
• JW Player
– https://github.com/jwplayer/jwplayer
IEEE ICME Tutorial - July 2017 105
SAND Implementations on GitHub
• SANDLibrary: https://github.com/fraunhoferfokus/SANDLibrary
• SANDServer: https://github.com/fraunhoferfokus/SANDServer
• TNO’s SAND demo: https://github.com/tnomedialab/sand
©MatthiasHeyde/FraunhoferFOKUS

Recommended

A Journey Towards Fully Immersive Media Access
A Journey Towards Fully Immersive Media AccessA Journey Towards Fully Immersive Media Access
A Journey Towards Fully Immersive Media AccessAlpen-Adria-Universität
 
Investigation of YouTube regarding Content Provisioning for HTTP Adaptive Str...
Investigation of YouTube regarding Content Provisioning for HTTP Adaptive Str...Investigation of YouTube regarding Content Provisioning for HTTP Adaptive Str...
Investigation of YouTube regarding Content Provisioning for HTTP Adaptive Str...Alpen-Adria-Universität
 
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming SystemsAutomated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming SystemsAlpen-Adria-Universität
 
Ultra-High-Definition Quality of Experience with MPEG-DASH
Ultra-High-Definition Quality of Experience with MPEG-DASHUltra-High-Definition Quality of Experience with MPEG-DASH
Ultra-High-Definition Quality of Experience with MPEG-DASHBitmovin Inc
 
CAdViSE: Cloud based Adaptive Video Streaming Evaluation Framework for the Au...
CAdViSE: Cloud based Adaptive Video Streaming Evaluation Framework for the Au...CAdViSE: Cloud based Adaptive Video Streaming Evaluation Framework for the Au...
CAdViSE: Cloud based Adaptive Video Streaming Evaluation Framework for the Au...Alpen-Adria-Universität
 
Quality of Experience for Inter-Destination Media Synchronization
Quality of Experience for Inter-Destination Media SynchronizationQuality of Experience for Inter-Destination Media Synchronization
Quality of Experience for Inter-Destination Media SynchronizationAlpen-Adria-Universität
 
CMAF 101 by Cyril Concolato
CMAF 101 by Cyril ConcolatoCMAF 101 by Cyril Concolato
CMAF 101 by Cyril ConcolatoErica Beavers
 

More Related Content

What's hot

CAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR SystemsCAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR SystemsAlpen-Adria-Universität
 
bitdash - Simple & Easy MPEG-DASH Player for Web and Mobile
bitdash - Simple & Easy MPEG-DASH Player for Web and Mobilebitdash - Simple & Easy MPEG-DASH Player for Web and Mobile
bitdash - Simple & Easy MPEG-DASH Player for Web and MobileBitmovin Inc
 
On Optimizing Resource Utilization in AVC-based Real-time Video Streaming
On Optimizing Resource Utilization in AVC-based Real-time Video StreamingOn Optimizing Resource Utilization in AVC-based Real-time Video Streaming
On Optimizing Resource Utilization in AVC-based Real-time Video StreamingAlpen-Adria-Universität
 
Towards Optimal Multirate Encoding for HTTP Adaptive Streaming
Towards Optimal Multirate Encoding for HTTP Adaptive StreamingTowards Optimal Multirate Encoding for HTTP Adaptive Streaming
Towards Optimal Multirate Encoding for HTTP Adaptive StreamingAlpen-Adria-Universität
 
Streaming tools comparison
Streaming tools comparisonStreaming tools comparison
Streaming tools comparisonCleveroad
 
Research Group Multimedia Communication (MMC)
Research Group Multimedia Communication (MMC)Research Group Multimedia Communication (MMC)
Research Group Multimedia Communication (MMC)Alpen-Adria-Universität
 

What's hot (7)

CAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR SystemsCAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR Systems
 
bitdash - Simple & Easy MPEG-DASH Player for Web and Mobile
bitdash - Simple & Easy MPEG-DASH Player for Web and Mobilebitdash - Simple & Easy MPEG-DASH Player for Web and Mobile
bitdash - Simple & Easy MPEG-DASH Player for Web and Mobile
 
On Optimizing Resource Utilization in AVC-based Real-time Video Streaming
On Optimizing Resource Utilization in AVC-based Real-time Video StreamingOn Optimizing Resource Utilization in AVC-based Real-time Video Streaming
On Optimizing Resource Utilization in AVC-based Real-time Video Streaming
 
Towards Optimal Multirate Encoding for HTTP Adaptive Streaming
Towards Optimal Multirate Encoding for HTTP Adaptive StreamingTowards Optimal Multirate Encoding for HTTP Adaptive Streaming
Towards Optimal Multirate Encoding for HTTP Adaptive Streaming
 
Streaming tools comparison
Streaming tools comparisonStreaming tools comparison
Streaming tools comparison
 
Research Group Multimedia Communication (MMC)
Research Group Multimedia Communication (MMC)Research Group Multimedia Communication (MMC)
Research Group Multimedia Communication (MMC)
 
HTTP Adaptive Streaming – Quo Vadis?
HTTP Adaptive Streaming – Quo Vadis?HTTP Adaptive Streaming – Quo Vadis?
HTTP Adaptive Streaming – Quo Vadis?
 

Similar to Adaptive Streaming of Traditional and Omnidirectional Media

Tutorial adaptive-streaming
Tutorial adaptive-streamingTutorial adaptive-streaming
Tutorial adaptive-streamingJohnGregory89
 
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional MediaDelivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional MediaAlpen-Adria-Universität
 
QoS for Media Networks
QoS for Media NetworksQoS for Media Networks
QoS for Media NetworksAmine Choukir
 
Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...
Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...
Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...Alpen-Adria-Universität
 
The Evolution of the Content Delivery Network
The Evolution of the Content Delivery NetworkThe Evolution of the Content Delivery Network
The Evolution of the Content Delivery NetworkCisco Service Provider
 
Premium Mobile and DSL Video with HEVC
Premium Mobile and DSL Video with HEVCPremium Mobile and DSL Video with HEVC
Premium Mobile and DSL Video with HEVCCisco Service Provider
 
IBC 2013 Multi-network Forum - Akamai
IBC 2013 Multi-network Forum - Akamai IBC 2013 Multi-network Forum - Akamai
IBC 2013 Multi-network Forum - Akamai Verimatrix
 
ARA JAGUAR-7000 Product Brief
ARA JAGUAR-7000 Product BriefARA JAGUAR-7000 Product Brief
ARA JAGUAR-7000 Product BriefChul-Woong Yang
 
Cloud or Not to Cloud, That is the Question!
Cloud or Not to Cloud, That is the Question!Cloud or Not to Cloud, That is the Question!
Cloud or Not to Cloud, That is the Question!Cisco Service Provider
 
Streaming Media West 2017 - HTML5 Workshop
Streaming Media West 2017 - HTML5 WorkshopStreaming Media West 2017 - HTML5 Workshop
Streaming Media West 2017 - HTML5 WorkshopErica Beavers
 
The OTT Challenge - Eric Leung
The OTT Challenge - Eric LeungThe OTT Challenge - Eric Leung
The OTT Challenge - Eric LeungMyNOG
 
Adaptive Media Streaming over Emerging Protocols
Adaptive Media Streaming over Emerging ProtocolsAdaptive Media Streaming over Emerging Protocols
Adaptive Media Streaming over Emerging ProtocolsAlpen-Adria-Universität
 
Implementing Fast IT Deploying Applications at the Pace of Innovation
Implementing Fast IT Deploying Applications at the Pace of Innovation Implementing Fast IT Deploying Applications at the Pace of Innovation
Implementing Fast IT Deploying Applications at the Pace of Innovation Cisco DevNet
 
S8 p2 2014-maputo-s8-pomy
S8 p2 2014-maputo-s8-pomyS8 p2 2014-maputo-s8-pomy
S8 p2 2014-maputo-s8-pomyDieu Tran Hoang
 
IBC Content Everywhere Hub Presentation: HTML5 And Fastest Encoding
IBC Content Everywhere Hub Presentation: HTML5 And Fastest EncodingIBC Content Everywhere Hub Presentation: HTML5 And Fastest Encoding
IBC Content Everywhere Hub Presentation: HTML5 And Fastest EncodingBitmovin Inc
 
Panel with IPv6 CE Vendors
Panel with IPv6 CE VendorsPanel with IPv6 CE Vendors
Panel with IPv6 CE VendorsAPNIC
 
Video Codecs and the Future by Vince Puglia
Video Codecs and the Future by Vince PugliaVideo Codecs and the Future by Vince Puglia
Video Codecs and the Future by Vince PugliaDialogic Inc.
 
Web Architecture and Technologies
Web Architecture and TechnologiesWeb Architecture and Technologies
Web Architecture and TechnologiesFulvio Corno
 
PLNOG 9: Marcin Strzyżewski, Marcin Wawrzyński - Videoscape Distribution Suite
PLNOG 9: Marcin Strzyżewski, Marcin Wawrzyński - Videoscape Distribution Suite PLNOG 9: Marcin Strzyżewski, Marcin Wawrzyński - Videoscape Distribution Suite
PLNOG 9: Marcin Strzyżewski, Marcin Wawrzyński - Videoscape Distribution Suite PROIDEA
 

Similar to Adaptive Streaming of Traditional and Omnidirectional Media (20)

Tutorial adaptive-streaming
Tutorial adaptive-streamingTutorial adaptive-streaming
Tutorial adaptive-streaming
 
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional MediaDelivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
 
QoS for Media Networks
QoS for Media NetworksQoS for Media Networks
QoS for Media Networks
 
Multimedia streaming
Multimedia streamingMultimedia streaming
Multimedia streaming
 
Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...
Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...
Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...
 
The Evolution of the Content Delivery Network
The Evolution of the Content Delivery NetworkThe Evolution of the Content Delivery Network
The Evolution of the Content Delivery Network
 
Premium Mobile and DSL Video with HEVC
Premium Mobile and DSL Video with HEVCPremium Mobile and DSL Video with HEVC
Premium Mobile and DSL Video with HEVC
 
IBC 2013 Multi-network Forum - Akamai
IBC 2013 Multi-network Forum - Akamai IBC 2013 Multi-network Forum - Akamai
IBC 2013 Multi-network Forum - Akamai
 
ARA JAGUAR-7000 Product Brief
ARA JAGUAR-7000 Product BriefARA JAGUAR-7000 Product Brief
ARA JAGUAR-7000 Product Brief
 
Cloud or Not to Cloud, That is the Question!
Cloud or Not to Cloud, That is the Question!Cloud or Not to Cloud, That is the Question!
Cloud or Not to Cloud, That is the Question!
 
Streaming Media West 2017 - HTML5 Workshop
Streaming Media West 2017 - HTML5 WorkshopStreaming Media West 2017 - HTML5 Workshop
Streaming Media West 2017 - HTML5 Workshop
 
The OTT Challenge - Eric Leung
The OTT Challenge - Eric LeungThe OTT Challenge - Eric Leung
The OTT Challenge - Eric Leung
 
Adaptive Media Streaming over Emerging Protocols
Adaptive Media Streaming over Emerging ProtocolsAdaptive Media Streaming over Emerging Protocols
Adaptive Media Streaming over Emerging Protocols
 
Implementing Fast IT Deploying Applications at the Pace of Innovation
Implementing Fast IT Deploying Applications at the Pace of Innovation Implementing Fast IT Deploying Applications at the Pace of Innovation
Implementing Fast IT Deploying Applications at the Pace of Innovation
 
S8 p2 2014-maputo-s8-pomy
S8 p2 2014-maputo-s8-pomyS8 p2 2014-maputo-s8-pomy
S8 p2 2014-maputo-s8-pomy
 
IBC Content Everywhere Hub Presentation: HTML5 And Fastest Encoding
IBC Content Everywhere Hub Presentation: HTML5 And Fastest EncodingIBC Content Everywhere Hub Presentation: HTML5 And Fastest Encoding
IBC Content Everywhere Hub Presentation: HTML5 And Fastest Encoding
 
Panel with IPv6 CE Vendors
Panel with IPv6 CE VendorsPanel with IPv6 CE Vendors
Panel with IPv6 CE Vendors
 
Video Codecs and the Future by Vince Puglia
Video Codecs and the Future by Vince PugliaVideo Codecs and the Future by Vince Puglia
Video Codecs and the Future by Vince Puglia
 
Web Architecture and Technologies
Web Architecture and TechnologiesWeb Architecture and Technologies
Web Architecture and Technologies
 
PLNOG 9: Marcin Strzyżewski, Marcin Wawrzyński - Videoscape Distribution Suite
PLNOG 9: Marcin Strzyżewski, Marcin Wawrzyński - Videoscape Distribution Suite PLNOG 9: Marcin Strzyżewski, Marcin Wawrzyński - Videoscape Distribution Suite
PLNOG 9: Marcin Strzyżewski, Marcin Wawrzyński - Videoscape Distribution Suite
 

More from Alpen-Adria-Universität

Content-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingContent-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingAlpen-Adria-Universität
 
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...Alpen-Adria-Universität
 
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Vid...
Empowerment of Atypical Viewers  via Low-Effort Personalized Modeling  of Vid...Empowerment of Atypical Viewers  via Low-Effort Personalized Modeling  of Vid...
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Vid...Alpen-Adria-Universität
 
Optimizing Video Streaming for Sustainability and Quality: The Role of Prese...
Optimizing Video Streaming  for Sustainability and Quality: The Role of Prese...Optimizing Video Streaming  for Sustainability and Quality: The Role of Prese...
Optimizing Video Streaming for Sustainability and Quality: The Role of Prese...Alpen-Adria-Universität
 
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...Alpen-Adria-Universität
 
Machine Learning Based Resource Utilization Prediction in the Computing Conti...
Machine Learning Based Resource Utilization Prediction in the Computing Conti...Machine Learning Based Resource Utilization Prediction in the Computing Conti...
Machine Learning Based Resource Utilization Prediction in the Computing Conti...Alpen-Adria-Universität
 
Evaluation of Quality of Experience of ABR Schemes in Gaming Stream
Evaluation of Quality of Experience of ABR Schemes in Gaming StreamEvaluation of Quality of Experience of ABR Schemes in Gaming Stream
Evaluation of Quality of Experience of ABR Schemes in Gaming StreamAlpen-Adria-Universität
 
Multi-access Edge Computing for Adaptive Video Streaming
Multi-access Edge Computing for Adaptive Video StreamingMulti-access Edge Computing for Adaptive Video Streaming
Multi-access Edge Computing for Adaptive Video StreamingAlpen-Adria-Universität
 
Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentAlpen-Adria-Universität
 
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...Alpen-Adria-Universität
 
Energy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and StrategiesEnergy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and StrategiesAlpen-Adria-Universität
 
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...Alpen-Adria-Universität
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningAlpen-Adria-Universität
 
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming ApplicationsSARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming ApplicationsAlpen-Adria-Universität
 
Immersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to HolographyImmersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to HolographyAlpen-Adria-Universität
 
LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...
LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...
LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...Alpen-Adria-Universität
 
HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)Alpen-Adria-Universität
 
How to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
How to Optimize Dynamic Adaptive Video Streaming? Challenges and SolutionsHow to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
How to Optimize Dynamic Adaptive Video Streaming? Challenges and SolutionsAlpen-Adria-Universität
 
MPEC2: Multilayer and Pipeline Video Encoding on the Computing Continuum
MPEC2: Multilayer and Pipeline Video Encoding on the Computing ContinuumMPEC2: Multilayer and Pipeline Video Encoding on the Computing Continuum
MPEC2: Multilayer and Pipeline Video Encoding on the Computing ContinuumAlpen-Adria-Universität
 
Collaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
Collaborative Edge-Assisted Systems for HTTP Adaptive Video StreamingCollaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
Collaborative Edge-Assisted Systems for HTTP Adaptive Video StreamingAlpen-Adria-Universität
 

More from Alpen-Adria-Universität (20)

Content-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive StreamingContent-adaptive Video Coding for HTTP Adaptive Streaming
Content-adaptive Video Coding for HTTP Adaptive Streaming
 
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Video...
 
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Vid...
Empowerment of Atypical Viewers  via Low-Effort Personalized Modeling  of Vid...Empowerment of Atypical Viewers  via Low-Effort Personalized Modeling  of Vid...
Empowerment of Atypical Viewers via Low-Effort Personalized Modeling of Vid...
 
Optimizing Video Streaming for Sustainability and Quality: The Role of Prese...
Optimizing Video Streaming  for Sustainability and Quality: The Role of Prese...Optimizing Video Streaming  for Sustainability and Quality: The Role of Prese...
Optimizing Video Streaming for Sustainability and Quality: The Role of Prese...
 
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
Energy-Efficient Multi-Codec Bitrate-Ladder Estimation for Adaptive Video Str...
 
Machine Learning Based Resource Utilization Prediction in the Computing Conti...
Machine Learning Based Resource Utilization Prediction in the Computing Conti...Machine Learning Based Resource Utilization Prediction in the Computing Conti...
Machine Learning Based Resource Utilization Prediction in the Computing Conti...
 
Evaluation of Quality of Experience of ABR Schemes in Gaming Stream
Evaluation of Quality of Experience of ABR Schemes in Gaming StreamEvaluation of Quality of Experience of ABR Schemes in Gaming Stream
Evaluation of Quality of Experience of ABR Schemes in Gaming Stream
 
Multi-access Edge Computing for Adaptive Video Streaming
Multi-access Edge Computing for Adaptive Video StreamingMulti-access Edge Computing for Adaptive Video Streaming
Multi-access Edge Computing for Adaptive Video Streaming
 
Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment
 
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
 
Energy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and StrategiesEnergy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and Strategies
 
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
 
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
 
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming ApplicationsSARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
 
Immersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to HolographyImmersive Video Delivery: From Omnidirectional Video to Holography
Immersive Video Delivery: From Omnidirectional Video to Holography
 
LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...
LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...
LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...
 
HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)HTTP Adaptive Streaming – Quo Vadis? (2023)
HTTP Adaptive Streaming – Quo Vadis? (2023)
 
How to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
How to Optimize Dynamic Adaptive Video Streaming? Challenges and SolutionsHow to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
How to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
 
MPEC2: Multilayer and Pipeline Video Encoding on the Computing Continuum
MPEC2: Multilayer and Pipeline Video Encoding on the Computing ContinuumMPEC2: Multilayer and Pipeline Video Encoding on the Computing Continuum
MPEC2: Multilayer and Pipeline Video Encoding on the Computing Continuum
 
Collaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
Collaborative Edge-Assisted Systems for HTTP Adaptive Video StreamingCollaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
Collaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
 

Recently uploaded

Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...Cprime
 
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...MichaelBenis1
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriSafe Software
 
AI improves software testing to be more fault tolerant, focused and efficient
AI improves software testing to be more fault tolerant, focused and efficientAI improves software testing to be more fault tolerant, focused and efficient
AI improves software testing to be more fault tolerant, focused and efficientKari Kakkonen
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueShapeBlue
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...ShapeBlue
 
eXtended Reality(XR) Basic introductions
eXtended Reality(XR) Basic introductionseXtended Reality(XR) Basic introductions
eXtended Reality(XR) Basic introductionsElanthirayan Madhavan
 
GDG Cloud Southlake 30 Brian Demers Breeding 10x Developers with Developer Pr...
GDG Cloud Southlake 30 Brian Demers Breeding 10x Developers with Developer Pr...GDG Cloud Southlake 30 Brian Demers Breeding 10x Developers with Developer Pr...
GDG Cloud Southlake 30 Brian Demers Breeding 10x Developers with Developer Pr...James Anderson
 
AI-Plugins-Planners-Persona-SemanticKernel.pptx
AI-Plugins-Planners-Persona-SemanticKernel.pptxAI-Plugins-Planners-Persona-SemanticKernel.pptx
AI-Plugins-Planners-Persona-SemanticKernel.pptxUdaiappa Ramachandran
 
Artificial Intelligence - AI For Everyone
Artificial Intelligence - AI For EveryoneArtificial Intelligence - AI For Everyone
Artificial Intelligence - AI For EveryoneSridhar Seshadri
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceVijayananda Mohire
 
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUGBoosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUGRick Ossendrijver
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingerssuser9354ce
 
Why Disability Justice should be at the core of your digital accessibility jo...
Why Disability Justice should be at the core of your digital accessibility jo...Why Disability Justice should be at the core of your digital accessibility jo...
Why Disability Justice should be at the core of your digital accessibility jo...Modality Co
 
AGFM - Toyota Coaster 1HZ Install Guide.pdf
AGFM - Toyota Coaster 1HZ Install Guide.pdfAGFM - Toyota Coaster 1HZ Install Guide.pdf
AGFM - Toyota Coaster 1HZ Install Guide.pdfRodneyThomas28
 
Low Latency at Extreme Scale: Proven Practices & Pitfalls
Low Latency at Extreme Scale: Proven Practices & PitfallsLow Latency at Extreme Scale: Proven Practices & Pitfalls
Low Latency at Extreme Scale: Proven Practices & PitfallsScyllaDB
 
Python For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ emPython For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ emNho Vĩnh
 
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...2toLead Limited
 
Communities, networking and developer culture
Communities, networking and developer cultureCommunities, networking and developer culture
Communities, networking and developer cultureRavi Sanghani
 
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024BookNet Canada
 

Recently uploaded (20)

Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
Improving IT Investment Decisions and Business Outcomes with Integrated Enter...
 
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...National Institute of Standards and Technology (NIST) Cybersecurity Framework...
National Institute of Standards and Technology (NIST) Cybersecurity Framework...
 
Geospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & EsriGeospatial Synergy: Amplifying Efficiency with FME & Esri
Geospatial Synergy: Amplifying Efficiency with FME & Esri
 
AI improves software testing to be more fault tolerant, focused and efficient
AI improves software testing to be more fault tolerant, focused and efficientAI improves software testing to be more fault tolerant, focused and efficient
AI improves software testing to be more fault tolerant, focused and efficient
 
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlueCloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
CloudStack Authentication Methods – Harikrishna Patnala, ShapeBlue
 
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
Elevating Cloud Infrastructure with Object Storage, DRS, VM Scheduling, and D...
 
eXtended Reality(XR) Basic introductions
eXtended Reality(XR) Basic introductionseXtended Reality(XR) Basic introductions
eXtended Reality(XR) Basic introductions
 
GDG Cloud Southlake 30 Brian Demers Breeding 10x Developers with Developer Pr...
GDG Cloud Southlake 30 Brian Demers Breeding 10x Developers with Developer Pr...GDG Cloud Southlake 30 Brian Demers Breeding 10x Developers with Developer Pr...
GDG Cloud Southlake 30 Brian Demers Breeding 10x Developers with Developer Pr...
 
AI-Plugins-Planners-Persona-SemanticKernel.pptx
AI-Plugins-Planners-Persona-SemanticKernel.pptxAI-Plugins-Planners-Persona-SemanticKernel.pptx
AI-Plugins-Planners-Persona-SemanticKernel.pptx
 
Artificial Intelligence - AI For Everyone
Artificial Intelligence - AI For EveryoneArtificial Intelligence - AI For Everyone
Artificial Intelligence - AI For Everyone
 
My Journey towards Artificial Intelligence
My Journey towards Artificial IntelligenceMy Journey towards Artificial Intelligence
My Journey towards Artificial Intelligence
 
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUGBoosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
Boosting Developer Effectiveness with a Java platform team 1.4 - ArnhemJUG
 
iOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostingeriOncologi_Pitch Deck_2024 slide show for hostinger
iOncologi_Pitch Deck_2024 slide show for hostinger
 
Why Disability Justice should be at the core of your digital accessibility jo...
Why Disability Justice should be at the core of your digital accessibility jo...Why Disability Justice should be at the core of your digital accessibility jo...
Why Disability Justice should be at the core of your digital accessibility jo...
 
AGFM - Toyota Coaster 1HZ Install Guide.pdf
AGFM - Toyota Coaster 1HZ Install Guide.pdfAGFM - Toyota Coaster 1HZ Install Guide.pdf
AGFM - Toyota Coaster 1HZ Install Guide.pdf
 
Low Latency at Extreme Scale: Proven Practices & Pitfalls
Low Latency at Extreme Scale: Proven Practices & PitfallsLow Latency at Extreme Scale: Proven Practices & Pitfalls
Low Latency at Extreme Scale: Proven Practices & Pitfalls
 
Python For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ emPython For Kids - Sách Lập trình cho trẻ em
Python For Kids - Sách Lập trình cho trẻ em
 
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
Microsoft x 2toLead Webinar Session 2 - How Employee Learning and Development...
 
Communities, networking and developer culture
Communities, networking and developer cultureCommunities, networking and developer culture
Communities, networking and developer culture
 
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
Trending now: Book subjects on the move in the Canadian market - Tech Forum 2024
 

Adaptive Streaming of Traditional and Omnidirectional Media

  • 1. Adaptive Streaming of Traditional and Omnidirectional Media Ali C. Begen, Comcast/OzU Christian Timmerer, AAU/Bitmovin IEEE ICME Tutorial – Hong Kong, July 2017
  • 2. Upon Attending This Tutorial, You Will Know About • Principles of HTTP adaptive streaming for the Web/HTML5 • Principles of omnidirectional (360°) media delivery • Content generation/distribution/consumption workflows for traditional and omnidirectional media • Standards and emerging technologies in the adaptive streaming space • Current and future research on traditional and omnidirectional media delivery This tutorial is public, however, some material might be copyrighted Use proper citation when using content from this tutorial (Thanks to T. Stockhammer, J. Simmons, K. Hughes, C. Concolato, S. Pham, W. Law and many others for helping with the material) IEEE ICME Tutorial - July 2017 2
  • 3. Ali C. Begen • Electrical engineering degree from Bilkent University (2001) • Ph.D. degree from Georgia Tech (2006) – Video delivery and multimedia communications • Research, development and standards at Cisco (2007-2015) – IPTV, content delivery, software clients – Transport and distribution over IP networks – Enterprise video • Consulting at Networked Media since 2016 • Assistant professor at OzU and IEEE Distinguished Lecturer since 2016 Christian Timmerer • Associate Professor at the Institute of Information Technology (ITEC), Multimedia Communication Group (MMC), Alpen-Adria- Universität Klagenfurt, Austria • Co-founder and CIO of Bitmovin • Research interests – Immersive multimedia communication – Streaming, adaptation, and – Quality of experience (QoE) • Blog: http://blog.timmerer.com; @timse7 Presenters Today IEEE ICME Tutorial - July 2017 3
  • 4. ACM MMSys 2018 CfP is Out! IEEE ICME Tutorial - July 2017 4 • City: Amsterdam • Dates: June 12-15, 2018 • Co-located with – NOSSDAV – MoVid – MMVE • http://www.mmsys2018.org/
  • 5. Agenda • Part I: Over-the-top (OTT) video and HTTP adaptive streaming – HTML5 standard and media extensions – HTTP adaptive streaming building blocks – Multi-bitrate encoding, encapsulation and encryption workflows – De jure and de facto standards (DASH, HLS, CMAF) – Experiences from the existing deployments • Part II: Omnidirectional (360°) media – Acquisition, projection, coding and packaging – Delivery, decoding and rendering – Emerging standards and industry practices – Current research trends, open issues, efforts that are underway in the streaming industry IEEE ICME Tutorial - July 2017 5
  • 6. IPTV vs. IP (Over-the-Top) Video First Things First IPTV IP Video Best-effort delivery Quality not guaranteed Mostly on demand Paid or ad-based service Managed delivery Emphasis on quality Mostly linear TV Always a paid service IEEE ICME Tutorial - July 2017 6
  • 7. Part I: OTT Video and HTTP Adaptive Streaming
  • 8. 8 Internet (IP aka OTT) Video Essentials • Reach all connected devicesReach • Enable live and on-demand delivery to the mass marketScale • Provide TV-like or even better and richer viewer experienceQuality of Experience • Enable revenue generation thru paid content, subscriptions or targeted advertising Business • Satisfy regulations for captioning, ratings, parental control and advertising Regulatory IEEE ICME Tutorial - July 2017
  • 9. 9 Creating Revenue – Attracting Eye Balls • High-end content – Hollywood movies, TV shows – Sports • Excellent quality – HD/3D/UHD audiovisual presentation w/o artifacts such as pixelization and rebuffering – Fast startup, fast zapping and low glass-to-glass delay • Usability – Navigation, content discovery, battery consumption, trick modes, social network integration • Service flexibility – Linear TV – Time-shifted and on-demand services • Reach – Any device, any time IEEE ICME Tutorial - July 2017
  • 10. One Request, One Response Progressive Download HTTP Request HTTP Response IEEE ICME Tutorial - July 2017 10
  • 11. What is Streaming? Streaming is transmission of a continuous content from a server to a client and its simultaneous consumption by the client Two Main Characteristics 1. Client consumption rate may be limited by real-time constraints as opposed to just bandwidth availability 2. Server transmission rate (loosely or tightly) matches to client consumption rate IEEE ICME Tutorial - July 2017 11
  • 12. Stalls, Slow Start-Up, Plug-In and DRM Issues IEEE ICME Tutorial - July 2017 12 Common Annoyances in Streaming • Unsupported/wrong – protocol – plug-in – codec – format – DRM • Slow start-up • Poor quality, quality variation • Frequent freezes/glitches • Lack of seeking
  • 13. IEEE ICME Tutorial - July 2017 13 Video Delivery over HTTP • Enables playback while still downloading • Server sends the file as fast as possible Progressive Download • Enables seeking via media indexing • Server paces transmission based on encoding rate Pseudo Streaming • Content is divided into short-duration chunks • Enables live streaming and ad insertion Chunked Streaming • Multiple versions of the content are created • Enables to adapt to network and device conditions Adaptive Streaming
  • 14. Adaptive Streaming over HTTP 14 Decoding and Presentation Streaming Client Media Buffer Content Ingest (Live or Pre-captured) Multi-rate Encoder Packager Origin (HTTP) Server … … … … ServerStorage HTTP GET Request Response IEEE ICME Tutorial - July 2017
  • 15. Adapt Video to Web Rather than Changing the Web 15 Adaptive Streaming over HTTP • Imitation of streaming via short downloads – Downloads small chunks to minimize bandwidth waste – Enables to monitor consumption and track the streaming clients • Adaptation to dynamic conditions and device capabilities – Adapts to dynamic conditions in the Internet and home network – Adapts to display resolution, CPU and memory resources of the streaming client à Facilitates “any device, anywhere, anytime” paradigm • Improved quality of experience (not always improved average quality) – Enables faster start-up and seeking, and quicker buffer fills – Reduces skips, freezes and stutters • Use of HTTP – Well-understood naming/addressing approach, and authentication/authorization infrastructure – Provides easy traversal for all kinds of middleboxes (e.g., NATs, firewalls) – Enables cloud access, leverages the existing and cheap HTTP caching infrastructure IEEE ICME Tutorial - July 2017
  • 16. Example Representations Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 2.2 Mbps 960 x 540 Rep. #3 1.4 Mbps 960 x 540 Rep. #4 900 Kbps 512 x 288 Rep. #5 600 Kbps 512 x 288 Rep. #6 400 Kbps 340 x 192 Rep. #7 200 Kbps 340 x 192 Source: Vertigo MIX10, Alex Zambelli’s Streaming Media Blog, Akamai Vancouver 2010 Sochi 2014 Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 1.95 Mbps 848 x 480 Rep. #3 1.25 Mbps 640 x 360 Rep. #4 900 Kbps 512 x 288 Rep. #5 600 Kbps 400 x 224 Rep. #6 400 Kbps 312 x 176 Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 2.2 Mbps 1024 x 576 Rep. #3 1.4 Mbps 768 x 432 Rep. #4 950 Kbps 640 x 360 Rep. #5 600 Kbps 512 x 288 Rep. #6 400 Kbps 384 x 216 Rep. #7 250 Kbps 384 x 216 Rep. #8 150 Kbps 256 x 144 FIFA 2014 IEEE ICME Tutorial - July 2017 16
  • 17. List of Accessible Segments and Their Timings An Example Manifest Format MPD Period id = 1 start = 0 s Period id = 3 start = 300 s Period id = 4 start = 850 s Period id = 2 start = 100 s Adaptation Set 0 subtitle turkish Adaptation Set 2 audio english Adaptation Set 1 BaseURL=http://abr.rocks.com/ Representation 2 Rate = 1 Mbps Representation 4 Rate = 3 Mbps Representation 1 Rate = 500 Kbps Representation 3 Rate = 2 Mbps Resolution = 720p Segment Info Duration = 10 s Template: 3/$Number$.mp4 Segment Access Initialization Segment http://abr.rocks.com/3/0.mp4 Media Segment 1 start = 0 s http://abr.rocks.com/3/1.mp4 Media Segment 2 start = 10 s http://abr.rocks.com/3/2.mp4 Adaptation Set 3 audio italian Adaptation Set 1 video Period id = 2 start = 100 s Representation 3 Rate = 2 Mbps Selection of components/tracks Well-defined media format Selection of representations Splicing of arbitrary content like ads Chunks with addresses and timing 17IEEE ICME Tutorial - July 2017
  • 18. Smart (aka Selfish and Greedy) Clients 18 Client manages - Manifest(s) - HTTP transport - TCP connection(s) Client monitors/measures - Playout buffer - Download times and throughput - Local resources (CPU, memory, screen, etc.) - Dropped frames Client performs adaptation Request IEEE ICME Tutorial - July 2017 Response
  • 19. End-to-End Workflow for OTT Production Preparation and Staging Distribution Consumption News Gathering Sport Events Premium Content Studio Multi-bitrate Encoding Encapsulation Protection Origin Servers VoD Content & Manifests Live Content & Manifests CDN IEEE ICME Tutorial - July 2017 19
  • 20. IEEE ICME Tutorial - July 2017 20
  • 21. IEEE ICME Tutorial - July 2017 21 • HTML5 is a set of technologies that allows more powerful Web sites and applications – Better semantics – Better connectivity – Offline and storage – Multimedia – 2D/3D graphics and effects – Performance and integration – Device access – Styling • Most interesting new elements – Semantic elements: <header>, <footer>, <article> and <section> – Graphic elements: <svg> and <canvas> – Multimedia elements: <audio> and <video> A Common Platform across Devices HTML5 Source: Mozilla MDN, w3schools.com
  • 22. • These apps can – leverage native APIs and offline capability – provide cross-device compatibility (iOS, Windows, Android, etc.) – be packaged and published to app stores • For streaming: – To a user, it looks like a regular native app they have downloaded – The app is actually a container and loads loads dynamically from the web server – Server-side changes propagate automatically to installed apps Hosted and Progressive Web Apps IEEE ICME Tutorial - July 2017 22 Source: Google PWA, Microsoft HWA
  • 23. IEEE ICME Tutorial - July 2017 23 Types of Browser-Based Playback Source: CTA WAVE Type 1: Minimum control architectural model for HTML5 support of adaptive streaming where manifest and heuristics are managed by the user agent Type 2: Adaptation control architectural model for HTML5 support of adaptive streaming providing script manageable features Type 3: Full media control architectural model for HTML5 support of adaptive streaming allowing script to explicitly send the media segments (MSE + EME)
  • 24. IEEE ICME Tutorial - July 2017 24 MSE Support in Web Browsers Source: http://caniuse.com/#search=mse
  • 25. IEEE ICME Tutorial - July 2017 25 Content Protection – Famous Quotes • “Digital files cannot be made uncopiable, any more than water can be made not wet” – Bruce Schneier (cryptographer), May 2001 • “We have Ph.D.'s here that know the stuff cold, and we don't believe it's possible to protect digital content” – Steve Jobs, December 2003 • “If we’re still talking about DRM in five years, please take me out and shoot me” – eMusic CEO David Pakman, February 2007 • “This is unethical” – Ian Hickson, HTML5 editor, upon learning of the Netflix-Google-Microsoft EME proposal - February 2012
  • 26. IEEE ICME Tutorial - July 2017 26 W3C Encrypted Media Extensions Source: https://www.w3.org/TR/encrypted-media/
  • 27. IEEE ICME Tutorial - July 2017 27 EME Support in Web Browsers Source: http://caniuse.com/#search=eme
  • 28. Web Video Ecosystem Encoding Encryption Rights Expression Web Video App Framework Decoding Decryption Rights Management The fundamental Digital Rights Management problem derives from a lack of interoperability which prevents mobility of experience The solution is to combine interoperable commercial Web video content with a cross- platform Web video app framework IEEE ICME Tutorial - July 2017 28 Source: John Simmons
  • 29. Contrary to a common misconception, with EME DRM functionality is not in the HTML/JS app There is no DRM in HTML5 with EME, and ECP requires that this be the case Digital Rights Management The HTML/JS app selects the DRM and controls key exchange between DRM client and server HTML/JavaScript Application Browser extends HTML5 media element to allow JavaScript handled key acquisition Browser A CDM exposes a key system to JavaScript; it is transparent whether the CDM is in the browser Content Decryption Module The Web, EME and Enhanced Content Protection (ECP) IEEE ICME Tutorial - July 2017 29 Source: John Simmons
  • 30. IEEE ICME Tutorial - July 2017 DRM Support on Desktop Browsers 30 Browser OS EME/CDM Flash Player NPAPI/ Silverlight 5 Chrome Win Widevine (Yes) No OS X (Yes) No Linux (Yes) No Firefox Win Widevine Yes No OS X Yes No Linux Yes No Safari > OS X Yosemite (Fairplay) Yes Yes < OS X Yosemite No Yes Yes IE/Edge < Win 7 No Yes Yes Win 10 PlayReady Yes No
  • 31. DRM Support on Mobile Platforms Platform Browser Native App / DRM iOS No MSE/EME yet HLS (AES-128 CBC) via <video> Native SDK and WebView: FairPlay Android MSE/EME Native + Android SDK (MediaDRM APIs) - (Widevine + OMA v2) ExoPlayer Native + WebView with Widevine Windows 10 MSE/EME WebViews/Hosted Web Apps with PlayReady IEEE ICME Tutorial - July 2017 31
  • 32. Ad Insertion Methods • Server-based – MPD is constructed or updated based on the ads – For multi-period MPDs, we can use • XLink • MPD chaining – For single-period MPDs • Ads are spliced into a continuous stream • Splicing can be offline or just-in-time • App-based – Requires additional support in the application – Uses DASH-specified user-defined events IEEE ICME Tutorial - July 2017 32
  • 33. Ad Insertion – XLink Resolution IEEE ICME Tutorial - July 2017 33
  • 34. IEEE ICME Tutorial - July 2017 34 Streaming over HTTP – The Promise • Leverage tried-and-true Web infrastructure for scaling – Video is just ordinary Web content! • Leverage tried-and-true TCP – Congestion avoidance – Reliability – No special QoS for video It should all “just work” J
  • 35. 35 Does It Just Work? • When streaming clients compete with other traffic, mostly yes • When streaming clients compete with each other, we begin to see problems: – The clients’ adaptation behaviors interact with each other – The competing clients form an “accidental” distributed control-feedback system • Unexpected behaviors will result in places like – Multiple screens within a household – ISP access and aggregation links – Small cells in stadiums and malls IEEE ICME Tutorial - July 2017
  • 36. A Single Microsoft Smooth Streaming Client under a Controlled Environment 36 Demystifying a Streaming Client 0 1 2 3 4 5 0 50 100 150 200 250 300 350 400 450 500 Bitrate(Mbps) Time (s) Available Bandwidth Requests Chunk Tput Average Tput Reading: “An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP,” ACM MMSys 2011 Buffer-filling State Back-to-back requests (Not pipelined) Steady State Periodic requests IEEE ICME Tutorial - July 2017
  • 37. 10 (Commercial) Streaming Clients Sharing a 10 Mbps Link IEEE ICME Tutorial - July 2017 37 Typical Example 0 200 400 600 800 1000 1200 1400 0 100 200 300 400 500 RequestedBitrate(Kbps) Time (s) Client1 Client2 Client3
  • 38. 38 Viewer Experience Statistics Selfishness Hurts Everyone Source: Conviva Viewer Experience Report, 2015 IEEE ICME Tutorial - July 2017
  • 39. Inner and Outer Control Loops 39 HTTP Server Manifest Media HTTP Origin Module TCP Sender Streaming Client Manifest Resource Monitors Streaming Application TCP ReceiverData / ACK Request Response There could be multiple TCPs destined to the same or different servers IEEE ICME Tutorial - July 2017
  • 40. IEEE ICME Tutorial - July 2017 40 Tradeoffs in Adaptive Streaming Overall quality Quality stability Proximity to live edge Stalls Zapping/seeking time
  • 41. Two Competing Clients 41 Understanding the Root Cause • Depending on the timing of the ON periods: – Unfairness, underutilization and/or instability may occur – Clients may grossly overestimate their fair share of the available bandwidth Clients cannot figure out how much bandwidth to use until they use too much Reading: “What happens when HTTP adaptive streaming players compete for bandwidth?,” ACM NOSSDAV 2012 IEEE ICME Tutorial - July 2017
  • 42. IEEE ICME Tutorial - July 2017 42 How to Solve the Issues? Solution Approaches Fix the clients Enable a control plane Get support from the network
  • 43. IEEE ICME Tutorial - July 2017 43 One Slide on Software Defined Networking (SDN) Control and data planes are decoupled, network intelligence and state are logically centralized, and the underlying network infrastructure is abstracted from the apps
  • 44. IEEE ICME Tutorial - July 2017 44 SDN-Based Bitrate Adaptation Reading: “SDNDASH: improving QoE of HTTP adaptive streaming using software defined networking,” ACM MM 2016
  • 45. Control Points in the Ecosystem • I want to make sure that my content is protected and looks awesome on any device • I want to make sure that my ads are viewed, trackable and measurable • I want to make sure that my servers are properly used and latency is low • I want to control the QoE of all my customers, differentiate my own services, make $$$ from OTT services • I want to make sure that my device/app provides the best possible video quality • I want the best quality for minimal $ ConsumerContent Provider ISP CDN ProviderAdvertiser Device/ App 45IEEE ICME Tutorial - July 2017
  • 46. IEEE ICME Tutorial - July 2017 46 DASH Part 5: Server and Network Assisted DASH (SAND) A Control Plane Approach DANE: DASH-assisting network element PER: Parameters for enhancing reception PED: Parameters for enhancing delivery Media PER Messages Metrics and Status Messages PED Messages DANE (Origin) DANE Regular Network Element DASH Client DANE (Analytics)
  • 47. Dead, Surviving and New-Born Technologies • Move Adaptive Stream (Long gone) – http://www.movenetworks.com • Microsoft Smooth Streaming (Legacy) – http://www.iis.net/expand/SmoothStreaming • Adobe Flash (Almost dead) – http://www.adobe.com/products/flashplayer.html • Adobe HTTP Dynamic Streaming (Legacy) – http://www.adobe.com/products/httpdynamicstreaming • Apple HTTP Live Streaming (The elephant in the room) – http://tools.ietf.org/html/draft-pantos-http-live-streaming (Soon to be an RFC) • MPEG DASH and CMAF (The standards) – http://mpeg.chiariglione.org/standards/mpeg-dash – http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format IEEE ICME Tutorial - July 2017 47
  • 48. • Fragmented architectures – Advertising, DRM, metadata, blackouts, etc. • Investing in more hardware and software – Increased CapEx and OpEx • Lack of consistent analytics • Preparing and delivering each asset in several incompatible formats – Higher storage and transport costs • Confusion due to the lack of skills to troubleshoot problems • Lack of common experience across devices for the same service – Tricks, captions, subtitles, ads, etc. What does Fragmentation Mean? Higher Costs Less Scalability Smaller Reach Frustration Skepticism Slow Adoption IEEE ICME Tutorial - July 2017 48
  • 49. 49 DASH intends to be to the Internet world … what MPEG2-TS has been to the broadcast world IEEE ICME Tutorial - July 2017
  • 50. 50 DASH intendsed to be to the Internet world … what MPEG2-TS has been to the broadcast world IEEE ICME Tutorial - July 2017
  • 51. What is HAPPENING in MPEG ?
  • 52. DASH Design Principles • DASH is not – system, protocol, presentation, codec, interactivity, DRM, client specification • DASH is an enabler – It provides formats to enable efficient and high-quality delivery of streaming services over the Internet – System definition is left to other organizations • Design choices – Enable reuse of existing technologies (containers, codecs, DRM, etc.) – Enable deployment on top of CDNs – Enable live and on-demand experiences – Move intelligence from network to client, enable client differentiation – Provide simple interoperability points (profiles) IEEE ICME Tutorial - July 2017 52
  • 53. 53 Shown in Red Scope of MPEG DASH HTTP Server DASH Client Control Engine MediaEngines HTTP ClientHTTP 1.1 Segment ParserMPD Transport MPD Parser MPD MPD MPD ... ... ... IEEE ICME Tutorial - July 2017
  • 54. IEEE ICME Tutorial - July 2017 54 DASH (ISO/IEC 23009) Parts • 23009-1: Media Presentation Description and Segment Formats – Amd. 1: NTP sync, extended profiles – Amd. 2: SRD, URL parameter insertion, role extensions – Amd. 3: External MPD link, period continuity, generalized HTTP header extensions/queries – Amd. 4: TV profile, MPD chaining/resetting, data URLs in MPD, switching across adaptation sets – à 3rd edition (FDIS), to be published soon • 23009-2: Conformance and Reference Software – 2nd edition (FDIS), to be published in 2017 – Amd. 1: SAND conformance rules • 23009-3: Implementation Guidelines (Informative) – 2nd edition Amd. 1: Multi-track content guidelines and rules, fix on dependent representations • 23009-4: Segment Encryption and Authentication – 2nd edition is in progress
  • 55. IEEE ICME Tutorial - July 2017 55 DASH (ISO/IEC 23009) Parts • 23009-5: Server and Network Assisted DASH (SAND) – 1st edition, published in Feb. 2017 • 23009-6: DASH over Full Duplex HTTP-based Protocols (FDH) – 1st edition (FDIS), to be published in 2017 • 23009-7: Delivery of CMAF Contents with DASH (Informative) – WD available, PDTR to be published after the July meeting
  • 56. IEEE ICME Tutorial - July 2017 56 Part 1 – 3rd Edition • TV profile – Broadcast TV content can be distributed over broadcast and/or broadband networks using DASH – Segments do not need to be a switching access point • MPD chaining – Multiple MPDs can be chained to each other for easier ad insertion • Data URLs in MPD – Initialization segments can be included in the MPD (only in the TV profile) • Switching across adaptation sets – A capable client can switch between AVC and HEVC tracks
  • 57. IEEE ICME Tutorial - July 2017 57 Part 6 – DASH over Full Duplex HTTP-Based Protocols • Part 6 describes how a client can signal various types of “push” behaviors to a server in HTTP/2 and for WebSockets • This is primarily targeted for low-latency live streaming • Similar activities – DASH over LTE broadcast (3GPP SA4) – DVB ABR multicast – ATSC 3.0 hybrid delivery
  • 58. IEEE ICME Tutorial - July 2017 58 Ongoing Work as of MPEG 119 (July 2017) • Core Experiments – High quality VR delivery with DASH (DASH-VR) • Technologies under Consideration – Support for Controlled Playback in DASH – Playback control – Owner defined content identifiers within MPDs – Usage of HEVC tile tracks in DASH – Relation between presentation time and EPT – Service-level service protection using segment encryption
  • 59. IEEE ICME Tutorial - July 2017 59 Common Media Application Format (CMAF) • Media delivery has three main components: – Media format – Manifest – Delivery • CMAF defines the media format only (Fragments, headers, segments, chunks, tracks) – CMAF does not specify a manifest format – CMAF does not specify a delivery method, either • CMAF uses ISO-BMFF and common encryption (CENC) – CENC means the media fragments can be decrypted/decoded by devices using different DRMs (aka multi-DRM support) – CMAF does not mandate CTR or CBC mode (which makes CMAF useful only for unencrypted content for now) • MPEG technologies (DASH and MMT) may be used for delivering CMAF content
  • 60. IEEE ICME Tutorial - July 2017 60 Common Media Application Format (CMAF) • CMAF defines media profiles for – Video – Audio – Subtitle • CMAF defines presentation profiles by selection a media profile from each category • Current status – iOS 10 supports CMAF with HLS manifests – PDAM1 (w16821) • Discussions on SHVC support • AAC 7.1 support – Conformance and interop is WIP with DASH-IF – Non-MPEG codecs are under discussion
  • 61. IEEE ICME Tutorial - July 2017 61 CMAF Development Timeline Feb. 2016 Working Draft June 2016 Committee Draft Nov. 2016 Draft International Standard Jan. 2017 Study of Draft International Standard May 2017 Final Draft International Standard July 2017 Approved International Standard Ballot Ballot Ballot
  • 62. IEEE ICME Tutorial - July 2017 62 CMAF ISO-BMFF Media Objects Encoder Encryption Packaging CMAF Header CMAF Fragment CMAF Fragment CMAF Chunk CMAF Chunk CMAF Chunk CMAF Fragment R A P R A P R A P R A P CMAF Fragment CMAF Segment CMAF Segment CMAF Track File • Manifests typically provide URLs to – CMAF track files – CMAF header + CMAF segments • single/multiple fragment(s) – CMAF header + CMAF chunk
  • 63. CMAF Chunks CMAF Header CMAF Content Consumption IEEE ICME Tutorial - July 2017 63 Un- packaging MSE CMAF Track File CMAF Fragments Switching can only happen seamlessly at CMAF Fragment boundaries CMAF Segments CMAF Header CMAF Header
  • 64. • Content format issues – Each asset is copied to multiple media formats • different video codecs • different audio codecs • different (regional) frame rates – Cost to content creators and distributors – Inefficiencies in CDNs and higher storage costs • Platform issues – Lack of consistent app behavior across platforms – Varying video features, APIs and semantics across platforms • Playback issues – Codec incompatibility – Partial profile support – Switching bitrate glitches – Audio discontinuities – Ad splicing problems – Long-term playback instability – Request protocol deficiencies – Memory problems, CPU weaknesses – Scaling (display) issues – Variable HDR support – Unknown capabilities Commercial OTT Issues 64IEEE ICME Tutorial - July 2017
  • 66. CTA Web Application Video Ecosystem (WAVE) Content specification based upcoming CMAF, compatible with DASH and HLS Testable requirements covering the most common device playback interoperability issues Reference application framework based on HTML5 providing functional guidelines for playback interop Content Specification HTML5 Reference Platform Device Playback Requirements Test Suite IEEE ICME Tutorial - July 2017 66
  • 67. Web Media API Community Group • Developers want to deploy their content on a range of devices and platforms, e.g., TVs, set-tops and mobile devices • To ensure a smooth user experience across devices, these user agents need to support a minimum set of Web technologies • This Community Group plans to specify such a set of Web technologies and additionally plans to provide guidance for developers and implementers • Web Media APIs 2017 – Details the Web APIs that should be included in device implementations in 2017 – https://www.w3.org/community/webmediaapi/ • Web Media Application Developer Guidelines – Companion guide to outline best practices for implementing Web media apps – https://w3c.github.io/webmediaguidelines/ IEEE ICME Tutorial - July 2017 67
  • 68. Many Definitions Do Exist – It is Like an Elephant What is Quality? IEEE ICME Tutorial - July 2017 68
  • 69. IEEE ICME Tutorial - July 2017 69
  • 70. Segments Have Different Complexities Bitrate Quality Video Segment #1 Equal Bitrate Allocation among Segments Consistent Quality Video Segment #2 70IEEE ICME Tutorial - July 2017
  • 71. Guidelines Limited Bitrate Variability to (Mostly) 10% So Far Adaptation Feature Does Not Deliver Consistent Quality Easy Moderate Easy Easy Easy Difficult Difficult Difficult Moderate Moderate Moderate Moderate … S Time (s) Segment Size 0 2 4 6 8 10 12 14 16 18 20 22 24 Segment Quality QCBR Small variation in encoding bitrate Large variation in quality If there is something worse than having to watch a video at a lousy quality, it is to watch that video with varying quality 71IEEE ICME Tutorial - July 2017
  • 72. What if We Encode in a More Subtle Fashion? Easy Moderate Easy Easy Easy Difficult Difficult Difficult Moderate Moderate Moderate Moderate … Time (s) Segment Size 0 2 4 6 8 10 12 14 16 18 20 22 24 Segment Quality QVBR While we spend the same total amount of bits, we not only increase average quality but also reduce quality variation Large variation in encoding bitrate Low variation in quality S HLS authoring spec for ATV allows 2x capping rate for VoD. For linear content, variability is limited to 10-25% range. 72IEEE ICME Tutorial - July 2017
  • 73. Content Aware Encoding Content Aware Streaming Generating VBR-encoded segments is easy, but streaming them is not! IEEE ICME Tutorial - July 2017 73
  • 74. Multiple Representations Naturally Enable “Cherry-Picking” 74 2.5 Mbps Network HTTP Server Streaming Client … … … … … 3 4 3 4 3 4 3 4 4 Mbps 3 Mbps 2 Mbps 1 Mbps 5 5 5 5 2 2 2 2 2 135 4 Time (s) SegmentSize … S SegmentQuality QVBR 0 2 4 6 8 10 IEEE ICME Tutorial - July 2017 Reading: “Streaming video over HTTP with consistent quality,” ACM MMSys 2014
  • 75. 75 Dimensions – In-Stream vs. Between-Streams • Same principle applies to both: – In-stream Case: Temporal bit shifting between segments – Between-streams Case: Bit shifting between streams sharing a bottleneck link Bitrate Quality Video Segment 1 Video Segment 2 CBR CQ Bitrate Quality Stream 1 (News) Stream 2 (Sports) Equal Bandwidth Sharing CQ IEEE ICME Tutorial - July 2017
  • 76. Deployment Challenges • Challenge 1: Development of quality metrics and temporal pooling models – A common metric that is suitable for a variety of content types – A temporal pooling model that will reliably work for different viewer profiles, devices and networks • Different viewers have different sensitivity levels to glitches for different content types, and they are also forgiving in different time scales – A young viewer (likely to have longer-term memory) watching sports on a big screen vs. an elder viewer (likely to have short-term memory) watching news on a smaller screen • Challenge 2: Integration into popular streaming client implementations – Many ecosystems are closed or proprietary, and one may not have access to the client algorithm to make the necessary changes (e.g., Apple HLS in iOS) – Standards bodies and industry consortiums may lead the way to develop certain guidelines 76IEEE ICME Tutorial - July 2017 Reading: “Quality-aware HTTP adaptive streaming,” IBC 2015
  • 77. Deployment Challenges • Challenge 3: Development of metadata standards – Computing the quality metric for each segment in each representation for each content is a tedious task, which is the easiest to deal with at the encoder or packager – Packing the metric values and conveying this information to all the clients in a timely and scalable manner is an equally important task – The timed metadata spec in MPEG (ISO/IEC 23001-10) is a good candidate for this task 77IEEE ICME Tutorial - July 2017
  • 78. Deployment Challenges • Challenge 4: Expansion to multi-client scenarios – We need controlled unfairness (which is fairness in quality not bitrate) among clients adaptively streaming the same or a different content over a network sharing resources (e.g., access network) • Easy scenario: One adult watching sports on a big screen vs. one adult watching a food show on a tablet • More complex scenario: One adult watching sports on a phone vs. three adults watching news on a big screen – The optimization across a number of streaming clients has to be done based on the utilities of the streamed videos, which depend on factors such as: • Spatial pooling model • Content types • Content features • Rendering devices • Audience profiles and sizes – SAND can help deploy controlled unfairness that we need in quality-aware streaming in multi- client scenarios 78IEEE ICME Tutorial - July 2017
  • 79. 79 Need for Analytics Service Provider: “Your video or CDN provider must be slow” Video/CDN Provider: “Your home network must be slow” Consumer: “The device or the app is slow” Device/App Vendor: “It must be the OS” OS Vendor: “Your Internet connection must be bad” IEEE ICME Tutorial - July 2017
  • 80. IEEE ICME Tutorial - July 2017 80 Four Major Areas of Further Exploration • How to choose bitrate/resolution pairs to make up/downshifts least visible • How to pick the segment durations • How to achieve content aware streaming and not just content aware encoding Content Preparation • What information could the network provide to streaming clients • How to achieve controlled unfairness • MPTCP or QUIC better than TCP? What about multicast and HTTP/2? Distribution and Delivery • How to model streaming dynamics for different genres • How to model the impact of faster zapping and trick modes on the QoE • Understanding the impact of QoE on viewer engagement QoE Modeling and Client Design • Understanding the interaction of adaptive streaming with caching in CDNs • Extracting actions based on real-time analytics • Fixing issues faster and remotely Analytics, Fault Isolation, Diagnostics
  • 81. Part II: Omnidirectional (360°) Media
  • 82. MPEG’s Definition What is Virtual Reality (VR) VR is a rendered environment (visual and acoustic, pre- dominantly real-world) providing an immersive experience to a user who can interact with it in a seemingly real or physical way using special electronic equipment IEEE ICME Tutorial - July 2017 82
  • 83. IEEE ICME Tutorial - July 2017 83 Virtual Reality (VR) Puts Us in a Virtual World Source: Phil Chou
  • 84. IEEE ICME Tutorial - July 2017 84 Augmented Reality (AR) Puts Virtual Objects in Our World Source: Phil Chou
  • 85. Delivering High-Quality VR in Economic Scale is Extremely Challenging The VR Challenge 30K x 24K x 36 x 60 x 2 / 600 ~5.2 Gbps Resolution Bit depth Frame rate Stereoscopic Compression gain IEEE ICME Tutorial - July 2017 85
  • 86. Ultimate Level of Immersion Interactions So intuitive that they become second nature Sounds So accurate that they are true to life Visuals So vibrant that they are eventually indistinguishable from the real world IEEE ICME Tutorial - July 2017 86
  • 87. Visual quality Sound quality Intuitive interactions Immersive VR Has Extreme Requirements Immersion High resolution audio Up to human hearing capabilities 3D audio Realistic 3D, positional, surround audio that is accurate to the real world Precise motion tracking Accurate on-device motion tracking Minimal latency Minimized system latency to remove perceptible lag Natural user interfaces Seamlessly interact with VR using natural movements, free from wires Extreme pixel quantity and quality Screen is very close to the eyes Spherical view Look anywhere with a full 360° spherical view Stereoscopic display Humans see in 3D IEEE ICME Tutorial - July 2017 87 Source: Thomas Stockhammer
  • 88. Need for Higher Resolutions on Mobile Phones? IEEE ICME Tutorial - July 2017 88 Source: Thomas Stockhammer
  • 89. The Human Eye can Only See High Resolution Where the Fovea is Focused Foveated Rendering Reduces Pixel Processing High resolution Low resolution The GPU renders a small rectangle at a high resolution and the rest of the FoV at a lower resolution IEEE ICME Tutorial - July 2017 89
  • 90. Generate Video • Simultaneously capture video with multiple cameras from different views to generate 360° spherical video • Use twice the cameras for stereoscopic video • Undistort, stitch together and map the discrete images to a equirectangular or cubemap format • Encode video Playback Video • Decode video • Apply an equirectangular or cubemap UV projection • Determine the pose and show appropriate view of 360° spherical video Generating and Consuming 360° Spherical Video Discrete unstitched camera images for 360° spherical view Equirectangular image Cubemap image Left eye VR headset view IEEE ICME Tutorial - July 2017 90
  • 91. IEEE ICME Tutorial - July 2017 91 Adaptive Streaming of VR Content • Required resolution of a panoramic video for achieving 4K resolution for viewport with 120° field of view (FoV)
  • 92. • 3-DoF – “In which direction I look,” look around the virtual world from a fixed point – Detect rotational movement • 6-DoF – “Where I am and in which direction I look,” move freely in the virtual world and look around corners – Detect rotational movement and translational movement Precise Motion Tracking of Head Movements and Gaze Z X Z Y Pitch Yaw Roll IEEE ICME Tutorial - July 2017 92
  • 93. Lag Prevents Immersion and Causes Discomfort Minimizing Motion-to Photon Latency is Important Low latency Noticeable latency IEEE ICME Tutorial - July 2017 93
  • 94. MPEG-I • New MPEG project – ISO/IEC 23090: Coded Representation of Immersive Media • Five parts are currently in progress: – Technical report: use cases, requirements and architectures – Omnidirectional media format (OMAF) – New and immersive video coding – New and immersive audio coding – Point cloud coding • Two more parts are considered – VR metadata – VR metrics IEEE ICME Tutorial - July 2017 94
  • 95. OMAF • Scope: 360° video and associated audio, 3-DoF only • Focus on the media format that enables VR video production, delivery and consumption – Projection and region-wise packing that are used to generate 2D video from the sphere signal – File format encapsulation and metadata signaling – DASH encapsulation and metadata signaling – Video/audio codecs and profiles/brands – Informative examples of viewport-dependent approaches IEEE ICME Tutorial - July 2017 95
  • 96. DASH/OMAF System Diagram Multi- camera capture Stitch Project HEVC encode OMAF composing DASH server DASH client network HEVC Decode Renderer Head mounted device (HMD) DASH encapsulation Head and eye tracking telemetry Request for tiles within FoV window Projection mapping metadata Projection mapping metadata authoring CDN VR player IEEE ICME Tutorial - July 2017 96
  • 97. Adaptive Streaming of VR Content IEEE ICME Tutorial - July 2017 97 Projection format: Equirectangular projection Coding format: AVC File format: ISOBMFF, M2TS Delivery format: DASH, HLS Web clients: HTML5, MSE, WebVR
  • 98. 2018 20202017 2019 2021 Internet Video Coding IoMT Media Orchestration CMAF CDVA (Video Analysis) Audio Wave Field Coding AR/VR Audio extensions New codec with Lightfield New & Immersive Video Coding Point Cloud Compression HDR TR HDR TR 2 Omnidirectional Media Format OMAF v2 Genome Compression Network-Distributed Video Coding Coding 2022 Systems and Tools Immersive Media AF Augmented Media AF Cross-platform media distribution VR360, on-demand and live (3-DoF) Immersive media with 6-DoF Combining natural and synthetic content MPEG’s Five-Year Roadmap IEEE ICME Tutorial - July 2017 98
  • 99. Tile-Based Streaming IEEE ICME Tutorial - July 2017 99 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Encoding & Packaging Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Delivery MPEG-HEVC/H.265 Tiles in ISOBMFF Adaptive Player Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile-based streaming of VR/360° content with MPEG-DASH SRD … … … … Adaptive Streaming using MPEG-DASH SRD Head Mounted Displays Browsers, Smart (Mobile) Devices (Stereo) 2D, (Stereo) 3D
  • 100. IEEE ICME Tutorial - July 2017 100 Tile-Based Streaming Options
  • 101. IEEE ICME Tutorial - July 2017 101 ODV Quality Metrics: S-PSNR
  • 102. IEEE ICME Tutorial - July 2017 102 ODV Quality Metrics: V-PSNR
  • 103. Thank You “Television! Teacher, mother, secret lover” – Homer Simpson IEEE ICME Tutorial - July 2017 103
  • 104. IEEE ICME Tutorial - July 2017 104 Source Code for Client Implementations • DASH Industry Forum – http://dashif.org/software/ • Open Source Implementations/Frameworks – http://dash.itec.aau.at/ – http://gpac.wp.mines-telecom.fr/ – https://github.com/google/shaka-player • JW Player – https://github.com/jwplayer/jwplayer
  • 105. IEEE ICME Tutorial - July 2017 105 SAND Implementations on GitHub • SANDLibrary: https://github.com/fraunhoferfokus/SANDLibrary • SANDServer: https://github.com/fraunhoferfokus/SANDServer • TNO’s SAND demo: https://github.com/tnomedialab/sand ©MatthiasHeyde/FraunhoferFOKUS