Delivering Traditional and Omnidirectional Media

Alpen-Adria-Universität
Alpen-Adria-UniversitätAssociate Professor at Alpen-Adria-Universität
IEEE ICME Tutorial – San Diego, CA July 2018
Delivering Traditional and Omnidirectional Media
Ali C. Begen, Liangping Ma and Christian Timmerer
http://www.slideshare.net/christian.timmerer
Upon Attending This Tutorial, You Will Know About
• Principles of HTTP adaptive streaming for the Web/HTML5
• Principles of omnidirectional (360°) media delivery
• Content generation/distribution/consumption workflows for traditional and
omnidirectional media
• Standards and emerging technologies in the adaptive streaming space
• 5G radio access and core networks, technologies and emerging multimedia use cases
This tutorial is public, however, some material might be copyrighted
Use proper citation when using content from this tutorial
(Thanks to T. Stockhammer, J. Simmons, K. Hughes, C. Concolato, S. Pham, W. Law, N. Weil, P. Chou
and many others for helping with the material)
IEEE ICME Tutorial - July 2018 2
Ali C. Begen
• Electrical engineering degree from Bilkent
University (2001)
• Ph.D. degree from Georgia Tech (2006)
– Video delivery and multimedia communications
• Research, development and standards at
Cisco (2007-2015)
– IPTV, content delivery, software clients
– Transport and distribution over IP networks
– Enterprise video
• Consulting at Networked Media since 2016
• Assistant professor at OzU and IEEE
Distinguished Lecturer since 2016
Christian Timmerer
• Associate professor at the Institute of
Information Technology (ITEC), Multimedia
Communication Group (MMC), Alpen-Adria-
Universität Klagenfurt, Austria
• Co-founder and CIO of Bitmovin
• Research interests
– Immersive multimedia communication
– Streaming, adaptation, and
– Quality of experience (QoE)
• Blog: http://blog.timmerer.com; @timse7
Presenters Today – Part I
IEEE ICME Tutorial - July 2018 3
Liangping Ma
• Member of Technical Staff, InterDigital (2009-)
• Research Engineer, San Diego Research Center
(2005-2007), Argon ST (2007-2009)
• PhD, Electrical Engineering, University of
Delaware (2004)
• BS, Physics, Wuhan University, China (1998)
• IEEE ComSoc Distinguished Lecturer since 2017
• Research interests
– Video communication
– 5G radio access network
– Cognitive radios
Presenters Today – Part II
IEEE ICME Tutorial - July 2018 4
• Two tutorials, 30 talks on
– HEVC, VVC and AV1
– Containers
– HDR
– Next-gen audio
– The latest and greatest in streaming
– Content protection and more
• Cost: Free
• Visit http://mile-high.video/ to register
Next week in Denver, CO
Mile High Video 2018
IEEE ICME Tutorial - July 2018 5
ACM MMSys 2019 Call for Papers and Special Sessions
• City: Amherst, MA
• Date: June 18-21, 2019
• Submission deadline: Oct. 30
• Co-located with
– NOSSDAV
– MMVE
– Packet Video
• http://www.mmsys2019.org/
IEEE ICME Tutorial - July 2018 6
Topics to Cover
• Part I: Streaming (Morning)
– HTTP adaptive streaming and workflows
– HTML5 video and media extensions
– Common issues in scaling and multi-screen/hybrid delivery
– Overview of Streaming Standards (DASH, CMAF and Apple HLS)
– Adaptive delivery of omnidirectional/360° video
– The developing MPEG-OMAF and MPEG-I standards
• Part II: Communications over 5G (Afternoon)
– 5G fundamentals: core network, radio access network and QoS
– Multimedia communication case studies
– Improving QoE in streaming
IEEE ICME Tutorial - July 2018 7
IPTV vs. IP (Over-the-Top) Video
First Things First
IPTV IP Video
Best-effort delivery
Quality not guaranteed
Mostly on demand
Paid or ad-based service
Managed delivery
Emphasis on quality
Mostly linear TV
Always a paid service
IEEE ICME Tutorial - July 2018 8
Part I: Streaming
HTTP Adaptive Streaming and Workflows
Internet (IP aka OTT) Video Essentials
Reach all connected devicesReach
Enable live and on-demand delivery to the mass marketScale
Provide TV-like consistent rich viewer experienceQuality of Experience
Enable revenue generation thru paid content, subscriptions,
ads, etc.Business
Satisfy regulations such as captioning, ratings and parental
controlRegulatory
IEEE ICME Tutorial - July 2018 10
Creating Revenue – Attracting Eye Balls
• High-end content
– Hollywood movies, TV shows
– Sports
• Excellent quality
– HD/3D/UHD audiovisual presentation w/o artifacts such as pixelization and rebuffering
– Fast startup, fast zapping and low glass-to-glass delay
• Usability
– Navigation, content discovery, battery consumption, trick modes, social network integration
• Service flexibility
– Linear TV
– Time-shifted and on-demand services
• Reach
– Any device, any time
IEEE ICME Tutorial - July 2018 11
One Request, One Response
Progressive Download
HTTP Request
HTTP Response
IEEE ICME Tutorial - July 2018 12
What is Streaming?
Streaming is transmission of a continuous content from a server
to a client and its simultaneous consumption by the client
Two Main Characteristics
1. Client consumption rate may be limited by real-time constraints as opposed
to just bandwidth availability
2. Server transmission rate (loosely or tightly) matches to client consumption
rate
IEEE ICME Tutorial - July 2018 13
Video Delivery over HTTP
• Enables
playback while
still downloading
• Server sends the
file as fast as
possible
Progressive
Download
• Enables seeking
via media indexing
• Server paces
transmission based
on encoding rate
Pseudo
Streaming
• Content is divided
into short-duration
chunks
• Enables live
streaming and ad
insertion
Chunked
Streaming
• Multiple versions
of the content are
created
• Enables to adapt
to network and
device conditions
Adaptive
Streaming
IEEE ICME Tutorial - July 2018 14
Adaptive Streaming over HTTP
Decoding and
Presentation
Streaming Client
Media Buffer
Content Ingest
(Live or Pre-captured)
Multi-rate
Encoder
Packager Origin (HTTP)
Server
…
…
…
…
ServerStorage
HTTP GET
Request
Response
IEEE ICME Tutorial - July 2018 15
Adapt Video to Web Rather than Changing the Web
Adaptive Streaming over HTTP
• Imitation of streaming via short downloads
– Downloads small chunks to minimize bandwidth waste
– Enables to monitor consumption and track the streaming clients
• Adaptation to dynamic conditions and device capabilities
– Adapts to dynamic conditions in the Internet and home network
– Adapts to display resolution, CPU and memory resources of the streaming client
à Facilitates “any device, anywhere, anytime” paradigm
• Improved quality of experience (not always improved average quality)
– Enables faster start-up and seeking, and quicker buffer fills
– Reduces skips, freezes and stutters
• Use of HTTP
– Well-understood naming/addressing approach, and authentication/authorization infrastructure
– Provides easy traversal for all kinds of middleboxes (e.g., NATs, firewalls)
– Enables cloud access, leverages the existing (cheap) HTTP caching infrastructure
IEEE ICME Tutorial - July 2018 16
Stalls, Slow Start-Up, Plug-In and DRM Issues
Common Annoyances in Streaming
• Unsupported/wrong
– protocol
– plug-in
– codec
– format
– DRM
• Slow start-up
• Poor quality, quality variation
• Frequent freezes/glitches
• Lack of seeking
IEEE ICME Tutorial - July 2018 17
Dead, Surviving, Maturing and Newborn Technologies
• Move Adaptive Stream (Long gone, but some components are in Slingbox)
– http://www.movenetworks.com
• Microsoft Smooth Streaming (Legacy)
– http://www.iis.net/expand/SmoothStreaming
• Adobe Flash (Almost dead)
– http://www.adobe.com/products/flashplayer.html
• Adobe HTTP Dynamic Streaming (Legacy)
– http://www.adobe.com/products/httpdynamicstreaming
• Apple HTTP Live Streaming (The elephant in the room)
– https://tools.ietf.org/html/rfc8216
– https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis
• MPEG DASH and CMAF (The standards)
– http://mpeg.chiariglione.org/standards/mpeg-dash
– http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format
IEEE ICME Tutorial - July 2018 18
List of Accessible Segments and Their Timings
An Example DASH Template-Based Manifest
MPD
Period id = 1
start = 0 s
Period id = 3
start = 300 s
Period id = 4
start = 850 s
Period id = 2
start = 100 s
Adaptation Set 0
subtitle turkish
Adaptation Set 2
audio english
Adaptation Set 1
BaseURL=http://abr.rocks.com/
Representation 2
Rate = 1 Mbps
Representation 4
Rate = 3 Mbps
Representation 1
Rate = 500 Kbps
Representation 3
Rate = 2 Mbps
Resolution = 720p
Segment Info
Duration = 10 s
Template:
3/$Number$.mp4
Segment Access
Initialization Segment
http://abr.rocks.com/3/0.mp4
Media Segment 1
start = 0 s
http://abr.rocks.com/3/1.mp4
Media Segment 2
start = 10 s
http://abr.rocks.com/3/2.mp4
Adaptation Set 3
audio italian
Adaptation Set 1
video
Period id = 2
start = 100 s
Representation 3
Rate = 2 Mbps
Selection of
components/tracks
Well-defined
media format
Selection of
representations
Splicing of arbitrary
content like ads
Chunks with addresses
and timing
IEEE ICME Tutorial - July 2018 19
An Example HLS Playlist-Based Manifest
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=232370,CODECS="mp4a.40.2, avc1.4d4015"
gear1/prog_index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=649879,CODECS="mp4a.40.2, avc1.4d401e"
gear2/prog_index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=41457,CODECS="mp4a.40.2"
gear0/prog_index.m3u8
master.m3u8
Source: https://developer.apple.com/streaming/examples/ and https://www.gpac-licensing.com/2014/12/01/apple-hls-comparing-versions/
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:9.97667,
fileSequence0.ts
#EXTINF:9.97667,
fileSequence1.ts
#EXTINF:9.97667,
fileSequence2.ts
.
.
.
#EXT-X-ENDLIST
gear1/prog_index.m3u8
IEEE ICME Tutorial - July 2018 20
Example Representations
Encoding
Bitrate
Resolution
Rep. #1 3.45 Mbps 1280 x 720
Rep. #2 2.2 Mbps 960 x 540
Rep. #3 1.4 Mbps 960 x 540
Rep. #4 900 Kbps 512 x 288
Rep. #5 600 Kbps 512 x 288
Rep. #6 400 Kbps 340 x 192
Rep. #7 200 Kbps 340 x 192
Source: Vertigo MIX10, Alex Zambelli’s Streaming Media Blog, Akamai, Comcast
Vancouver 2010 Sochi 2014
Encoding
Bitrate
Resolution
Rep. #1 3.45 Mbps 1280 x 720
Rep. #2 1.95 Mbps 848 x 480
Rep. #3 1.25 Mbps 640 x 360
Rep. #4 900 Kbps 512 x 288
Rep. #5 600 Kbps 400 x 224
Rep. #6 400 Kbps 312 x 176
Encoding
Bitrate
Resolution
Rep. #1 18 Mbps 4K (60p)
Rep. #2 12.2 Mbps 2560x1440 (60p)
Rep. #3 4.7 Mbps 2K (60p)
Rep. #4 3.5 Mbps 1280x720 (60p)
Rep. #5 2 Mbps 1280 x 720
Rep. #6 1.2 Mbps 768 x 432
Rep. #7 750 Kbps 640 x 360
Rep. #8 500 Kbps 512 x 288
Rep. #9 300 Kbps 320 x 180
Rep. #10 200 Kbps 320 x 180
PyeongChang 2018
IEEE ICME Tutorial - July 2018 21
Smart and Selfish Clients
HAS Working Principle - Client fetches and parses the manifest
- Client uses the OS-provided HTTP stack
(HTTP may run over TCP or QUIC)
- Client uses the required decryption tools
for the protected content
Client monitors and measures
- Size of the playout buffer (both in bytes and seconds)
- Chunk download times and throughput
- Local resources (CPU, memory, window size, etc.)
- Dropped frames
Client performs adaptation
Request
Response
HTTPServer
Client
Client measures and reports metrics for analytics
(One can also multicast
media segments)
IEEE ICME Tutorial - July 2018 22
Tradeoffs in Adaptive Streaming
Overall quality
Quality stability
Proximity to live edge
Stalls
Zapping/seeking time
IEEE ICME Tutorial - July 2018 23
End-to-End Workflow for OTT
Production Preparation and Staging Distribution Consumption
News
Gathering
Sport Events
Premium
Content
Studio
Multi-bitrate
Encoding
Encapsulation
Protection
Origin Servers
VoD
Content &
Manifests
Live
Content &
Manifests
CDN
IEEE ICME Tutorial - July 2018 24
Part I: Streaming
HTML5 Video and Media Extensions
• HTML5 is a set of technologies that
allows more powerful Web sites and
applications
– Better semantics
– Better connectivity
– Offline and storage
– Multimedia
– 2D/3D graphics and effects
– Performance and integration
– Device access
– Styling
• Most interesting new elements
– Semantic elements: <header>, <footer>,
<article> and <section>
– Graphic elements: <svg> and <canvas>
– Multimedia elements: <audio> and <video>
A Common Platform across Devices
HTML5
Source: Mozilla MDN, w3schools.com
IEEE ICME Tutorial - July 2018 26
• These apps can
– leverage native APIs and offline capability
– provide cross-device compatibility (iOS,
Windows, Android, etc.)
– be packaged and published to app stores
• For streaming:
– To a user, it looks like a regular native app
they have downloaded
– The app is actually a container and loads
loads dynamically from the web server
– Server-side changes propagate
automatically to installed apps
Hosted and Progressive Web Apps
Source: Google PWA, Microsoft HWA
IEEE ICME Tutorial - July 2018 27
Types of Browser-Based Playback
Source: CTA WAVE
Type 1: Minimum control architectural model for HTML5 support of adaptive
streaming where manifest and heuristics are managed by the user agent
Type 2: Adaptation control architectural model for HTML5 support of
adaptive streaming providing script manageable features
Type 3: Full media control architectural model for HTML5 support of adaptive
streaming allowing script to explicitly send the media segments (MSE + EME)
IEEE ICME Tutorial - July 2018 28
MSE Support in Web Browsers (as of July’18)
Source: http://caniuse.com/#search=mse
IEEE ICME Tutorial - July 2018 29
Content Protection – Famous Quotes
Digital files cannot be made uncopiable,
any more than water can be made not wet
Bruce Schneier (cryptographer), 2001
We have Ph.D.'s here that know the stuff
cold, and we don't believe it's possible to
protect digital content
Steve Jobs, 2003
If we’re still talking about DRM in five
years, please take me out and shoot me
eMusic CEO David Pakman, 2007
IEEE ICME Tutorial - July 2018 30
W3C Encrypted Media Extensions
Source: https://www.w3.org/TR/encrypted-media/
IEEE ICME Tutorial - July 2018 31
EME Support in Web Browsers (as of July’18)
Source: http://caniuse.com/#search=eme
IEEE ICME Tutorial - July 2018 32
Web Video Ecosystem
Encoding
Encryption
Rights Expression
Web Video App Framework
Decoding
Decryption
Rights Management
The fundamental Digital Rights Management
problem derives from a lack of interoperability
which prevents mobility of experience
The solution is to combine interoperable
commercial Web video content with a cross-
platform Web video app framework
Source: John Simmons
IEEE ICME Tutorial - July 2018 33
Contrary to a common misconception, with EME DRM
functionality is not in the HTML/JS app
There is no DRM in HTML5 with EME, and ECP
requires that this be the case
The HTML/JS app selects the DRM and controls key
exchange between DRM client and server
Browser extends HTML5 media element to allow
JavaScript handled key acquisition
Browser
A CDM exposes a key system to JavaScript; it is
transparent whether the CDM is in the browser
The Web, EME and Enhanced Content Protection (ECP)
Source: John Simmons
IEEE ICME Tutorial - July 2018 34
DRM Support on Desktop Browsers
Browser OS EME/CDM Flash Player NPAPI/ Silverlight 5
Chrome Win Widevine (Yes) No
OS X (Yes) No
Linux (Yes) No
Firefox Win Widevine Yes No
OS X Yes No
Linux Yes No
Safari > OS X Yosemite (FairPlay) Yes Yes
< OS X Yosemite No Yes Yes
IE/Edge < Win 7 No Yes Yes
Win 10 PlayReady Yes No
IEEE ICME Tutorial - July 2018 35
DRM Support on Mobile Platforms
Platform Browser Native App / DRM
iOS No MSE/EME yet
HLS (AES-128 CBC) via <video>
Native SDK and WebView: FairPlay
Android MSE/EME Native + Android SDK (MediaDRM APIs) - (Widevine
+ OMA v2)
ExoPlayer
Native + WebView with Widevine
Windows 10 MSE/EME WebViews/Hosted Web Apps with PlayReady
IEEE ICME Tutorial - July 2018 36
• Server-side ad insertion (SSAI)
– Mostly used for linear/dynamic ad insertion (ads
are still targeted)
– Easier for less capable devices
– Smoother transition between the content and ads
– Uses XLink or MPD chaining in DASH for multi-
period MPDs
– Splices ads into a continuous stream (offline or
just-in-time) for single-period MPDs
• Client-side ad insertion
– Requires additional support in the application
– Uses events in DASH
Ad Insertion Methods
Preroll Idle Midroll Idle Postroll
Preroll Content Midroll Content Postroll Pause Content Pause Content Pause
Primary Stream Primary Stream
Ad Stream
IEEE ICME Tutorial - July 2018 37
Ad Insertion – XLink Resolution
IEEE ICME Tutorial - July 2018 38
Part I: Streaming
Common Issues in Scaling and Multi-Screen/Hybrid Delivery
Streaming over HTTP – The Promise
• Leverage tried-and-true Web infrastructure for scaling
– Video is just ordinary Web content!
• Leverage tried-and-true TCP
– Congestion avoidance
– Reliability
– No special QoS for video
THERE
IT SHOULD
JUST WORK
IEEE ICME Tutorial - July 2018 40
Doesitjustwork?
Mostly yes, when streaming
clients compete with other
types of traffic
Not really, when streaming
clients compete with each
other
Streaming clients interact with each
other forming an “accidental”
distributed control-feedback system
• Multiple screens within a household
• ISP access/aggregation links
• Small cells in stadiums and malls
IEEE ICME Tutorial - July 2018 41
A Single Microsoft Smooth Streaming Client under a Controlled Environment
Demystifying a Streaming Client
0
1
2
3
4
5
0 50 100 150 200 250 300 350 400 450 500
Bitrate(Mbps)
Time (s)
Available Bandwidth Requests Chunk Tput Average Tput
Reading: “An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP,” ACM MMSys 2011
Buffer-filling State
Back-to-back requests
Steady State
Periodic requests
IEEE ICME Tutorial - July 2018 42
10 (Commercial) Streaming Clients Sharing a 10 Mbps Link
Selfishness Hurts Everyone
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500
RequestedBitrate(Kbps)
Time (s)
Client1 Client2 Client3
IEEE ICME Tutorial - July 2018 43
Viewer Experience Statistics
Selfishness Hurts Everyone
Source: Conviva Viewer Experience Report, 2015
IEEE ICME Tutorial - July 2018 44
Inner and Outer Control Loops
HTTP Server
Manifest
Media
HTTP
Origin
Module
TCP Sender
Streaming Client
Manifest
Resource
Monitors
Streaming
Application
TCP ReceiverData / ACK
There could be multiple TCPs destined to the same or different servers
Request / Response
IEEE ICME Tutorial - July 2018 45
Streaming with Multiple TCP Connections
• Using multiple concurrent TCPs
– Can help mitigate head-of-line blocking
– Allows fetching multiple (sub)segments in parallel
– Allows to quickly abandon a non-working connection without having to slow-start a new one
System performance deteriorates very quickly if many clients adopt
this approach without limiting the aggregated bandwidth consumption
IEEE ICME Tutorial - July 2018 46
Two Competing Clients
Understanding the Root Cause
• Depending on the timing of the ON periods:
– Unfairness, underutilization and/or instability may occur
– Clients may grossly overestimate their fair share of the available bandwidth
Clients cannot figure out how much bandwidth to use until they use too much
(Just like TCP)
Reading: “What happens when HTTP adaptive streaming players compete for bandwidth?,” ACM NOSSDAV 2012
IEEE ICME Tutorial - July 2018 47
How to Solve the Issues?
• Use a better adaptation algorithm like PANDA or BOLA
• Use machine learning or deep learning like Pensieve
• Improve the HTTP/TCP stack, try out the alternatives
• Adopt ideas from game/consensus theory (GTA)
Fix the clients and/or the transport
• QoS in the core/edge
• SDN
Get support from the network
• Assist the clients and network elements thru metrics and analytics
Enable a control plane
IEEE ICME Tutorial - July 2018 48
Bitrate Adaptation Schemes
Bitrate
Adaptation
Schemes
Client-
based
Adaptation
Bandwidth-
based
Buffer-
based
Mixed
adaptation
Proprietary
solutions
MDP-based
Server-
based
Adaptation
Network-
assisted
Adaptation
Hybrid
Adaptation
SDN-based
Server and
network-
assisted
Reading: “A survey on bitrate adaptation schemes for streaming media over HTTP,” IEEE Commun. Surveys Tuts., to appear
IEEE ICME Tutorial - July 2018 49
High-Level Comparison between Different Schemes
• The client-based adaptation schemes
– Show a good performance in single and few-client scenarios
– Largely fail in multi-client scenarios
• The server-based adaptation schemes
– Require custom servers
– More effective in eliminating the bitrate oscillation problems
– Less scalable due to increased complexity on the servers
• The network-assisted adaptation and hybrid schemes
– Show a good performance in both small and large populations
– Require modifications on the clients, server and/or network devices
– Pose practicality issues for deployment
IEEE ICME Tutorial - July 2018 50
A Note on the Reproducibility
• A performance comparison between different algorithms is often difficult due to
– Lack of the source codes
– Lack of a unified QoE framework and universally accepted metrics
• Schemes have their own parameters and assumptions, and work well only under
certain circumstances
IEEE ICME Tutorial - July 2018 51
Game/Consensus Theory
• Structuring a set of players in cooperative groups
• Enables a strong level of cooperation among the players
• Model and analyze fairness, stability and efficiency
Coalition Formation
• Players that are interested in reaching an agreement among themselves over a
common objective (how to reach this agreement and on the terms of the agreement)
• Concept solution: Nash Bargaining Solution (NBS)
Bargaining Solution
• Exploited together with NBS and coalition-formation game
• Relied on a strong coordination among the players
Consensus Theory
IEEE ICME Tutorial - July 2018 52
GTA: A Game Theory Based ABR Scheme
• Designed based on a cooperative game in the form of static formation-based coalitions
• Formulates the ABR decision problem as a bargaining process and consensus mechanism
• Outputs the optimal decision by reaching the Pareto optimal (PO) Nash bargaining solution (NBS)
Game Theory
Cooperative
Coalition
Formation
Bargaining
Solution
Consensus
Theory
Strategic
GTA
GTA player
Coalition
Bargaining process
IEEE ICME Tutorial - July 2018 53
• PANDA & CS2P Estimators
– Throughput estimation
• SAND Enabler
– Implements various SAND-enabled communication
interfaces
• SSIMplus MAP Model
– Maps content type, display resolution and service
plan type into one common perceptual quality
space in which a set of clusters are constructed
• QoE Metrics
– QoE model and various metrics
• GT Agent
– Develops the GTA decision rules for selecting the
best bitrate
GTA Components
Reading: “Want to play DASH? a game theoretic approach for adaptive streaming over HTTP,” ACM MMSys 2018
IEEE ICME Tutorial - July 2018 54
Component #1: Cooperation Enabler
GTA Design
Non-Cooperative Mode
• Mix of GTA and non-GTA players
• Greedy interaction with non-GTA
players
• GTA clients may start in non-
cooperative mode and then learn
the accurate number of clients
through potential and wonderful
life utility (WLU)
Cooperative Mode
• Multiple GTA players
• GTA players are aggregated into
a set of clusters based on
SSIMplus-based MAP
IEEE ICME Tutorial - July 2018 55
Component #2: QoE Calculator
GTA Design
Avg. Quality Avg. Quality Switch Stalls Startup Delay
IEEE ICME Tutorial - July 2018 56
Component #3 and #4: GT Strategy Calculator and GT (Dis)agreement Calculator
GTA Design
• GT Strategy Calculator
– Compute the GT strategy (i.e., the action-utility R(A) relationship set) which is defined over the
strategy space S as follows:
• GT (Dis)agreement Calculator
– Compute the GT disagreement decision
• i.e., bargaining outcome disagreements (W)
P = {p1,...,pN} is the set of N players
R is the set of possible utilities when actions
from A are taken
• r is the player utility when action a is
taken
• k is the current step, K is total number of
segments of video u
R(A) is the set of possible actions with their
achievable utilities
R-(A-) is the set of pessimal actions with their
resulting utilities
S and W are defined by the function F which
is defined over the space: F: (A → R) ∪ {W}
PO (NBS): max. (S – W)
AgreementpointDisagreementpoint
MAX
IEEE ICME Tutorial - July 2018 57
Component #5: QoE Optimizer
GTA Design
• GTA rule for ABR selection can be formulated as a network utility maximization
– Strictly increasing concave
– Solver: Dual decomposition + dynamic programming
• Find the bargaining solution by solving the Problem(S,W)
– Bargaining Process and consensus decision problem is defined as a pair Problem(S,W)
– The bargaining solution is defined by a function F over the space F : (S, W ) → Rn
– F specifies a unique Pareto optimal (PO) bargaining outcome (O* = F(S,W)) for every Problem(S,W) using Nash Bargaining
Solution (NBS)
– O* is the set that contains only the optimal bargaining outcomes
C.1 Maintain the current buffer occupancy
between the min and max buffer
thresholds
C.2 Satisfy the SSIMplus MAP model
C.3 Lead to a unique (PO) NBS
C.4 Chunk time constraint where max
time needed to download the
corresponding chunk ≦ chunk duration
IEEE ICME Tutorial - July 2018 58
Source Code is Available at http://streaming.university/GTA/
dash.js Integration
Gray boxes indicate the components that have been developed or modified
IEEE ICME Tutorial - July 2018 59
Setup for Performance Evaluation
• VoD experimental setup
– Two machines (running Ubuntu 16.04 LTS), one for the dash.js player and one for the server
– tc-NetEm to replicate the throughput profiles
– PANDA parameters as described in the original paper
– Throughput smoothing: the mean of the last three throughput estimations
– The fastMPC horizon was set to the next three chunks
– The min and max buffer occupancy thresholds were set to 8 and 32 seconds
– CT = animation (BBB), chunk duration of 4 s, total duration of 600 s
– 10 bitrate levels: [250, 300, 400, 700, 900, 1500, 2000, 3000, 3500, 4000] Kbps
IEEE ICME Tutorial - July 2018 60
Results: Video Stability
IEEE ICME Tutorial - July 2018 61
Results: QoE
IEEE ICME Tutorial - July 2018 62
Numbers Indicate GTA Improvement over Other Schemes
GTA Outperforms the Existing ABR Schemes
BBA BOLA Rate-based Hybrid Quetra ELASTIC QDASH SARA FESTIVE
QoE 55% 33.5% 35.5% 19.5% 39% 46.5% 47% 40.5% 30.5%
Video Stability 60% 72% 37.5% 63.5% 66.5% 70% 66% 53% 63.5%
Stall Events 13.6% 1.1% 6.1% 1.7% 3.87% 7.87% 5.87% 2.9% 3.6%
IEEE ICME Tutorial - July 2018 63
http://streaming.university/GTA/
GTA Demo and Source Codes
Offline Demo Online Demo
IEEE ICME Tutorial - July 2018 64
Quickly Starting Media Streams Using QUIC
QUIC: Multiplexed transport protocol over UDP
• Reduced handshake latency
• Improved congestion control
• Improved loss recovery
• Multiplexing streams
Can QUIC help improve viewer experience in
HTTP adaptive streaming?
Reading: “Quickly starting media streams using QUIC,” PV 2018
IEEE ICME Tutorial - July 2018 65
• Start media streams more
quickly
• Reduce seeking latency
• Cope better with frequent
connection changes?
Can QUIC help us
IEEE ICME Tutorial - July 2018 66
Previous Research
• Some reported
– QUIC does not impact streaming performance
– QUIC does not provide a boost to HAS
• Others reported
– QUIC’s 0-RTT performed better than the other protocols
– QUIC provided better streaming, but only for high-quality video
• Google said QUIC reduced YouTube rebuffer rates by 15%
QUIC code evolves rapidly and 3rd-party implementations may
not necessarily reflect protocol’s real performance
IEEE ICME Tutorial - July 2018 67
Comparison between Our and Prior Work
1 WiFi, 4G/LTE and 3G, 2 Based on the third-party implementation version at the time of research, 3 Latest at the time of research.
IEEE ICME Tutorial - July 2018 68
• Player Features
– Python based
– TCP and QUIC support integrated
as subprocess
– Scripted frame-seek (fast-
forward) ability
– BASIC, SARA, BBA-2 adaptation
algorithms
• Source Code
– github.com/sevketarisu/quic-
streaming
github.com/sevketarisu/quic-streaming
IEEE ICME Tutorial - July 2018 69
• Environment Setup
– Public Internet
– No traffic shaping
– Each test repeated 10 times
• Server
– Apache HTTP server on Amazon
EC2
– Located in Frankfurt, DE
• Client
– Runs Google’s proto-quic library
– Located in Istanbul, TR
– Connected via WiFi, LTE or 3G
Chromium QUIC
(proto-quic library)
WiFi, LTE, 3G
Amazon EC2
libcurl
IEEE ICME Tutorial - July 2018 70
Evaluated Metrics
• Average playback bitrate
– Average of the bitrates of the downloaded segments
• Average wait time after seeking
– Time from the frame-seek request to the playback of the requested media
– A rule of thumb is to keep this time under two seconds
• Rebuffer rate
IEEE ICME Tutorial - July 2018 71
Frame-Seek (Fast-Forward) Feature Implemented in the Player
Frame-Seek Scenarios
IEEE ICME Tutorial - July 2018 72
Frame-Seek Scenario Results
2.80
2.99
2.862.89
3.05 2.99
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
WiFi LTE 3G
AveragePlaybackBitrate(Mbps)
TCP QUIC
Results are averaged for BASIC, SARA and BBA-2 algorithms.
QUIC provided a higher (or at least an
equal) average playback bitrate in all
cases and for all algorithms
Max Bitrate Available: 3.9 Mbps
IEEE ICME Tutorial - July 2018 73
Frame-Seek Scenario Results
2.09 2.16
2.38
1.28 1.29
1.42
0.0
0.5
1.0
1.5
2.0
2.5
3.0
WiFi LTE 3G
AverageWaitTimeafterSeeking(s)
TCP QUIC
Results are averaged for BASIC, SARA and BBA-2 algorithms.
QUIC reduced wait times to
less than 1.5 seconds
IEEE ICME Tutorial - July 2018 74
Frame-Seek Scenario Results
3.67 3.67
5.67
1.00
1.33
2.00
0
1
2
3
4
5
6
WiFi LTE 3G
RebufferRate(%)
TCP QUIC
QUIC’s benefits are bigger
when delays are higher
Results are averaged for BASIC, SARA and BBA-2 algorithms.
IEEE ICME Tutorial - July 2018 75
Connection-Switch Simulation
IEEE ICME Tutorial - July 2018 76
Connection-Switch Scenario
IEEE ICME Tutorial - July 2018 77
WiFi - LTE Switches
Connection-Switch Results
3.00
4.00
5.00
2.00
3.00
4.00
0
1
2
3
4
5
6
BASIC SARA BBA-2
RebufferRate(%)
TCP QUIC
QUIC provided a higher (or at least
an equal) average playback bitrate
for all algorithms
IEEE ICME Tutorial - July 2018 78
WiFi - 3G Switches
Connection-Switch Results
13.00
2.00
5.00
6.00
1.00
4.00
0
2
4
6
8
10
12
14
BASIC SARA BBA-2
RebufferRate(%)
TCP QUIC
QUIC provided a higher (or at least
an equal) average playback bitrate
for all algorithms
IEEE ICME Tutorial - July 2018 79
But
QUIC is still evolving in the IETF;
should there be significant
changes, the tests will need to be
repeated
– https://quicwg.org/
– https://github.com/quicwg
Conclusions So Far
• QUIC reduces wait times and
rebuffer rates without reducing
playback bitrate
• QUIC outperforms TCP when
frequent network changes occur
• QUIC’s benefits are greater in
networks with larger delay (e.g.,
early generation 3G networks)
IEEE ICME Tutorial - July 2018 80
One Slide on Software Defined Networking (SDN)
Control and data planes are decoupled, network intelligence and state are logically
centralized, and the underlying network infrastructure is abstracted from the apps
IEEE ICME Tutorial - July 2018 81
SDN-Based Bitrate Adaptation
Reading: “SDNDASH: improving QoE of HTTP adaptive streaming using software defined networking,” ACM MM 2016
IEEE ICME Tutorial - July 2018 82
Server and Network-Assisted Adaptation
• A control plane approach that offers asynchronous client-to-network, network-to-client
and network-to-network communications
– Collect metrics and status information from various entities in the system
– Share this information to assist the clients and aware elements to improve media delivery
• DASH part 5 (23009-5) outlines the server and network-assisted DASH (SAND)
framework along with the necessary interfaces
IEEE ICME Tutorial - July 2018 83
Conflicting Goals in the Media Delivery Chain
• I want to make sure that my content is protected and looks awesome on any device
• I want to make sure that my ads are viewed, trackable and measurable
• I want to make sure that my servers are properly used and latency is low
• I want to control the QoE of my customers, make $ from and differentiate my own video services
• I want to make sure that my device/app provides the best possible video quality
• I want the best quality for minimal $
ConsumerContent
Provider
ISP
CDN
ProviderAdvertiser
Device/
App
IEEE ICME Tutorial - July 2018 84
Introducing DASH-Assisting Network Elements (DANE)
A Control Plane Approach
Origin (HTTP)
Server
Encoder/
Transcoder Packager
CDN
RG
Clients
Media
Parameters for enhancing reception (PER)
Metrics and status messages
Parameters for enhancing delivery (PED)
Analytics
Server
IEEE ICME Tutorial - July 2018 85
Why is Latency Important?
IEEE ICME Tutorial - July 2018 86
Contributors to the Latency
• Video encoding pipeline duration
• Ingest and packaging operations
• Network propagation delays
• CDN buffer delays
• Media segment duration
• Player behavior
– Buffering
– Playhead positioning
– Resilience
Current Implementations
• Smooth Streaming: 2-second segments,
usually 10 seconds of latency
• DASH: 2-second segments supported by most
players, usually 8 to 10 seconds of latency
• HLS
– Until mid-2016: 10-second segments, usually 30
seconds of latency
– Since mid-2016: 6-second segments, usually 18 to
20 seconds of latency
– Safari Mobile in iOS11: Autostart for live streams
and support for short segments
– App Store & iOS applications: 6-second segments
are still mandatory
Latency in One Slide
IEEE ICME Tutorial - July 2018 87
Stream Start Time ≠ Latency
Time
Live encoder
producing 2-second
segments
iOS (3 segments)
Last available
Lowest latency
1 2 3 4
Start Now
2 3 4
4
5
Latency: 7 s
Latency: 3 s
Latency: 2 s
6 seconds of buffer
~0 seconds of startup*
2 seconds of buffer
~0 seconds of startup*
2 seconds of buffer
1 second of startup*
Segment fetching time is assumed to be negligible in this example
5
IEEE ICME Tutorial - July 2018 88
Latency Facts
• Stream start time: Time it takes for the video to start playing after pressing ‘play’
• Hand–waving (glass-to-glass) latency: Time delay between an action occurring in front
of the camera and the same action being visible on the display
• Stream start time is not equal to latency
– A stream may take 5 s to start and only be 2 s behind live
– Conversely, a stream can start in 2 s and be 5 s behind live
• A player can start quickly by choosing to start behind the live point (which increases
latency) and build its buffer by fetching segments faster than real time till the live edge
• Low latency is always a trade-off against robustness of the playback
IEEE ICME Tutorial - July 2018 89
Part I: Streaming
Overview of Streaming Standards (DASH, CMAF and Apple HLS)
Dead, Surviving, Maturing and Newborn Technologies
• Move Adaptive Stream (Long gone, but some components are in Slingbox)
– http://www.movenetworks.com
• Microsoft Smooth Streaming (Legacy)
– http://www.iis.net/expand/SmoothStreaming
• Adobe Flash (Almost dead)
– http://www.adobe.com/products/flashplayer.html
• Adobe HTTP Dynamic Streaming (Legacy)
– http://www.adobe.com/products/httpdynamicstreaming
• Apple HTTP Live Streaming (The elephant in the room)
– https://tools.ietf.org/html/rfc8216
– https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis
• MPEG DASH and CMAF (The standards)
– http://mpeg.chiariglione.org/standards/mpeg-dash
– http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format
IEEE ICME Tutorial - July 2018 91
• Fragmented architectures
– Advertising, DRM, metadata, blackouts, etc.
• Investing in more hardware and
software
– Increased CapEx and OpEx
• Lack of consistent analytics
• Preparing and delivering each asset in
several incompatible formats
– Higher storage and transport costs
• Confusion due to the lack of skills to
troubleshoot problems
• Lack of common experience across
devices for the same service
– Tricks, captions, subtitles, ads, etc.
What does Fragmentation Mean?
Higher Costs
Less Scalability
Smaller Reach
Frustration
Skepticism
Slow Adoption
IEEE ICME Tutorial - July 2018 92
DASH intends to be to
the Internet world …
what MPEG2-TS has been to
the broadcast world
IEEE ICME Tutorial - July 2018 93
DASH intendsed to be to
the Internet world …
what MPEG2-TS has been to
the broadcast world
IEEE ICME Tutorial - July 2018 94
Dynamic Adaptive Streaming over HTTP (DASH)
• DASH is not
– system, protocol, presentation, codec, interactivity, DRM, client specification
• DASH is an enabler
– It provides formats to enable efficient and high-quality delivery of streaming services
– System definition is left to other organizations
• Design choices
– Reuse the existing technologies (containers, codecs, DRM, etc.)
– Enable deployment on top of CDNs
– Move intelligence from network to client, enable client differentiation
– Provide simple interoperability points (profiles)
IEEE ICME Tutorial - July 2018 95
Shown in Red
Scope of MPEG DASH
HTTP Server DASH Client
Control Engine
MediaEngines
HTTP ClientHTTP/1.1
Segment
ParserMPD Transport
MPD
Parser
MPD
MPD
MPD
...
...
...
IEEE ICME Tutorial - July 2018 96
Brief History of DASH at MPEG
• Amd. 1: NTP sync, extended profiles
• Amd. 2: SRD, URL parameter insertion, role extensions
• Amd. 3: External MPD link, period continuity, generalized HTTP header extensions/queries
• Amd. 4: TV profile, MPD chaining/resetting, data URLs in MPD, switching across adaptation sets
• 3rd edition (FDIS) in ballot
• Amd. 5 (WiP): Device information, quality equivalence descriptor, timed text roles, announcing
popular content, flexible IOP signaling, early available periods, signaling missing/alternative
segments
23009-1: Media Presentation Description and Segment Formats
• 2nd edition was published in Oct. 2017
• Amd. 1 (WiP): SAND conformance rules
23009-2: Conformance and Reference Software
• 3rd edition (WD) is in progress
23009-3: Implementation Guidelines (Informative)
IEEE ICME Tutorial - July 2018 97
Brief History of DASH at MPEG
• 2nd edition (FDIS) in ballot
23009-4: Segment Encryption and Authentication
• 1st edition, published in Feb. 2017
23009-5: Server and Network Assisted DASH (SAND)
• 1st edition, published in Dec. 2017
23009-6: DASH over Full Duplex HTTP-Based Protocols (FDH)
• 1st edition (WD) is work in progress
23009-7: Delivery of CMAF Contents with DASH (Informative)
IEEE ICME Tutorial - July 2018 98
• Describes various types of “push”
behaviors in HTTP/2 and WebSockets
• Primarily targeted for low-latency live
streaming
• Similar activities
– DASH over LTE broadcast (3GPP SA4)
– DVB ABR multicast
– ATSC 3.0 hybrid delivery
Part 6 – DASH over Full Duplex HTTP-Based Protocols
IEEE ICME Tutorial - July 2018 99
Ongoing Work as of MPEG 123 (July 2018)
• Technologies under Consideration (w17812)
– Usage of HEVC tile tracks in DASH
– Annotation and client model for content selection
– Signaling for quality control
– Announcing popular content in DASH
– Using segment templates for forensic watermarking
– Using DASH MPD chaining for mid-roll ads
– DASH playlist description
– Mixed MPD
– Event processing model
– Patch method for MPD updates
IEEE ICME Tutorial - July 2018 100
Common Media Application Format (CMAF)
• Media delivery has three main components:
– Media format
– Manifest
– Delivery
• CMAF defines the media format only (Fragments, headers, segments, chunks, tracks)
– No manifest format or a delivery method is specified
• CMAF uses ISO-BMFF and common encryption (CENC)
– CENC means the media fragments can be decrypted/decoded by devices using different DRMs
– CMAF does not mandate CTR or CBC mode
• This makes CMAF useful only for unencrypted content for now, but industry is moving towards CBCS
• Any delivery method may be used for delivering CMAF content
– HTTP
– RTP multicast/unicast
– LTE broadcast
IEEE ICME Tutorial - July 2018 101
CMAF (ISO/IEC 23000-19)
• CMAF defines media profiles for
– Video
– Audio
– Subtitle
• CMAF defines presentation profiles by selection a media profile from each category
• Current status
– 1st edition was published in Jan. 2018
– Supported in iOS 10+ (with HLS playlists)
– Amd. 1: SHVC media profile and additional audio media profiles
– Amd. 2: xHE-AAC and other media profiles
IEEE ICME Tutorial - July 2018 102
Each Frame can Be a CMAF Chunk
CMAF ISO-BMFF Media Objects
RAP …
RAP
…
RAP
…
RAP
…
Fragment Fragment Fragment Fragment
Segment Segment
Track File
…
…
…
Encoding
Packaging
Encryption
CMAF
Header
Seamless switching can only happen
at fragment boundaries
Manifests may provide URLs to
• Track files
• CMAF header + segments
• CMAF header + chunks
IEEE ICME Tutorial - July 2018 103
ismc
Smooth
mpd
DASH
m3u8
HLS
Source
Encoder
f4m
HDS
m3u8
m3u8
mpd
mpd
ismc
ismc
f4m
f4m
Credit: Akamai
IEEE ICME Tutorial - July 2018 104
ismc
Smooth
mpd
DASH
m3u8
HLS
Source
Encoder
f4m
HDS
m3u8
m3u8
mpd
mpd
ismc
ismc
f4m
f4m
Credit: Akamai
More packaging $
More storage $
Less efficient caching
IEEE ICME Tutorial - July 2018 105
Multi-Platform OTT Workflow
Efficient Distribution in the CDN
CMAF
Source
Encoder
m3u8
mpd
Credit: Akamai
m3u8
mpd
m3u8
mpd
m3u8
mpd
m3u8
mpd
m3u8
mpd
IEEE ICME Tutorial - July 2018 106
Encode and Package Once, Deliver Efficiently
Origin (HTTP)
Server
Encoder/
Transcoder
LinearIngest
On-demandContent
Pick your
favorite codec
Pick your chunk and
segment sizes
CIF
(SCTE)
to client manifest
translation
withEBPmarkers
Packager
IEEE ICME Tutorial - July 2018 107
Non-HTTP Transport Options
RTP Multicast
- Define RTP payload format for CMAF chunks/fragments/segments
- Use RFC 4588 and 6285 for loss recovery and rapid acquisition
Broadcast
- TS 26.346 and TS 26.347: Delivering DASH content via MBMS
- A/331:2017: Carrying DASH content over broadcast and/or broadband networks
Peer-to-Peer
- WebRTC: Clients fetch DASH/CMAF pieces from each other in addition to CDN servers
- This helps reduce the load on the CDN as well as the stream start times
IEEE ICME Tutorial - July 2018 108
• Content format issues
– Each asset is copied to multiple media
formats
• different video codecs
• different audio codecs
• different (regional) frame rates
– Cost to content creators and distributors
– Inefficiencies in CDNs and higher storage
costs
• Platform issues
– Lack of consistent app behavior across
platforms
– Varying video features, APIs and semantics
across platforms
• Playback issues
– Codec incompatibility
– Partial profile support
– Switching bitrate glitches
– Audio discontinuities
– Ad splicing problems
– Long-term playback instability
– Request protocol deficiencies
– Memory problems, CPU weaknesses
– Scaling (display) issues
– Variable HDR support
– Unknown capabilities
Commercial OTT Issues
IEEE ICME Tutorial - July 2018 109
CTA Web Application Video Ecosystem (WAVE)
Steering
Committee
Technical
Working
Group
Content Specification Task Force
Content specification based upcoming CMAF,
compatible with DASH and HLS
Device Playback Capabilities Task Force
Testable requirements covering the most
common device playback interoperability
issues
HTML5 API Task Force
Reference application framework based on
HTML5 providing functional guidelines for
playback interop
IEEE ICME Tutorial - July 2018 110
Web Media API Snapshot 2017
• First annual API Snapshot published in Dec. 2017
– Lists key APIs supported in 2017 in all major HTML code bases
– CTA-W3C agreement to co-publish this spec
– Plan to propose Community Group spec as a W3C standards-track spec
– Available at https://www.w3.org/2017/12/webmediaapi.html
• CTA WAVE issued RFP to create a test suite for all listed APIs based on W3C API
tests
– Test suite will enable manufacturers to test that their HTML support is up-to-date
• Web Media Application Developer Guidelines
– Companion guide to outline best practices for implementing Web media apps
– https://w3c.github.io/webmediaguidelines/
IEEE ICME Tutorial - July 2018 111
WAVE Current and Future Publications
• Published
– “Web Media API Snapshot 2017”, Final Community Group Report, Dec. 2017,
https://www.w3.org/2017/12/webmediaapi.html
– “Web Application Video Ecosystem – Content Specification”, Apr. 2018,
https://members.cta.tech/ctaWAVE
• Pending
– Event Messages in WAVE (white paper)
– Web Application Video Ecosystem (WAVE) Device Playback Capabilities
– Web Media Application Developer Guidelines
– Web Media Porting Specification
IEEE ICME Tutorial - July 2018 112
Part I: Streaming
Adaptive Delivery of Omnidirectional/360° Video
$44,000,000
Example Platform/Infrastructure
https://bitmovin.com/
MPEG’s Definition
What is Virtual Reality (VR)
VR is a rendered environment (visual and acoustic, pre-
dominantly real-world) providing an immersive experience to a
user who can interact with it in a seemingly real or physical way
using special electronic equipment
IEEE ICME Tutorial - July 2018 116
Virtual Reality (VR) puts us in a Virtual World
Source: Phil Chou
IEEE ICME Tutorial - July 2018 117
Augmented Reality (AR) puts Virtual Objects in our World
Source: Phil Chou
IEEE ICME Tutorial - July 2018 118
Delivering High-Quality VR in Economic Scale is Extremely Challenging
The VR Challenge
30K x 24K x 36 x 60 x 2 / 600
~5.2 Gbps
Resolution
Bit depth Frame rate Stereoscopic
Compression gain
IEEE ICME Tutorial - July 2018 119
Ultimate Level of Immersion
Interactions
So intuitive that they
become second nature
Sounds
So accurate that
they are true to life
Visuals
So vibrant that they are eventually
indistinguishable from the real world
IEEE ICME Tutorial - July 2018 120
Visual
quality
Sound
quality
Intuitive
interactions
Immersive VR Has Extreme Requirements
Immersion
High resolution audio
Up to human hearing capabilities
3D audio
Realistic 3D, positional, surround
audio that is accurate to the real world
Precise motion tracking
Accurate on-device motion tracking
Minimal latency
Minimized system latency
to remove perceptible lag
Natural user interfaces
Seamlessly interact with VR using
natural movements, free from wires
Extreme pixel
quantity and quality
Screen is very close to the eyes
Spherical view
Look anywhere with a
full 360° spherical view
Stereoscopic display
Humans see in 3D
Source: Thomas Stockhammer
IEEE ICME Tutorial - July 2018 121
Need for Higher Resolutions on Mobile Phones?
Source: Thomas Stockhammer
IEEE ICME Tutorial - July 2018 122
Omnidirectional/360° Video
Capturing
Devices
• Stitching, projection formats
• Encoding, encryption, encapsulation
• Storage, content distribution, delivery
• Processing, decoding, rendering
Consumer
Devices
IEEE ICME Tutorial - July 2018 123
Adaptive Streaming of VR Content
• Required resolution of a panoramic video for achieving 4K resolution for viewport with
120° field of view (FoV)
IEEE ICME Tutorial - July 2018 124
Functional Architecture
Encode,
Encrypt,
Encapsulate
Decode,
Decrypt,
Decapsulate
Project &
Render
Fuse, Stitch
& Edit
①
② ③ ④
⑤
Store & Deliver
Capture Consume
Content Creation NetworkServer Client
Encoding
Encryption
Encapsulation
Storage
Delivery Decryption
Decapsulation
Decoding RenderingEditing
Processing Processing
Capture
Acquisition
Consumption
Distribution
Adaptive Delivery of Omnidirectional Video
Interaction
From ecosystem…
To building blocks…
IEEE ICME Tutorial - July 2018 125
Functional Architecture
Encode,
Encrypt,
Encapsulate
Decode,
Decrypt,
Decapsulate
Project &
Render
Fuse, Stitch
& Edit
①
② ③ ④
⑤
Store & Deliver
Capture Consume
Content Creation NetworkServer Client
Encoding
Encryption
Encapsulation
Storage
Delivery Decryption
Decapsulation
Decoding RenderingEditing
Processing Processing
Capture
Acquisition
Consumption
Distribution
Adaptive Delivery of Omnidirectional Video
Interaction
From ecosystem…
To building blocks…
IEEE ICME Tutorial - July 2018 126
Adaptive Streaming Options (1)
• Traditional, viewport-agnostic
streaming
– ERP – handle like reg. video
– Simple, easy, deployed today
– Bandwidth waste, quality issues
• Viewport-adaptive streaming
– Multiple versions for predefined
viewports
– Various projection techniques (pyramid)
– Bandwidth waste reduced, increased
storage and CDN costs, limited flexibility X. Corbillon, et al., "Viewport-adaptive navigable 360-degree
video delivery," 2017 IEEE International Conference on
Communications (ICC), Paris, 2017,
https://doi.org/10.1109/ICC.2017.7996611
IEEE ICME Tutorial - July 2018 127
Adaptive Streaming Options (2)
• Tile-based streaming
– Use tiling technique of
modern video codecs
– High complexity, full flexibility
– Multiple challenges
M. Graf, et al. 2017. Towards Bandwidth Efficient Adaptive
Streaming of Omnidirectional Video over HTTP: Design,
Implementation, and Evaluation. Proc. ACM MMSys'17.
https://doi.org/10.1145/3083187.3084016
C. Concolato, et al., "Adaptive Streaming of
HEVC Tiled Videos using MPEG-DASH,"
IEEE TCSVT, 2017.
https://doi.org/10.1109/TCSVT.2017.2688491
IEEE ICME Tutorial - July 2018 128
System Overview
Tile-Based Adaptive Streaming
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Encoding &
Packaging
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
DifferentQ
uality
Representations
Delivery
MPEG-HEVC/H.265
Tiles in ISOBMFF
Adaptive
Player
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile-based streaming of VR/360°
content with MPEG-DASH SRD / OMAF
…
…
…
…
Adaptive Streaming using
MPEG-DASH SRD / OMAF
Head Mounted Displays
Browsers, Smart (Mobile) Devices
(Stereo) 2D, (Stereo) 3D
IEEE ICME Tutorial - July 2018 129
Encoding Options
• AVC dominates the market
• HEVC, VP9, AV1 support tiles
– Divides a picture into
independent, rectangular regions
– Tradeoff: bitrate, quality, flexibility
• Multiple tiling options available
– Uniform vs. non-uniform tiling
– Same vs. mixed resolutions
• New quality metrics, mostly based on PSNR but subjective quality
assessments/metrics increasing
https://bitmovin.com/2017-video-developer-report/
I. D.D. Curcio, et al. 2017. Bandwidth Reduction of Omnidirectional Viewport-Dependent Video
Streaming via Subjective Quality Assessment. Proc. AltMM'17.
https://doi.org/10.1145/3132361.3132364
A. Zare, et al. 2016. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual
Reality Applications. Proc. ACM MM'16. http://dx.doi.org/10.1145/2964284.2967292
IEEE ICME Tutorial - July 2018 130
Quality Metrics: S-PSNR
M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015
IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 31–36.
DOI:http://dx.doi.org/10.1109/ISMAR.2015.12
IEEE ICME Tutorial - July 2018 131
Quality Metrics: V-PSNR
M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015
IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 31–36.
DOI:http://dx.doi.org/10.1109/ISMAR.2015.12
IEEE ICME Tutorial - July 2018 132
Dataset
• Segment length / intra period
– 1s (tiled content) vs 1, 2, 4s (monolithic content)
• Tiling pattern (columns × rows): 1×1, (i.e., tiles monolithic), 3×2, 5×3, 6×4, and 8×5
• Resolution: 1920×960, 3840×1920 and 7680×3840
• Map projection: equirectangular format
• Quantization parameter: QP={22,27,32,37,42}
• Head motion recordings for V-PSNR evaluation
IEEE ICME Tutorial - July 2018 133
Bitrate Overhead due to Tiling
500 1000 1500 2000 2500
384042444648
Bitrate [kbps]
Y−PSNR[dB]
●
●
●
●
●
●
● 1x1 tiles
3x2 tiles
5x3 tiles
6x4 tiles
8x5 tiles
Tile Overhead for Resolution: 1920x960
Sequence: AssassinsCreed
IEEE ICME Tutorial - July 2018 134
Bitrate Overhead due to Tiling
2000 4000 6000 8000
30323436384042
Bitrate [kbps]
Y−PSNR[dB]
●
●
●
●
●
●
● 1x1 tiles
3x2 tiles
5x3 tiles
6x4 tiles
8x5 tiles
Tile Overhead for Resolution: 1920x960
Sequence: ExploreTheWorld
IEEE ICME Tutorial - July 2018 135
Bandwidth Requirements
IEEE ICME Tutorial - July 2018 136
Adaptive Streaming Issues for (Tiled) 360° Video
• Increasing number of segment/tile requests
– HTTP/2 server push, query parameters, proprietary protocols
– Additional functionality at server – breaks fundamental HAS requirements
• Low latency streaming (see also next slide)
– Reducing segment size impacts coding efficiency (1s vs. 4s)
– CMAF chunks + other enhancements to enable sub-second latency
– Remember: live internet video will grow 15-fold from 2016 to 2021
• Viewport prediction
– Allows prefetching (caching) but cannot predict to much into future (1s)
– Impact on segment size but situation will get better the more data is available –
machine learning/AI will help
• QoE (see also following slide)
– Still in its infancy but situation much better than one year ago
– Requires datasets, subjective studies, quality models, metrics
M. Xu, et al., "A subjective visual quality assessment method of
panoramic videos," Proc. ICME’17.
https://doi.org/10.1109/ICME.2017.8019351
R. Schatz, et al., "Towards subjective quality of experience
assessment for omnidirectional video streaming," Proc.
QoMEX’17, https://doi.org/10.1109/QoMEX.2017.7965657
Y. Rai, at al. 2017. A Dataset of Head and Eye Movements for 360
Degree Images. Proc. ACM MMSys’17.
https://doi.org/10.1145/3083187.3083218
S. Petrangeli, et al. 2017. An HTTP/2-Based Adaptive Streaming
Framework for 360° Virtual Reality Videos. Proc. ACM MM'17.
https://doi.org/10.1145/3123266.3123453
N. Bouzakaria, et al., "Overhead and performance of low latency
live streaming using MPEG-DASH," Proc. IISA’14.
https://doi.org/10.1109/IISA.2014.6878732
C.-L. Fan, et al. 2017. Fixation Prediction for 360° Video Streaming in Head-
Mounted Virtual Reality. Proc. NOSSDAV'17.
https://doi.org/10.1145/3083165.3083180
IEEE ICME Tutorial - July 2018 137
Lag Prevents Immersion and Causes Discomfort
Minimizing Motion-to Photon Latency is Important
Low latency Noticeable latency
IEEE ICME Tutorial - July 2018 138
Motivation
QoE of Omnidirectional Video
• QoE of Omnidirectional Video (OV, alt.: 360º Movie) Streaming consumed via HMD
– Immersive panorama video (or image) surrounding the user
– User can turn head to changes gaze direction
• OV streaming = content class with
dramatically increasing popularity
– Cisco VNI: globally, virtual–reality traffic will
increase 61-fold between 2015 and 2020
Immersive
Media
VR
Responsive
VR
OV
MR AR
R. Schatz, et al., "Towards subjective quality of experience
assessment for omnidirectional video streaming," Proc.
QoMEX’17, https://doi.org/10.1109/QoMEX.2017.7965657
IEEE ICME Tutorial - July 2018 139
Motivation
QoE of Omnidirectional Video
• Gaps in current state of the art
– Lots of knowledge on 2D & 3D Video quality
– Lots of knowledge on human factors in responsive AR/VR
– BUT: lack of quantitative models and subj. testing methods for OV streaming
Particularly as regards quality perception and the
Impact of stalling on OV QoE
QoE = m * c2
QoE = m * c2
?
IEEE ICME Tutorial - July 2018 140
IEEE ICME Tutorial - July 2018 141
Research Questions
RQ1: How does stalling impact the QoE of HMD-based omnidirectional video
(OV)?
RQ2: What are QoE impact differences between HMD-based OV and
traditional TV-based video consumption with respect to stalling?
RQ3: What is the impact of different content motion levels on OV stalling
perception?
IEEE ICME Tutorial - July 2018 142
Method: Subjective Lab Study
• Test task: watch short video clips (duration 60s) impaired by stalling
• Two Devices / Watching Modes:
– 2D video on large TV screen (65”, Full HD)
– Stereoscopic OV on an Oculus Rift DK2 HMD (with headphones and 3-axis orientation sensor)
vs
IEEE ICME Tutorial - July 2018 143
Test Subjects
• Recruited from Vienna City region, received monetary incentives
• Recruited: 27 participants, but …
… removed 5 subjects due to vision problems, lack of HMD fit and cybersickness
à N=22
– 12 male, 10 female
– 4 had some limited VR experience
– Age: mean 37, median 31.5, min 22, max 65 years
IEEE ICME Tutorial - July 2018 144
Test Content (SRCs)
• 2D / TV
– Sintel movie (© Blender Foundation)
– One 60s clip
• OV / HMD
– Vulkane – Terra X
(© ZDF)
– Two 60s clips
extracted
• Both feature:
– Animation
– Storyline
Test Setup
IEEE ICME Tutorial - July 2018 145
Impairments (HRCs) and Test Conditions (PVSs)
• PVS:
– 2D/TV: 6 HRCs (Sintel movie) (+ 1 Training condition)
– OV/HMD: 6 HRCs x 2 SRCs (static camera, moving camera) (1 Training condition)
• Post-stimulus electronic questionnaire with 4/7 (TV/HMD) rating questions related to
quality, immersion, etc.
IEEE ICME Tutorial - July 2018 146
OV move: Four users could not complete the OV move conditions, i.e. N=18
How Annoying were Interruptions (If Any) of the Video Playback?
Results: Impact of Stalling on User Annoyance
Imperceptible
Very annoying
IEEE ICME Tutorial - July 2018 147
I Was Completely Captivated by the Virtual Environment
Impact of Stalling on Perceived Involvement (IPQ INV4*)
Fully agree
Full disagree
* from Igroup Presence Questionnaire (IPQ)
IEEE ICME Tutorial - July 2018 148
Results / Findings
• RQ1 (stalling QoE impact):
– Strong impact of stalling number and duration (similar to previous results)
– Existing stalling models can be reused
• RQ2 (differences 2D/OV):
– No significant differences between TV/2D and HMD/OV (à unexpected)
• RQ3 (impact of OV content motion):
– No significant differences between moving and static camera, just some trends (à unexpected)
• In addition:
– Splits yield some differences in rating variance and stalling detection
Influence on variance: sex, prior vr experience
Influence on stalling detection: commented on HMD resolution
IEEE ICME Tutorial - July 2018 149
AdViSE
• Adaptive Media Content
[DASH, HLS, CMAF]
• Players/Algorithms
• Network Parameters
Impaired
Media Sequences
Generate
Impaired Media
Sequences
Templates [Startup
Delay, Stalling, …]
WESP
QoE Evaluation
Parameters
[Questionnaire,
Methodology,
Crowdsourcing
Platform, …]
QoS/QoE
Metrics
Subjective
Results + Other
Data
Reports
Analysis
AdViSE: Anatoliy Zabrovskiy, Evgeny Kuzmin, Evgeny Petrov, Christian Timmerer, and Christopher Mueller. 2017. AdViSE: Adaptive Video Streaming
Evaluation Framework for the Automated Testing of Media Players. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM,
New York, NY, USA, 217-220. DOI: https://doi.org/10.1145/3083187.3083221
WESP: Benjamin Rainer, Markus Waltl, Christian Timmerer, A Web based Subjective Evaluation Platform, In Proceedings of the 5th International Workshop
on Quality of Multimedia Experience (QoMEX'13) (Christian Timmerer, Patrick Le Callet, Martin Varela, Stefan Winkler, Tiago H Falk, eds.), IEEE, Los
Alamitos, CA, USA, pp. 24-25, 2013. https://doi.org/10.1109/QoMEX.2013.6603196
Objective and Subjective Evaluation Platform
How to Evaluate HAS Systems?
①
②
③④
⑤
Log of
Segment
Requests
IEEE ICME Tutorial - July 2018 150
AdViSE: Adaptive Video Streaming Evaluation
• Scalable, end-to-end HAS evaluation
through emulation w/ a plethora of
– content configurations
– players/algorithms (including for player
competition)
– network parameters/traces
• Real content and network settings with
real dynamic, adaptive streaming
including rendering!
• Collection of various metrics from
players: API or directly from the
algorithms/HTML5
• Derived metrics and utilize QoE models
proposed in the literature
• Segment request log to generate
impaired media sequence as perceived
by end users for subjective quality
testing
IEEE ICME Tutorial - July 2018 151
WESP: Web-Based Subjective Evaluation Platform
• Subjective quality assessments
(SQA) are vital tool, reliable
results, but cost-intensive
• Utilize crowdsourcing to reduce
costs
• WESP
– Enables easy and simple
configuration of SQAs including
possible integration of third-party
tools for online surveys
– Provides means to conduct SQAs
using the existing crowdsourcing
platforms taking into account best
practice
– Allows for the analysis of the results
IEEE ICME Tutorial - July 2018 152
C. Timmerer, A. Zabrovskiy, and Ali C. Begen, Automated Objective and
Subjective Evaluation of HTTP Adaptive Streaming Systems, Proceedings
of the 1st IEEE International Conference on Multimedia Information
Processing and Retrieval (MIPR), Miami, FL, USA, April 2018.
Example Evaluation Results
• Test sequence encoded 15 different representation (Amazon Prime configuration:
400x224@100Kbps – 1920x1080@15Mbps) with 4s segment length
• Bandwidth trajectory based on prior work
proposed in literature; network delay 70 ms
IEEE ICME Tutorial - July 2018 153
Bandwidth Index vs. QoE
• Bandwidth index
– Average bitrate
– Efficiency
– Stability
• QoE model
– Startup delay
– Stalls
C. Timmerer, A. Zabrovskiy, E. Kuzmin, E. Petrov, Quality of experience of commercially deployed adaptive media players, In 2017 21st
Conference of Open Innovations Association (FRUCT) (Sergey Balandin, ed.), pp. 330-335, 2017.
https://doi.org/10.23919/FRUCT.2017.8250200
IEEE ICME Tutorial - July 2018 154
Part I: Streaming
The Developing MPEG-OMAF and MPEG-I Standards
Standardization Overview
C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in
IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017.
https://doi.org/10.1109/MCOMSTD.2017.1700038
Data Representations and Formats
JPEG MPEG IEEE
System Standards and APIs
VR-IF 3GPP DVB Khronos W3C
Quality of Experience
QUALINET ITU-T VQEG
Guidelines
DASH-IF
CTASVA SMPTE ETSI
https://multimediacommunication.blogspot.com/2017/04/vr360-streaming-standardization-related.html
IETF/IRTF
IEEE ICME Tutorial - July 2018 156
Standardization Overview
C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in
IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017.
https://doi.org/10.1109/MCOMSTD.2017.1700038
Data Representations and Formats
JPEG MPEG IEEE
System Standards and APIs
VR-IF 3GPP DVB Khronos W3C
Quality of Experience
QUALINET ITU-T VQEG
Guidelines
DASH-IF
CTASVA SMPTE ETSI
https://multimediacommunication.blogspot.com/2017/04/vr360-streaming-standardization-related.html
IETF/IRTF
IEEE ICME Tutorial - July 2018 157
Coded Representation of Immersive Media
MPEG-I (ISO/IEC 23090)
• Part 1: Architectures
• Part 2: Omnidirectional media format (OMAF)
• Part 3: Versatile Video Coding (VVC)
• Part 4: Immersive audio
• Part 5: Point cloud compression
• Part 6: Immersive media metrics
• Part 7: Immersive media metadata
• Part 8: Network-based media processing (NBMP)
IEEE ICME Tutorial - July 2018 158
Scope
Omnidirectional Media Format (OMAF)
• 360° video, images, audio and associated timed text – 3DoF only
• Specifies
– A coordinate system
• Consists of a unit sphere and the x (back-to-front), the y (lateral, side-to-side), and the z (vertical, up) axes
– Projection and rectangular region-wise packing methods
• Used for conversion of a spherical video/image into a two-dimensional rectangular video/image
– The sphere signal is the result of stitching of video signals captured by multiple cameras
– A special case: fisheye video
– Storage of omnidirectional media and the associated metadata using ISO-BMFF
– Encapsulation, signaling and streaming of omnidirectional media in DASH and MMT
– Media profiles and presentation profiles
• Provide interoperable and conformance points for media codecs as well as media coding and encapsulation
configurations
• Provides some informative viewport-dependent 360° video processing approaches
IEEE ICME Tutorial - July 2018 159
A: Real-world scene
B: Multiple-sensors-captured video or audio
D/D’: Projected/packed video
E/E’: Coded video or audio bitstream
F/F’: ISOBMFF file/segment
OMAF player
Acquisition Audio encoding
Video encoding
Image encoding
File/segment
encapsulation
Delivery
File/segment
decapsulation
Audio decoding
Video decoding
Image decoding
Image rendering
Audio rendering
Loudspeakers/
headphones
Display
Head/eye
tracking
Fileplayback
Orientation/
viewport metadata
Orientation/viewport metadata
Metadata
Metadata
A Ba
Bi D
Ea
Ei
Ev Fs
F
F’
F’s
E’a
E’v
E’iD’
B’a
A’i
A’a
Image stitching,
rotation, projection,
and region-wise
packing
https://mpeg.chiariglione.org/standards/mpeg-i/omnidirectional-media-format
R. Skupin, et al., "Standardization Status of 360 degree Video Coding and Delivery,” Proc. IEEE VCIP’17
OMAFArchitecture
DASH/OMAF System Diagram
Multi-
camera
capture
Stitch Project
HEVC
encode
OMAF
composing
DASH server
DASH
client
HEVC
Decode Renderer
Head mounted
device (HMD)
DASH
encapsulation
Head and eye tracking telemetry
Request for
tiles within
FoV window
Projection mapping metadata
Projection mapping metadata
Authoring
CDN
VR player
IEEE ICME Tutorial - July 2018 161
MPEG-
1,2,4,H,I
MPEG-7
MPEG-21
MPEG-A
MPEG-
B,C,D,
DASH
MPEG-
E,M
MPEG-
U,V
Compression of
video, audio
and 3DG
Description of video,
audio and multimedia
for content search
Technologies for content
e-commerce
Multimedia application
formats (combinations
of content formats)
Systems, video,
audio and transport
Multimedia platform
technologies
Device and application
interfaces
https://mpeg.chiariglione.org/meetings/123
MPEG’s Areas of Activity
IEEE ICME Tutorial - July 2018 162
1990 2000 20101995 2005 2015
MP3
MPEG-2
Video &
Systems AVC HEVC
MP4 FF MMTDASH
AAC
Internet
Audio
Digital
Television
Music
Distribution
Media Storage and
Streaming
UHD & Immersive
Services
New Forms of
Digital
Television
Unified Media
Streaming
OFF
HD DistributionMobile Video
Custom Fonts on
Web & DigiPub
MPEG-4
Video
CMAF
3D Audio
2020
MajorMPEG
Standards
6DoF Application Format?
Genome Compression v2
PCC Extensions?
OMAF v2
2018 20202017 2019 2021
IoMT
Media Orchestration
Descriptors for Video Analysis (CDVA)
6DoF Audio
Point Cloud Compression
OMAF v1
Genome Compression
Network-based Media Processing
Coding
2022
Scene Description for Immersive Media
Versatile Video Coding
2023
MIAF
Systems
and ToolsWeb Resource Tracks
Dense Representation of Light Field Video
3DoF+ Video
MPEGRoadmap
Part II: Communications over 5G
5G Fundamentals: Core Network, Radio Access Network and QoS
5G fundamentals: Core
Network, Radio Access
Network and QoS
ICME 2018 Tutorial
Liangping Ma
July 23, 2018
© 2018 InterDigital, Inc. All rights reserved. 166
167
Who is InterDigital?
Invention, Collaboration, Contribution
© 2018 InterDigital, Inc. All rights reserved.
Berlin
Montreal, QC (R&D)
London (R&D)
Seoul
Buffalo, NY
Melville, NY (R&D)
Conshohocken, PA (R&D)
Wilmington, DE (HQ)
San Diego, CA (R&D)
• Four decades of discovery
and innovation in wireless
• Pioneer in digital wireless
technologies
• Key contributions to global
wireless standards
• Inventing solutions for more
efficient broadband networks
• ~190 engineers (~80% of
whom hold advanced
degrees)
168
What is 5G?
© 2018 InterDigital, Inc. All rights reserved.
169
5G Use Cases and Key Requirements
Enhanced Mobile Broadband (eMBB)
§ Peak data rates: 20 Gbps (DL) and 10 Gbps (UL)
§ Peak spectral efficiency:
30 bps/Hz (DL) and 15 bps/Hz (UL)
§ 4 ms user plane latency
§ Indoor/hotspot and enhanced wide-area
coverage
Massive Machine Type
Communications (mMTC)
§ Low data rates
(1 to 100 kbps)
§ High device density
(up to 1,000,000 /km2)
§ Latency: Seconds to hours
§ Low power: Up to 15 years
battery life
Ultra-Reliable and Low Latency
Communications (URLLC)
§ Low to medium data rates
(50 kbps to 10 Mbps)
§ 0.5 ms user plane latency
§ 99.999% reliability and availability
within 1 ms
§ High mobility
Key challenge for 5G design: support for different services having diverging requirements
Source: ITU-R SG5 WP-5D
© 2018 InterDigital, Inc. All rights reserved.
170
What Differentiates 5G from Previous Generations?
Uses Cases
& Services
Voice + SMS
Voice +
Small Data
Mobile
Broadband
Enhanced Mobile Broadband
Massive Machine Type Communications
Ultra Reliable and Low Latency Communications
Spectrum
200 KHz
Channels
Below 2 GHz
5 MHz
Channels
Below 3.6 GHz
Up to 20 MHz
Channels
Below 3.8 GHz
Up to 1 GHz Channels
Below 100 GHz
Radio
Technology
GSM/GPRS
(Single RAT )
UMTS/HSPA
(Single RAT)
LTE/LTE-A
(Single RAT)
Multiple Radio Access Technologies
Integrated in a 5G Network:
5G New Radio, LTE Advanced Pro, NB-IoT, Wi-Fi, …
Network
Topology
Macro Cells
Macro and
Small Cells
Heterogeneous
Macro and
Small Cells
Challenging Traditional Cell and Network Concepts:
Small Cells, Mobile Edge Computing,
Network Slicing, …
2G 4G 5G3G
© 2018 InterDigital, Inc. All rights reserved.
2021
171
5G 3GPP Standardization Timeline
2017 2018 2019 2020
Rel-17 Work Items
LTE Release 17LTE Release 15R14
NR SI
Full NR RAN stand-
alone and CT
Specifications
approved
Rel-16 specification
approved with some
additional enhancements
NR R15 first drop
specification completed
(NR non-standalone
and CN standalone)
Phased approach enables early commercial deployment of Phase 1 in 2019 and Phase 2 in 2022+
IMT-2020 Proposal
Submission
NR Rel-15
5G LTE + NB-IoT
(required for
mMTC use cases
for IMT-2020)
5G New Radio
(NR)
…
Allows for very early commercial implementation – No
more hardware changes envisioned for R-15
Rel-16 SIs
(RAN/SA)
Rel-17 Study Items
LTE Release 16
Additional Rel-16 SIs
approved
Rel-16 Work Items (R16)
3rd drop
© 2018 InterDigital, Inc. All rights reserved.
Focus on completion of the protocol design
§ NAS protocol between the UE and the CN
§ Internal Core Network interfaces based on the
service based architecture (SBA)
§ Interconnection of the Core Network with
external networks
First set of NR specifications completed
§ Ready for Hardware Implementation and for first
phase of deployment - Non-Standalone NR only
(i.e. connected and controlled by LTE)
§ RAN1 PHY and RAN2 User Plane completed
for stand-alone NR
§ Support for Enhanced Mobile Broadband use
cases only and low latency
172
5G New Radio Status – Rel-15
Core
Network
and
Non Access
Stratum
Radio
Access
Network
Next Gen Core(NGC) architecture completed
§ New fully flexible architecture based on Network
Slicing concept and further separation of control
and user planes
§ Key functions: Session Management, Mobility
Management, QoS and access agnostic CN
Focus on completion stand-alone NR
§ Support for stand-alone NR connected directly
to a 5G CN (RAN2/RAN3/RAN4)
§ Add/complete additional enhancements to
support ultra-reliable and low latency use cases
(URLLC) (RAN1/RAN2)
December 2017 June 2018
© 2018 InterDigital, Inc. All rights reserved.
173
5G Design Principles
© 2018 InterDigital, Inc. All rights reserved.
174
5G NR Design Principles
Architecture & Network
• Flexible, modular and scalable
architecture
• Support high level of
programmability and
automation in network
• Built in support of network
customization via slicing
• Easy integration of new services
5G is designed to be flexible and adaptable to meet wide range of current and future requirements
Radio Access Network
• Minimize always-on signaling
• Flexible frame structure and
numerology
• Support wide range of frequency
bands
• Built-in support for beamforming
• Easy integration of future
features
© 2018 InterDigital, Inc. All rights reserved.
175
5G Technologies
© 2018 InterDigital, Inc. All rights reserved.
176
3GPP 5G Deployment Scenarios
5G
RAN
LTE/4G
Network
5G
Network
LTE
Base Station
New Radio RAN
Internet
Non
Standalone
(NSA) 5G 5G
New Radio
4G EPC
5G Core Network
Slice #3 (URLLC)
Slice #2 (IoT)
Slice #1 (eMBB)
SGW
MME
PGW
HSS
AMF
Non-Standalone (NSA) 5G enables large-scale trials and deployments starting in 2019
3 Main Deployment Scenarios
§ NSA deployment - utilizes
LTE radio and 4G EPC as
an anchor and NR Radio
as hotspot
§ Allows earlier
implementability
§ Stand-alone NR – 5G RAN
connecting to 5G CN only
§ LTE radio connected to 5G
CN network
5G CN designed to interwork
with the current EPC network
to enable seamless migration
for the operators
© 2018 InterDigital, Inc. All rights reserved.
177
Architecture options and
specification timelines
© 2018 InterDigital, Inc. All rights reserved.
EPC
NextGen
Core
LTE NR
3
Non-standalone
Drop 1: Dec. 2017
NR
2
LTE
5
NextGen
Core
NextGen
Core
Standalone NR
Drop 2: June 2018
EPC
NextGen
Core
LTE NR
4
EPC
NextGen
Core
LTE NR
7
LTE connected to 5G CN
Drop 3: Dec 2018 – NR +
178
Control and User Plane Separation
© 2018 InterDigital, Inc. All rights reserved.
Control and User plane separation modularizes the 5G Core Network Design
§ Reduces Latency on application service
§ By relocating UPFs which are closer to the RAN;
§ Or by selecting UPFs more appropriate for the
intended UE usage type without increasing the
number of control plane nodes
§ Supports increase of Data Traffic, by enabling
to add user plane nodes without changing
the number of CP resources in the network
§ Independent evolution of the CP and UP functions
§ Enables SDN routing to deliver user plane
data more efficiently
5G Core Network Architecture
179
Service Based Architecture (SBA)
§ 5G CN has been revolutionized with Service Based
interfaces for core network interactions
§ CN interfaces have traditionally been Point-to-Point
(P2P) e.g. MAP, Diameter
§ Network Functions (NFs) in 5G CN offer their
services to other NFs via APIs based on web-based
technology (e.g. HTTP).
§ Makes deployment easy as libraries, development
tools, and other components are broadly available.
§ Flexible deployments of new services: possible to
connect to other components without new specific
interfaces
§ Network Repository Function (NRF) allows NFs to
discover the services of other NFs.
© 2018 InterDigital, Inc. All rights reserved.
SBA shifts the paradigm from telecom-style protocol interfaces to web-based APIs
UDMNSSF PCFNRF
AUSF
AF
SMFAMF
5G CN Service Based Control Plane
Network Slice
• Logical end-to-end network
• Tailored based on service it offers
(e.g.eMBB, IoT, URLCCC etc.)
Slicing offers customized network resources for different verticals
© 2018 InterDigital, Inc. All rights reserved. 180
Network Slicing
Benefits of slicing
• Flexible and modular network deployment
• Isolation between dedicated network resources
• New products and services can be brought to market
rapidly and adapted to fast-changing demands
Virtual (Cloud) RAN Enablers
ØStandards efforts to enable virtualization in multi-
vendor environments
• Central Unit / Distributed Unit split
• CU/DU split below the PDCP
• CU performs user management and user data processing
• DU manages the radio and cell
• Control Plane/User plane split
• Central unit further split into virtual entities:
• User data processing and user/mobility control
• Support of lower layer split – no conclusion
• Some of the radio control function performed by the
central unit
• Challenging due to ties with implementation
Virtualization gives operators a flexible network design reducing cost
181© 2018 InterDigital, Inc. All rights reserved.
CU
DU
Control Data
gNB
DU
182
Network Function Virtualization (NFV)
© 2018 InterDigital, Inc. All rights reserved.
Sources: Network Functions Virtualisation -- White Paper on NFV priorities for 5G, ETSI, Feb. 2017.
Network Functions Virtualisation – Introductory White Paper, ETSI, Oct. 2012.
NFV is not limited to 5G
Use cases for 5G:
- Virtualization of the core network
- Virtualization of the RAN
Decouple network functions from hardware
183
Numerology
A numerology is defined by sub-carrier spacing and CP overhead
ØOFDM waveform in both UL &DL
• Optimized CP-OFDM with high spectrum utilization
• Spectrum confinement techniques such as filtering and windowing
• Simplifies design for D2D, Massive MIMO reciprocity & IAB.
• DFT-s-FDM for coverage-limited scenarios in UL
ØScalable Numerology
• Subcarrier Spacing (SCS) scales with the BW (15#$% ∗ 2(
)
• To support diverse spectrum bands and services
• To avoid processing complexity associated with a large FFT size
• Higher SCS -> Higher bands -> Shorter OFDM symbols
• To address RF impairments in mmWave bands (e.g., Phase Noise),
enable analog beamforming in mmWave bands and, lower latency
• Lower SCS -> Larger cyclic prefix
• More robust to the channel delay spread (e.g., Broadcast Service)
• To support narrow band transmissions for mMTC services
SCS [kHz] 15 30 60 120 240
OFDM Duration [µs] 66.67 33.33 16.67 8.33 4.17
CP Duration [µs] 4.69 2.34 1.17 0.58 0.29
Total Symbol [µs] 71.36 35.67 17.84 8.91 4.46
Slot Duration [µs] 1000 500 250 125 62.5
Band [GHz] < 6 < 6 < 6 < > 6 > 6
© 2018 InterDigital, Inc. All rights reserved.
15 kHz
30 kHz
60 kHz
120 kHz
50 MHz
400 MHz
Max BW
100 MHz
200 MHz
184
Frame Structure
Flexible frame structure adapting to various bands, duplexing mode and service requirements
UL
ULCtrl
DL
UL
ULCtrl
DL
ULCtrl
Downlink-Centric Slot
ØFlexible Frame Structure
• Flexible TTI Duration
§ Scalable TTI enables efficient service multiplexing
o Symbols across different SCSs are aligned at the symbol boundaries
§ Shorter TTI to support low latency application (e.g., URLLC)
o User can also be schedule on a fraction of a slot (i.e., Mini-slot)
§ Longer TTI to achieve higher spectral efficiency (e.g., eMBB)
o User can be scheduled on multiple-slots
• Flexible Scheduling
§ Both dynamic TDD and FDD are supported
§ OFDM symbols in a slot can be DL, UL or flexible
§ Flexible symbols can be used as gap or for forward compatibility
Uplink-Centric Slot
1
30 kHz
120 kHz
0
0
20 12 13
1 2 12 13
1 2 60 kHz
15 kHz
1 ms slot with 14 OFDM symbols
500 µs
250 µs
125 µs
OFDM
symbol
Mini-Slot
Slot with load balancing
© 2018 InterDigital, Inc. All rights reserved.
Dynamic TDD
185
Bandwidth Adaptation
Adapting UE’s receive and transmit bandwidth irrespective of gNB’s total supported bandwidth
ØFlexible Bandwidth Allocation
• NR supports a maximum BW of 400MHz per Component Carrier
(CC) with up to 16 CC in Carrier Aggregation (CA)
• Supporting bandwidth smaller than the CC bandwidth enabled
by configuring a Bandwidth Part (BWP)
• BWP Benefits include:
§ Enables UE power savings
§ Supports UE with lower maximum bandwidth capability
§ Single BPW is active at a time
• Increases scheduling flexibility
• Allow service multiplexing in frequency domain for use-
cases with different numerology BWP1: 40 MHz and subcarrier spacing of 15 kHz
BWP2: 10 MHz and subcarrier spacing of 15 kHz
BWP3: 10 MHz and subcarrier spacing of 60 kHz
© 2018 InterDigital, Inc. All rights reserved.
Frequency
Time
BWP1 BWP2
BWP3
BWP2 BWP1
Carrier
186
Antenna
© 2018 InterDigital, Inc. All rights reserved.
• More bandwidth leads to higher throughput
• Need to go to higher bands
• But higher bands have larger path loss than lower bands
Free space path loss – Friis equation !" = !$%$ %"
&
4()*
+
Sources: W. L. Stutzman and G. A. Thiele, Antenna Theory and Design, Wiley, 2013.
Example: When f increases from 2.8GHz to 28 GHz, the path loss increases by 20 dB.
frequencyReceived power
R
Pt Pr
Why? The maximum effective area (effective aperture) or “collecting area” of an antenna is related to the frequency
e.g. for an ideal (short) dipole antenna, maximum effective area =
,
-./0
Beamforming to close the link at higher frequencies
187
Hybrid Beamforming
© 2018 InterDigital, Inc. All rights reserved.
• Analog beamforming:
phase shifter
• Digital beamforming:
precoding matrix BB
RF
RF
BB
RF
RF! = # $ + &
where $ = +, = +-.+//,
s
+-.
+//
Phase shifter
Tradeoff between cost and flexibility
Massive MIMO
Ø Conjugate Beamforming
• Coherent combining at receiver
Ø For TDD, no performance loss
when the number of antennas is
much larger than the number of
users (M/K → ∞)
• Leverage channel reciprocity
• Result of Law of Large Numbers
Ø Pilot contamination
• In mobile scenarios the channel
changes over time
• Contamination occurs when two
users use the same pilot
Original Work by Thomas L. Marzetta (2006)
© 2018 InterDigital, Inc. All rights reserved.
k
188
Massive MIMO
Ø Enhanced CSI Feedback
• Multi-resolution CSI
• Low spatial resolution with a linear precoder targeting SU-MIMO
operation (Type-I CSI) à low latency and overhead
• High spatial resolution with a linear combination precoder targeting
MU-MIMO operation (Type-II CSI) à better performance
• Flexible CSI reporting timing
• CSI reporting timing can be dynamically indicated to optimize
latency based on CSI computational complexity
• Hybrid beamforming support
• Joint CSI reporting of beam index and precoding matrix index (PMI)
Ø Enhanced UL Transmission
• Non-Codebook based UL transmission
• Introduced in NR to enable UL reciprocity functionality in TDD
• UE determines its PUSCH precoder with some gNB assistance
Work under 3GPP
: DL control to trigger aperiodic CSI reporting
: UL channel for aperiodic CSI reporting
Low latency CSI High latency CSI
Slot #n Slot #n+1 Slot #n+2 Slot #n+3 Slot #n+4
• gNB can dynamically indicate the CSI reporting timing
© 2018 InterDigital, Inc. All rights reserved.
1) CSI-RS
3) DL Control
2) Sounding Signal
4) Precoded UL Data
Precoder
Calculation
189
190
Beam Management
Ø Beam management for high frequency
• Support of high frequency spectrums (up to 52.6GHz) in NR
which is different from LTE
• Beam management is used to maintain optimal Tx/Rx beam
pairing for control and data channels
Ø Beamformed Initial Access
• Fundamental difference with LTE RACH lies on beamforming
• Beam association between RACH and SS/PBCH blocks
• UE selects an associated UL RACH resource of a selected DL
SS block based on L1-RSRP measurements
Management of analog beams at Tx/Rx is newly introduced in NR
© 2018 InterDigital, Inc. All rights reserved.
191
Beam Failure Recovery
Ø Recovery from beam failure
• Beam-based system is susceptible to blockage, UE rotation,
and mobility
• Link fails when Tx-Rx beams for DL control channel are
mismatched
• RLF has been used in LTE when link fails but that requires a restart
from initial access procedure à high latency
• Beam recovery recovers beams for DL control channel
• UE monitors quality of DL control channel beam and indicates new
candidate beam using a dedicated PRACH resource when link failed
• Beam recovery is used to recover the link before RLF à low latency
A new mechanism introduced to handle frequent Tx/Rx beam mismatch
Beam failure due to blockage
New candidate beam
PRACHfor
beam1
PRACHfor
beam2
PRACHfor
beam3
UE sends PRACH
resource associated
with new candidate
beam to trigger
beam recovery
© 2018 InterDigital, Inc. All rights reserved.
Ultra-Reliable and Low-Latency Communications
NR introduced several mechanisms to enable URLLC
192© 2018 InterDigital, Inc. All rights reserved.
Low Latency
DL Data
ULCtrl
Gap
HARQ-ACK
Slot
• Self-contained slot
Transmitting HARQ-ACK in the self-
contained slot immediately after
receiving the DL data
• Mini-Slot
Slot
Mini-Slot
Smaller time units
for data scheduling
• Scalable subcarrier spacing
• Grant Free UL transmission
• Very low periodicity SR
• Pre-emption
High Reliability
• Diversity via duplication
f1
f2
Same PDCP PDU on
two carriers
Carrier-Aggregation
LTE NR
Same PDCP PDU over
different RAT (DC)
Dual-Connectivity (EN-DC)
• Control channel duplication
• Polar coding for control information
• Optimized CQI/MCS tables
193
Channel Coding: Data Channel
Ø Enhanced LDPC Codes for eMBB Data Channel
• Advantages of LDPC codes
• Support high data rates (NR peak data rate: 20 Gbps DL/10 Gbps UL)
• High area and energy efficiency and good BLER performance
• Quasi-cyclic LDPC code enables high degree of parallelism
• Parity-check matrix is lifted from small base graph
• Two base graphs
• Optimized for different regions of code rates and code block sizes
• Length and rate flexibility
• Shortening and lifting to achieve 1-bit granularity on code block size
• Circular buffer with puncturing and repetition to achieve codeword
length flexibility;
• Support for both CC-HARQ and IR-HARQ
ØFuture Enhancements
• Potential modifications for URLLC data channel to address the error floor
and achieve high reliability
Channel coding was entirely redesigned for NR
LTE: Turbo NR: LDPC
Throughput Medium-low High
Area efficiency Low High
Energy efficiency Low High
Implementation complexity comparison (R1-167567)
Coverage regions of 2 LDPC base graphs
© 2018 InterDigital, Inc. All rights reserved.
R=1/4
R=2/3
R=0.95
308 3840 Code
block size
Code
rate
8448
Base Graph 1
Base Graph 2
Channel Coding: LDPC
• Generator matrix
• Parity check matrix
© 2018 InterDigital, Inc. All rights reserved. 194
A primer on linear block codes and Tanner graph representation
! =
#$$ #$% … #$'
⋮ ⋮ ⋮ ⋮
#)$ #)% … #)'
=
*$
⋮
*)
*$, *%, …, *) are linearly independent vectors of length ,
k information bits - = (/$, /%, …,/)) mapped to a ,-bit codeword:
1 = - ! = /$*$ + /%*%+ …+/)*)
3: (, − 6)×, matrix, each row is orthogonal to rows of !
Any codeword 1 satisEies: 13F = 0
• Tanner graph
3 =
1 0 1 0
0 1 1 1
x1+ x3 = 0
x2 + x3 + x4 = 0
x1
x2
x3
x4
Channel Coding: LDPC
• Binary Erasure Channel
© 2018 InterDigital, Inc. All rights reserved. 195
An example of message passing decoding
x1+ x3 = 0
x2 + x3 + x4 = 0
1
x2=?
x3=?
0
1
0
1
1
1
Channel Coding: LDPC
• Parity check matrix H
• Base graph (matrix of 0’s and 1’s)
• Replace 1 with a cyclically shifted ZxZ identify matrix
• Replace 0 with a ZxZ all-zero matrix
Example:
© 2018 InterDigital, Inc. All rights reserved. 196
1 1 0 1 1 0
0 1 1 0 1 0
1 0 1 0 0 1
3 2 N/A 2 1 N/A
N/A 3 1 N/A 0 N/A
1 N/A 3 N/A N/A 0
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
0 0 0 1
1 0 0 0
0 1 0 0
0 0 1 0
Shift 3 positions
0 0 0 1
1 0 0 0
0 1 0 0
0 0 1 0
0 0 1 0
0 0 0 1
1 0 0 0
0 1 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 1 0
0 0 0 1
1 0 0 0
0 1 0 0
0 1 0 0
0 0 1 0
0 0 0 1
1 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
First row of base graph results in the first 4 rows of H:
Source: 3GPP TR 38.802, Study on New Radio Access Technology -- Physical Layer Aspects V14.0.0 (2017-03)
This construction is sometimes called lifting.
Parity check matrix for high throughput
197
Channel Coding: Control Channel
Ø Polar Codes for Control Channel and Broadcast Channel
• Advantages of Polar codes
• Good BLER performance at small block sizes and no error floor
• Polar codes are used to encode DCI, UCI (>11 bits) and MIB
• A single nested polar sequence
• Short polar sequence is derived from long polar sequence
• Optimized Rate Matching (RM) scheme
• Select among three RM schemes: puncturing, shortening, repetition
• Unified circular buffer design for all three RM schemes
• Triangular channel interleaver (UCI only)
• Advanced polar code construction
• Distributed CRC for DCI and MIB to achieve early termination
• Parity check polar code for small size UCI for better performance
ØFuture Enhancements
• Extension of polar code for URLLC data channel
NR is the first commercial system adopting Polar codes
Unified circular buffer design for three RM schemes
Polar
Encoder
(Mother
code
length N)
Sub-
block
0
Sub-
block
1
Sub-
block
31
Sub-
block
0
Sub-
block
1
Sub-
block
31
Sub-block wise
interleaving
Circular
buffer
Shortening
Repetition
Puncturing
Selecting bits from the
unified circular buffer
Rate
matching
© 2018 InterDigital, Inc. All rights reserved.
Channel Coding: Polar code
• Invented by Erdal Arikan in 2007
• Based on the phenomena of channel polarization
© 2018 InterDigital, Inc. All rights reserved. 198
W
X Y
1. Binary Discrete Memoryless Channel (DMC)
2. Consider N copies of the DMC, and channel code X = GU
W
X1 Y1
W
X2 Y2
W
XN YN
...
G
U1
U2
UN
...
...
Information bits codeword ! "#, "%, … , "'; )#, )%, … , )'
= +
,-#
'
! ",; )#, )%, … , )' "#, … , ",.#
= +
,-#
'
!(",; )#, )%, … , )', "#, … , ",.#)
12,: ", → ()#, )%, … , )',"#, … , ",.#New channels
Transform scalar channels into equivalent vector channels
Channel Coding: Polar code
• N=8
© 2018 InterDigital, Inc. All rights reserved. 199
Transform scalar channels into equivalent vector channels (Example)
• N=2
• N=4
W
W
Y1
Y2
U1
U2
W
W
W
W
Y1
Y2
Y3
Y4
U1
U2
U3
U4
W
W
W
W
W
W
W
W
U1'
U2'
U3'
U4'
U1"
U2"
U3"
U4"
Y1
Y2
Y3
Y4
Y5
Y6
Y7
Y8
U1
U2
U3
U4
U5
U6
U7
U8
QoS Provisioning
• 4G: there are 9 pre-defined QoS Class Identifier (QCI) values
© 2018 InterDigital, Inc. All rights reserved. 200
By a combination of QCI, ARP, GBR/Non-GBR
Source: 3GPP TS 23.203, v11.2.0, “Policy and charging control architecture”, June, 2011.
QCI Resource
Type
Priority Packet Delay
Budget
Packet Error Loss
Rate
Example Services
1
GBR
2 100 ms 10-2 Conversational Voice
2 4 150 ms 10-3 Conversational Video (Live Streaming)
3 3 50 ms 10-3 Real Time Gaming
4 5 300 ms 10-6 Non-Conversational Video (Buffered Streaming)
5 1 100 ms 10-6 IMS Signalling
6
Non-GBR 6 300 ms 10-6
Video (Buffered Streaming)
TCP-based (e.g., www, e-mail, chat, ftp, p2p file
sharing, progressive video, etc.)
7
7 100 ms 10-3
Voice,
Video (Live Streaming)
Interactive Gaming
8
8 300 ms 10-6
Video (Buffered Streaming)
TCP-based (e.g., www, e-mail, chat, ftp, p2p file
9 9 sharing, progressive video, etc.)
GBR: Guaranteed Bit Rate
ARP: Allocation and Retention Priority
QoS Provisioning
• 5G: Added new QCI values
© 2018 InterDigital, Inc. All rights reserved. 201
More flexible QoS differentiation and more efficient signaling
Source: 3GPP TS 38.300, v15.3.0, “Policy and charging control architecture”, June, 2018.
QCI Resource Type Priority
Level
Packet Delay
Budget
(NOTE 13)
Packet Error
Loss
Rate (NOTE 2)
Example Services
65 GBR 0.7 75 ms
10-2
Mission Critical user plane Push To Talk voice (e.g.,
MCPTT)
67 1.5 100 ms 10-3 Mission Critical Video user plane
75 2.5 50 ms 10-2 V2X messages
69 Non-GBR 0.5 60 ms 10-6 Mission Critical delay sensitive signalling (e.g., MC-
PTT signalling, MC Video signalling)
70 5.5 200 ms 10-6 Mission Critical Data (e.g. example services are the
same as QCI 6/8/9)
79 6.5 50 ms 10-2 V2X messages
80 6.8 10 ms 10-6 Low latency eMBB applications (TCP/UDP-based);
Augmented Reality
QoS Provisioning
• 5G: Added new QCI values
© 2018 InterDigital, Inc. All rights reserved. 202
More flexible QoS differentiation and more efficient signaling
Source: 3GPP TS 38.300, v15.3.0, “Policy and charging control architecture”, June, 2018.
QCI Resource
Type
Priority
Level
Packet Delay
Budget
Packet Error
Loss
Rate
Maximum Burst
Size
Data Rate
Averaging
Window
Example Services
82 GBR 1.9 10 ms 10-4 255 bytes 2 s Discrete Automation
83 GBR 2.2 10 ms 10-4 1358 bytes 2 s Discrete Automation
84 GBR 2.4 30 ms 10-5 1358 bytes 2 s Intelligent Transport
Systems
85 GBR 2.1 5 ms 10-5 255 bytes 2 s
Electricity Distribution-
high voltage
QoS Provisioning
• 5G added SDAP (service data adaptation protocol) sublayer
• 5G introduced reflective QoS
• Derive uplink QoS from downlink QoS
© 2018 InterDigital, Inc. All rights reserved. 203
More flexible QoS differentiation and more efficient signaling
RByRBx
IP Packet
H SDAP SDU
PDCP SDUH
RLC SDUH
MAC SDUH H
IP Packet
H SDAP SDU
PDCP SDUH
RLC SDUH
MAC SDU
IP Packet
H SDAP SDU
PDCP SDUH
SDU SegmentH
MAC SDU
SDU SegmentH
H H MAC SDU
...
...
...
...
MAC PDU – Transport Block
PDCP
RLC
SDAP
MAC
n n+1 m
Source: 3GPP TS 38.300, v15.2.0, “NR; NR and NG-RAN Overall Description”, June, 2018.
RB: Radio Bearer
PDCP: Packet Data Convergence Protocol
204
On-going and Future Work
© 2018 InterDigital, Inc. All rights reserved.
Outlook for Release 16 Core Network
© 2018 InterDigital, Inc. All rights reserved.
205
Release 16 aims to enable services intended to be supported by the 5G Core Network
205
5G Massive IoT (MIoT)
§ Enable MTC and Cellular IoT functionalities in the 5G Core Network
§ MTC related features from Rel-10 to Rel-14 (e.g. Congestion
Control, Monitoring, Small Data Transfer, CP/UP Optimizations,
Non-IP Data Delivery, etc…) shall be supported in 5GC
§ Enhancements the 5G system to address Massive IoT (MIoT) use
cases based on the initial 5G requirements
V2X services in 5G
§ System enhancements of EPS and 5G to support advanced
V2X Use cases:
§ Platooning, extended sensor sharing, enhanced
positioning accuracy, advanced driving and remote
driving
Access Traffic Steering Switch and Splitting
§ Multi-access (e.g. 3GPP and WLAN) PDU session management
§ Support traffic splitting between multiple accesses (e.g. 3GPP and
WLAN)
§ Switch traffic between multiple accesses
5G LAN Services
§ Enhancements to the 5G system to support 5GLAN service
§ Support for 3GPP network that is not for public use and
for which service continuity and roaming with a PLMN is
possible (a type network).
§ Support an isolated 3GPP network that does not interact
with a PLMN (b type network)
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
Delivering Traditional and Omnidirectional Media
1 of 258

Recommended

Dynamic Adaptive Point Cloud Streaming by
Dynamic Adaptive Point Cloud StreamingDynamic Adaptive Point Cloud Streaming
Dynamic Adaptive Point Cloud StreamingAlpen-Adria-Universität
1.6K views28 slides
HTTP Adaptive Streaming State of the Art and Challenges Ahead by
HTTP Adaptive StreamingState of the Art and Challenges AheadHTTP Adaptive StreamingState of the Art and Challenges Ahead
HTTP Adaptive Streaming State of the Art and Challenges AheadAlpen-Adria-Universität
1.5K views34 slides
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems by
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming SystemsAutomated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems
Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming SystemsAlpen-Adria-Universität
878 views13 slides
An Introduction to OMNeT++ 5.4 by
An Introduction to OMNeT++ 5.4An Introduction to OMNeT++ 5.4
An Introduction to OMNeT++ 5.4Alpen-Adria-Universität
1.8K views57 slides
Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th... by
Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...
Video Coding for Large-Scale HTTP Adaptive Streaming Deployments: State of th...Alpen-Adria-Universität
1.6K views21 slides
A Journey Towards Fully Immersive Media Access by
A Journey Towards Fully Immersive Media AccessA Journey Towards Fully Immersive Media Access
A Journey Towards Fully Immersive Media AccessAlpen-Adria-Universität
1.9K views158 slides

More Related Content

What's hot

Adaptive Streaming of Traditional and Omnidirectional Media by
Adaptive Streaming of Traditional and Omnidirectional MediaAdaptive Streaming of Traditional and Omnidirectional Media
Adaptive Streaming of Traditional and Omnidirectional MediaAlpen-Adria-Universität
2.5K views105 slides
Docker-Based Evaluation Framework for Video Streaming QoE in Broadband Networks by
 Docker-Based Evaluation Framework for Video Streaming QoE in Broadband Networks Docker-Based Evaluation Framework for Video Streaming QoE in Broadband Networks
Docker-Based Evaluation Framework for Video Streaming QoE in Broadband NetworksAlpen-Adria-Universität
346 views17 slides
What’s new in MPEG? by
What’s new in MPEG?What’s new in MPEG?
What’s new in MPEG?Alpen-Adria-Universität
2.6K views53 slides
CAdViSE or how to find the Sweet Spots of ABR Systems by
CAdViSE or how to find the Sweet Spots of ABR SystemsCAdViSE or how to find the Sweet Spots of ABR Systems
CAdViSE or how to find the Sweet Spots of ABR SystemsAlpen-Adria-Universität
674 views19 slides
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo... by
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...Alpen-Adria-Universität
12.9K views25 slides
PEMWN'21 - ANGELA by
PEMWN'21 - ANGELAPEMWN'21 - ANGELA
PEMWN'21 - ANGELAJesus Aguilar
738 views15 slides

What's hot(20)

Docker-Based Evaluation Framework for Video Streaming QoE in Broadband Networks by Alpen-Adria-Universität
 Docker-Based Evaluation Framework for Video Streaming QoE in Broadband Networks Docker-Based Evaluation Framework for Video Streaming QoE in Broadband Networks
Docker-Based Evaluation Framework for Video Streaming QoE in Broadband Networks
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo... by Alpen-Adria-Universität
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...
Tile-based Streaming of 8K Omnidirectional Video: Subjective and Objective Qo...
H2BR: An HTTP/2-based Retransmission Technique to Improve the QoE of Adaptive... by Alpen-Adria-Universität
H2BR: An HTTP/2-based Retransmission Technique to Improve the QoE of Adaptive...H2BR: An HTTP/2-based Retransmission Technique to Improve the QoE of Adaptive...
H2BR: An HTTP/2-based Retransmission Technique to Improve the QoE of Adaptive...
ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq... by Alpen-Adria-Universität
ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...
ComplexCTTP: Complexity Class Based Transcoding Time Prediction for Video Seq...
WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D... by Minh Nguyen
WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...
WISH: User-centric Bitrate Adaptation for HTTP Adaptive Streaming on Mobile D...
Minh Nguyen716 views
Where to Encode: A Performance Analysis of Intel x86 and Arm-based Amazon EC2... by Alpen-Adria-Universität
Where to Encode: A Performance Analysis of Intel x86 and Arm-based Amazon EC2...Where to Encode: A Performance Analysis of Intel x86 and Arm-based Amazon EC2...
Where to Encode: A Performance Analysis of Intel x86 and Arm-based Amazon EC2...
Understanding Quality of Experience of Heuristic-based HTTP Adaptive Bitrate ... by Alpen-Adria-Universität
Understanding Quality of Experience of Heuristic-based HTTP Adaptive Bitrate ...Understanding Quality of Experience of Heuristic-based HTTP Adaptive Bitrate ...
Understanding Quality of Experience of Heuristic-based HTTP Adaptive Bitrate ...
Policy-driven Dynamic HTTP Adaptive Streaming Player Environment by Minh Nguyen
Policy-driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-driven Dynamic HTTP Adaptive Streaming Player Environment
Minh Nguyen32 views
ES-HAS: An Edge- and SDN-Assisted Framework for HTTP Adaptive Video Streaming by Alpen-Adria-Universität
ES-HAS: An Edge- and SDN-Assisted Framework for HTTP Adaptive Video StreamingES-HAS: An Edge- and SDN-Assisted Framework for HTTP Adaptive Video Streaming
ES-HAS: An Edge- and SDN-Assisted Framework for HTTP Adaptive Video Streaming

Similar to Delivering Traditional and Omnidirectional Media

Tutorial adaptive-streaming by
Tutorial adaptive-streamingTutorial adaptive-streaming
Tutorial adaptive-streamingJohnGregory89
104 views101 slides
Web Architecture and Technologies by
Web Architecture and TechnologiesWeb Architecture and Technologies
Web Architecture and TechnologiesFulvio Corno
3.5K views93 slides
QoS for Media Networks by
QoS for Media NetworksQoS for Media Networks
QoS for Media NetworksAmine Choukir
1K views59 slides
Multimedia streaming by
Multimedia streamingMultimedia streaming
Multimedia streamingSelvaraj Kesavan
295 views12 slides
An Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular Environments by
An Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular EnvironmentsAn Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular Environments
An Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular EnvironmentsAlpen-Adria-Universität
3.2K views21 slides
ARA JAGUAR-7000 Product Brief by
ARA JAGUAR-7000 Product BriefARA JAGUAR-7000 Product Brief
ARA JAGUAR-7000 Product BriefChul-Woong Yang
27 views17 slides

Similar to Delivering Traditional and Omnidirectional Media(20)

Tutorial adaptive-streaming by JohnGregory89
Tutorial adaptive-streamingTutorial adaptive-streaming
Tutorial adaptive-streaming
JohnGregory89104 views
Web Architecture and Technologies by Fulvio Corno
Web Architecture and TechnologiesWeb Architecture and Technologies
Web Architecture and Technologies
Fulvio Corno3.5K views
An Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular Environments by Alpen-Adria-Universität
An Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular EnvironmentsAn Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular Environments
An Evaluation of Dynamic Adaptive Streaming over HTTP in Vehicular Environments
1. A Survey on Bitrate Adaptation Schemes for Streaming Media over HTTP.pdf by AliIssa53
1. A Survey on Bitrate Adaptation Schemes for Streaming Media over HTTP.pdf1. A Survey on Bitrate Adaptation Schemes for Streaming Media over HTTP.pdf
1. A Survey on Bitrate Adaptation Schemes for Streaming Media over HTTP.pdf
AliIssa534 views
KITE Network Instrumentation: Advanced WebRTC Testing by Alexandre Gouaillard
KITE Network Instrumentation: Advanced WebRTC TestingKITE Network Instrumentation: Advanced WebRTC Testing
KITE Network Instrumentation: Advanced WebRTC Testing
Spotlight on the petroleum and energy vertical by FileCatalyst
Spotlight on the petroleum and energy vertical Spotlight on the petroleum and energy vertical
Spotlight on the petroleum and energy vertical
FileCatalyst618 views
Practical Routers & Switches for Electrical Engineers by Living Online
Practical Routers & Switches for Electrical EngineersPractical Routers & Switches for Electrical Engineers
Practical Routers & Switches for Electrical Engineers
Living Online1.2K views
A brief history of streaming video in the Internet by Stenio Fernandes
A brief history of streaming video in the InternetA brief history of streaming video in the Internet
A brief history of streaming video in the Internet
Stenio Fernandes1.4K views
2. A Survey of Rate Adaptation Techniques for Dynamic Adaptive Streaming over... by AliIssa53
2. A Survey of Rate Adaptation Techniques for Dynamic Adaptive Streaming over...2. A Survey of Rate Adaptation Techniques for Dynamic Adaptive Streaming over...
2. A Survey of Rate Adaptation Techniques for Dynamic Adaptive Streaming over...
AliIssa534 views
Partner webinar featuring CatDV by FileCatalyst
Partner webinar featuring CatDVPartner webinar featuring CatDV
Partner webinar featuring CatDV
FileCatalyst730 views
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming by DanieleLorenzi6
QoE- and Energy-aware Content Consumption for HTTP Adaptive StreamingQoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
QoE- and Energy-aware Content Consumption for HTTP Adaptive Streaming
DanieleLorenzi615 views
MMSys'21 - Multi-access edge computing for adaptive bitrate video streaming by Jesus Aguilar
MMSys'21 - Multi-access edge computing for adaptive bitrate video streamingMMSys'21 - Multi-access edge computing for adaptive bitrate video streaming
MMSys'21 - Multi-access edge computing for adaptive bitrate video streaming
Jesus Aguilar53 views
Web Services for the Internet of Things by Markku Laine
Web Services for the Internet of ThingsWeb Services for the Internet of Things
Web Services for the Internet of Things
Markku Laine4.2K views
testppt ch01(1) by ryaekle
testppt ch01(1)testppt ch01(1)
testppt ch01(1)
ryaekle650 views

More from Alpen-Adria-Universität

Multi-access Edge Computing for Adaptive Video Streaming by
Multi-access Edge Computing for Adaptive Video StreamingMulti-access Edge Computing for Adaptive Video Streaming
Multi-access Edge Computing for Adaptive Video StreamingAlpen-Adria-Universität
507 views36 slides
Policy-Driven Dynamic HTTP Adaptive Streaming Player Environment by
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentPolicy-Driven Dynamic HTTP Adaptive Streaming Player Environment
Policy-Driven Dynamic HTTP Adaptive Streaming Player EnvironmentAlpen-Adria-Universität
163 views47 slides
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In... by
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...Alpen-Adria-Universität
183 views17 slides
Energy Consumption in Video Streaming: Components, Measurements, and Strategies by
Energy Consumption in Video Streaming: Components, Measurements, and StrategiesEnergy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and StrategiesAlpen-Adria-Universität
658 views32 slides
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ... by
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...Alpen-Adria-Universität
515 views27 slides
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning by
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningAlpen-Adria-Universität
280 views63 slides

More from Alpen-Adria-Universität(20)

VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In... by Alpen-Adria-Universität
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
VE-Match: Video Encoding Matching-based Model for Cloud and Edge Computing In...
Energy Consumption in Video Streaming: Components, Measurements, and Strategies by Alpen-Adria-Universität
Energy Consumption in Video Streaming: Components, Measurements, and StrategiesEnergy Consumption in Video Streaming: Components, Measurements, and Strategies
Energy Consumption in Video Streaming: Components, Measurements, and Strategies
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ... by Alpen-Adria-Universität
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Exploring the Energy Consumption of Video Streaming: Components, Challenges, ...
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning by Alpen-Adria-Universität
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine LearningVideo Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
Video Coding Enhancements for HTTP Adaptive Streaming Using Machine Learning
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications by Alpen-Adria-Universität
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming ApplicationsSARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
SARENA: SFC-Enabled Architecture for Adaptive Video Streaming Applications
LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation... by Alpen-Adria-Universität
LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...
LLL-CAdViSE: Live Low-Latency Cloud-based Adaptive Video Streaming Evaluation...
How to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions by Alpen-Adria-Universität
How to Optimize Dynamic Adaptive Video Streaming? Challenges and SolutionsHow to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
How to Optimize Dynamic Adaptive Video Streaming? Challenges and Solutions
MPEC2: Multilayer and Pipeline Video Encoding on the Computing Continuum by Alpen-Adria-Universität
MPEC2: Multilayer and Pipeline Video Encoding on the Computing ContinuumMPEC2: Multilayer and Pipeline Video Encoding on the Computing Continuum
MPEC2: Multilayer and Pipeline Video Encoding on the Computing Continuum
Collaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming by Alpen-Adria-Universität
Collaborative Edge-Assisted Systems for HTTP Adaptive Video StreamingCollaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
Collaborative Edge-Assisted Systems for HTTP Adaptive Video Streaming
OTEC: An Optimized Transcoding Task Scheduler for Cloud and Fog Environments by Alpen-Adria-Universität
OTEC: An Optimized Transcoding Task Scheduler for Cloud and Fog EnvironmentsOTEC: An Optimized Transcoding Task Scheduler for Cloud and Fog Environments
OTEC: An Optimized Transcoding Task Scheduler for Cloud and Fog Environments
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming by Alpen-Adria-Universität
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live StreamingETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming
ETPS: Efficient Two-pass Encoding Scheme for Adaptive Live Streaming

Recently uploaded

Combining Orchestration and Choreography for a Clean Architecture by
Combining Orchestration and Choreography for a Clean ArchitectureCombining Orchestration and Choreography for a Clean Architecture
Combining Orchestration and Choreography for a Clean ArchitectureThomasHeinrichs1
69 views24 slides
Roadmap to Become Experts.pptx by
Roadmap to Become Experts.pptxRoadmap to Become Experts.pptx
Roadmap to Become Experts.pptxdscwidyatamanew
11 views45 slides
Transcript: The Details of Description Techniques tips and tangents on altern... by
Transcript: The Details of Description Techniques tips and tangents on altern...Transcript: The Details of Description Techniques tips and tangents on altern...
Transcript: The Details of Description Techniques tips and tangents on altern...BookNet Canada
130 views15 slides
STPI OctaNE CoE Brochure.pdf by
STPI OctaNE CoE Brochure.pdfSTPI OctaNE CoE Brochure.pdf
STPI OctaNE CoE Brochure.pdfmadhurjyapb
12 views1 slide
Uni Systems for Power Platform.pptx by
Uni Systems for Power Platform.pptxUni Systems for Power Platform.pptx
Uni Systems for Power Platform.pptxUni Systems S.M.S.A.
50 views21 slides

Recently uploaded(20)

Combining Orchestration and Choreography for a Clean Architecture by ThomasHeinrichs1
Combining Orchestration and Choreography for a Clean ArchitectureCombining Orchestration and Choreography for a Clean Architecture
Combining Orchestration and Choreography for a Clean Architecture
ThomasHeinrichs169 views
Transcript: The Details of Description Techniques tips and tangents on altern... by BookNet Canada
Transcript: The Details of Description Techniques tips and tangents on altern...Transcript: The Details of Description Techniques tips and tangents on altern...
Transcript: The Details of Description Techniques tips and tangents on altern...
BookNet Canada130 views
STPI OctaNE CoE Brochure.pdf by madhurjyapb
STPI OctaNE CoE Brochure.pdfSTPI OctaNE CoE Brochure.pdf
STPI OctaNE CoE Brochure.pdf
madhurjyapb12 views
AMAZON PRODUCT RESEARCH.pdf by JerikkLaureta
AMAZON PRODUCT RESEARCH.pdfAMAZON PRODUCT RESEARCH.pdf
AMAZON PRODUCT RESEARCH.pdf
JerikkLaureta15 views
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen... by NUS-ISS
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...
Upskilling the Evolving Workforce with Digital Fluency for Tomorrow's Challen...
NUS-ISS28 views
How the World's Leading Independent Automotive Distributor is Reinventing Its... by NUS-ISS
How the World's Leading Independent Automotive Distributor is Reinventing Its...How the World's Leading Independent Automotive Distributor is Reinventing Its...
How the World's Leading Independent Automotive Distributor is Reinventing Its...
NUS-ISS15 views
Emerging & Future Technology - How to Prepare for the Next 10 Years of Radica... by NUS-ISS
Emerging & Future Technology - How to Prepare for the Next 10 Years of Radica...Emerging & Future Technology - How to Prepare for the Next 10 Years of Radica...
Emerging & Future Technology - How to Prepare for the Next 10 Years of Radica...
NUS-ISS16 views
Architecting CX Measurement Frameworks and Ensuring CX Metrics are fit for Pu... by NUS-ISS
Architecting CX Measurement Frameworks and Ensuring CX Metrics are fit for Pu...Architecting CX Measurement Frameworks and Ensuring CX Metrics are fit for Pu...
Architecting CX Measurement Frameworks and Ensuring CX Metrics are fit for Pu...
NUS-ISS37 views
Future of Learning - Yap Aye Wee.pdf by NUS-ISS
Future of Learning - Yap Aye Wee.pdfFuture of Learning - Yap Aye Wee.pdf
Future of Learning - Yap Aye Wee.pdf
NUS-ISS41 views
DALI Basics Course 2023 by Ivory Egg
DALI Basics Course  2023DALI Basics Course  2023
DALI Basics Course 2023
Ivory Egg14 views
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum... by NUS-ISS
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...
Beyond the Hype: What Generative AI Means for the Future of Work - Damien Cum...
NUS-ISS34 views
Understanding GenAI/LLM and What is Google Offering - Felix Goh by NUS-ISS
Understanding GenAI/LLM and What is Google Offering - Felix GohUnderstanding GenAI/LLM and What is Google Offering - Felix Goh
Understanding GenAI/LLM and What is Google Offering - Felix Goh
NUS-ISS41 views
Igniting Next Level Productivity with AI-Infused Data Integration Workflows by Safe Software
Igniting Next Level Productivity with AI-Infused Data Integration Workflows Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Igniting Next Level Productivity with AI-Infused Data Integration Workflows
Safe Software225 views
[2023] Putting the R! in R&D.pdf by Eleanor McHugh
[2023] Putting the R! in R&D.pdf[2023] Putting the R! in R&D.pdf
[2023] Putting the R! in R&D.pdf
Eleanor McHugh38 views
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors by sugiuralab
TouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective SensorsTouchLog: Finger Micro Gesture Recognition  Using Photo-Reflective Sensors
TouchLog: Finger Micro Gesture Recognition Using Photo-Reflective Sensors
sugiuralab15 views

Delivering Traditional and Omnidirectional Media

  • 1. IEEE ICME Tutorial – San Diego, CA July 2018 Delivering Traditional and Omnidirectional Media Ali C. Begen, Liangping Ma and Christian Timmerer http://www.slideshare.net/christian.timmerer
  • 2. Upon Attending This Tutorial, You Will Know About • Principles of HTTP adaptive streaming for the Web/HTML5 • Principles of omnidirectional (360°) media delivery • Content generation/distribution/consumption workflows for traditional and omnidirectional media • Standards and emerging technologies in the adaptive streaming space • 5G radio access and core networks, technologies and emerging multimedia use cases This tutorial is public, however, some material might be copyrighted Use proper citation when using content from this tutorial (Thanks to T. Stockhammer, J. Simmons, K. Hughes, C. Concolato, S. Pham, W. Law, N. Weil, P. Chou and many others for helping with the material) IEEE ICME Tutorial - July 2018 2
  • 3. Ali C. Begen • Electrical engineering degree from Bilkent University (2001) • Ph.D. degree from Georgia Tech (2006) – Video delivery and multimedia communications • Research, development and standards at Cisco (2007-2015) – IPTV, content delivery, software clients – Transport and distribution over IP networks – Enterprise video • Consulting at Networked Media since 2016 • Assistant professor at OzU and IEEE Distinguished Lecturer since 2016 Christian Timmerer • Associate professor at the Institute of Information Technology (ITEC), Multimedia Communication Group (MMC), Alpen-Adria- Universität Klagenfurt, Austria • Co-founder and CIO of Bitmovin • Research interests – Immersive multimedia communication – Streaming, adaptation, and – Quality of experience (QoE) • Blog: http://blog.timmerer.com; @timse7 Presenters Today – Part I IEEE ICME Tutorial - July 2018 3
  • 4. Liangping Ma • Member of Technical Staff, InterDigital (2009-) • Research Engineer, San Diego Research Center (2005-2007), Argon ST (2007-2009) • PhD, Electrical Engineering, University of Delaware (2004) • BS, Physics, Wuhan University, China (1998) • IEEE ComSoc Distinguished Lecturer since 2017 • Research interests – Video communication – 5G radio access network – Cognitive radios Presenters Today – Part II IEEE ICME Tutorial - July 2018 4
  • 5. • Two tutorials, 30 talks on – HEVC, VVC and AV1 – Containers – HDR – Next-gen audio – The latest and greatest in streaming – Content protection and more • Cost: Free • Visit http://mile-high.video/ to register Next week in Denver, CO Mile High Video 2018 IEEE ICME Tutorial - July 2018 5
  • 6. ACM MMSys 2019 Call for Papers and Special Sessions • City: Amherst, MA • Date: June 18-21, 2019 • Submission deadline: Oct. 30 • Co-located with – NOSSDAV – MMVE – Packet Video • http://www.mmsys2019.org/ IEEE ICME Tutorial - July 2018 6
  • 7. Topics to Cover • Part I: Streaming (Morning) – HTTP adaptive streaming and workflows – HTML5 video and media extensions – Common issues in scaling and multi-screen/hybrid delivery – Overview of Streaming Standards (DASH, CMAF and Apple HLS) – Adaptive delivery of omnidirectional/360° video – The developing MPEG-OMAF and MPEG-I standards • Part II: Communications over 5G (Afternoon) – 5G fundamentals: core network, radio access network and QoS – Multimedia communication case studies – Improving QoE in streaming IEEE ICME Tutorial - July 2018 7
  • 8. IPTV vs. IP (Over-the-Top) Video First Things First IPTV IP Video Best-effort delivery Quality not guaranteed Mostly on demand Paid or ad-based service Managed delivery Emphasis on quality Mostly linear TV Always a paid service IEEE ICME Tutorial - July 2018 8
  • 9. Part I: Streaming HTTP Adaptive Streaming and Workflows
  • 10. Internet (IP aka OTT) Video Essentials Reach all connected devicesReach Enable live and on-demand delivery to the mass marketScale Provide TV-like consistent rich viewer experienceQuality of Experience Enable revenue generation thru paid content, subscriptions, ads, etc.Business Satisfy regulations such as captioning, ratings and parental controlRegulatory IEEE ICME Tutorial - July 2018 10
  • 11. Creating Revenue – Attracting Eye Balls • High-end content – Hollywood movies, TV shows – Sports • Excellent quality – HD/3D/UHD audiovisual presentation w/o artifacts such as pixelization and rebuffering – Fast startup, fast zapping and low glass-to-glass delay • Usability – Navigation, content discovery, battery consumption, trick modes, social network integration • Service flexibility – Linear TV – Time-shifted and on-demand services • Reach – Any device, any time IEEE ICME Tutorial - July 2018 11
  • 12. One Request, One Response Progressive Download HTTP Request HTTP Response IEEE ICME Tutorial - July 2018 12
  • 13. What is Streaming? Streaming is transmission of a continuous content from a server to a client and its simultaneous consumption by the client Two Main Characteristics 1. Client consumption rate may be limited by real-time constraints as opposed to just bandwidth availability 2. Server transmission rate (loosely or tightly) matches to client consumption rate IEEE ICME Tutorial - July 2018 13
  • 14. Video Delivery over HTTP • Enables playback while still downloading • Server sends the file as fast as possible Progressive Download • Enables seeking via media indexing • Server paces transmission based on encoding rate Pseudo Streaming • Content is divided into short-duration chunks • Enables live streaming and ad insertion Chunked Streaming • Multiple versions of the content are created • Enables to adapt to network and device conditions Adaptive Streaming IEEE ICME Tutorial - July 2018 14
  • 15. Adaptive Streaming over HTTP Decoding and Presentation Streaming Client Media Buffer Content Ingest (Live or Pre-captured) Multi-rate Encoder Packager Origin (HTTP) Server … … … … ServerStorage HTTP GET Request Response IEEE ICME Tutorial - July 2018 15
  • 16. Adapt Video to Web Rather than Changing the Web Adaptive Streaming over HTTP • Imitation of streaming via short downloads – Downloads small chunks to minimize bandwidth waste – Enables to monitor consumption and track the streaming clients • Adaptation to dynamic conditions and device capabilities – Adapts to dynamic conditions in the Internet and home network – Adapts to display resolution, CPU and memory resources of the streaming client à Facilitates “any device, anywhere, anytime” paradigm • Improved quality of experience (not always improved average quality) – Enables faster start-up and seeking, and quicker buffer fills – Reduces skips, freezes and stutters • Use of HTTP – Well-understood naming/addressing approach, and authentication/authorization infrastructure – Provides easy traversal for all kinds of middleboxes (e.g., NATs, firewalls) – Enables cloud access, leverages the existing (cheap) HTTP caching infrastructure IEEE ICME Tutorial - July 2018 16
  • 17. Stalls, Slow Start-Up, Plug-In and DRM Issues Common Annoyances in Streaming • Unsupported/wrong – protocol – plug-in – codec – format – DRM • Slow start-up • Poor quality, quality variation • Frequent freezes/glitches • Lack of seeking IEEE ICME Tutorial - July 2018 17
  • 18. Dead, Surviving, Maturing and Newborn Technologies • Move Adaptive Stream (Long gone, but some components are in Slingbox) – http://www.movenetworks.com • Microsoft Smooth Streaming (Legacy) – http://www.iis.net/expand/SmoothStreaming • Adobe Flash (Almost dead) – http://www.adobe.com/products/flashplayer.html • Adobe HTTP Dynamic Streaming (Legacy) – http://www.adobe.com/products/httpdynamicstreaming • Apple HTTP Live Streaming (The elephant in the room) – https://tools.ietf.org/html/rfc8216 – https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis • MPEG DASH and CMAF (The standards) – http://mpeg.chiariglione.org/standards/mpeg-dash – http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format IEEE ICME Tutorial - July 2018 18
  • 19. List of Accessible Segments and Their Timings An Example DASH Template-Based Manifest MPD Period id = 1 start = 0 s Period id = 3 start = 300 s Period id = 4 start = 850 s Period id = 2 start = 100 s Adaptation Set 0 subtitle turkish Adaptation Set 2 audio english Adaptation Set 1 BaseURL=http://abr.rocks.com/ Representation 2 Rate = 1 Mbps Representation 4 Rate = 3 Mbps Representation 1 Rate = 500 Kbps Representation 3 Rate = 2 Mbps Resolution = 720p Segment Info Duration = 10 s Template: 3/$Number$.mp4 Segment Access Initialization Segment http://abr.rocks.com/3/0.mp4 Media Segment 1 start = 0 s http://abr.rocks.com/3/1.mp4 Media Segment 2 start = 10 s http://abr.rocks.com/3/2.mp4 Adaptation Set 3 audio italian Adaptation Set 1 video Period id = 2 start = 100 s Representation 3 Rate = 2 Mbps Selection of components/tracks Well-defined media format Selection of representations Splicing of arbitrary content like ads Chunks with addresses and timing IEEE ICME Tutorial - July 2018 19
  • 20. An Example HLS Playlist-Based Manifest #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=232370,CODECS="mp4a.40.2, avc1.4d4015" gear1/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=649879,CODECS="mp4a.40.2, avc1.4d401e" gear2/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=41457,CODECS="mp4a.40.2" gear0/prog_index.m3u8 master.m3u8 Source: https://developer.apple.com/streaming/examples/ and https://www.gpac-licensing.com/2014/12/01/apple-hls-comparing-versions/ #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-PLAYLIST-TYPE:VOD #EXTINF:9.97667, fileSequence0.ts #EXTINF:9.97667, fileSequence1.ts #EXTINF:9.97667, fileSequence2.ts . . . #EXT-X-ENDLIST gear1/prog_index.m3u8 IEEE ICME Tutorial - July 2018 20
  • 21. Example Representations Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 2.2 Mbps 960 x 540 Rep. #3 1.4 Mbps 960 x 540 Rep. #4 900 Kbps 512 x 288 Rep. #5 600 Kbps 512 x 288 Rep. #6 400 Kbps 340 x 192 Rep. #7 200 Kbps 340 x 192 Source: Vertigo MIX10, Alex Zambelli’s Streaming Media Blog, Akamai, Comcast Vancouver 2010 Sochi 2014 Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 1.95 Mbps 848 x 480 Rep. #3 1.25 Mbps 640 x 360 Rep. #4 900 Kbps 512 x 288 Rep. #5 600 Kbps 400 x 224 Rep. #6 400 Kbps 312 x 176 Encoding Bitrate Resolution Rep. #1 18 Mbps 4K (60p) Rep. #2 12.2 Mbps 2560x1440 (60p) Rep. #3 4.7 Mbps 2K (60p) Rep. #4 3.5 Mbps 1280x720 (60p) Rep. #5 2 Mbps 1280 x 720 Rep. #6 1.2 Mbps 768 x 432 Rep. #7 750 Kbps 640 x 360 Rep. #8 500 Kbps 512 x 288 Rep. #9 300 Kbps 320 x 180 Rep. #10 200 Kbps 320 x 180 PyeongChang 2018 IEEE ICME Tutorial - July 2018 21
  • 22. Smart and Selfish Clients HAS Working Principle - Client fetches and parses the manifest - Client uses the OS-provided HTTP stack (HTTP may run over TCP or QUIC) - Client uses the required decryption tools for the protected content Client monitors and measures - Size of the playout buffer (both in bytes and seconds) - Chunk download times and throughput - Local resources (CPU, memory, window size, etc.) - Dropped frames Client performs adaptation Request Response HTTPServer Client Client measures and reports metrics for analytics (One can also multicast media segments) IEEE ICME Tutorial - July 2018 22
  • 23. Tradeoffs in Adaptive Streaming Overall quality Quality stability Proximity to live edge Stalls Zapping/seeking time IEEE ICME Tutorial - July 2018 23
  • 24. End-to-End Workflow for OTT Production Preparation and Staging Distribution Consumption News Gathering Sport Events Premium Content Studio Multi-bitrate Encoding Encapsulation Protection Origin Servers VoD Content & Manifests Live Content & Manifests CDN IEEE ICME Tutorial - July 2018 24
  • 25. Part I: Streaming HTML5 Video and Media Extensions
  • 26. • HTML5 is a set of technologies that allows more powerful Web sites and applications – Better semantics – Better connectivity – Offline and storage – Multimedia – 2D/3D graphics and effects – Performance and integration – Device access – Styling • Most interesting new elements – Semantic elements: <header>, <footer>, <article> and <section> – Graphic elements: <svg> and <canvas> – Multimedia elements: <audio> and <video> A Common Platform across Devices HTML5 Source: Mozilla MDN, w3schools.com IEEE ICME Tutorial - July 2018 26
  • 27. • These apps can – leverage native APIs and offline capability – provide cross-device compatibility (iOS, Windows, Android, etc.) – be packaged and published to app stores • For streaming: – To a user, it looks like a regular native app they have downloaded – The app is actually a container and loads loads dynamically from the web server – Server-side changes propagate automatically to installed apps Hosted and Progressive Web Apps Source: Google PWA, Microsoft HWA IEEE ICME Tutorial - July 2018 27
  • 28. Types of Browser-Based Playback Source: CTA WAVE Type 1: Minimum control architectural model for HTML5 support of adaptive streaming where manifest and heuristics are managed by the user agent Type 2: Adaptation control architectural model for HTML5 support of adaptive streaming providing script manageable features Type 3: Full media control architectural model for HTML5 support of adaptive streaming allowing script to explicitly send the media segments (MSE + EME) IEEE ICME Tutorial - July 2018 28
  • 29. MSE Support in Web Browsers (as of July’18) Source: http://caniuse.com/#search=mse IEEE ICME Tutorial - July 2018 29
  • 30. Content Protection – Famous Quotes Digital files cannot be made uncopiable, any more than water can be made not wet Bruce Schneier (cryptographer), 2001 We have Ph.D.'s here that know the stuff cold, and we don't believe it's possible to protect digital content Steve Jobs, 2003 If we’re still talking about DRM in five years, please take me out and shoot me eMusic CEO David Pakman, 2007 IEEE ICME Tutorial - July 2018 30
  • 31. W3C Encrypted Media Extensions Source: https://www.w3.org/TR/encrypted-media/ IEEE ICME Tutorial - July 2018 31
  • 32. EME Support in Web Browsers (as of July’18) Source: http://caniuse.com/#search=eme IEEE ICME Tutorial - July 2018 32
  • 33. Web Video Ecosystem Encoding Encryption Rights Expression Web Video App Framework Decoding Decryption Rights Management The fundamental Digital Rights Management problem derives from a lack of interoperability which prevents mobility of experience The solution is to combine interoperable commercial Web video content with a cross- platform Web video app framework Source: John Simmons IEEE ICME Tutorial - July 2018 33
  • 34. Contrary to a common misconception, with EME DRM functionality is not in the HTML/JS app There is no DRM in HTML5 with EME, and ECP requires that this be the case The HTML/JS app selects the DRM and controls key exchange between DRM client and server Browser extends HTML5 media element to allow JavaScript handled key acquisition Browser A CDM exposes a key system to JavaScript; it is transparent whether the CDM is in the browser The Web, EME and Enhanced Content Protection (ECP) Source: John Simmons IEEE ICME Tutorial - July 2018 34
  • 35. DRM Support on Desktop Browsers Browser OS EME/CDM Flash Player NPAPI/ Silverlight 5 Chrome Win Widevine (Yes) No OS X (Yes) No Linux (Yes) No Firefox Win Widevine Yes No OS X Yes No Linux Yes No Safari > OS X Yosemite (FairPlay) Yes Yes < OS X Yosemite No Yes Yes IE/Edge < Win 7 No Yes Yes Win 10 PlayReady Yes No IEEE ICME Tutorial - July 2018 35
  • 36. DRM Support on Mobile Platforms Platform Browser Native App / DRM iOS No MSE/EME yet HLS (AES-128 CBC) via <video> Native SDK and WebView: FairPlay Android MSE/EME Native + Android SDK (MediaDRM APIs) - (Widevine + OMA v2) ExoPlayer Native + WebView with Widevine Windows 10 MSE/EME WebViews/Hosted Web Apps with PlayReady IEEE ICME Tutorial - July 2018 36
  • 37. • Server-side ad insertion (SSAI) – Mostly used for linear/dynamic ad insertion (ads are still targeted) – Easier for less capable devices – Smoother transition between the content and ads – Uses XLink or MPD chaining in DASH for multi- period MPDs – Splices ads into a continuous stream (offline or just-in-time) for single-period MPDs • Client-side ad insertion – Requires additional support in the application – Uses events in DASH Ad Insertion Methods Preroll Idle Midroll Idle Postroll Preroll Content Midroll Content Postroll Pause Content Pause Content Pause Primary Stream Primary Stream Ad Stream IEEE ICME Tutorial - July 2018 37
  • 38. Ad Insertion – XLink Resolution IEEE ICME Tutorial - July 2018 38
  • 39. Part I: Streaming Common Issues in Scaling and Multi-Screen/Hybrid Delivery
  • 40. Streaming over HTTP – The Promise • Leverage tried-and-true Web infrastructure for scaling – Video is just ordinary Web content! • Leverage tried-and-true TCP – Congestion avoidance – Reliability – No special QoS for video THERE IT SHOULD JUST WORK IEEE ICME Tutorial - July 2018 40
  • 41. Doesitjustwork? Mostly yes, when streaming clients compete with other types of traffic Not really, when streaming clients compete with each other Streaming clients interact with each other forming an “accidental” distributed control-feedback system • Multiple screens within a household • ISP access/aggregation links • Small cells in stadiums and malls IEEE ICME Tutorial - July 2018 41
  • 42. A Single Microsoft Smooth Streaming Client under a Controlled Environment Demystifying a Streaming Client 0 1 2 3 4 5 0 50 100 150 200 250 300 350 400 450 500 Bitrate(Mbps) Time (s) Available Bandwidth Requests Chunk Tput Average Tput Reading: “An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP,” ACM MMSys 2011 Buffer-filling State Back-to-back requests Steady State Periodic requests IEEE ICME Tutorial - July 2018 42
  • 43. 10 (Commercial) Streaming Clients Sharing a 10 Mbps Link Selfishness Hurts Everyone 0 200 400 600 800 1000 1200 1400 0 100 200 300 400 500 RequestedBitrate(Kbps) Time (s) Client1 Client2 Client3 IEEE ICME Tutorial - July 2018 43
  • 44. Viewer Experience Statistics Selfishness Hurts Everyone Source: Conviva Viewer Experience Report, 2015 IEEE ICME Tutorial - July 2018 44
  • 45. Inner and Outer Control Loops HTTP Server Manifest Media HTTP Origin Module TCP Sender Streaming Client Manifest Resource Monitors Streaming Application TCP ReceiverData / ACK There could be multiple TCPs destined to the same or different servers Request / Response IEEE ICME Tutorial - July 2018 45
  • 46. Streaming with Multiple TCP Connections • Using multiple concurrent TCPs – Can help mitigate head-of-line blocking – Allows fetching multiple (sub)segments in parallel – Allows to quickly abandon a non-working connection without having to slow-start a new one System performance deteriorates very quickly if many clients adopt this approach without limiting the aggregated bandwidth consumption IEEE ICME Tutorial - July 2018 46
  • 47. Two Competing Clients Understanding the Root Cause • Depending on the timing of the ON periods: – Unfairness, underutilization and/or instability may occur – Clients may grossly overestimate their fair share of the available bandwidth Clients cannot figure out how much bandwidth to use until they use too much (Just like TCP) Reading: “What happens when HTTP adaptive streaming players compete for bandwidth?,” ACM NOSSDAV 2012 IEEE ICME Tutorial - July 2018 47
  • 48. How to Solve the Issues? • Use a better adaptation algorithm like PANDA or BOLA • Use machine learning or deep learning like Pensieve • Improve the HTTP/TCP stack, try out the alternatives • Adopt ideas from game/consensus theory (GTA) Fix the clients and/or the transport • QoS in the core/edge • SDN Get support from the network • Assist the clients and network elements thru metrics and analytics Enable a control plane IEEE ICME Tutorial - July 2018 48
  • 49. Bitrate Adaptation Schemes Bitrate Adaptation Schemes Client- based Adaptation Bandwidth- based Buffer- based Mixed adaptation Proprietary solutions MDP-based Server- based Adaptation Network- assisted Adaptation Hybrid Adaptation SDN-based Server and network- assisted Reading: “A survey on bitrate adaptation schemes for streaming media over HTTP,” IEEE Commun. Surveys Tuts., to appear IEEE ICME Tutorial - July 2018 49
  • 50. High-Level Comparison between Different Schemes • The client-based adaptation schemes – Show a good performance in single and few-client scenarios – Largely fail in multi-client scenarios • The server-based adaptation schemes – Require custom servers – More effective in eliminating the bitrate oscillation problems – Less scalable due to increased complexity on the servers • The network-assisted adaptation and hybrid schemes – Show a good performance in both small and large populations – Require modifications on the clients, server and/or network devices – Pose practicality issues for deployment IEEE ICME Tutorial - July 2018 50
  • 51. A Note on the Reproducibility • A performance comparison between different algorithms is often difficult due to – Lack of the source codes – Lack of a unified QoE framework and universally accepted metrics • Schemes have their own parameters and assumptions, and work well only under certain circumstances IEEE ICME Tutorial - July 2018 51
  • 52. Game/Consensus Theory • Structuring a set of players in cooperative groups • Enables a strong level of cooperation among the players • Model and analyze fairness, stability and efficiency Coalition Formation • Players that are interested in reaching an agreement among themselves over a common objective (how to reach this agreement and on the terms of the agreement) • Concept solution: Nash Bargaining Solution (NBS) Bargaining Solution • Exploited together with NBS and coalition-formation game • Relied on a strong coordination among the players Consensus Theory IEEE ICME Tutorial - July 2018 52
  • 53. GTA: A Game Theory Based ABR Scheme • Designed based on a cooperative game in the form of static formation-based coalitions • Formulates the ABR decision problem as a bargaining process and consensus mechanism • Outputs the optimal decision by reaching the Pareto optimal (PO) Nash bargaining solution (NBS) Game Theory Cooperative Coalition Formation Bargaining Solution Consensus Theory Strategic GTA GTA player Coalition Bargaining process IEEE ICME Tutorial - July 2018 53
  • 54. • PANDA & CS2P Estimators – Throughput estimation • SAND Enabler – Implements various SAND-enabled communication interfaces • SSIMplus MAP Model – Maps content type, display resolution and service plan type into one common perceptual quality space in which a set of clusters are constructed • QoE Metrics – QoE model and various metrics • GT Agent – Develops the GTA decision rules for selecting the best bitrate GTA Components Reading: “Want to play DASH? a game theoretic approach for adaptive streaming over HTTP,” ACM MMSys 2018 IEEE ICME Tutorial - July 2018 54
  • 55. Component #1: Cooperation Enabler GTA Design Non-Cooperative Mode • Mix of GTA and non-GTA players • Greedy interaction with non-GTA players • GTA clients may start in non- cooperative mode and then learn the accurate number of clients through potential and wonderful life utility (WLU) Cooperative Mode • Multiple GTA players • GTA players are aggregated into a set of clusters based on SSIMplus-based MAP IEEE ICME Tutorial - July 2018 55
  • 56. Component #2: QoE Calculator GTA Design Avg. Quality Avg. Quality Switch Stalls Startup Delay IEEE ICME Tutorial - July 2018 56
  • 57. Component #3 and #4: GT Strategy Calculator and GT (Dis)agreement Calculator GTA Design • GT Strategy Calculator – Compute the GT strategy (i.e., the action-utility R(A) relationship set) which is defined over the strategy space S as follows: • GT (Dis)agreement Calculator – Compute the GT disagreement decision • i.e., bargaining outcome disagreements (W) P = {p1,...,pN} is the set of N players R is the set of possible utilities when actions from A are taken • r is the player utility when action a is taken • k is the current step, K is total number of segments of video u R(A) is the set of possible actions with their achievable utilities R-(A-) is the set of pessimal actions with their resulting utilities S and W are defined by the function F which is defined over the space: F: (A → R) ∪ {W} PO (NBS): max. (S – W) AgreementpointDisagreementpoint MAX IEEE ICME Tutorial - July 2018 57
  • 58. Component #5: QoE Optimizer GTA Design • GTA rule for ABR selection can be formulated as a network utility maximization – Strictly increasing concave – Solver: Dual decomposition + dynamic programming • Find the bargaining solution by solving the Problem(S,W) – Bargaining Process and consensus decision problem is defined as a pair Problem(S,W) – The bargaining solution is defined by a function F over the space F : (S, W ) → Rn – F specifies a unique Pareto optimal (PO) bargaining outcome (O* = F(S,W)) for every Problem(S,W) using Nash Bargaining Solution (NBS) – O* is the set that contains only the optimal bargaining outcomes C.1 Maintain the current buffer occupancy between the min and max buffer thresholds C.2 Satisfy the SSIMplus MAP model C.3 Lead to a unique (PO) NBS C.4 Chunk time constraint where max time needed to download the corresponding chunk ≦ chunk duration IEEE ICME Tutorial - July 2018 58
  • 59. Source Code is Available at http://streaming.university/GTA/ dash.js Integration Gray boxes indicate the components that have been developed or modified IEEE ICME Tutorial - July 2018 59
  • 60. Setup for Performance Evaluation • VoD experimental setup – Two machines (running Ubuntu 16.04 LTS), one for the dash.js player and one for the server – tc-NetEm to replicate the throughput profiles – PANDA parameters as described in the original paper – Throughput smoothing: the mean of the last three throughput estimations – The fastMPC horizon was set to the next three chunks – The min and max buffer occupancy thresholds were set to 8 and 32 seconds – CT = animation (BBB), chunk duration of 4 s, total duration of 600 s – 10 bitrate levels: [250, 300, 400, 700, 900, 1500, 2000, 3000, 3500, 4000] Kbps IEEE ICME Tutorial - July 2018 60
  • 61. Results: Video Stability IEEE ICME Tutorial - July 2018 61
  • 62. Results: QoE IEEE ICME Tutorial - July 2018 62
  • 63. Numbers Indicate GTA Improvement over Other Schemes GTA Outperforms the Existing ABR Schemes BBA BOLA Rate-based Hybrid Quetra ELASTIC QDASH SARA FESTIVE QoE 55% 33.5% 35.5% 19.5% 39% 46.5% 47% 40.5% 30.5% Video Stability 60% 72% 37.5% 63.5% 66.5% 70% 66% 53% 63.5% Stall Events 13.6% 1.1% 6.1% 1.7% 3.87% 7.87% 5.87% 2.9% 3.6% IEEE ICME Tutorial - July 2018 63
  • 64. http://streaming.university/GTA/ GTA Demo and Source Codes Offline Demo Online Demo IEEE ICME Tutorial - July 2018 64
  • 65. Quickly Starting Media Streams Using QUIC QUIC: Multiplexed transport protocol over UDP • Reduced handshake latency • Improved congestion control • Improved loss recovery • Multiplexing streams Can QUIC help improve viewer experience in HTTP adaptive streaming? Reading: “Quickly starting media streams using QUIC,” PV 2018 IEEE ICME Tutorial - July 2018 65
  • 66. • Start media streams more quickly • Reduce seeking latency • Cope better with frequent connection changes? Can QUIC help us IEEE ICME Tutorial - July 2018 66
  • 67. Previous Research • Some reported – QUIC does not impact streaming performance – QUIC does not provide a boost to HAS • Others reported – QUIC’s 0-RTT performed better than the other protocols – QUIC provided better streaming, but only for high-quality video • Google said QUIC reduced YouTube rebuffer rates by 15% QUIC code evolves rapidly and 3rd-party implementations may not necessarily reflect protocol’s real performance IEEE ICME Tutorial - July 2018 67
  • 68. Comparison between Our and Prior Work 1 WiFi, 4G/LTE and 3G, 2 Based on the third-party implementation version at the time of research, 3 Latest at the time of research. IEEE ICME Tutorial - July 2018 68
  • 69. • Player Features – Python based – TCP and QUIC support integrated as subprocess – Scripted frame-seek (fast- forward) ability – BASIC, SARA, BBA-2 adaptation algorithms • Source Code – github.com/sevketarisu/quic- streaming github.com/sevketarisu/quic-streaming IEEE ICME Tutorial - July 2018 69
  • 70. • Environment Setup – Public Internet – No traffic shaping – Each test repeated 10 times • Server – Apache HTTP server on Amazon EC2 – Located in Frankfurt, DE • Client – Runs Google’s proto-quic library – Located in Istanbul, TR – Connected via WiFi, LTE or 3G Chromium QUIC (proto-quic library) WiFi, LTE, 3G Amazon EC2 libcurl IEEE ICME Tutorial - July 2018 70
  • 71. Evaluated Metrics • Average playback bitrate – Average of the bitrates of the downloaded segments • Average wait time after seeking – Time from the frame-seek request to the playback of the requested media – A rule of thumb is to keep this time under two seconds • Rebuffer rate IEEE ICME Tutorial - July 2018 71
  • 72. Frame-Seek (Fast-Forward) Feature Implemented in the Player Frame-Seek Scenarios IEEE ICME Tutorial - July 2018 72
  • 73. Frame-Seek Scenario Results 2.80 2.99 2.862.89 3.05 2.99 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 WiFi LTE 3G AveragePlaybackBitrate(Mbps) TCP QUIC Results are averaged for BASIC, SARA and BBA-2 algorithms. QUIC provided a higher (or at least an equal) average playback bitrate in all cases and for all algorithms Max Bitrate Available: 3.9 Mbps IEEE ICME Tutorial - July 2018 73
  • 74. Frame-Seek Scenario Results 2.09 2.16 2.38 1.28 1.29 1.42 0.0 0.5 1.0 1.5 2.0 2.5 3.0 WiFi LTE 3G AverageWaitTimeafterSeeking(s) TCP QUIC Results are averaged for BASIC, SARA and BBA-2 algorithms. QUIC reduced wait times to less than 1.5 seconds IEEE ICME Tutorial - July 2018 74
  • 75. Frame-Seek Scenario Results 3.67 3.67 5.67 1.00 1.33 2.00 0 1 2 3 4 5 6 WiFi LTE 3G RebufferRate(%) TCP QUIC QUIC’s benefits are bigger when delays are higher Results are averaged for BASIC, SARA and BBA-2 algorithms. IEEE ICME Tutorial - July 2018 75
  • 76. Connection-Switch Simulation IEEE ICME Tutorial - July 2018 76
  • 77. Connection-Switch Scenario IEEE ICME Tutorial - July 2018 77
  • 78. WiFi - LTE Switches Connection-Switch Results 3.00 4.00 5.00 2.00 3.00 4.00 0 1 2 3 4 5 6 BASIC SARA BBA-2 RebufferRate(%) TCP QUIC QUIC provided a higher (or at least an equal) average playback bitrate for all algorithms IEEE ICME Tutorial - July 2018 78
  • 79. WiFi - 3G Switches Connection-Switch Results 13.00 2.00 5.00 6.00 1.00 4.00 0 2 4 6 8 10 12 14 BASIC SARA BBA-2 RebufferRate(%) TCP QUIC QUIC provided a higher (or at least an equal) average playback bitrate for all algorithms IEEE ICME Tutorial - July 2018 79
  • 80. But QUIC is still evolving in the IETF; should there be significant changes, the tests will need to be repeated – https://quicwg.org/ – https://github.com/quicwg Conclusions So Far • QUIC reduces wait times and rebuffer rates without reducing playback bitrate • QUIC outperforms TCP when frequent network changes occur • QUIC’s benefits are greater in networks with larger delay (e.g., early generation 3G networks) IEEE ICME Tutorial - July 2018 80
  • 81. One Slide on Software Defined Networking (SDN) Control and data planes are decoupled, network intelligence and state are logically centralized, and the underlying network infrastructure is abstracted from the apps IEEE ICME Tutorial - July 2018 81
  • 82. SDN-Based Bitrate Adaptation Reading: “SDNDASH: improving QoE of HTTP adaptive streaming using software defined networking,” ACM MM 2016 IEEE ICME Tutorial - July 2018 82
  • 83. Server and Network-Assisted Adaptation • A control plane approach that offers asynchronous client-to-network, network-to-client and network-to-network communications – Collect metrics and status information from various entities in the system – Share this information to assist the clients and aware elements to improve media delivery • DASH part 5 (23009-5) outlines the server and network-assisted DASH (SAND) framework along with the necessary interfaces IEEE ICME Tutorial - July 2018 83
  • 84. Conflicting Goals in the Media Delivery Chain • I want to make sure that my content is protected and looks awesome on any device • I want to make sure that my ads are viewed, trackable and measurable • I want to make sure that my servers are properly used and latency is low • I want to control the QoE of my customers, make $ from and differentiate my own video services • I want to make sure that my device/app provides the best possible video quality • I want the best quality for minimal $ ConsumerContent Provider ISP CDN ProviderAdvertiser Device/ App IEEE ICME Tutorial - July 2018 84
  • 85. Introducing DASH-Assisting Network Elements (DANE) A Control Plane Approach Origin (HTTP) Server Encoder/ Transcoder Packager CDN RG Clients Media Parameters for enhancing reception (PER) Metrics and status messages Parameters for enhancing delivery (PED) Analytics Server IEEE ICME Tutorial - July 2018 85
  • 86. Why is Latency Important? IEEE ICME Tutorial - July 2018 86
  • 87. Contributors to the Latency • Video encoding pipeline duration • Ingest and packaging operations • Network propagation delays • CDN buffer delays • Media segment duration • Player behavior – Buffering – Playhead positioning – Resilience Current Implementations • Smooth Streaming: 2-second segments, usually 10 seconds of latency • DASH: 2-second segments supported by most players, usually 8 to 10 seconds of latency • HLS – Until mid-2016: 10-second segments, usually 30 seconds of latency – Since mid-2016: 6-second segments, usually 18 to 20 seconds of latency – Safari Mobile in iOS11: Autostart for live streams and support for short segments – App Store & iOS applications: 6-second segments are still mandatory Latency in One Slide IEEE ICME Tutorial - July 2018 87
  • 88. Stream Start Time ≠ Latency Time Live encoder producing 2-second segments iOS (3 segments) Last available Lowest latency 1 2 3 4 Start Now 2 3 4 4 5 Latency: 7 s Latency: 3 s Latency: 2 s 6 seconds of buffer ~0 seconds of startup* 2 seconds of buffer ~0 seconds of startup* 2 seconds of buffer 1 second of startup* Segment fetching time is assumed to be negligible in this example 5 IEEE ICME Tutorial - July 2018 88
  • 89. Latency Facts • Stream start time: Time it takes for the video to start playing after pressing ‘play’ • Hand–waving (glass-to-glass) latency: Time delay between an action occurring in front of the camera and the same action being visible on the display • Stream start time is not equal to latency – A stream may take 5 s to start and only be 2 s behind live – Conversely, a stream can start in 2 s and be 5 s behind live • A player can start quickly by choosing to start behind the live point (which increases latency) and build its buffer by fetching segments faster than real time till the live edge • Low latency is always a trade-off against robustness of the playback IEEE ICME Tutorial - July 2018 89
  • 90. Part I: Streaming Overview of Streaming Standards (DASH, CMAF and Apple HLS)
  • 91. Dead, Surviving, Maturing and Newborn Technologies • Move Adaptive Stream (Long gone, but some components are in Slingbox) – http://www.movenetworks.com • Microsoft Smooth Streaming (Legacy) – http://www.iis.net/expand/SmoothStreaming • Adobe Flash (Almost dead) – http://www.adobe.com/products/flashplayer.html • Adobe HTTP Dynamic Streaming (Legacy) – http://www.adobe.com/products/httpdynamicstreaming • Apple HTTP Live Streaming (The elephant in the room) – https://tools.ietf.org/html/rfc8216 – https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis • MPEG DASH and CMAF (The standards) – http://mpeg.chiariglione.org/standards/mpeg-dash – http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format IEEE ICME Tutorial - July 2018 91
  • 92. • Fragmented architectures – Advertising, DRM, metadata, blackouts, etc. • Investing in more hardware and software – Increased CapEx and OpEx • Lack of consistent analytics • Preparing and delivering each asset in several incompatible formats – Higher storage and transport costs • Confusion due to the lack of skills to troubleshoot problems • Lack of common experience across devices for the same service – Tricks, captions, subtitles, ads, etc. What does Fragmentation Mean? Higher Costs Less Scalability Smaller Reach Frustration Skepticism Slow Adoption IEEE ICME Tutorial - July 2018 92
  • 93. DASH intends to be to the Internet world … what MPEG2-TS has been to the broadcast world IEEE ICME Tutorial - July 2018 93
  • 94. DASH intendsed to be to the Internet world … what MPEG2-TS has been to the broadcast world IEEE ICME Tutorial - July 2018 94
  • 95. Dynamic Adaptive Streaming over HTTP (DASH) • DASH is not – system, protocol, presentation, codec, interactivity, DRM, client specification • DASH is an enabler – It provides formats to enable efficient and high-quality delivery of streaming services – System definition is left to other organizations • Design choices – Reuse the existing technologies (containers, codecs, DRM, etc.) – Enable deployment on top of CDNs – Move intelligence from network to client, enable client differentiation – Provide simple interoperability points (profiles) IEEE ICME Tutorial - July 2018 95
  • 96. Shown in Red Scope of MPEG DASH HTTP Server DASH Client Control Engine MediaEngines HTTP ClientHTTP/1.1 Segment ParserMPD Transport MPD Parser MPD MPD MPD ... ... ... IEEE ICME Tutorial - July 2018 96
  • 97. Brief History of DASH at MPEG • Amd. 1: NTP sync, extended profiles • Amd. 2: SRD, URL parameter insertion, role extensions • Amd. 3: External MPD link, period continuity, generalized HTTP header extensions/queries • Amd. 4: TV profile, MPD chaining/resetting, data URLs in MPD, switching across adaptation sets • 3rd edition (FDIS) in ballot • Amd. 5 (WiP): Device information, quality equivalence descriptor, timed text roles, announcing popular content, flexible IOP signaling, early available periods, signaling missing/alternative segments 23009-1: Media Presentation Description and Segment Formats • 2nd edition was published in Oct. 2017 • Amd. 1 (WiP): SAND conformance rules 23009-2: Conformance and Reference Software • 3rd edition (WD) is in progress 23009-3: Implementation Guidelines (Informative) IEEE ICME Tutorial - July 2018 97
  • 98. Brief History of DASH at MPEG • 2nd edition (FDIS) in ballot 23009-4: Segment Encryption and Authentication • 1st edition, published in Feb. 2017 23009-5: Server and Network Assisted DASH (SAND) • 1st edition, published in Dec. 2017 23009-6: DASH over Full Duplex HTTP-Based Protocols (FDH) • 1st edition (WD) is work in progress 23009-7: Delivery of CMAF Contents with DASH (Informative) IEEE ICME Tutorial - July 2018 98
  • 99. • Describes various types of “push” behaviors in HTTP/2 and WebSockets • Primarily targeted for low-latency live streaming • Similar activities – DASH over LTE broadcast (3GPP SA4) – DVB ABR multicast – ATSC 3.0 hybrid delivery Part 6 – DASH over Full Duplex HTTP-Based Protocols IEEE ICME Tutorial - July 2018 99
  • 100. Ongoing Work as of MPEG 123 (July 2018) • Technologies under Consideration (w17812) – Usage of HEVC tile tracks in DASH – Annotation and client model for content selection – Signaling for quality control – Announcing popular content in DASH – Using segment templates for forensic watermarking – Using DASH MPD chaining for mid-roll ads – DASH playlist description – Mixed MPD – Event processing model – Patch method for MPD updates IEEE ICME Tutorial - July 2018 100
  • 101. Common Media Application Format (CMAF) • Media delivery has three main components: – Media format – Manifest – Delivery • CMAF defines the media format only (Fragments, headers, segments, chunks, tracks) – No manifest format or a delivery method is specified • CMAF uses ISO-BMFF and common encryption (CENC) – CENC means the media fragments can be decrypted/decoded by devices using different DRMs – CMAF does not mandate CTR or CBC mode • This makes CMAF useful only for unencrypted content for now, but industry is moving towards CBCS • Any delivery method may be used for delivering CMAF content – HTTP – RTP multicast/unicast – LTE broadcast IEEE ICME Tutorial - July 2018 101
  • 102. CMAF (ISO/IEC 23000-19) • CMAF defines media profiles for – Video – Audio – Subtitle • CMAF defines presentation profiles by selection a media profile from each category • Current status – 1st edition was published in Jan. 2018 – Supported in iOS 10+ (with HLS playlists) – Amd. 1: SHVC media profile and additional audio media profiles – Amd. 2: xHE-AAC and other media profiles IEEE ICME Tutorial - July 2018 102
  • 103. Each Frame can Be a CMAF Chunk CMAF ISO-BMFF Media Objects RAP … RAP … RAP … RAP … Fragment Fragment Fragment Fragment Segment Segment Track File … … … Encoding Packaging Encryption CMAF Header Seamless switching can only happen at fragment boundaries Manifests may provide URLs to • Track files • CMAF header + segments • CMAF header + chunks IEEE ICME Tutorial - July 2018 103
  • 105. ismc Smooth mpd DASH m3u8 HLS Source Encoder f4m HDS m3u8 m3u8 mpd mpd ismc ismc f4m f4m Credit: Akamai More packaging $ More storage $ Less efficient caching IEEE ICME Tutorial - July 2018 105
  • 106. Multi-Platform OTT Workflow Efficient Distribution in the CDN CMAF Source Encoder m3u8 mpd Credit: Akamai m3u8 mpd m3u8 mpd m3u8 mpd m3u8 mpd m3u8 mpd IEEE ICME Tutorial - July 2018 106
  • 107. Encode and Package Once, Deliver Efficiently Origin (HTTP) Server Encoder/ Transcoder LinearIngest On-demandContent Pick your favorite codec Pick your chunk and segment sizes CIF (SCTE) to client manifest translation withEBPmarkers Packager IEEE ICME Tutorial - July 2018 107
  • 108. Non-HTTP Transport Options RTP Multicast - Define RTP payload format for CMAF chunks/fragments/segments - Use RFC 4588 and 6285 for loss recovery and rapid acquisition Broadcast - TS 26.346 and TS 26.347: Delivering DASH content via MBMS - A/331:2017: Carrying DASH content over broadcast and/or broadband networks Peer-to-Peer - WebRTC: Clients fetch DASH/CMAF pieces from each other in addition to CDN servers - This helps reduce the load on the CDN as well as the stream start times IEEE ICME Tutorial - July 2018 108
  • 109. • Content format issues – Each asset is copied to multiple media formats • different video codecs • different audio codecs • different (regional) frame rates – Cost to content creators and distributors – Inefficiencies in CDNs and higher storage costs • Platform issues – Lack of consistent app behavior across platforms – Varying video features, APIs and semantics across platforms • Playback issues – Codec incompatibility – Partial profile support – Switching bitrate glitches – Audio discontinuities – Ad splicing problems – Long-term playback instability – Request protocol deficiencies – Memory problems, CPU weaknesses – Scaling (display) issues – Variable HDR support – Unknown capabilities Commercial OTT Issues IEEE ICME Tutorial - July 2018 109
  • 110. CTA Web Application Video Ecosystem (WAVE) Steering Committee Technical Working Group Content Specification Task Force Content specification based upcoming CMAF, compatible with DASH and HLS Device Playback Capabilities Task Force Testable requirements covering the most common device playback interoperability issues HTML5 API Task Force Reference application framework based on HTML5 providing functional guidelines for playback interop IEEE ICME Tutorial - July 2018 110
  • 111. Web Media API Snapshot 2017 • First annual API Snapshot published in Dec. 2017 – Lists key APIs supported in 2017 in all major HTML code bases – CTA-W3C agreement to co-publish this spec – Plan to propose Community Group spec as a W3C standards-track spec – Available at https://www.w3.org/2017/12/webmediaapi.html • CTA WAVE issued RFP to create a test suite for all listed APIs based on W3C API tests – Test suite will enable manufacturers to test that their HTML support is up-to-date • Web Media Application Developer Guidelines – Companion guide to outline best practices for implementing Web media apps – https://w3c.github.io/webmediaguidelines/ IEEE ICME Tutorial - July 2018 111
  • 112. WAVE Current and Future Publications • Published – “Web Media API Snapshot 2017”, Final Community Group Report, Dec. 2017, https://www.w3.org/2017/12/webmediaapi.html – “Web Application Video Ecosystem – Content Specification”, Apr. 2018, https://members.cta.tech/ctaWAVE • Pending – Event Messages in WAVE (white paper) – Web Application Video Ecosystem (WAVE) Device Playback Capabilities – Web Media Application Developer Guidelines – Web Media Porting Specification IEEE ICME Tutorial - July 2018 112
  • 113. Part I: Streaming Adaptive Delivery of Omnidirectional/360° Video
  • 116. MPEG’s Definition What is Virtual Reality (VR) VR is a rendered environment (visual and acoustic, pre- dominantly real-world) providing an immersive experience to a user who can interact with it in a seemingly real or physical way using special electronic equipment IEEE ICME Tutorial - July 2018 116
  • 117. Virtual Reality (VR) puts us in a Virtual World Source: Phil Chou IEEE ICME Tutorial - July 2018 117
  • 118. Augmented Reality (AR) puts Virtual Objects in our World Source: Phil Chou IEEE ICME Tutorial - July 2018 118
  • 119. Delivering High-Quality VR in Economic Scale is Extremely Challenging The VR Challenge 30K x 24K x 36 x 60 x 2 / 600 ~5.2 Gbps Resolution Bit depth Frame rate Stereoscopic Compression gain IEEE ICME Tutorial - July 2018 119
  • 120. Ultimate Level of Immersion Interactions So intuitive that they become second nature Sounds So accurate that they are true to life Visuals So vibrant that they are eventually indistinguishable from the real world IEEE ICME Tutorial - July 2018 120
  • 121. Visual quality Sound quality Intuitive interactions Immersive VR Has Extreme Requirements Immersion High resolution audio Up to human hearing capabilities 3D audio Realistic 3D, positional, surround audio that is accurate to the real world Precise motion tracking Accurate on-device motion tracking Minimal latency Minimized system latency to remove perceptible lag Natural user interfaces Seamlessly interact with VR using natural movements, free from wires Extreme pixel quantity and quality Screen is very close to the eyes Spherical view Look anywhere with a full 360° spherical view Stereoscopic display Humans see in 3D Source: Thomas Stockhammer IEEE ICME Tutorial - July 2018 121
  • 122. Need for Higher Resolutions on Mobile Phones? Source: Thomas Stockhammer IEEE ICME Tutorial - July 2018 122
  • 123. Omnidirectional/360° Video Capturing Devices • Stitching, projection formats • Encoding, encryption, encapsulation • Storage, content distribution, delivery • Processing, decoding, rendering Consumer Devices IEEE ICME Tutorial - July 2018 123
  • 124. Adaptive Streaming of VR Content • Required resolution of a panoramic video for achieving 4K resolution for viewport with 120° field of view (FoV) IEEE ICME Tutorial - July 2018 124
  • 125. Functional Architecture Encode, Encrypt, Encapsulate Decode, Decrypt, Decapsulate Project & Render Fuse, Stitch & Edit ① ② ③ ④ ⑤ Store & Deliver Capture Consume Content Creation NetworkServer Client Encoding Encryption Encapsulation Storage Delivery Decryption Decapsulation Decoding RenderingEditing Processing Processing Capture Acquisition Consumption Distribution Adaptive Delivery of Omnidirectional Video Interaction From ecosystem… To building blocks… IEEE ICME Tutorial - July 2018 125
  • 126. Functional Architecture Encode, Encrypt, Encapsulate Decode, Decrypt, Decapsulate Project & Render Fuse, Stitch & Edit ① ② ③ ④ ⑤ Store & Deliver Capture Consume Content Creation NetworkServer Client Encoding Encryption Encapsulation Storage Delivery Decryption Decapsulation Decoding RenderingEditing Processing Processing Capture Acquisition Consumption Distribution Adaptive Delivery of Omnidirectional Video Interaction From ecosystem… To building blocks… IEEE ICME Tutorial - July 2018 126
  • 127. Adaptive Streaming Options (1) • Traditional, viewport-agnostic streaming – ERP – handle like reg. video – Simple, easy, deployed today – Bandwidth waste, quality issues • Viewport-adaptive streaming – Multiple versions for predefined viewports – Various projection techniques (pyramid) – Bandwidth waste reduced, increased storage and CDN costs, limited flexibility X. Corbillon, et al., "Viewport-adaptive navigable 360-degree video delivery," 2017 IEEE International Conference on Communications (ICC), Paris, 2017, https://doi.org/10.1109/ICC.2017.7996611 IEEE ICME Tutorial - July 2018 127
  • 128. Adaptive Streaming Options (2) • Tile-based streaming – Use tiling technique of modern video codecs – High complexity, full flexibility – Multiple challenges M. Graf, et al. 2017. Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation. Proc. ACM MMSys'17. https://doi.org/10.1145/3083187.3084016 C. Concolato, et al., "Adaptive Streaming of HEVC Tiled Videos using MPEG-DASH," IEEE TCSVT, 2017. https://doi.org/10.1109/TCSVT.2017.2688491 IEEE ICME Tutorial - July 2018 128
  • 129. System Overview Tile-Based Adaptive Streaming Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Encoding & Packaging Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 DifferentQ uality Representations Delivery MPEG-HEVC/H.265 Tiles in ISOBMFF Adaptive Player Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile-based streaming of VR/360° content with MPEG-DASH SRD / OMAF … … … … Adaptive Streaming using MPEG-DASH SRD / OMAF Head Mounted Displays Browsers, Smart (Mobile) Devices (Stereo) 2D, (Stereo) 3D IEEE ICME Tutorial - July 2018 129
  • 130. Encoding Options • AVC dominates the market • HEVC, VP9, AV1 support tiles – Divides a picture into independent, rectangular regions – Tradeoff: bitrate, quality, flexibility • Multiple tiling options available – Uniform vs. non-uniform tiling – Same vs. mixed resolutions • New quality metrics, mostly based on PSNR but subjective quality assessments/metrics increasing https://bitmovin.com/2017-video-developer-report/ I. D.D. Curcio, et al. 2017. Bandwidth Reduction of Omnidirectional Viewport-Dependent Video Streaming via Subjective Quality Assessment. Proc. AltMM'17. https://doi.org/10.1145/3132361.3132364 A. Zare, et al. 2016. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications. Proc. ACM MM'16. http://dx.doi.org/10.1145/2964284.2967292 IEEE ICME Tutorial - July 2018 130
  • 131. Quality Metrics: S-PSNR M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 31–36. DOI:http://dx.doi.org/10.1109/ISMAR.2015.12 IEEE ICME Tutorial - July 2018 131
  • 132. Quality Metrics: V-PSNR M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 31–36. DOI:http://dx.doi.org/10.1109/ISMAR.2015.12 IEEE ICME Tutorial - July 2018 132
  • 133. Dataset • Segment length / intra period – 1s (tiled content) vs 1, 2, 4s (monolithic content) • Tiling pattern (columns × rows): 1×1, (i.e., tiles monolithic), 3×2, 5×3, 6×4, and 8×5 • Resolution: 1920×960, 3840×1920 and 7680×3840 • Map projection: equirectangular format • Quantization parameter: QP={22,27,32,37,42} • Head motion recordings for V-PSNR evaluation IEEE ICME Tutorial - July 2018 133
  • 134. Bitrate Overhead due to Tiling 500 1000 1500 2000 2500 384042444648 Bitrate [kbps] Y−PSNR[dB] ● ● ● ● ● ● ● 1x1 tiles 3x2 tiles 5x3 tiles 6x4 tiles 8x5 tiles Tile Overhead for Resolution: 1920x960 Sequence: AssassinsCreed IEEE ICME Tutorial - July 2018 134
  • 135. Bitrate Overhead due to Tiling 2000 4000 6000 8000 30323436384042 Bitrate [kbps] Y−PSNR[dB] ● ● ● ● ● ● ● 1x1 tiles 3x2 tiles 5x3 tiles 6x4 tiles 8x5 tiles Tile Overhead for Resolution: 1920x960 Sequence: ExploreTheWorld IEEE ICME Tutorial - July 2018 135
  • 136. Bandwidth Requirements IEEE ICME Tutorial - July 2018 136
  • 137. Adaptive Streaming Issues for (Tiled) 360° Video • Increasing number of segment/tile requests – HTTP/2 server push, query parameters, proprietary protocols – Additional functionality at server – breaks fundamental HAS requirements • Low latency streaming (see also next slide) – Reducing segment size impacts coding efficiency (1s vs. 4s) – CMAF chunks + other enhancements to enable sub-second latency – Remember: live internet video will grow 15-fold from 2016 to 2021 • Viewport prediction – Allows prefetching (caching) but cannot predict to much into future (1s) – Impact on segment size but situation will get better the more data is available – machine learning/AI will help • QoE (see also following slide) – Still in its infancy but situation much better than one year ago – Requires datasets, subjective studies, quality models, metrics M. Xu, et al., "A subjective visual quality assessment method of panoramic videos," Proc. ICME’17. https://doi.org/10.1109/ICME.2017.8019351 R. Schatz, et al., "Towards subjective quality of experience assessment for omnidirectional video streaming," Proc. QoMEX’17, https://doi.org/10.1109/QoMEX.2017.7965657 Y. Rai, at al. 2017. A Dataset of Head and Eye Movements for 360 Degree Images. Proc. ACM MMSys’17. https://doi.org/10.1145/3083187.3083218 S. Petrangeli, et al. 2017. An HTTP/2-Based Adaptive Streaming Framework for 360° Virtual Reality Videos. Proc. ACM MM'17. https://doi.org/10.1145/3123266.3123453 N. Bouzakaria, et al., "Overhead and performance of low latency live streaming using MPEG-DASH," Proc. IISA’14. https://doi.org/10.1109/IISA.2014.6878732 C.-L. Fan, et al. 2017. Fixation Prediction for 360° Video Streaming in Head- Mounted Virtual Reality. Proc. NOSSDAV'17. https://doi.org/10.1145/3083165.3083180 IEEE ICME Tutorial - July 2018 137
  • 138. Lag Prevents Immersion and Causes Discomfort Minimizing Motion-to Photon Latency is Important Low latency Noticeable latency IEEE ICME Tutorial - July 2018 138
  • 139. Motivation QoE of Omnidirectional Video • QoE of Omnidirectional Video (OV, alt.: 360º Movie) Streaming consumed via HMD – Immersive panorama video (or image) surrounding the user – User can turn head to changes gaze direction • OV streaming = content class with dramatically increasing popularity – Cisco VNI: globally, virtual–reality traffic will increase 61-fold between 2015 and 2020 Immersive Media VR Responsive VR OV MR AR R. Schatz, et al., "Towards subjective quality of experience assessment for omnidirectional video streaming," Proc. QoMEX’17, https://doi.org/10.1109/QoMEX.2017.7965657 IEEE ICME Tutorial - July 2018 139
  • 140. Motivation QoE of Omnidirectional Video • Gaps in current state of the art – Lots of knowledge on 2D & 3D Video quality – Lots of knowledge on human factors in responsive AR/VR – BUT: lack of quantitative models and subj. testing methods for OV streaming Particularly as regards quality perception and the Impact of stalling on OV QoE QoE = m * c2 QoE = m * c2 ? IEEE ICME Tutorial - July 2018 140
  • 141. IEEE ICME Tutorial - July 2018 141 Research Questions RQ1: How does stalling impact the QoE of HMD-based omnidirectional video (OV)? RQ2: What are QoE impact differences between HMD-based OV and traditional TV-based video consumption with respect to stalling? RQ3: What is the impact of different content motion levels on OV stalling perception?
  • 142. IEEE ICME Tutorial - July 2018 142 Method: Subjective Lab Study • Test task: watch short video clips (duration 60s) impaired by stalling • Two Devices / Watching Modes: – 2D video on large TV screen (65”, Full HD) – Stereoscopic OV on an Oculus Rift DK2 HMD (with headphones and 3-axis orientation sensor) vs
  • 143. IEEE ICME Tutorial - July 2018 143 Test Subjects • Recruited from Vienna City region, received monetary incentives • Recruited: 27 participants, but … … removed 5 subjects due to vision problems, lack of HMD fit and cybersickness à N=22 – 12 male, 10 female – 4 had some limited VR experience – Age: mean 37, median 31.5, min 22, max 65 years
  • 144. IEEE ICME Tutorial - July 2018 144 Test Content (SRCs) • 2D / TV – Sintel movie (© Blender Foundation) – One 60s clip • OV / HMD – Vulkane – Terra X (© ZDF) – Two 60s clips extracted • Both feature: – Animation – Storyline
  • 145. Test Setup IEEE ICME Tutorial - July 2018 145
  • 146. Impairments (HRCs) and Test Conditions (PVSs) • PVS: – 2D/TV: 6 HRCs (Sintel movie) (+ 1 Training condition) – OV/HMD: 6 HRCs x 2 SRCs (static camera, moving camera) (1 Training condition) • Post-stimulus electronic questionnaire with 4/7 (TV/HMD) rating questions related to quality, immersion, etc. IEEE ICME Tutorial - July 2018 146
  • 147. OV move: Four users could not complete the OV move conditions, i.e. N=18 How Annoying were Interruptions (If Any) of the Video Playback? Results: Impact of Stalling on User Annoyance Imperceptible Very annoying IEEE ICME Tutorial - July 2018 147
  • 148. I Was Completely Captivated by the Virtual Environment Impact of Stalling on Perceived Involvement (IPQ INV4*) Fully agree Full disagree * from Igroup Presence Questionnaire (IPQ) IEEE ICME Tutorial - July 2018 148
  • 149. Results / Findings • RQ1 (stalling QoE impact): – Strong impact of stalling number and duration (similar to previous results) – Existing stalling models can be reused • RQ2 (differences 2D/OV): – No significant differences between TV/2D and HMD/OV (à unexpected) • RQ3 (impact of OV content motion): – No significant differences between moving and static camera, just some trends (à unexpected) • In addition: – Splits yield some differences in rating variance and stalling detection Influence on variance: sex, prior vr experience Influence on stalling detection: commented on HMD resolution IEEE ICME Tutorial - July 2018 149
  • 150. AdViSE • Adaptive Media Content [DASH, HLS, CMAF] • Players/Algorithms • Network Parameters Impaired Media Sequences Generate Impaired Media Sequences Templates [Startup Delay, Stalling, …] WESP QoE Evaluation Parameters [Questionnaire, Methodology, Crowdsourcing Platform, …] QoS/QoE Metrics Subjective Results + Other Data Reports Analysis AdViSE: Anatoliy Zabrovskiy, Evgeny Kuzmin, Evgeny Petrov, Christian Timmerer, and Christopher Mueller. 2017. AdViSE: Adaptive Video Streaming Evaluation Framework for the Automated Testing of Media Players. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM, New York, NY, USA, 217-220. DOI: https://doi.org/10.1145/3083187.3083221 WESP: Benjamin Rainer, Markus Waltl, Christian Timmerer, A Web based Subjective Evaluation Platform, In Proceedings of the 5th International Workshop on Quality of Multimedia Experience (QoMEX'13) (Christian Timmerer, Patrick Le Callet, Martin Varela, Stefan Winkler, Tiago H Falk, eds.), IEEE, Los Alamitos, CA, USA, pp. 24-25, 2013. https://doi.org/10.1109/QoMEX.2013.6603196 Objective and Subjective Evaluation Platform How to Evaluate HAS Systems? ① ② ③④ ⑤ Log of Segment Requests IEEE ICME Tutorial - July 2018 150
  • 151. AdViSE: Adaptive Video Streaming Evaluation • Scalable, end-to-end HAS evaluation through emulation w/ a plethora of – content configurations – players/algorithms (including for player competition) – network parameters/traces • Real content and network settings with real dynamic, adaptive streaming including rendering! • Collection of various metrics from players: API or directly from the algorithms/HTML5 • Derived metrics and utilize QoE models proposed in the literature • Segment request log to generate impaired media sequence as perceived by end users for subjective quality testing IEEE ICME Tutorial - July 2018 151
  • 152. WESP: Web-Based Subjective Evaluation Platform • Subjective quality assessments (SQA) are vital tool, reliable results, but cost-intensive • Utilize crowdsourcing to reduce costs • WESP – Enables easy and simple configuration of SQAs including possible integration of third-party tools for online surveys – Provides means to conduct SQAs using the existing crowdsourcing platforms taking into account best practice – Allows for the analysis of the results IEEE ICME Tutorial - July 2018 152
  • 153. C. Timmerer, A. Zabrovskiy, and Ali C. Begen, Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems, Proceedings of the 1st IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR), Miami, FL, USA, April 2018. Example Evaluation Results • Test sequence encoded 15 different representation (Amazon Prime configuration: 400x224@100Kbps – 1920x1080@15Mbps) with 4s segment length • Bandwidth trajectory based on prior work proposed in literature; network delay 70 ms IEEE ICME Tutorial - July 2018 153
  • 154. Bandwidth Index vs. QoE • Bandwidth index – Average bitrate – Efficiency – Stability • QoE model – Startup delay – Stalls C. Timmerer, A. Zabrovskiy, E. Kuzmin, E. Petrov, Quality of experience of commercially deployed adaptive media players, In 2017 21st Conference of Open Innovations Association (FRUCT) (Sergey Balandin, ed.), pp. 330-335, 2017. https://doi.org/10.23919/FRUCT.2017.8250200 IEEE ICME Tutorial - July 2018 154
  • 155. Part I: Streaming The Developing MPEG-OMAF and MPEG-I Standards
  • 156. Standardization Overview C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017. https://doi.org/10.1109/MCOMSTD.2017.1700038 Data Representations and Formats JPEG MPEG IEEE System Standards and APIs VR-IF 3GPP DVB Khronos W3C Quality of Experience QUALINET ITU-T VQEG Guidelines DASH-IF CTASVA SMPTE ETSI https://multimediacommunication.blogspot.com/2017/04/vr360-streaming-standardization-related.html IETF/IRTF IEEE ICME Tutorial - July 2018 156
  • 157. Standardization Overview C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017. https://doi.org/10.1109/MCOMSTD.2017.1700038 Data Representations and Formats JPEG MPEG IEEE System Standards and APIs VR-IF 3GPP DVB Khronos W3C Quality of Experience QUALINET ITU-T VQEG Guidelines DASH-IF CTASVA SMPTE ETSI https://multimediacommunication.blogspot.com/2017/04/vr360-streaming-standardization-related.html IETF/IRTF IEEE ICME Tutorial - July 2018 157
  • 158. Coded Representation of Immersive Media MPEG-I (ISO/IEC 23090) • Part 1: Architectures • Part 2: Omnidirectional media format (OMAF) • Part 3: Versatile Video Coding (VVC) • Part 4: Immersive audio • Part 5: Point cloud compression • Part 6: Immersive media metrics • Part 7: Immersive media metadata • Part 8: Network-based media processing (NBMP) IEEE ICME Tutorial - July 2018 158
  • 159. Scope Omnidirectional Media Format (OMAF) • 360° video, images, audio and associated timed text – 3DoF only • Specifies – A coordinate system • Consists of a unit sphere and the x (back-to-front), the y (lateral, side-to-side), and the z (vertical, up) axes – Projection and rectangular region-wise packing methods • Used for conversion of a spherical video/image into a two-dimensional rectangular video/image – The sphere signal is the result of stitching of video signals captured by multiple cameras – A special case: fisheye video – Storage of omnidirectional media and the associated metadata using ISO-BMFF – Encapsulation, signaling and streaming of omnidirectional media in DASH and MMT – Media profiles and presentation profiles • Provide interoperable and conformance points for media codecs as well as media coding and encapsulation configurations • Provides some informative viewport-dependent 360° video processing approaches IEEE ICME Tutorial - July 2018 159
  • 160. A: Real-world scene B: Multiple-sensors-captured video or audio D/D’: Projected/packed video E/E’: Coded video or audio bitstream F/F’: ISOBMFF file/segment OMAF player Acquisition Audio encoding Video encoding Image encoding File/segment encapsulation Delivery File/segment decapsulation Audio decoding Video decoding Image decoding Image rendering Audio rendering Loudspeakers/ headphones Display Head/eye tracking Fileplayback Orientation/ viewport metadata Orientation/viewport metadata Metadata Metadata A Ba Bi D Ea Ei Ev Fs F F’ F’s E’a E’v E’iD’ B’a A’i A’a Image stitching, rotation, projection, and region-wise packing https://mpeg.chiariglione.org/standards/mpeg-i/omnidirectional-media-format R. Skupin, et al., "Standardization Status of 360 degree Video Coding and Delivery,” Proc. IEEE VCIP’17 OMAFArchitecture
  • 161. DASH/OMAF System Diagram Multi- camera capture Stitch Project HEVC encode OMAF composing DASH server DASH client HEVC Decode Renderer Head mounted device (HMD) DASH encapsulation Head and eye tracking telemetry Request for tiles within FoV window Projection mapping metadata Projection mapping metadata Authoring CDN VR player IEEE ICME Tutorial - July 2018 161
  • 162. MPEG- 1,2,4,H,I MPEG-7 MPEG-21 MPEG-A MPEG- B,C,D, DASH MPEG- E,M MPEG- U,V Compression of video, audio and 3DG Description of video, audio and multimedia for content search Technologies for content e-commerce Multimedia application formats (combinations of content formats) Systems, video, audio and transport Multimedia platform technologies Device and application interfaces https://mpeg.chiariglione.org/meetings/123 MPEG’s Areas of Activity IEEE ICME Tutorial - July 2018 162
  • 163. 1990 2000 20101995 2005 2015 MP3 MPEG-2 Video & Systems AVC HEVC MP4 FF MMTDASH AAC Internet Audio Digital Television Music Distribution Media Storage and Streaming UHD & Immersive Services New Forms of Digital Television Unified Media Streaming OFF HD DistributionMobile Video Custom Fonts on Web & DigiPub MPEG-4 Video CMAF 3D Audio 2020 MajorMPEG Standards
  • 164. 6DoF Application Format? Genome Compression v2 PCC Extensions? OMAF v2 2018 20202017 2019 2021 IoMT Media Orchestration Descriptors for Video Analysis (CDVA) 6DoF Audio Point Cloud Compression OMAF v1 Genome Compression Network-based Media Processing Coding 2022 Scene Description for Immersive Media Versatile Video Coding 2023 MIAF Systems and ToolsWeb Resource Tracks Dense Representation of Light Field Video 3DoF+ Video MPEGRoadmap
  • 165. Part II: Communications over 5G 5G Fundamentals: Core Network, Radio Access Network and QoS
  • 166. 5G fundamentals: Core Network, Radio Access Network and QoS ICME 2018 Tutorial Liangping Ma July 23, 2018 © 2018 InterDigital, Inc. All rights reserved. 166
  • 167. 167 Who is InterDigital? Invention, Collaboration, Contribution © 2018 InterDigital, Inc. All rights reserved. Berlin Montreal, QC (R&D) London (R&D) Seoul Buffalo, NY Melville, NY (R&D) Conshohocken, PA (R&D) Wilmington, DE (HQ) San Diego, CA (R&D) • Four decades of discovery and innovation in wireless • Pioneer in digital wireless technologies • Key contributions to global wireless standards • Inventing solutions for more efficient broadband networks • ~190 engineers (~80% of whom hold advanced degrees)
  • 168. 168 What is 5G? © 2018 InterDigital, Inc. All rights reserved.
  • 169. 169 5G Use Cases and Key Requirements Enhanced Mobile Broadband (eMBB) § Peak data rates: 20 Gbps (DL) and 10 Gbps (UL) § Peak spectral efficiency: 30 bps/Hz (DL) and 15 bps/Hz (UL) § 4 ms user plane latency § Indoor/hotspot and enhanced wide-area coverage Massive Machine Type Communications (mMTC) § Low data rates (1 to 100 kbps) § High device density (up to 1,000,000 /km2) § Latency: Seconds to hours § Low power: Up to 15 years battery life Ultra-Reliable and Low Latency Communications (URLLC) § Low to medium data rates (50 kbps to 10 Mbps) § 0.5 ms user plane latency § 99.999% reliability and availability within 1 ms § High mobility Key challenge for 5G design: support for different services having diverging requirements Source: ITU-R SG5 WP-5D © 2018 InterDigital, Inc. All rights reserved.
  • 170. 170 What Differentiates 5G from Previous Generations? Uses Cases & Services Voice + SMS Voice + Small Data Mobile Broadband Enhanced Mobile Broadband Massive Machine Type Communications Ultra Reliable and Low Latency Communications Spectrum 200 KHz Channels Below 2 GHz 5 MHz Channels Below 3.6 GHz Up to 20 MHz Channels Below 3.8 GHz Up to 1 GHz Channels Below 100 GHz Radio Technology GSM/GPRS (Single RAT ) UMTS/HSPA (Single RAT) LTE/LTE-A (Single RAT) Multiple Radio Access Technologies Integrated in a 5G Network: 5G New Radio, LTE Advanced Pro, NB-IoT, Wi-Fi, … Network Topology Macro Cells Macro and Small Cells Heterogeneous Macro and Small Cells Challenging Traditional Cell and Network Concepts: Small Cells, Mobile Edge Computing, Network Slicing, … 2G 4G 5G3G © 2018 InterDigital, Inc. All rights reserved.
  • 171. 2021 171 5G 3GPP Standardization Timeline 2017 2018 2019 2020 Rel-17 Work Items LTE Release 17LTE Release 15R14 NR SI Full NR RAN stand- alone and CT Specifications approved Rel-16 specification approved with some additional enhancements NR R15 first drop specification completed (NR non-standalone and CN standalone) Phased approach enables early commercial deployment of Phase 1 in 2019 and Phase 2 in 2022+ IMT-2020 Proposal Submission NR Rel-15 5G LTE + NB-IoT (required for mMTC use cases for IMT-2020) 5G New Radio (NR) … Allows for very early commercial implementation – No more hardware changes envisioned for R-15 Rel-16 SIs (RAN/SA) Rel-17 Study Items LTE Release 16 Additional Rel-16 SIs approved Rel-16 Work Items (R16) 3rd drop © 2018 InterDigital, Inc. All rights reserved.
  • 172. Focus on completion of the protocol design § NAS protocol between the UE and the CN § Internal Core Network interfaces based on the service based architecture (SBA) § Interconnection of the Core Network with external networks First set of NR specifications completed § Ready for Hardware Implementation and for first phase of deployment - Non-Standalone NR only (i.e. connected and controlled by LTE) § RAN1 PHY and RAN2 User Plane completed for stand-alone NR § Support for Enhanced Mobile Broadband use cases only and low latency 172 5G New Radio Status – Rel-15 Core Network and Non Access Stratum Radio Access Network Next Gen Core(NGC) architecture completed § New fully flexible architecture based on Network Slicing concept and further separation of control and user planes § Key functions: Session Management, Mobility Management, QoS and access agnostic CN Focus on completion stand-alone NR § Support for stand-alone NR connected directly to a 5G CN (RAN2/RAN3/RAN4) § Add/complete additional enhancements to support ultra-reliable and low latency use cases (URLLC) (RAN1/RAN2) December 2017 June 2018 © 2018 InterDigital, Inc. All rights reserved.
  • 173. 173 5G Design Principles © 2018 InterDigital, Inc. All rights reserved.
  • 174. 174 5G NR Design Principles Architecture & Network • Flexible, modular and scalable architecture • Support high level of programmability and automation in network • Built in support of network customization via slicing • Easy integration of new services 5G is designed to be flexible and adaptable to meet wide range of current and future requirements Radio Access Network • Minimize always-on signaling • Flexible frame structure and numerology • Support wide range of frequency bands • Built-in support for beamforming • Easy integration of future features © 2018 InterDigital, Inc. All rights reserved.
  • 175. 175 5G Technologies © 2018 InterDigital, Inc. All rights reserved.
  • 176. 176 3GPP 5G Deployment Scenarios 5G RAN LTE/4G Network 5G Network LTE Base Station New Radio RAN Internet Non Standalone (NSA) 5G 5G New Radio 4G EPC 5G Core Network Slice #3 (URLLC) Slice #2 (IoT) Slice #1 (eMBB) SGW MME PGW HSS AMF Non-Standalone (NSA) 5G enables large-scale trials and deployments starting in 2019 3 Main Deployment Scenarios § NSA deployment - utilizes LTE radio and 4G EPC as an anchor and NR Radio as hotspot § Allows earlier implementability § Stand-alone NR – 5G RAN connecting to 5G CN only § LTE radio connected to 5G CN network 5G CN designed to interwork with the current EPC network to enable seamless migration for the operators © 2018 InterDigital, Inc. All rights reserved.
  • 177. 177 Architecture options and specification timelines © 2018 InterDigital, Inc. All rights reserved. EPC NextGen Core LTE NR 3 Non-standalone Drop 1: Dec. 2017 NR 2 LTE 5 NextGen Core NextGen Core Standalone NR Drop 2: June 2018 EPC NextGen Core LTE NR 4 EPC NextGen Core LTE NR 7 LTE connected to 5G CN Drop 3: Dec 2018 – NR +
  • 178. 178 Control and User Plane Separation © 2018 InterDigital, Inc. All rights reserved. Control and User plane separation modularizes the 5G Core Network Design § Reduces Latency on application service § By relocating UPFs which are closer to the RAN; § Or by selecting UPFs more appropriate for the intended UE usage type without increasing the number of control plane nodes § Supports increase of Data Traffic, by enabling to add user plane nodes without changing the number of CP resources in the network § Independent evolution of the CP and UP functions § Enables SDN routing to deliver user plane data more efficiently 5G Core Network Architecture
  • 179. 179 Service Based Architecture (SBA) § 5G CN has been revolutionized with Service Based interfaces for core network interactions § CN interfaces have traditionally been Point-to-Point (P2P) e.g. MAP, Diameter § Network Functions (NFs) in 5G CN offer their services to other NFs via APIs based on web-based technology (e.g. HTTP). § Makes deployment easy as libraries, development tools, and other components are broadly available. § Flexible deployments of new services: possible to connect to other components without new specific interfaces § Network Repository Function (NRF) allows NFs to discover the services of other NFs. © 2018 InterDigital, Inc. All rights reserved. SBA shifts the paradigm from telecom-style protocol interfaces to web-based APIs UDMNSSF PCFNRF AUSF AF SMFAMF 5G CN Service Based Control Plane
  • 180. Network Slice • Logical end-to-end network • Tailored based on service it offers (e.g.eMBB, IoT, URLCCC etc.) Slicing offers customized network resources for different verticals © 2018 InterDigital, Inc. All rights reserved. 180 Network Slicing Benefits of slicing • Flexible and modular network deployment • Isolation between dedicated network resources • New products and services can be brought to market rapidly and adapted to fast-changing demands
  • 181. Virtual (Cloud) RAN Enablers ØStandards efforts to enable virtualization in multi- vendor environments • Central Unit / Distributed Unit split • CU/DU split below the PDCP • CU performs user management and user data processing • DU manages the radio and cell • Control Plane/User plane split • Central unit further split into virtual entities: • User data processing and user/mobility control • Support of lower layer split – no conclusion • Some of the radio control function performed by the central unit • Challenging due to ties with implementation Virtualization gives operators a flexible network design reducing cost 181© 2018 InterDigital, Inc. All rights reserved. CU DU Control Data gNB DU
  • 182. 182 Network Function Virtualization (NFV) © 2018 InterDigital, Inc. All rights reserved. Sources: Network Functions Virtualisation -- White Paper on NFV priorities for 5G, ETSI, Feb. 2017. Network Functions Virtualisation – Introductory White Paper, ETSI, Oct. 2012. NFV is not limited to 5G Use cases for 5G: - Virtualization of the core network - Virtualization of the RAN Decouple network functions from hardware
  • 183. 183 Numerology A numerology is defined by sub-carrier spacing and CP overhead ØOFDM waveform in both UL &DL • Optimized CP-OFDM with high spectrum utilization • Spectrum confinement techniques such as filtering and windowing • Simplifies design for D2D, Massive MIMO reciprocity & IAB. • DFT-s-FDM for coverage-limited scenarios in UL ØScalable Numerology • Subcarrier Spacing (SCS) scales with the BW (15#$% ∗ 2( ) • To support diverse spectrum bands and services • To avoid processing complexity associated with a large FFT size • Higher SCS -> Higher bands -> Shorter OFDM symbols • To address RF impairments in mmWave bands (e.g., Phase Noise), enable analog beamforming in mmWave bands and, lower latency • Lower SCS -> Larger cyclic prefix • More robust to the channel delay spread (e.g., Broadcast Service) • To support narrow band transmissions for mMTC services SCS [kHz] 15 30 60 120 240 OFDM Duration [µs] 66.67 33.33 16.67 8.33 4.17 CP Duration [µs] 4.69 2.34 1.17 0.58 0.29 Total Symbol [µs] 71.36 35.67 17.84 8.91 4.46 Slot Duration [µs] 1000 500 250 125 62.5 Band [GHz] < 6 < 6 < 6 < > 6 > 6 © 2018 InterDigital, Inc. All rights reserved. 15 kHz 30 kHz 60 kHz 120 kHz 50 MHz 400 MHz Max BW 100 MHz 200 MHz
  • 184. 184 Frame Structure Flexible frame structure adapting to various bands, duplexing mode and service requirements UL ULCtrl DL UL ULCtrl DL ULCtrl Downlink-Centric Slot ØFlexible Frame Structure • Flexible TTI Duration § Scalable TTI enables efficient service multiplexing o Symbols across different SCSs are aligned at the symbol boundaries § Shorter TTI to support low latency application (e.g., URLLC) o User can also be schedule on a fraction of a slot (i.e., Mini-slot) § Longer TTI to achieve higher spectral efficiency (e.g., eMBB) o User can be scheduled on multiple-slots • Flexible Scheduling § Both dynamic TDD and FDD are supported § OFDM symbols in a slot can be DL, UL or flexible § Flexible symbols can be used as gap or for forward compatibility Uplink-Centric Slot 1 30 kHz 120 kHz 0 0 20 12 13 1 2 12 13 1 2 60 kHz 15 kHz 1 ms slot with 14 OFDM symbols 500 µs 250 µs 125 µs OFDM symbol Mini-Slot Slot with load balancing © 2018 InterDigital, Inc. All rights reserved. Dynamic TDD
  • 185. 185 Bandwidth Adaptation Adapting UE’s receive and transmit bandwidth irrespective of gNB’s total supported bandwidth ØFlexible Bandwidth Allocation • NR supports a maximum BW of 400MHz per Component Carrier (CC) with up to 16 CC in Carrier Aggregation (CA) • Supporting bandwidth smaller than the CC bandwidth enabled by configuring a Bandwidth Part (BWP) • BWP Benefits include: § Enables UE power savings § Supports UE with lower maximum bandwidth capability § Single BPW is active at a time • Increases scheduling flexibility • Allow service multiplexing in frequency domain for use- cases with different numerology BWP1: 40 MHz and subcarrier spacing of 15 kHz BWP2: 10 MHz and subcarrier spacing of 15 kHz BWP3: 10 MHz and subcarrier spacing of 60 kHz © 2018 InterDigital, Inc. All rights reserved. Frequency Time BWP1 BWP2 BWP3 BWP2 BWP1 Carrier
  • 186. 186 Antenna © 2018 InterDigital, Inc. All rights reserved. • More bandwidth leads to higher throughput • Need to go to higher bands • But higher bands have larger path loss than lower bands Free space path loss – Friis equation !" = !$%$ %" & 4()* + Sources: W. L. Stutzman and G. A. Thiele, Antenna Theory and Design, Wiley, 2013. Example: When f increases from 2.8GHz to 28 GHz, the path loss increases by 20 dB. frequencyReceived power R Pt Pr Why? The maximum effective area (effective aperture) or “collecting area” of an antenna is related to the frequency e.g. for an ideal (short) dipole antenna, maximum effective area = , -./0 Beamforming to close the link at higher frequencies
  • 187. 187 Hybrid Beamforming © 2018 InterDigital, Inc. All rights reserved. • Analog beamforming: phase shifter • Digital beamforming: precoding matrix BB RF RF BB RF RF! = # $ + & where $ = +, = +-.+//, s +-. +// Phase shifter Tradeoff between cost and flexibility
  • 188. Massive MIMO Ø Conjugate Beamforming • Coherent combining at receiver Ø For TDD, no performance loss when the number of antennas is much larger than the number of users (M/K → ∞) • Leverage channel reciprocity • Result of Law of Large Numbers Ø Pilot contamination • In mobile scenarios the channel changes over time • Contamination occurs when two users use the same pilot Original Work by Thomas L. Marzetta (2006) © 2018 InterDigital, Inc. All rights reserved. k 188
  • 189. Massive MIMO Ø Enhanced CSI Feedback • Multi-resolution CSI • Low spatial resolution with a linear precoder targeting SU-MIMO operation (Type-I CSI) à low latency and overhead • High spatial resolution with a linear combination precoder targeting MU-MIMO operation (Type-II CSI) à better performance • Flexible CSI reporting timing • CSI reporting timing can be dynamically indicated to optimize latency based on CSI computational complexity • Hybrid beamforming support • Joint CSI reporting of beam index and precoding matrix index (PMI) Ø Enhanced UL Transmission • Non-Codebook based UL transmission • Introduced in NR to enable UL reciprocity functionality in TDD • UE determines its PUSCH precoder with some gNB assistance Work under 3GPP : DL control to trigger aperiodic CSI reporting : UL channel for aperiodic CSI reporting Low latency CSI High latency CSI Slot #n Slot #n+1 Slot #n+2 Slot #n+3 Slot #n+4 • gNB can dynamically indicate the CSI reporting timing © 2018 InterDigital, Inc. All rights reserved. 1) CSI-RS 3) DL Control 2) Sounding Signal 4) Precoded UL Data Precoder Calculation 189
  • 190. 190 Beam Management Ø Beam management for high frequency • Support of high frequency spectrums (up to 52.6GHz) in NR which is different from LTE • Beam management is used to maintain optimal Tx/Rx beam pairing for control and data channels Ø Beamformed Initial Access • Fundamental difference with LTE RACH lies on beamforming • Beam association between RACH and SS/PBCH blocks • UE selects an associated UL RACH resource of a selected DL SS block based on L1-RSRP measurements Management of analog beams at Tx/Rx is newly introduced in NR © 2018 InterDigital, Inc. All rights reserved.
  • 191. 191 Beam Failure Recovery Ø Recovery from beam failure • Beam-based system is susceptible to blockage, UE rotation, and mobility • Link fails when Tx-Rx beams for DL control channel are mismatched • RLF has been used in LTE when link fails but that requires a restart from initial access procedure à high latency • Beam recovery recovers beams for DL control channel • UE monitors quality of DL control channel beam and indicates new candidate beam using a dedicated PRACH resource when link failed • Beam recovery is used to recover the link before RLF à low latency A new mechanism introduced to handle frequent Tx/Rx beam mismatch Beam failure due to blockage New candidate beam PRACHfor beam1 PRACHfor beam2 PRACHfor beam3 UE sends PRACH resource associated with new candidate beam to trigger beam recovery © 2018 InterDigital, Inc. All rights reserved.
  • 192. Ultra-Reliable and Low-Latency Communications NR introduced several mechanisms to enable URLLC 192© 2018 InterDigital, Inc. All rights reserved. Low Latency DL Data ULCtrl Gap HARQ-ACK Slot • Self-contained slot Transmitting HARQ-ACK in the self- contained slot immediately after receiving the DL data • Mini-Slot Slot Mini-Slot Smaller time units for data scheduling • Scalable subcarrier spacing • Grant Free UL transmission • Very low periodicity SR • Pre-emption High Reliability • Diversity via duplication f1 f2 Same PDCP PDU on two carriers Carrier-Aggregation LTE NR Same PDCP PDU over different RAT (DC) Dual-Connectivity (EN-DC) • Control channel duplication • Polar coding for control information • Optimized CQI/MCS tables
  • 193. 193 Channel Coding: Data Channel Ø Enhanced LDPC Codes for eMBB Data Channel • Advantages of LDPC codes • Support high data rates (NR peak data rate: 20 Gbps DL/10 Gbps UL) • High area and energy efficiency and good BLER performance • Quasi-cyclic LDPC code enables high degree of parallelism • Parity-check matrix is lifted from small base graph • Two base graphs • Optimized for different regions of code rates and code block sizes • Length and rate flexibility • Shortening and lifting to achieve 1-bit granularity on code block size • Circular buffer with puncturing and repetition to achieve codeword length flexibility; • Support for both CC-HARQ and IR-HARQ ØFuture Enhancements • Potential modifications for URLLC data channel to address the error floor and achieve high reliability Channel coding was entirely redesigned for NR LTE: Turbo NR: LDPC Throughput Medium-low High Area efficiency Low High Energy efficiency Low High Implementation complexity comparison (R1-167567) Coverage regions of 2 LDPC base graphs © 2018 InterDigital, Inc. All rights reserved. R=1/4 R=2/3 R=0.95 308 3840 Code block size Code rate 8448 Base Graph 1 Base Graph 2
  • 194. Channel Coding: LDPC • Generator matrix • Parity check matrix © 2018 InterDigital, Inc. All rights reserved. 194 A primer on linear block codes and Tanner graph representation ! = #$$ #$% … #$' ⋮ ⋮ ⋮ ⋮ #)$ #)% … #)' = *$ ⋮ *) *$, *%, …, *) are linearly independent vectors of length , k information bits - = (/$, /%, …,/)) mapped to a ,-bit codeword: 1 = - ! = /$*$ + /%*%+ …+/)*) 3: (, − 6)×, matrix, each row is orthogonal to rows of ! Any codeword 1 satisEies: 13F = 0 • Tanner graph 3 = 1 0 1 0 0 1 1 1 x1+ x3 = 0 x2 + x3 + x4 = 0 x1 x2 x3 x4
  • 195. Channel Coding: LDPC • Binary Erasure Channel © 2018 InterDigital, Inc. All rights reserved. 195 An example of message passing decoding x1+ x3 = 0 x2 + x3 + x4 = 0 1 x2=? x3=? 0 1 0 1 1 1
  • 196. Channel Coding: LDPC • Parity check matrix H • Base graph (matrix of 0’s and 1’s) • Replace 1 with a cyclically shifted ZxZ identify matrix • Replace 0 with a ZxZ all-zero matrix Example: © 2018 InterDigital, Inc. All rights reserved. 196 1 1 0 1 1 0 0 1 1 0 1 0 1 0 1 0 0 1 3 2 N/A 2 1 N/A N/A 3 1 N/A 0 N/A 1 N/A 3 N/A N/A 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 0 1 0 0 0 0 1 0 Shift 3 positions 0 0 0 1 1 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 First row of base graph results in the first 4 rows of H: Source: 3GPP TR 38.802, Study on New Radio Access Technology -- Physical Layer Aspects V14.0.0 (2017-03) This construction is sometimes called lifting. Parity check matrix for high throughput
  • 197. 197 Channel Coding: Control Channel Ø Polar Codes for Control Channel and Broadcast Channel • Advantages of Polar codes • Good BLER performance at small block sizes and no error floor • Polar codes are used to encode DCI, UCI (>11 bits) and MIB • A single nested polar sequence • Short polar sequence is derived from long polar sequence • Optimized Rate Matching (RM) scheme • Select among three RM schemes: puncturing, shortening, repetition • Unified circular buffer design for all three RM schemes • Triangular channel interleaver (UCI only) • Advanced polar code construction • Distributed CRC for DCI and MIB to achieve early termination • Parity check polar code for small size UCI for better performance ØFuture Enhancements • Extension of polar code for URLLC data channel NR is the first commercial system adopting Polar codes Unified circular buffer design for three RM schemes Polar Encoder (Mother code length N) Sub- block 0 Sub- block 1 Sub- block 31 Sub- block 0 Sub- block 1 Sub- block 31 Sub-block wise interleaving Circular buffer Shortening Repetition Puncturing Selecting bits from the unified circular buffer Rate matching © 2018 InterDigital, Inc. All rights reserved.
  • 198. Channel Coding: Polar code • Invented by Erdal Arikan in 2007 • Based on the phenomena of channel polarization © 2018 InterDigital, Inc. All rights reserved. 198 W X Y 1. Binary Discrete Memoryless Channel (DMC) 2. Consider N copies of the DMC, and channel code X = GU W X1 Y1 W X2 Y2 W XN YN ... G U1 U2 UN ... ... Information bits codeword ! "#, "%, … , "'; )#, )%, … , )' = + ,-# ' ! ",; )#, )%, … , )' "#, … , ",.# = + ,-# ' !(",; )#, )%, … , )', "#, … , ",.#) 12,: ", → ()#, )%, … , )',"#, … , ",.#New channels Transform scalar channels into equivalent vector channels
  • 199. Channel Coding: Polar code • N=8 © 2018 InterDigital, Inc. All rights reserved. 199 Transform scalar channels into equivalent vector channels (Example) • N=2 • N=4 W W Y1 Y2 U1 U2 W W W W Y1 Y2 Y3 Y4 U1 U2 U3 U4 W W W W W W W W U1' U2' U3' U4' U1" U2" U3" U4" Y1 Y2 Y3 Y4 Y5 Y6 Y7 Y8 U1 U2 U3 U4 U5 U6 U7 U8
  • 200. QoS Provisioning • 4G: there are 9 pre-defined QoS Class Identifier (QCI) values © 2018 InterDigital, Inc. All rights reserved. 200 By a combination of QCI, ARP, GBR/Non-GBR Source: 3GPP TS 23.203, v11.2.0, “Policy and charging control architecture”, June, 2011. QCI Resource Type Priority Packet Delay Budget Packet Error Loss Rate Example Services 1 GBR 2 100 ms 10-2 Conversational Voice 2 4 150 ms 10-3 Conversational Video (Live Streaming) 3 3 50 ms 10-3 Real Time Gaming 4 5 300 ms 10-6 Non-Conversational Video (Buffered Streaming) 5 1 100 ms 10-6 IMS Signalling 6 Non-GBR 6 300 ms 10-6 Video (Buffered Streaming) TCP-based (e.g., www, e-mail, chat, ftp, p2p file sharing, progressive video, etc.) 7 7 100 ms 10-3 Voice, Video (Live Streaming) Interactive Gaming 8 8 300 ms 10-6 Video (Buffered Streaming) TCP-based (e.g., www, e-mail, chat, ftp, p2p file 9 9 sharing, progressive video, etc.) GBR: Guaranteed Bit Rate ARP: Allocation and Retention Priority
  • 201. QoS Provisioning • 5G: Added new QCI values © 2018 InterDigital, Inc. All rights reserved. 201 More flexible QoS differentiation and more efficient signaling Source: 3GPP TS 38.300, v15.3.0, “Policy and charging control architecture”, June, 2018. QCI Resource Type Priority Level Packet Delay Budget (NOTE 13) Packet Error Loss Rate (NOTE 2) Example Services 65 GBR 0.7 75 ms 10-2 Mission Critical user plane Push To Talk voice (e.g., MCPTT) 67 1.5 100 ms 10-3 Mission Critical Video user plane 75 2.5 50 ms 10-2 V2X messages 69 Non-GBR 0.5 60 ms 10-6 Mission Critical delay sensitive signalling (e.g., MC- PTT signalling, MC Video signalling) 70 5.5 200 ms 10-6 Mission Critical Data (e.g. example services are the same as QCI 6/8/9) 79 6.5 50 ms 10-2 V2X messages 80 6.8 10 ms 10-6 Low latency eMBB applications (TCP/UDP-based); Augmented Reality
  • 202. QoS Provisioning • 5G: Added new QCI values © 2018 InterDigital, Inc. All rights reserved. 202 More flexible QoS differentiation and more efficient signaling Source: 3GPP TS 38.300, v15.3.0, “Policy and charging control architecture”, June, 2018. QCI Resource Type Priority Level Packet Delay Budget Packet Error Loss Rate Maximum Burst Size Data Rate Averaging Window Example Services 82 GBR 1.9 10 ms 10-4 255 bytes 2 s Discrete Automation 83 GBR 2.2 10 ms 10-4 1358 bytes 2 s Discrete Automation 84 GBR 2.4 30 ms 10-5 1358 bytes 2 s Intelligent Transport Systems 85 GBR 2.1 5 ms 10-5 255 bytes 2 s Electricity Distribution- high voltage
  • 203. QoS Provisioning • 5G added SDAP (service data adaptation protocol) sublayer • 5G introduced reflective QoS • Derive uplink QoS from downlink QoS © 2018 InterDigital, Inc. All rights reserved. 203 More flexible QoS differentiation and more efficient signaling RByRBx IP Packet H SDAP SDU PDCP SDUH RLC SDUH MAC SDUH H IP Packet H SDAP SDU PDCP SDUH RLC SDUH MAC SDU IP Packet H SDAP SDU PDCP SDUH SDU SegmentH MAC SDU SDU SegmentH H H MAC SDU ... ... ... ... MAC PDU – Transport Block PDCP RLC SDAP MAC n n+1 m Source: 3GPP TS 38.300, v15.2.0, “NR; NR and NG-RAN Overall Description”, June, 2018. RB: Radio Bearer PDCP: Packet Data Convergence Protocol
  • 204. 204 On-going and Future Work © 2018 InterDigital, Inc. All rights reserved.
  • 205. Outlook for Release 16 Core Network © 2018 InterDigital, Inc. All rights reserved. 205 Release 16 aims to enable services intended to be supported by the 5G Core Network 205 5G Massive IoT (MIoT) § Enable MTC and Cellular IoT functionalities in the 5G Core Network § MTC related features from Rel-10 to Rel-14 (e.g. Congestion Control, Monitoring, Small Data Transfer, CP/UP Optimizations, Non-IP Data Delivery, etc…) shall be supported in 5GC § Enhancements the 5G system to address Massive IoT (MIoT) use cases based on the initial 5G requirements V2X services in 5G § System enhancements of EPS and 5G to support advanced V2X Use cases: § Platooning, extended sensor sharing, enhanced positioning accuracy, advanced driving and remote driving Access Traffic Steering Switch and Splitting § Multi-access (e.g. 3GPP and WLAN) PDU session management § Support traffic splitting between multiple accesses (e.g. 3GPP and WLAN) § Switch traffic between multiple accesses 5G LAN Services § Enhancements to the 5G system to support 5GLAN service § Support for 3GPP network that is not for public use and for which service continuity and roaming with a PLMN is possible (a type network). § Support an isolated 3GPP network that does not interact with a PLMN (b type network)