IEEE ICME Tutorial - Streaming and 5G Fundamentals

IEEE ICME Tutorial – San Diego, CA July 2018
Delivering Traditional and Omnidirectional Media
Ali C. Begen, Liangping Ma and Christian Timmerer
http://www.slideshare.net/christian.timmerer

Upon Attending This Tutorial, You Will Know About
• Principles of HTTP adaptive streaming for the Web/HTML5
• Principles of omnidirectional (360°) media delivery
• Content generation/distribution/consumption workflows for traditional and
omnidirectional media
• Standards and emerging technologies in the adaptive streaming space
• 5G radio access and core networks, technologies and emerging multimedia use cases
This tutorial is public, however, some material might be copyrighted
Use proper citation when using content from this tutorial
(Thanks to T. Stockhammer, J. Simmons, K. Hughes, C. Concolato, S. Pham, W. Law, N. Weil, P. Chou
and many others for helping with the material)
IEEE ICME Tutorial - July 2018 2

Ali C. Begen
• Electrical engineering degree from Bilkent
University (2001)
• Ph.D. degree from Georgia Tech (2006)
– Video delivery and multimedia communications
• Research, development and standards at
Cisco (2007-2015)
– IPTV, content delivery, software clients
– Transport and distribution over IP networks
– Enterprise video
• Consulting at Networked Media since 2016
• Assistant professor at OzU and IEEE
Distinguished Lecturer since 2016
Christian Timmerer
• Associate professor at the Institute of
Information Technology (ITEC), Multimedia
Communication Group (MMC), Alpen-Adria-
Universität Klagenfurt, Austria
• Co-founder and CIO of Bitmovin
• Research interests
– Immersive multimedia communication
– Streaming, adaptation, and
– Quality of experience (QoE)
• Blog: http://blog.timmerer.com; @timse7
Presenters Today – Part I

Liangping Ma
• Member of Technical Staff, InterDigital (2009-)
• Research Engineer, San Diego Research Center
(2005-2007), Argon ST (2007-2009)
• PhD, Electrical Engineering, University of
Delaware (2004)
• BS, Physics, Wuhan University, China (1998)
• IEEE ComSoc Distinguished Lecturer since 2017
• Research interests
– Video communication
– 5G radio access network
– Cognitive radios
Presenters Today – Part II

• Two tutorials, 30 talks on
– HEVC, VVC and AV1
– Containers
– HDR
– Next-gen audio
– The latest and greatest in streaming
– Content protection and more
• Cost: Free
• Visit http://mile-high.video/ to register
Next week in Denver, CO
Mile High Video 2018

ACM MMSys 2019 Call for Papers and Special Sessions
• City: Amherst, MA
• Date: June 18-21, 2019
• Submission deadline: Oct. 30
• Co-located with
– NOSSDAV
– MMVE
– Packet Video
• http://www.mmsys2019.org/

Topics to Cover
• Part I: Streaming (Morning)
– HTTP adaptive streaming and workflows
– HTML5 video and media extensions
– Common issues in scaling and multi-screen/hybrid delivery
– Overview of Streaming Standards (DASH, CMAF and Apple HLS)
– Adaptive delivery of omnidirectional/360° video
– The developing MPEG-OMAF and MPEG-I standards
• Part II: Communications over 5G (Afternoon)
– 5G fundamentals: core network, radio access network and QoS
– Multimedia communication case studies
– Improving QoE in streaming

IPTV vs. IP (Over-the-Top) Video
First Things First
IPTV IP Video
Best-effort delivery
Quality not guaranteed
Mostly on demand
Paid or ad-based service
Managed delivery
Emphasis on quality
Mostly linear TV
Always a paid service

Part I: Streaming
HTTP Adaptive Streaming and Workflows

Internet (IP aka OTT) Video Essentials
Reach all connected devicesReach
Enable live and on-demand delivery to the mass marketScale
Provide TV-like consistent rich viewer experienceQuality of Experience
Enable revenue generation thru paid content, subscriptions,
ads, etc.Business
Satisfy regulations such as captioning, ratings and parental
controlRegulatory

Creating Revenue – Attracting Eye Balls
• High-end content
– Hollywood movies, TV shows
– Sports
• Excellent quality
– HD/3D/UHD audiovisual presentation w/o artifacts such as pixelization and rebuffering
– Fast startup, fast zapping and low glass-to-glass delay
• Usability
– Navigation, content discovery, battery consumption, trick modes, social network integration
• Service flexibility
– Linear TV
– Time-shifted and on-demand services
• Reach
– Any device, any time

One Request, One Response
Progressive Download
HTTP Request
HTTP Response

What is Streaming?
Streaming is transmission of a continuous content from a server
to a client and its simultaneous consumption by the client
Two Main Characteristics
1. Client consumption rate may be limited by real-time constraints as opposed
to just bandwidth availability
2. Server transmission rate (loosely or tightly) matches to client consumption
rate

Video Delivery over HTTP
• Enables
playback while
still downloading
• Server sends the
file as fast as
possible
Progressive
Download
• Enables seeking
via media indexing
• Server paces
transmission based
on encoding rate
Pseudo
Streaming
• Content is divided
into short-duration
chunks
• Enables live
streaming and ad
insertion
Chunked
Streaming
• Multiple versions
of the content are
created
• Enables to adapt
to network and
device conditions
Adaptive
Streaming

Adaptive Streaming over HTTP
Decoding and
Presentation
Streaming Client
Media Buffer
Content Ingest
(Live or Pre-captured)
Multi-rate
Encoder
Packager Origin (HTTP)
Server
…
…
…
…
ServerStorage
HTTP GET
Request
Response

Adapt Video to Web Rather than Changing the Web
Adaptive Streaming over HTTP
• Imitation of streaming via short downloads
– Downloads small chunks to minimize bandwidth waste
– Enables to monitor consumption and track the streaming clients
• Adaptation to dynamic conditions and device capabilities
– Adapts to dynamic conditions in the Internet and home network
– Adapts to display resolution, CPU and memory resources of the streaming client
à Facilitates “any device, anywhere, anytime” paradigm
• Improved quality of experience (not always improved average quality)
– Enables faster start-up and seeking, and quicker buffer fills
– Reduces skips, freezes and stutters
• Use of HTTP
– Well-understood naming/addressing approach, and authentication/authorization infrastructure
– Provides easy traversal for all kinds of middleboxes (e.g., NATs, firewalls)
– Enables cloud access, leverages the existing (cheap) HTTP caching infrastructure

Stalls, Slow Start-Up, Plug-In and DRM Issues
Common Annoyances in Streaming
• Unsupported/wrong
– protocol
– plug-in
– codec
– format
– DRM
• Slow start-up
• Poor quality, quality variation
• Frequent freezes/glitches
• Lack of seeking

Dead, Surviving, Maturing and Newborn Technologies
• Move Adaptive Stream (Long gone, but some components are in Slingbox)
– http://www.movenetworks.com
• Microsoft Smooth Streaming (Legacy)
– http://www.iis.net/expand/SmoothStreaming
• Adobe Flash (Almost dead)
– http://www.adobe.com/products/flashplayer.html
• Adobe HTTP Dynamic Streaming (Legacy)
– http://www.adobe.com/products/httpdynamicstreaming
• Apple HTTP Live Streaming (The elephant in the room)
– https://tools.ietf.org/html/rfc8216
– https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis
• MPEG DASH and CMAF (The standards)
– http://mpeg.chiariglione.org/standards/mpeg-dash
– http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format

List of Accessible Segments and Their Timings
An Example DASH Template-Based Manifest
MPD
Period id = 1
start = 0 s
Period id = 3
start = 300 s
Period id = 4
start = 850 s
Period id = 2
start = 100 s
Adaptation Set 0
subtitle turkish
Adaptation Set 2
audio english
Adaptation Set 1
BaseURL=http://abr.rocks.com/
Representation 2
Rate = 1 Mbps
Representation 4
Rate = 3 Mbps
Representation 1
Rate = 500 Kbps
Representation 3
Rate = 2 Mbps
Resolution = 720p
Segment Info
Duration = 10 s
Template:
3/$Number$.mp4
Segment Access
Initialization Segment
http://abr.rocks.com/3/0.mp4
Media Segment 1
start = 0 s
Media Segment 2
start = 10 s
Adaptation Set 3
audio italian
Adaptation Set 1
video
Period id = 2
start = 100 s
Representation 3
Rate = 2 Mbps
Selection of
components/tracks
Well-defined
media format
Selection of
representations
Splicing of arbitrary
content like ads
Chunks with addresses
and timing

An Example HLS Playlist-Based Manifest
#EXTM3U
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=232370,CODECS="mp4a.40.2, avc1.4d4015"
gear1/prog_index.m3u8
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=649879,CODECS="mp4a.40.2, avc1.4d401e"
#EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=41457,CODECS="mp4a.40.2"
master.m3u8
Source: https://developer.apple.com/streaming/examples/ and https://www.gpac-licensing.com/2014/12/01/apple-hls-comparing-versions/
#EXTM3U
#EXT-X-TARGETDURATION:10
#EXT-X-VERSION:3
#EXT-X-MEDIA-SEQUENCE:0
#EXT-X-PLAYLIST-TYPE:VOD
#EXTINF:9.97667,
fileSequence0.ts
#EXTINF:9.97667,
fileSequence1.ts
#EXTINF:9.97667,
fileSequence2.ts
.
.
.
#EXT-X-ENDLIST

Example Representations
Encoding
Bitrate
Resolution
Rep. #1 3.45 Mbps 1280 x 720
Rep. #2 2.2 Mbps 960 x 540
Rep. #3 1.4 Mbps 960 x 540
Rep. #4 900 Kbps 512 x 288
Rep. #5 600 Kbps 512 x 288
Rep. #6 400 Kbps 340 x 192
Rep. #7 200 Kbps 340 x 192
Source: Vertigo MIX10, Alex Zambelli’s Streaming Media Blog, Akamai, Comcast
Vancouver 2010 Sochi 2014
Encoding
Bitrate
Resolution
Rep. #1 3.45 Mbps 1280 x 720
Rep. #2 1.95 Mbps 848 x 480
Rep. #3 1.25 Mbps 640 x 360
Rep. #4 900 Kbps 512 x 288
Rep. #5 600 Kbps 400 x 224
Rep. #6 400 Kbps 312 x 176
Encoding
Bitrate
Resolution
Rep. #1 18 Mbps 4K (60p)
Rep. #2 12.2 Mbps 2560x1440 (60p)
Rep. #3 4.7 Mbps 2K (60p)
Rep. #4 3.5 Mbps 1280x720 (60p)
Rep. #5 2 Mbps 1280 x 720
Rep. #6 1.2 Mbps 768 x 432
Rep. #7 750 Kbps 640 x 360
Rep. #8 500 Kbps 512 x 288
Rep. #9 300 Kbps 320 x 180
Rep. #10 200 Kbps 320 x 180
PyeongChang 2018

Smart and Selfish Clients
HAS Working Principle - Client fetches and parses the manifest
- Client uses the OS-provided HTTP stack
(HTTP may run over TCP or QUIC)
- Client uses the required decryption tools
for the protected content
Client monitors and measures
- Size of the playout buffer (both in bytes and seconds)
- Chunk download times and throughput
- Local resources (CPU, memory, window size, etc.)
- Dropped frames
Client performs adaptation
Request
Response
HTTPServer
Client
Client measures and reports metrics for analytics
(One can also multicast
media segments)

Tradeoffs in Adaptive Streaming
Overall quality
Quality stability
Proximity to live edge
Stalls
Zapping/seeking time

End-to-End Workflow for OTT
Production Preparation and Staging Distribution Consumption
News
Gathering
Sport Events
Premium
Content
Studio
Multi-bitrate
Encoding
Encapsulation
Protection
Origin Servers
VoD
Content &
Manifests
Live
Content &
Manifests
CDN

Part I: Streaming
HTML5 Video and Media Extensions

• HTML5 is a set of technologies that
allows more powerful Web sites and
applications
– Better semantics
– Better connectivity
– Offline and storage
– Multimedia
– 2D/3D graphics and effects
– Performance and integration
– Device access
– Styling
• Most interesting new elements
– Semantic elements: <header>, <footer>,
<article> and <section>
– Graphic elements: <svg> and <canvas>
– Multimedia elements: <audio> and <video>
A Common Platform across Devices
HTML5
Source: Mozilla MDN, w3schools.com

• These apps can
– leverage native APIs and offline capability
– provide cross-device compatibility (iOS,
Windows, Android, etc.)
– be packaged and published to app stores
• For streaming:
– To a user, it looks like a regular native app
they have downloaded
– The app is actually a container and loads
loads dynamically from the web server
– Server-side changes propagate
automatically to installed apps
Hosted and Progressive Web Apps
Source: Google PWA, Microsoft HWA

Types of Browser-Based Playback
Source: CTA WAVE
Type 1: Minimum control architectural model for HTML5 support of adaptive
streaming where manifest and heuristics are managed by the user agent
Type 2: Adaptation control architectural model for HTML5 support of
adaptive streaming providing script manageable features
Type 3: Full media control architectural model for HTML5 support of adaptive
streaming allowing script to explicitly send the media segments (MSE + EME)

MSE Support in Web Browsers (as of July’18)
Source: http://caniuse.com/#search=mse

Content Protection – Famous Quotes
Digital files cannot be made uncopiable,
any more than water can be made not wet
Bruce Schneier (cryptographer), 2001
We have Ph.D.'s here that know the stuff
cold, and we don't believe it's possible to
protect digital content
Steve Jobs, 2003
If we’re still talking about DRM in five
years, please take me out and shoot me
eMusic CEO David Pakman, 2007

W3C Encrypted Media Extensions
Source: https://www.w3.org/TR/encrypted-media/

EME Support in Web Browsers (as of July’18)
Source: http://caniuse.com/#search=eme

Web Video Ecosystem
Encoding
Encryption
Rights Expression
Web Video App Framework
Decoding
Decryption
Rights Management
The fundamental Digital Rights Management
problem derives from a lack of interoperability
which prevents mobility of experience
The solution is to combine interoperable
commercial Web video content with a cross-
platform Web video app framework
Source: John Simmons

Contrary to a common misconception, with EME DRM
functionality is not in the HTML/JS app
There is no DRM in HTML5 with EME, and ECP
requires that this be the case
The HTML/JS app selects the DRM and controls key
exchange between DRM client and server
Browser extends HTML5 media element to allow
JavaScript handled key acquisition
Browser
A CDM exposes a key system to JavaScript; it is
transparent whether the CDM is in the browser
The Web, EME and Enhanced Content Protection (ECP)
Source: John Simmons

DRM Support on Desktop Browsers
Browser OS EME/CDM Flash Player NPAPI/ Silverlight 5
Chrome Win Widevine (Yes) No
OS X (Yes) No
Linux (Yes) No
Firefox Win Widevine Yes No
OS X Yes No
Linux Yes No
Safari > OS X Yosemite (FairPlay) Yes Yes
< OS X Yosemite No Yes Yes
IE/Edge < Win 7 No Yes Yes
Win 10 PlayReady Yes No

DRM Support on Mobile Platforms
Platform Browser Native App / DRM
iOS No MSE/EME yet
HLS (AES-128 CBC) via <video>
Native SDK and WebView: FairPlay
Android MSE/EME Native + Android SDK (MediaDRM APIs) - (Widevine
+ OMA v2)
ExoPlayer
Native + WebView with Widevine
Windows 10 MSE/EME WebViews/Hosted Web Apps with PlayReady

• Server-side ad insertion (SSAI)
– Mostly used for linear/dynamic ad insertion (ads
are still targeted)
– Easier for less capable devices
– Smoother transition between the content and ads
– Uses XLink or MPD chaining in DASH for multi-
period MPDs
– Splices ads into a continuous stream (offline or
just-in-time) for single-period MPDs
• Client-side ad insertion
– Requires additional support in the application
– Uses events in DASH
Ad Insertion Methods
Preroll Idle Midroll Idle Postroll
Preroll Content Midroll Content Postroll Pause Content Pause Content Pause
Primary Stream Primary Stream
Ad Stream

Ad Insertion – XLink Resolution

Part I: Streaming
Common Issues in Scaling and Multi-Screen/Hybrid Delivery

Streaming over HTTP – The Promise
• Leverage tried-and-true Web infrastructure for scaling
– Video is just ordinary Web content!
• Leverage tried-and-true TCP
– Congestion avoidance
– Reliability
– No special QoS for video
THERE
IT SHOULD
JUST WORK

Doesitjustwork?
Mostly yes, when streaming
clients compete with other
types of traffic
Not really, when streaming
clients compete with each
other
Streaming clients interact with each
other forming an “accidental”
distributed control-feedback system
• Multiple screens within a household
• ISP access/aggregation links
• Small cells in stadiums and malls

A Single Microsoft Smooth Streaming Client under a Controlled Environment
Demystifying a Streaming Client
0
1
2
3
4
5
0 50 100 150 200 250 300 350 400 450 500
Bitrate(Mbps)
Time (s)
Available Bandwidth Requests Chunk Tput Average Tput
Reading: “An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP,” ACM MMSys 2011
Buffer-filling State
Back-to-back requests
Steady State
Periodic requests

10 (Commercial) Streaming Clients Sharing a 10 Mbps Link
Selfishness Hurts Everyone
0
200
400
600
800
1000
1200
1400
0 100 200 300 400 500
RequestedBitrate(Kbps)
Time (s)
Client1 Client2 Client3

Viewer Experience Statistics
Selfishness Hurts Everyone
Source: Conviva Viewer Experience Report, 2015

Inner and Outer Control Loops
HTTP Server
Manifest
Media
HTTP
Origin
Module
TCP Sender
Streaming Client
Manifest
Resource
Monitors
Streaming
Application
TCP ReceiverData / ACK
There could be multiple TCPs destined to the same or different servers
Request / Response

Streaming with Multiple TCP Connections
• Using multiple concurrent TCPs
– Can help mitigate head-of-line blocking
– Allows fetching multiple (sub)segments in parallel
– Allows to quickly abandon a non-working connection without having to slow-start a new one
System performance deteriorates very quickly if many clients adopt
this approach without limiting the aggregated bandwidth consumption

Two Competing Clients
Understanding the Root Cause
• Depending on the timing of the ON periods:
– Unfairness, underutilization and/or instability may occur
– Clients may grossly overestimate their fair share of the available bandwidth
Clients cannot figure out how much bandwidth to use until they use too much
(Just like TCP)
Reading: “What happens when HTTP adaptive streaming players compete for bandwidth?,” ACM NOSSDAV 2012

How to Solve the Issues?
• Use a better adaptation algorithm like PANDA or BOLA
• Use machine learning or deep learning like Pensieve
• Improve the HTTP/TCP stack, try out the alternatives
• Adopt ideas from game/consensus theory (GTA)
Fix the clients and/or the transport
• QoS in the core/edge
• SDN
Get support from the network
• Assist the clients and network elements thru metrics and analytics
Enable a control plane

Bitrate Adaptation Schemes
Bitrate
Adaptation
Schemes
Client-
based
Adaptation
Bandwidth-
based
Buffer-
based
Mixed
adaptation
Proprietary
solutions
MDP-based
Server-
based
Adaptation
Network-
assisted
Adaptation
Hybrid
Adaptation
SDN-based
Server and
network-
assisted
Reading: “A survey on bitrate adaptation schemes for streaming media over HTTP,” IEEE Commun. Surveys Tuts., to appear

High-Level Comparison between Different Schemes
• The client-based adaptation schemes
– Show a good performance in single and few-client scenarios
– Largely fail in multi-client scenarios
• The server-based adaptation schemes
– Require custom servers
– More effective in eliminating the bitrate oscillation problems
– Less scalable due to increased complexity on the servers
• The network-assisted adaptation and hybrid schemes
– Show a good performance in both small and large populations
– Require modifications on the clients, server and/or network devices
– Pose practicality issues for deployment

A Note on the Reproducibility
• A performance comparison between different algorithms is often difficult due to
– Lack of the source codes
– Lack of a unified QoE framework and universally accepted metrics
• Schemes have their own parameters and assumptions, and work well only under
certain circumstances

Game/Consensus Theory
• Structuring a set of players in cooperative groups
• Enables a strong level of cooperation among the players
• Model and analyze fairness, stability and efficiency
Coalition Formation
• Players that are interested in reaching an agreement among themselves over a
common objective (how to reach this agreement and on the terms of the agreement)
• Concept solution: Nash Bargaining Solution (NBS)
Bargaining Solution
• Exploited together with NBS and coalition-formation game
• Relied on a strong coordination among the players
Consensus Theory

GTA: A Game Theory Based ABR Scheme
• Designed based on a cooperative game in the form of static formation-based coalitions
• Formulates the ABR decision problem as a bargaining process and consensus mechanism
• Outputs the optimal decision by reaching the Pareto optimal (PO) Nash bargaining solution (NBS)
Game Theory
Cooperative
Coalition
Formation
Bargaining
Solution
Consensus
Theory
Strategic
GTA
GTA player
Coalition
Bargaining process

• PANDA & CS2P Estimators
– Throughput estimation
• SAND Enabler
– Implements various SAND-enabled communication
interfaces
• SSIMplus MAP Model
– Maps content type, display resolution and service
plan type into one common perceptual quality
space in which a set of clusters are constructed
• QoE Metrics
– QoE model and various metrics
• GT Agent
– Develops the GTA decision rules for selecting the
best bitrate
GTA Components
Reading: “Want to play DASH? a game theoretic approach for adaptive streaming over HTTP,” ACM MMSys 2018

Component #1: Cooperation Enabler
GTA Design
Non-Cooperative Mode
• Mix of GTA and non-GTA players
• Greedy interaction with non-GTA
players
• GTA clients may start in non-
cooperative mode and then learn
the accurate number of clients
through potential and wonderful
life utility (WLU)
Cooperative Mode
• Multiple GTA players
• GTA players are aggregated into
a set of clusters based on
SSIMplus-based MAP

Component #2: QoE Calculator
GTA Design
Avg. Quality Avg. Quality Switch Stalls Startup Delay

Component #3 and #4: GT Strategy Calculator and GT (Dis)agreement Calculator
GTA Design
• GT Strategy Calculator
– Compute the GT strategy (i.e., the action-utility R(A) relationship set) which is defined over the
strategy space S as follows:
• GT (Dis)agreement Calculator
– Compute the GT disagreement decision
• i.e., bargaining outcome disagreements (W)
P = {p1,...,pN} is the set of N players
R is the set of possible utilities when actions
from A are taken
• r is the player utility when action a is
taken
• k is the current step, K is total number of
segments of video u
R(A) is the set of possible actions with their
achievable utilities
R-(A-) is the set of pessimal actions with their
resulting utilities
S and W are defined by the function F which
is defined over the space: F: (A → R) ∪ {W}
PO (NBS): max. (S – W)
AgreementpointDisagreementpoint
MAX

Component #5: QoE Optimizer
GTA Design
• GTA rule for ABR selection can be formulated as a network utility maximization
– Strictly increasing concave
– Solver: Dual decomposition + dynamic programming
• Find the bargaining solution by solving the Problem(S,W)
– Bargaining Process and consensus decision problem is defined as a pair Problem(S,W)
– The bargaining solution is defined by a function F over the space F : (S, W ) → Rn
– F specifies a unique Pareto optimal (PO) bargaining outcome (O* = F(S,W)) for every Problem(S,W) using Nash Bargaining
Solution (NBS)
– O* is the set that contains only the optimal bargaining outcomes
C.1 Maintain the current buffer occupancy
between the min and max buffer
thresholds
C.2 Satisfy the SSIMplus MAP model
C.3 Lead to a unique (PO) NBS
C.4 Chunk time constraint where max
time needed to download the
corresponding chunk ≦ chunk duration

Source Code is Available at http://streaming.university/GTA/
dash.js Integration
Gray boxes indicate the components that have been developed or modified

Setup for Performance Evaluation
• VoD experimental setup
– Two machines (running Ubuntu 16.04 LTS), one for the dash.js player and one for the server
– tc-NetEm to replicate the throughput profiles
– PANDA parameters as described in the original paper
– Throughput smoothing: the mean of the last three throughput estimations
– The fastMPC horizon was set to the next three chunks
– The min and max buffer occupancy thresholds were set to 8 and 32 seconds
– CT = animation (BBB), chunk duration of 4 s, total duration of 600 s
– 10 bitrate levels: [250, 300, 400, 700, 900, 1500, 2000, 3000, 3500, 4000] Kbps

Results: Video Stability

Results: QoE

Numbers Indicate GTA Improvement over Other Schemes
GTA Outperforms the Existing ABR Schemes
BBA BOLA Rate-based Hybrid Quetra ELASTIC QDASH SARA FESTIVE
QoE 55% 33.5% 35.5% 19.5% 39% 46.5% 47% 40.5% 30.5%
Video Stability 60% 72% 37.5% 63.5% 66.5% 70% 66% 53% 63.5%
Stall Events 13.6% 1.1% 6.1% 1.7% 3.87% 7.87% 5.87% 2.9% 3.6%

http://streaming.university/GTA/
GTA Demo and Source Codes
Offline Demo Online Demo

Quickly Starting Media Streams Using QUIC
QUIC: Multiplexed transport protocol over UDP
• Reduced handshake latency
• Improved congestion control
• Improved loss recovery
• Multiplexing streams
Can QUIC help improve viewer experience in
HTTP adaptive streaming?
Reading: “Quickly starting media streams using QUIC,” PV 2018

• Start media streams more
quickly
• Reduce seeking latency
• Cope better with frequent
connection changes?
Can QUIC help us

Previous Research
• Some reported
– QUIC does not impact streaming performance
– QUIC does not provide a boost to HAS
• Others reported
– QUIC’s 0-RTT performed better than the other protocols
– QUIC provided better streaming, but only for high-quality video
• Google said QUIC reduced YouTube rebuffer rates by 15%
QUIC code evolves rapidly and 3rd-party implementations may
not necessarily reflect protocol’s real performance

Comparison between Our and Prior Work
1 WiFi, 4G/LTE and 3G, 2 Based on the third-party implementation version at the time of research, 3 Latest at the time of research.

• Player Features
– Python based
– TCP and QUIC support integrated
as subprocess
– Scripted frame-seek (fast-
forward) ability
– BASIC, SARA, BBA-2 adaptation
algorithms
• Source Code
– github.com/sevketarisu/quic-
streaming
github.com/sevketarisu/quic-streaming

• Environment Setup
– Public Internet
– No traffic shaping
– Each test repeated 10 times
• Server
– Apache HTTP server on Amazon
EC2
– Located in Frankfurt, DE
• Client
– Runs Google’s proto-quic library
– Located in Istanbul, TR
– Connected via WiFi, LTE or 3G
Chromium QUIC
(proto-quic library)
WiFi, LTE, 3G
Amazon EC2
libcurl

Evaluated Metrics
• Average playback bitrate
– Average of the bitrates of the downloaded segments
• Average wait time after seeking
– Time from the frame-seek request to the playback of the requested media
– A rule of thumb is to keep this time under two seconds
• Rebuffer rate

Frame-Seek (Fast-Forward) Feature Implemented in the Player
Frame-Seek Scenarios

Frame-Seek Scenario Results
2.80
2.99
2.862.89
3.05 2.99
0.0
0.5
1.0
1.5
2.0
2.5
3.0
3.5
4.0
WiFi LTE 3G
AveragePlaybackBitrate(Mbps)
TCP QUIC
Results are averaged for BASIC, SARA and BBA-2 algorithms.
QUIC provided a higher (or at least an
equal) average playback bitrate in all
cases and for all algorithms
Max Bitrate Available: 3.9 Mbps

2.09 2.16
2.38
1.28 1.29
1.42
0.0
0.5
1.0
1.5
2.0
2.5
3.0
WiFi LTE 3G
AverageWaitTimeafterSeeking(s)
TCP QUIC
QUIC reduced wait times to
less than 1.5 seconds

3.67 3.67
5.67
1.00
1.33
2.00
0
1
2
3
4
5
6
WiFi LTE 3G
RebufferRate(%)
TCP QUIC
QUIC’s benefits are bigger
when delays are higher

Connection-Switch Simulation

Connection-Switch Scenario

WiFi - LTE Switches
Connection-Switch Results
3.00
4.00
5.00
2.00
3.00
4.00
0
1
2
3
4
5
6
BASIC SARA BBA-2
RebufferRate(%)
TCP QUIC
QUIC provided a higher (or at least
an equal) average playback bitrate
for all algorithms

WiFi - 3G Switches
Connection-Switch Results
13.00
2.00
5.00
6.00
1.00
4.00
0
2
4
6
8
10
12
14
BASIC SARA BBA-2
RebufferRate(%)
TCP QUIC
QUIC provided a higher (or at least
an equal) average playback bitrate
for all algorithms

But
QUIC is still evolving in the IETF;
should there be significant
changes, the tests will need to be
repeated
– https://quicwg.org/
– https://github.com/quicwg
Conclusions So Far
• QUIC reduces wait times and
rebuffer rates without reducing
playback bitrate
• QUIC outperforms TCP when
frequent network changes occur
• QUIC’s benefits are greater in
networks with larger delay (e.g.,
early generation 3G networks)

One Slide on Software Defined Networking (SDN)
Control and data planes are decoupled, network intelligence and state are logically
centralized, and the underlying network infrastructure is abstracted from the apps

SDN-Based Bitrate Adaptation
Reading: “SDNDASH: improving QoE of HTTP adaptive streaming using software defined networking,” ACM MM 2016

Server and Network-Assisted Adaptation
• A control plane approach that offers asynchronous client-to-network, network-to-client
and network-to-network communications
– Collect metrics and status information from various entities in the system
– Share this information to assist the clients and aware elements to improve media delivery
• DASH part 5 (23009-5) outlines the server and network-assisted DASH (SAND)
framework along with the necessary interfaces

Conflicting Goals in the Media Delivery Chain
• I want to make sure that my content is protected and looks awesome on any device
• I want to make sure that my ads are viewed, trackable and measurable
• I want to make sure that my servers are properly used and latency is low
• I want to control the QoE of my customers, make $ from and differentiate my own video services
• I want to make sure that my device/app provides the best possible video quality
• I want the best quality for minimal $
ConsumerContent
Provider
ISP
CDN
ProviderAdvertiser
Device/
App

Introducing DASH-Assisting Network Elements (DANE)
A Control Plane Approach
Origin (HTTP)
Server
Encoder/
Transcoder Packager
CDN
RG
Clients
Media
Parameters for enhancing reception (PER)
Metrics and status messages
Parameters for enhancing delivery (PED)
Analytics
Server

Why is Latency Important?

Contributors to the Latency
• Video encoding pipeline duration
• Ingest and packaging operations
• Network propagation delays
• CDN buffer delays
• Media segment duration
• Player behavior
– Buffering
– Playhead positioning
– Resilience
Current Implementations
• Smooth Streaming: 2-second segments,
usually 10 seconds of latency
• DASH: 2-second segments supported by most
players, usually 8 to 10 seconds of latency
• HLS
– Until mid-2016: 10-second segments, usually 30
seconds of latency
– Since mid-2016: 6-second segments, usually 18 to
20 seconds of latency
– Safari Mobile in iOS11: Autostart for live streams
and support for short segments
– App Store & iOS applications: 6-second segments
are still mandatory
Latency in One Slide

Stream Start Time ≠ Latency
Time
Live encoder
producing 2-second
segments
iOS (3 segments)
Last available
Lowest latency
1 2 3 4
Start Now
2 3 4
4
5
Latency: 7 s
Latency: 3 s
Latency: 2 s
6 seconds of buffer
~0 seconds of startup*
2 seconds of buffer
~0 seconds of startup*
2 seconds of buffer
1 second of startup*
Segment fetching time is assumed to be negligible in this example
5

Latency Facts
• Stream start time: Time it takes for the video to start playing after pressing ‘play’
• Hand–waving (glass-to-glass) latency: Time delay between an action occurring in front
of the camera and the same action being visible on the display
• Stream start time is not equal to latency
– A stream may take 5 s to start and only be 2 s behind live
– Conversely, a stream can start in 2 s and be 5 s behind live
• A player can start quickly by choosing to start behind the live point (which increases
latency) and build its buffer by fetching segments faster than real time till the live edge
• Low latency is always a trade-off against robustness of the playback

Part I: Streaming
Overview of Streaming Standards (DASH, CMAF and Apple HLS)

Dead, Surviving, Maturing and Newborn Technologies
• Move Adaptive Stream (Long gone, but some components are in Slingbox)
– http://www.movenetworks.com
• Microsoft Smooth Streaming (Legacy)
– http://www.iis.net/expand/SmoothStreaming
• Adobe Flash (Almost dead)
– http://www.adobe.com/products/flashplayer.html
• Adobe HTTP Dynamic Streaming (Legacy)
– http://www.adobe.com/products/httpdynamicstreaming
• Apple HTTP Live Streaming (The elephant in the room)
– https://tools.ietf.org/html/rfc8216
– https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis
• MPEG DASH and CMAF (The standards)
– http://mpeg.chiariglione.org/standards/mpeg-dash
– http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format

• Fragmented architectures
– Advertising, DRM, metadata, blackouts, etc.
• Investing in more hardware and
software
– Increased CapEx and OpEx
• Lack of consistent analytics
• Preparing and delivering each asset in
several incompatible formats
– Higher storage and transport costs
• Confusion due to the lack of skills to
troubleshoot problems
• Lack of common experience across
devices for the same service
– Tricks, captions, subtitles, ads, etc.
What does Fragmentation Mean?
Higher Costs
Less Scalability
Smaller Reach
Frustration
Skepticism
Slow Adoption

DASH intends to be to
the Internet world …
what MPEG2-TS has been to
the broadcast world

DASH intendsed to be to
the Internet world …
what MPEG2-TS has been to
the broadcast world

Dynamic Adaptive Streaming over HTTP (DASH)
• DASH is not
– system, protocol, presentation, codec, interactivity, DRM, client specification
• DASH is an enabler
– It provides formats to enable efficient and high-quality delivery of streaming services
– System definition is left to other organizations
• Design choices
– Reuse the existing technologies (containers, codecs, DRM, etc.)
– Enable deployment on top of CDNs
– Move intelligence from network to client, enable client differentiation
– Provide simple interoperability points (profiles)

Shown in Red
Scope of MPEG DASH
HTTP Server DASH Client
Control Engine
MediaEngines
HTTP ClientHTTP/1.1
Segment
ParserMPD Transport
MPD
Parser
MPD
MPD
MPD
...
...
...

Brief History of DASH at MPEG
• Amd. 1: NTP sync, extended profiles
• Amd. 2: SRD, URL parameter insertion, role extensions
• Amd. 3: External MPD link, period continuity, generalized HTTP header extensions/queries
• Amd. 4: TV profile, MPD chaining/resetting, data URLs in MPD, switching across adaptation sets
• 3rd edition (FDIS) in ballot
• Amd. 5 (WiP): Device information, quality equivalence descriptor, timed text roles, announcing
popular content, flexible IOP signaling, early available periods, signaling missing/alternative
segments
23009-1: Media Presentation Description and Segment Formats
• 2nd edition was published in Oct. 2017
• Amd. 1 (WiP): SAND conformance rules
23009-2: Conformance and Reference Software
• 3rd edition (WD) is in progress
23009-3: Implementation Guidelines (Informative)

Brief History of DASH at MPEG
• 2nd edition (FDIS) in ballot
23009-4: Segment Encryption and Authentication
• 1st edition, published in Feb. 2017
23009-5: Server and Network Assisted DASH (SAND)
• 1st edition, published in Dec. 2017
23009-6: DASH over Full Duplex HTTP-Based Protocols (FDH)
• 1st edition (WD) is work in progress
23009-7: Delivery of CMAF Contents with DASH (Informative)

• Describes various types of “push”
behaviors in HTTP/2 and WebSockets
• Primarily targeted for low-latency live
streaming
• Similar activities
– DASH over LTE broadcast (3GPP SA4)
– DVB ABR multicast
– ATSC 3.0 hybrid delivery
Part 6 – DASH over Full Duplex HTTP-Based Protocols

Ongoing Work as of MPEG 123 (July 2018)
• Technologies under Consideration (w17812)
– Usage of HEVC tile tracks in DASH
– Annotation and client model for content selection
– Signaling for quality control
– Announcing popular content in DASH
– Using segment templates for forensic watermarking
– Using DASH MPD chaining for mid-roll ads
– DASH playlist description
– Mixed MPD
– Event processing model
– Patch method for MPD updates

Common Media Application Format (CMAF)
• Media delivery has three main components:
– Media format
– Manifest
– Delivery
• CMAF defines the media format only (Fragments, headers, segments, chunks, tracks)
– No manifest format or a delivery method is specified
• CMAF uses ISO-BMFF and common encryption (CENC)
– CENC means the media fragments can be decrypted/decoded by devices using different DRMs
– CMAF does not mandate CTR or CBC mode
• This makes CMAF useful only for unencrypted content for now, but industry is moving towards CBCS
• Any delivery method may be used for delivering CMAF content
– HTTP
– RTP multicast/unicast
– LTE broadcast

CMAF (ISO/IEC 23000-19)
• CMAF defines media profiles for
– Video
– Audio
– Subtitle
• CMAF defines presentation profiles by selection a media profile from each category
• Current status
– 1st edition was published in Jan. 2018
– Supported in iOS 10+ (with HLS playlists)
– Amd. 1: SHVC media profile and additional audio media profiles
– Amd. 2: xHE-AAC and other media profiles

Each Frame can Be a CMAF Chunk
CMAF ISO-BMFF Media Objects
RAP …
RAP
…
RAP
…
RAP
…
Fragment Fragment Fragment Fragment
Segment Segment
Track File
…
…
…
Encoding
Packaging
Encryption
CMAF
Header
Seamless switching can only happen
at fragment boundaries
Manifests may provide URLs to
• Track files
• CMAF header + segments
• CMAF header + chunks

ismc
Smooth
mpd
DASH
m3u8
HLS
Source
Encoder
f4m
HDS
m3u8
m3u8
mpd
mpd
ismc
ismc
f4m
f4m
Credit: Akamai

ismc
Smooth
mpd
DASH
m3u8
HLS
Source
Encoder
f4m
HDS
m3u8
m3u8
mpd
mpd
ismc
ismc
f4m
f4m
Credit: Akamai
More packaging $
More storage $
Less efficient caching

Multi-Platform OTT Workflow
Efficient Distribution in the CDN
CMAF
Source
Encoder
m3u8
mpd
Credit: Akamai
m3u8
mpd
m3u8
mpd
m3u8
mpd
m3u8
mpd
m3u8
mpd

Encode and Package Once, Deliver Efficiently
Origin (HTTP)
Server
Encoder/
Transcoder
LinearIngest
On-demandContent
Pick your
favorite codec
Pick your chunk and
segment sizes
CIF
(SCTE)
to client manifest
translation
withEBPmarkers
Packager

Non-HTTP Transport Options
RTP Multicast
- Define RTP payload format for CMAF chunks/fragments/segments
- Use RFC 4588 and 6285 for loss recovery and rapid acquisition
Broadcast
- TS 26.346 and TS 26.347: Delivering DASH content via MBMS
- A/331:2017: Carrying DASH content over broadcast and/or broadband networks
Peer-to-Peer
- WebRTC: Clients fetch DASH/CMAF pieces from each other in addition to CDN servers
- This helps reduce the load on the CDN as well as the stream start times

• Content format issues
– Each asset is copied to multiple media
formats
• different video codecs
• different audio codecs
• different (regional) frame rates
– Cost to content creators and distributors
– Inefficiencies in CDNs and higher storage
costs
• Platform issues
– Lack of consistent app behavior across
platforms
– Varying video features, APIs and semantics
across platforms
• Playback issues
– Codec incompatibility
– Partial profile support
– Switching bitrate glitches
– Audio discontinuities
– Ad splicing problems
– Long-term playback instability
– Request protocol deficiencies
– Memory problems, CPU weaknesses
– Scaling (display) issues
– Variable HDR support
– Unknown capabilities
Commercial OTT Issues

CTA Web Application Video Ecosystem (WAVE)
Steering
Committee
Technical
Working
Group
Content Specification Task Force
Content specification based upcoming CMAF,
compatible with DASH and HLS
Device Playback Capabilities Task Force
Testable requirements covering the most
common device playback interoperability
issues
HTML5 API Task Force
Reference application framework based on
HTML5 providing functional guidelines for
playback interop

Web Media API Snapshot 2017
• First annual API Snapshot published in Dec. 2017
– Lists key APIs supported in 2017 in all major HTML code bases
– CTA-W3C agreement to co-publish this spec
– Plan to propose Community Group spec as a W3C standards-track spec
– Available at https://www.w3.org/2017/12/webmediaapi.html
• CTA WAVE issued RFP to create a test suite for all listed APIs based on W3C API
tests
– Test suite will enable manufacturers to test that their HTML support is up-to-date
• Web Media Application Developer Guidelines
– Companion guide to outline best practices for implementing Web media apps
– https://w3c.github.io/webmediaguidelines/

WAVE Current and Future Publications
• Published
– “Web Media API Snapshot 2017”, Final Community Group Report, Dec. 2017,
https://www.w3.org/2017/12/webmediaapi.html
– “Web Application Video Ecosystem – Content Specification”, Apr. 2018,
https://members.cta.tech/ctaWAVE
• Pending
– Event Messages in WAVE (white paper)
– Web Application Video Ecosystem (WAVE) Device Playback Capabilities
– Web Media Application Developer Guidelines
– Web Media Porting Specification

Part I: Streaming
Adaptive Delivery of Omnidirectional/360° Video

Example Platform/Infrastructure
https://bitmovin.com/

MPEG’s Definition
What is Virtual Reality (VR)
VR is a rendered environment (visual and acoustic, pre-
dominantly real-world) providing an immersive experience to a
user who can interact with it in a seemingly real or physical way
using special electronic equipment

Virtual Reality (VR) puts us in a Virtual World
Source: Phil Chou

Augmented Reality (AR) puts Virtual Objects in our World
Source: Phil Chou

Delivering High-Quality VR in Economic Scale is Extremely Challenging
The VR Challenge
30K x 24K x 36 x 60 x 2 / 600
~5.2 Gbps
Resolution
Bit depth Frame rate Stereoscopic
Compression gain

Ultimate Level of Immersion
Interactions
So intuitive that they
become second nature
Sounds
So accurate that
they are true to life
Visuals
So vibrant that they are eventually
indistinguishable from the real world

Visual
quality
Sound
quality
Intuitive
interactions
Immersive VR Has Extreme Requirements
Immersion
High resolution audio
Up to human hearing capabilities
3D audio
Realistic 3D, positional, surround
audio that is accurate to the real world
Precise motion tracking
Accurate on-device motion tracking
Minimal latency
Minimized system latency
to remove perceptible lag
Natural user interfaces
Seamlessly interact with VR using
natural movements, free from wires
Extreme pixel
quantity and quality
Screen is very close to the eyes
Spherical view
Look anywhere with a
full 360° spherical view
Stereoscopic display
Humans see in 3D
Source: Thomas Stockhammer

Need for Higher Resolutions on Mobile Phones?
Source: Thomas Stockhammer

Omnidirectional/360° Video
Capturing
Devices
• Stitching, projection formats
• Encoding, encryption, encapsulation
• Storage, content distribution, delivery
• Processing, decoding, rendering
Consumer
Devices

Adaptive Streaming of VR Content
• Required resolution of a panoramic video for achieving 4K resolution for viewport with
120° field of view (FoV)

Functional Architecture
Encode,
Encrypt,
Encapsulate
Decode,
Decrypt,
Decapsulate
Project &
Render
Fuse, Stitch
& Edit
①
② ③ ④
⑤
Store & Deliver
Capture Consume
Content Creation NetworkServer Client
Encoding
Encryption
Encapsulation
Storage
Delivery Decryption
Decapsulation
Decoding RenderingEditing
Processing Processing
Capture
Acquisition
Consumption
Distribution
Adaptive Delivery of Omnidirectional Video
Interaction
From ecosystem…
To building blocks…

Adaptive Streaming Options (1)
• Traditional, viewport-agnostic
streaming
– ERP – handle like reg. video
– Simple, easy, deployed today
– Bandwidth waste, quality issues
• Viewport-adaptive streaming
– Multiple versions for predefined
viewports
– Various projection techniques (pyramid)
– Bandwidth waste reduced, increased
storage and CDN costs, limited flexibility X. Corbillon, et al., "Viewport-adaptive navigable 360-degree
video delivery," 2017 IEEE International Conference on
Communications (ICC), Paris, 2017,
https://doi.org/10.1109/ICC.2017.7996611

Adaptive Streaming Options (2)
• Tile-based streaming
– Use tiling technique of
modern video codecs
– High complexity, full flexibility
– Multiple challenges
M. Graf, et al. 2017. Towards Bandwidth Efficient Adaptive
Streaming of Omnidirectional Video over HTTP: Design,
Implementation, and Evaluation. Proc. ACM MMSys'17.
https://doi.org/10.1145/3083187.3084016
C. Concolato, et al., "Adaptive Streaming of
HEVC Tiled Videos using MPEG-DASH,"
IEEE TCSVT, 2017.
https://doi.org/10.1109/TCSVT.2017.2688491

System Overview
Tile-Based Adaptive Streaming
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Encoding &
Packaging
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
DifferentQ
uality
Representations
Delivery
MPEG-HEVC/H.265
Tiles in ISOBMFF
Adaptive
Player
Tile
10
Tile
11
Tile
12
Tile
13
Tile
14
Tile
15
Tile
16
Tile
17
Tile
18
Tile
19
Tile
20
Tile
21
Tile
22
Tile
23
Tile
24
Tile-based streaming of VR/360°
content with MPEG-DASH SRD / OMAF
…
…
…
…
Adaptive Streaming using
MPEG-DASH SRD / OMAF
Head Mounted Displays
Browsers, Smart (Mobile) Devices
(Stereo) 2D, (Stereo) 3D

Encoding Options
• AVC dominates the market
• HEVC, VP9, AV1 support tiles
– Divides a picture into
independent, rectangular regions
– Tradeoff: bitrate, quality, flexibility
• Multiple tiling options available
– Uniform vs. non-uniform tiling
– Same vs. mixed resolutions
• New quality metrics, mostly based on PSNR but subjective quality
assessments/metrics increasing
https://bitmovin.com/2017-video-developer-report/
I. D.D. Curcio, et al. 2017. Bandwidth Reduction of Omnidirectional Viewport-Dependent Video
Streaming via Subjective Quality Assessment. Proc. AltMM'17.
https://doi.org/10.1145/3132361.3132364
A. Zare, et al. 2016. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual
Reality Applications. Proc. ACM MM'16. http://dx.doi.org/10.1145/2964284.2967292

Quality Metrics: S-PSNR
M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015
IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 31–36.
DOI:http://dx.doi.org/10.1109/ISMAR.2015.12

Quality Metrics: V-PSNR
M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015
IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 31–36.
DOI:http://dx.doi.org/10.1109/ISMAR.2015.12

Dataset
• Segment length / intra period
– 1s (tiled content) vs 1, 2, 4s (monolithic content)
• Tiling pattern (columns × rows): 1×1, (i.e., tiles monolithic), 3×2, 5×3, 6×4, and 8×5
• Resolution: 1920×960, 3840×1920 and 7680×3840
• Map projection: equirectangular format
• Quantization parameter: QP={22,27,32,37,42}
• Head motion recordings for V-PSNR evaluation

Bitrate Overhead due to Tiling
500 1000 1500 2000 2500
384042444648
Bitrate [kbps]
Y−PSNR[dB]
●
●
●
●
●
●
● 1x1 tiles
3x2 tiles
5x3 tiles
6x4 tiles
8x5 tiles
Tile Overhead for Resolution: 1920x960
Sequence: AssassinsCreed

Bitrate Overhead due to Tiling
2000 4000 6000 8000
30323436384042
Bitrate [kbps]
Y−PSNR[dB]
●
●
●
●
●
●
● 1x1 tiles
3x2 tiles
5x3 tiles
6x4 tiles
8x5 tiles
Tile Overhead for Resolution: 1920x960
Sequence: ExploreTheWorld

Bandwidth Requirements

Adaptive Streaming Issues for (Tiled) 360° Video
• Increasing number of segment/tile requests
– HTTP/2 server push, query parameters, proprietary protocols
– Additional functionality at server – breaks fundamental HAS requirements
• Low latency streaming (see also next slide)
– Reducing segment size impacts coding efficiency (1s vs. 4s)
– CMAF chunks + other enhancements to enable sub-second latency
– Remember: live internet video will grow 15-fold from 2016 to 2021
• Viewport prediction
– Allows prefetching (caching) but cannot predict to much into future (1s)
– Impact on segment size but situation will get better the more data is available –
machine learning/AI will help
• QoE (see also following slide)
– Still in its infancy but situation much better than one year ago
– Requires datasets, subjective studies, quality models, metrics
M. Xu, et al., "A subjective visual quality assessment method of
panoramic videos," Proc. ICME’17.
https://doi.org/10.1109/ICME.2017.8019351
R. Schatz, et al., "Towards subjective quality of experience
assessment for omnidirectional video streaming," Proc.
QoMEX’17, https://doi.org/10.1109/QoMEX.2017.7965657
Y. Rai, at al. 2017. A Dataset of Head and Eye Movements for 360
Degree Images. Proc. ACM MMSys’17.
https://doi.org/10.1145/3083187.3083218
S. Petrangeli, et al. 2017. An HTTP/2-Based Adaptive Streaming
Framework for 360° Virtual Reality Videos. Proc. ACM MM'17.
https://doi.org/10.1145/3123266.3123453
N. Bouzakaria, et al., "Overhead and performance of low latency
live streaming using MPEG-DASH," Proc. IISA’14.
https://doi.org/10.1109/IISA.2014.6878732
C.-L. Fan, et al. 2017. Fixation Prediction for 360° Video Streaming in Head-
Mounted Virtual Reality. Proc. NOSSDAV'17.
https://doi.org/10.1145/3083165.3083180

Lag Prevents Immersion and Causes Discomfort
Minimizing Motion-to Photon Latency is Important
Low latency Noticeable latency

Motivation
QoE of Omnidirectional Video
• QoE of Omnidirectional Video (OV, alt.: 360º Movie) Streaming consumed via HMD
– Immersive panorama video (or image) surrounding the user
– User can turn head to changes gaze direction
• OV streaming = content class with
dramatically increasing popularity
– Cisco VNI: globally, virtual–reality traffic will
increase 61-fold between 2015 and 2020
Immersive
Media
VR
Responsive
VR
OV
MR AR
R. Schatz, et al., "Towards subjective quality of experience
assessment for omnidirectional video streaming," Proc.
QoMEX’17, https://doi.org/10.1109/QoMEX.2017.7965657

Motivation
QoE of Omnidirectional Video
• Gaps in current state of the art
– Lots of knowledge on 2D & 3D Video quality
– Lots of knowledge on human factors in responsive AR/VR
– BUT: lack of quantitative models and subj. testing methods for OV streaming
Particularly as regards quality perception and the
Impact of stalling on OV QoE
QoE = m * c2
QoE = m * c2
?

Research Questions
RQ1: How does stalling impact the QoE of HMD-based omnidirectional video
(OV)?
RQ2: What are QoE impact differences between HMD-based OV and
traditional TV-based video consumption with respect to stalling?
RQ3: What is the impact of different content motion levels on OV stalling
perception?

Method: Subjective Lab Study
• Test task: watch short video clips (duration 60s) impaired by stalling
• Two Devices / Watching Modes:
– 2D video on large TV screen (65”, Full HD)
– Stereoscopic OV on an Oculus Rift DK2 HMD (with headphones and 3-axis orientation sensor)
vs

Test Subjects
• Recruited from Vienna City region, received monetary incentives
• Recruited: 27 participants, but …
… removed 5 subjects due to vision problems, lack of HMD fit and cybersickness
à N=22
– 12 male, 10 female
– 4 had some limited VR experience
– Age: mean 37, median 31.5, min 22, max 65 years

Test Content (SRCs)
• 2D / TV
– Sintel movie (© Blender Foundation)
– One 60s clip
• OV / HMD
– Vulkane – Terra X
(© ZDF)
– Two 60s clips
extracted
• Both feature:
– Animation
– Storyline

Test Setup

Impairments (HRCs) and Test Conditions (PVSs)
• PVS:
– 2D/TV: 6 HRCs (Sintel movie) (+ 1 Training condition)
– OV/HMD: 6 HRCs x 2 SRCs (static camera, moving camera) (1 Training condition)
• Post-stimulus electronic questionnaire with 4/7 (TV/HMD) rating questions related to
quality, immersion, etc.

OV move: Four users could not complete the OV move conditions, i.e. N=18
How Annoying were Interruptions (If Any) of the Video Playback?
Results: Impact of Stalling on User Annoyance
Imperceptible
Very annoying

I Was Completely Captivated by the Virtual Environment
Impact of Stalling on Perceived Involvement (IPQ INV4*)
Fully agree
Full disagree
* from Igroup Presence Questionnaire (IPQ)

Results / Findings
• RQ1 (stalling QoE impact):
– Strong impact of stalling number and duration (similar to previous results)
– Existing stalling models can be reused
• RQ2 (differences 2D/OV):
– No significant differences between TV/2D and HMD/OV (à unexpected)
• RQ3 (impact of OV content motion):
– No significant differences between moving and static camera, just some trends (à unexpected)
• In addition:
– Splits yield some differences in rating variance and stalling detection
Influence on variance: sex, prior vr experience
Influence on stalling detection: commented on HMD resolution

AdViSE
• Adaptive Media Content
[DASH, HLS, CMAF]
• Players/Algorithms
• Network Parameters
Impaired
Media Sequences
Generate
Impaired Media
Sequences
Templates [Startup
Delay, Stalling, …]
WESP
QoE Evaluation
Parameters
[Questionnaire,
Methodology,
Crowdsourcing
Platform, …]
QoS/QoE
Metrics
Subjective
Results + Other
Data
Reports
Analysis
AdViSE: Anatoliy Zabrovskiy, Evgeny Kuzmin, Evgeny Petrov, Christian Timmerer, and Christopher Mueller. 2017. AdViSE: Adaptive Video Streaming
Evaluation Framework for the Automated Testing of Media Players. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM,
New York, NY, USA, 217-220. DOI: https://doi.org/10.1145/3083187.3083221
WESP: Benjamin Rainer, Markus Waltl, Christian Timmerer, A Web based Subjective Evaluation Platform, In Proceedings of the 5th International Workshop
on Quality of Multimedia Experience (QoMEX'13) (Christian Timmerer, Patrick Le Callet, Martin Varela, Stefan Winkler, Tiago H Falk, eds.), IEEE, Los
Alamitos, CA, USA, pp. 24-25, 2013. https://doi.org/10.1109/QoMEX.2013.6603196
Objective and Subjective Evaluation Platform
How to Evaluate HAS Systems?
①
②
③④
⑤
Log of
Segment
Requests

AdViSE: Adaptive Video Streaming Evaluation
• Scalable, end-to-end HAS evaluation
through emulation w/ a plethora of
– content configurations
– players/algorithms (including for player
competition)
– network parameters/traces
• Real content and network settings with
real dynamic, adaptive streaming
including rendering!
• Collection of various metrics from
players: API or directly from the
algorithms/HTML5
• Derived metrics and utilize QoE models
proposed in the literature
• Segment request log to generate
impaired media sequence as perceived
by end users for subjective quality
testing

WESP: Web-Based Subjective Evaluation Platform
• Subjective quality assessments
(SQA) are vital tool, reliable
results, but cost-intensive
• Utilize crowdsourcing to reduce
costs
• WESP
– Enables easy and simple
configuration of SQAs including
possible integration of third-party
tools for online surveys
– Provides means to conduct SQAs
using the existing crowdsourcing
platforms taking into account best
practice
– Allows for the analysis of the results

C. Timmerer, A. Zabrovskiy, and Ali C. Begen, Automated Objective and
Subjective Evaluation of HTTP Adaptive Streaming Systems, Proceedings
of the 1st IEEE International Conference on Multimedia Information
Processing and Retrieval (MIPR), Miami, FL, USA, April 2018.
Example Evaluation Results
• Test sequence encoded 15 different representation (Amazon Prime configuration:
400x224@100Kbps – 1920x1080@15Mbps) with 4s segment length
• Bandwidth trajectory based on prior work
proposed in literature; network delay 70 ms

Bandwidth Index vs. QoE
• Bandwidth index
– Average bitrate
– Efficiency
– Stability
• QoE model
– Startup delay
– Stalls
C. Timmerer, A. Zabrovskiy, E. Kuzmin, E. Petrov, Quality of experience of commercially deployed adaptive media players, In 2017 21st
Conference of Open Innovations Association (FRUCT) (Sergey Balandin, ed.), pp. 330-335, 2017.
https://doi.org/10.23919/FRUCT.2017.8250200

Part I: Streaming
The Developing MPEG-OMAF and MPEG-I Standards

Standardization Overview
C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in
IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017.
https://doi.org/10.1109/MCOMSTD.2017.1700038
Data Representations and Formats
JPEG MPEG IEEE
System Standards and APIs
VR-IF 3GPP DVB Khronos W3C
Quality of Experience
QUALINET ITU-T VQEG
Guidelines
DASH-IF
CTASVA SMPTE ETSI
https://multimediacommunication.blogspot.com/2017/04/vr360-streaming-standardization-related.html
IETF/IRTF

Coded Representation of Immersive Media
MPEG-I (ISO/IEC 23090)
• Part 1: Architectures
• Part 2: Omnidirectional media format (OMAF)
• Part 3: Versatile Video Coding (VVC)
• Part 4: Immersive audio
• Part 5: Point cloud compression
• Part 6: Immersive media metrics
• Part 7: Immersive media metadata
• Part 8: Network-based media processing (NBMP)

Scope
Omnidirectional Media Format (OMAF)
• 360° video, images, audio and associated timed text – 3DoF only
• Specifies
– A coordinate system
• Consists of a unit sphere and the x (back-to-front), the y (lateral, side-to-side), and the z (vertical, up) axes
– Projection and rectangular region-wise packing methods
• Used for conversion of a spherical video/image into a two-dimensional rectangular video/image
– The sphere signal is the result of stitching of video signals captured by multiple cameras
– A special case: fisheye video
– Storage of omnidirectional media and the associated metadata using ISO-BMFF
– Encapsulation, signaling and streaming of omnidirectional media in DASH and MMT
– Media profiles and presentation profiles
• Provide interoperable and conformance points for media codecs as well as media coding and encapsulation
configurations
• Provides some informative viewport-dependent 360° video processing approaches

A: Real-world scene
B: Multiple-sensors-captured video or audio
D/D’: Projected/packed video
E/E’: Coded video or audio bitstream
F/F’: ISOBMFF file/segment
OMAF player
Acquisition Audio encoding
Video encoding
Image encoding
File/segment
encapsulation
Delivery
File/segment
decapsulation
Audio decoding
Video decoding
Image decoding
Image rendering
Audio rendering
Loudspeakers/
headphones
Display
Head/eye
tracking
Fileplayback
Orientation/
viewport metadata
Orientation/viewport metadata
Metadata
Metadata
A Ba
Bi D
Ea
Ei
Ev Fs
F
F’
F’s
E’a
E’v
E’iD’
B’a
A’i
A’a
Image stitching,
rotation, projection,
and region-wise
packing
https://mpeg.chiariglione.org/standards/mpeg-i/omnidirectional-media-format
R. Skupin, et al., "Standardization Status of 360 degree Video Coding and Delivery,” Proc. IEEE VCIP’17
OMAFArchitecture

DASH/OMAF System Diagram
Multi-
camera
capture
Stitch Project
HEVC
encode
OMAF
composing
DASH server
DASH
client
HEVC
Decode Renderer
Head mounted
device (HMD)
DASH
encapsulation
Head and eye tracking telemetry
Request for
tiles within
FoV window
Projection mapping metadata
Projection mapping metadata
Authoring
CDN
VR player

MPEG-
1,2,4,H,I
MPEG-7
MPEG-21
MPEG-A
MPEG-
B,C,D,
DASH
MPEG-
E,M
MPEG-
U,V
Compression of
video, audio
and 3DG
Description of video,
audio and multimedia
for content search
Technologies for content
e-commerce
Multimedia application
formats (combinations
of content formats)
Systems, video,
audio and transport
Multimedia platform
technologies
Device and application
interfaces
https://mpeg.chiariglione.org/meetings/123
MPEG’s Areas of Activity

1990 2000 20101995 2005 2015
MP3
MPEG-2
Video &
Systems AVC HEVC
MP4 FF MMTDASH
AAC
Internet
Audio
Digital
Television
Music
Distribution
Media Storage and
Streaming
UHD & Immersive
Services
New Forms of
Digital
Television
Unified Media
Streaming
OFF
HD DistributionMobile Video
Custom Fonts on
Web & DigiPub
MPEG-4
Video
CMAF
3D Audio
2020
MajorMPEG
Standards

6DoF Application Format?
Genome Compression v2
PCC Extensions?
OMAF v2
2018 20202017 2019 2021
IoMT
Media Orchestration
Descriptors for Video Analysis (CDVA)
6DoF Audio
Point Cloud Compression
OMAF v1
Genome Compression
Network-based Media Processing
Coding
2022
Scene Description for Immersive Media
Versatile Video Coding
2023
MIAF
Systems
and ToolsWeb Resource Tracks
Dense Representation of Light Field Video
3DoF+ Video
MPEGRoadmap

Part II: Communications over 5G
5G Fundamentals: Core Network, Radio Access Network and QoS

167
Who is InterDigital?
Invention, Collaboration, Contribution
© 2018 InterDigital, Inc. All rights reserved.
Berlin
Montreal, QC (R&D)
London (R&D)
Seoul
Buffalo, NY
Melville, NY (R&D)
Conshohocken, PA (R&D)
Wilmington, DE (HQ)
San Diego, CA (R&D)
• Four decades of discovery
and innovation in wireless
• Pioneer in digital wireless
technologies
• Key contributions to global
wireless standards
• Inventing solutions for more
efficient broadband networks
• ~190 engineers (~80% of
whom hold advanced
degrees)

168
What is 5G?

169
5G Use Cases and Key Requirements
Enhanced Mobile Broadband (eMBB)
§ Peak data rates: 20 Gbps (DL) and 10 Gbps (UL)
§ Peak spectral efficiency:
30 bps/Hz (DL) and 15 bps/Hz (UL)
§ 4 ms user plane latency
§ Indoor/hotspot and enhanced wide-area
coverage
Massive Machine Type
Communications (mMTC)
§ Low data rates
(1 to 100 kbps)
§ High device density
(up to 1,000,000 /km2)
§ Latency: Seconds to hours
§ Low power: Up to 15 years
battery life
Ultra-Reliable and Low Latency
Communications (URLLC)
§ Low to medium data rates
(50 kbps to 10 Mbps)
§ 0.5 ms user plane latency
§ 99.999% reliability and availability
within 1 ms
§ High mobility
Key challenge for 5G design: support for different services having diverging requirements
Source: ITU-R SG5 WP-5D

170
What Differentiates 5G from Previous Generations?
Uses Cases
& Services
Voice + SMS
Voice +
Small Data
Mobile
Broadband
Enhanced Mobile Broadband
Massive Machine Type Communications
Ultra Reliable and Low Latency Communications
Spectrum
200 KHz
Channels
Below 2 GHz
5 MHz
Channels
Below 3.6 GHz
Up to 20 MHz
Channels
Below 3.8 GHz
Up to 1 GHz Channels
Below 100 GHz
Radio
Technology
GSM/GPRS
(Single RAT )
UMTS/HSPA
(Single RAT)
LTE/LTE-A
(Single RAT)
Multiple Radio Access Technologies
Integrated in a 5G Network:
5G New Radio, LTE Advanced Pro, NB-IoT, Wi-Fi, …
Network
Topology
Macro Cells
Macro and
Small Cells
Heterogeneous
Macro and
Small Cells
Challenging Traditional Cell and Network Concepts:
Small Cells, Mobile Edge Computing,
Network Slicing, …
2G 4G 5G3G

2021
171
5G 3GPP Standardization Timeline
2017 2018 2019 2020
Rel-17 Work Items
LTE Release 17LTE Release 15R14
NR SI
Full NR RAN stand-
alone and CT
Specifications
approved
Rel-16 specification
approved with some
additional enhancements
NR R15 first drop
specification completed
(NR non-standalone
and CN standalone)
Phased approach enables early commercial deployment of Phase 1 in 2019 and Phase 2 in 2022+
IMT-2020 Proposal
Submission
NR Rel-15
5G LTE + NB-IoT
(required for
mMTC use cases
for IMT-2020)
5G New Radio
(NR)
…
Allows for very early commercial implementation – No
more hardware changes envisioned for R-15
Rel-16 SIs
(RAN/SA)
Rel-17 Study Items
LTE Release 16
Additional Rel-16 SIs
approved
Rel-16 Work Items (R16)
3rd drop

Focus on completion of the protocol design
§ NAS protocol between the UE and the CN
§ Internal Core Network interfaces based on the
service based architecture (SBA)
§ Interconnection of the Core Network with
external networks
First set of NR specifications completed
§ Ready for Hardware Implementation and for first
phase of deployment - Non-Standalone NR only
(i.e. connected and controlled by LTE)
§ RAN1 PHY and RAN2 User Plane completed
for stand-alone NR
§ Support for Enhanced Mobile Broadband use
cases only and low latency
172
5G New Radio Status – Rel-15
Core
Network
and
Non Access
Stratum
Radio
Access
Network
Next Gen Core(NGC) architecture completed
§ New fully flexible architecture based on Network
Slicing concept and further separation of control
and user planes
§ Key functions: Session Management, Mobility
Management, QoS and access agnostic CN
Focus on completion stand-alone NR
§ Support for stand-alone NR connected directly
to a 5G CN (RAN2/RAN3/RAN4)
§ Add/complete additional enhancements to
support ultra-reliable and low latency use cases
(URLLC) (RAN1/RAN2)
December 2017 June 2018

173
5G Design Principles

174
5G NR Design Principles
Architecture & Network
• Flexible, modular and scalable
architecture
• Support high level of
programmability and
automation in network
• Built in support of network
customization via slicing
• Easy integration of new services
5G is designed to be flexible and adaptable to meet wide range of current and future requirements
Radio Access Network
• Minimize always-on signaling
• Flexible frame structure and
numerology
• Support wide range of frequency
bands
• Built-in support for beamforming
• Easy integration of future
features

175
5G Technologies

176
3GPP 5G Deployment Scenarios
5G
RAN
LTE/4G
Network
5G
Network
LTE
Base Station
New Radio RAN
Internet
Non
Standalone
(NSA) 5G 5G
New Radio
4G EPC
5G Core Network
Slice #3 (URLLC)
Slice #2 (IoT)
Slice #1 (eMBB)
SGW
MME
PGW
HSS
AMF
Non-Standalone (NSA) 5G enables large-scale trials and deployments starting in 2019
3 Main Deployment Scenarios
§ NSA deployment - utilizes
LTE radio and 4G EPC as
an anchor and NR Radio
as hotspot
§ Allows earlier
implementability
§ Stand-alone NR – 5G RAN
connecting to 5G CN only
§ LTE radio connected to 5G
CN network
5G CN designed to interwork
with the current EPC network
to enable seamless migration
for the operators

177
Architecture options and
specification timelines
EPC
NextGen
Core
LTE NR
3
Non-standalone
Drop 1: Dec. 2017
NR
2
LTE
5
NextGen
Core
NextGen
Core
Standalone NR
Drop 2: June 2018
EPC
NextGen
Core
LTE NR
4
EPC
NextGen
Core
LTE NR
7
LTE connected to 5G CN
Drop 3: Dec 2018 – NR +

178
Control and User Plane Separation
Control and User plane separation modularizes the 5G Core Network Design
§ Reduces Latency on application service
§ By relocating UPFs which are closer to the RAN;
§ Or by selecting UPFs more appropriate for the
intended UE usage type without increasing the
number of control plane nodes
§ Supports increase of Data Traffic, by enabling
to add user plane nodes without changing
the number of CP resources in the network
§ Independent evolution of the CP and UP functions
§ Enables SDN routing to deliver user plane
data more efficiently
5G Core Network Architecture

179
Service Based Architecture (SBA)
§ 5G CN has been revolutionized with Service Based
interfaces for core network interactions
§ CN interfaces have traditionally been Point-to-Point
(P2P) e.g. MAP, Diameter
§ Network Functions (NFs) in 5G CN offer their
services to other NFs via APIs based on web-based
technology (e.g. HTTP).
§ Makes deployment easy as libraries, development
tools, and other components are broadly available.
§ Flexible deployments of new services: possible to
connect to other components without new specific
interfaces
§ Network Repository Function (NRF) allows NFs to
discover the services of other NFs.
SBA shifts the paradigm from telecom-style protocol interfaces to web-based APIs
UDMNSSF PCFNRF
AUSF
AF
SMFAMF
5G CN Service Based Control Plane

Network Slice
• Logical end-to-end network
• Tailored based on service it offers
(e.g.eMBB, IoT, URLCCC etc.)
Slicing offers customized network resources for different verticals
Network Slicing
Benefits of slicing
• Flexible and modular network deployment
• Isolation between dedicated network resources
• New products and services can be brought to market
rapidly and adapted to fast-changing demands

Virtual (Cloud) RAN Enablers
ØStandards efforts to enable virtualization in multi-
vendor environments
• Central Unit / Distributed Unit split
• CU/DU split below the PDCP
• CU performs user management and user data processing
• DU manages the radio and cell
• Control Plane/User plane split
• Central unit further split into virtual entities:
• User data processing and user/mobility control
• Support of lower layer split – no conclusion
• Some of the radio control function performed by the
central unit
• Challenging due to ties with implementation
Virtualization gives operators a flexible network design reducing cost
181© 2018 InterDigital, Inc. All rights reserved.
CU
DU
Control Data
gNB
DU

182
Network Function Virtualization (NFV)
Sources: Network Functions Virtualisation -- White Paper on NFV priorities for 5G, ETSI, Feb. 2017.
Network Functions Virtualisation – Introductory White Paper, ETSI, Oct. 2012.
NFV is not limited to 5G
Use cases for 5G:
- Virtualization of the core network
- Virtualization of the RAN
Decouple network functions from hardware

183
Numerology
A numerology is defined by sub-carrier spacing and CP overhead
ØOFDM waveform in both UL &DL
• Optimized CP-OFDM with high spectrum utilization
• Spectrum confinement techniques such as filtering and windowing
• Simplifies design for D2D, Massive MIMO reciprocity & IAB.
• DFT-s-FDM for coverage-limited scenarios in UL
ØScalable Numerology
• Subcarrier Spacing (SCS) scales with the BW (15#$% ∗ 2(
)
• To support diverse spectrum bands and services
• To avoid processing complexity associated with a large FFT size
• Higher SCS -> Higher bands -> Shorter OFDM symbols
• To address RF impairments in mmWave bands (e.g., Phase Noise),
enable analog beamforming in mmWave bands and, lower latency
• Lower SCS -> Larger cyclic prefix
• More robust to the channel delay spread (e.g., Broadcast Service)
• To support narrow band transmissions for mMTC services
SCS [kHz] 15 30 60 120 240
OFDM Duration [µs] 66.67 33.33 16.67 8.33 4.17
CP Duration [µs] 4.69 2.34 1.17 0.58 0.29
Total Symbol [µs] 71.36 35.67 17.84 8.91 4.46
Slot Duration [µs] 1000 500 250 125 62.5
Band [GHz] < 6 < 6 < 6 < > 6 > 6
15 kHz
30 kHz
60 kHz
120 kHz
50 MHz
400 MHz
Max BW
100 MHz
200 MHz

184
Frame Structure
Flexible frame structure adapting to various bands, duplexing mode and service requirements
UL
ULCtrl
DL
UL
ULCtrl
DL
ULCtrl
Downlink-Centric Slot
ØFlexible Frame Structure
• Flexible TTI Duration
§ Scalable TTI enables efficient service multiplexing
o Symbols across different SCSs are aligned at the symbol boundaries
§ Shorter TTI to support low latency application (e.g., URLLC)
o User can also be schedule on a fraction of a slot (i.e., Mini-slot)
§ Longer TTI to achieve higher spectral efficiency (e.g., eMBB)
o User can be scheduled on multiple-slots
• Flexible Scheduling
§ Both dynamic TDD and FDD are supported
§ OFDM symbols in a slot can be DL, UL or flexible
§ Flexible symbols can be used as gap or for forward compatibility
Uplink-Centric Slot
1
30 kHz
120 kHz
0
0
20 12 13
1 2 12 13
1 2 60 kHz
15 kHz
1 ms slot with 14 OFDM symbols
500 µs
250 µs
125 µs
OFDM
symbol
Mini-Slot
Slot with load balancing
Dynamic TDD

185
Bandwidth Adaptation
Adapting UE’s receive and transmit bandwidth irrespective of gNB’s total supported bandwidth
ØFlexible Bandwidth Allocation
• NR supports a maximum BW of 400MHz per Component Carrier
(CC) with up to 16 CC in Carrier Aggregation (CA)
• Supporting bandwidth smaller than the CC bandwidth enabled
by configuring a Bandwidth Part (BWP)
• BWP Benefits include:
§ Enables UE power savings
§ Supports UE with lower maximum bandwidth capability
§ Single BPW is active at a time
• Increases scheduling flexibility
• Allow service multiplexing in frequency domain for use-
cases with different numerology BWP1: 40 MHz and subcarrier spacing of 15 kHz
BWP2: 10 MHz and subcarrier spacing of 15 kHz
BWP3: 10 MHz and subcarrier spacing of 60 kHz
Frequency
Time
BWP1 BWP2
BWP3
BWP2 BWP1
Carrier

186
Antenna
• More bandwidth leads to higher throughput
• Need to go to higher bands
• But higher bands have larger path loss than lower bands
Free space path loss – Friis equation !" = !$%$ %"
&
4()*
+
Sources: W. L. Stutzman and G. A. Thiele, Antenna Theory and Design, Wiley, 2013.
Example: When f increases from 2.8GHz to 28 GHz, the path loss increases by 20 dB.
frequencyReceived power
R
Pt Pr
Why? The maximum effective area (effective aperture) or “collecting area” of an antenna is related to the frequency
e.g. for an ideal (short) dipole antenna, maximum effective area =
,
-./0
Beamforming to close the link at higher frequencies

187
Hybrid Beamforming
• Analog beamforming:
phase shifter
• Digital beamforming:
precoding matrix BB
RF
RF
BB
RF
RF! = # $ + &
where $ = +, = +-.+//,
s
+-.
+//
Phase shifter
Tradeoff between cost and flexibility

Massive MIMO
Ø Conjugate Beamforming
• Coherent combining at receiver
Ø For TDD, no performance loss
when the number of antennas is
much larger than the number of
users (M/K → ∞)
• Leverage channel reciprocity
• Result of Law of Large Numbers
Ø Pilot contamination
• In mobile scenarios the channel
changes over time
• Contamination occurs when two
users use the same pilot
Original Work by Thomas L. Marzetta (2006)
k
188

Massive MIMO
Ø Enhanced CSI Feedback
• Multi-resolution CSI
• Low spatial resolution with a linear precoder targeting SU-MIMO
operation (Type-I CSI) à low latency and overhead
• High spatial resolution with a linear combination precoder targeting
MU-MIMO operation (Type-II CSI) à better performance
• Flexible CSI reporting timing
• CSI reporting timing can be dynamically indicated to optimize
latency based on CSI computational complexity
• Hybrid beamforming support
• Joint CSI reporting of beam index and precoding matrix index (PMI)
Ø Enhanced UL Transmission
• Non-Codebook based UL transmission
• Introduced in NR to enable UL reciprocity functionality in TDD
• UE determines its PUSCH precoder with some gNB assistance
Work under 3GPP
: DL control to trigger aperiodic CSI reporting
: UL channel for aperiodic CSI reporting
Low latency CSI High latency CSI
Slot #n Slot #n+1 Slot #n+2 Slot #n+3 Slot #n+4
• gNB can dynamically indicate the CSI reporting timing
1) CSI-RS
3) DL Control
2) Sounding Signal
4) Precoded UL Data
Precoder
Calculation
189

190
Beam Management
Ø Beam management for high frequency
• Support of high frequency spectrums (up to 52.6GHz) in NR
which is different from LTE
• Beam management is used to maintain optimal Tx/Rx beam
pairing for control and data channels
Ø Beamformed Initial Access
• Fundamental difference with LTE RACH lies on beamforming
• Beam association between RACH and SS/PBCH blocks
• UE selects an associated UL RACH resource of a selected DL
SS block based on L1-RSRP measurements
Management of analog beams at Tx/Rx is newly introduced in NR

191
Beam Failure Recovery
Ø Recovery from beam failure
• Beam-based system is susceptible to blockage, UE rotation,
and mobility
• Link fails when Tx-Rx beams for DL control channel are
mismatched
• RLF has been used in LTE when link fails but that requires a restart
from initial access procedure à high latency
• Beam recovery recovers beams for DL control channel
• UE monitors quality of DL control channel beam and indicates new
candidate beam using a dedicated PRACH resource when link failed
• Beam recovery is used to recover the link before RLF à low latency
A new mechanism introduced to handle frequent Tx/Rx beam mismatch
Beam failure due to blockage
New candidate beam
PRACHfor
beam1
PRACHfor
beam2
PRACHfor
beam3
UE sends PRACH
resource associated
with new candidate
beam to trigger
beam recovery

Ultra-Reliable and Low-Latency Communications
NR introduced several mechanisms to enable URLLC
192© 2018 InterDigital, Inc. All rights reserved.
Low Latency
DL Data
ULCtrl
Gap
HARQ-ACK
Slot
• Self-contained slot
Transmitting HARQ-ACK in the self-
contained slot immediately after
receiving the DL data
• Mini-Slot
Slot
Mini-Slot
Smaller time units
for data scheduling
• Scalable subcarrier spacing
• Grant Free UL transmission
• Very low periodicity SR
• Pre-emption
High Reliability
• Diversity via duplication
f1
f2
Same PDCP PDU on
two carriers
Carrier-Aggregation
LTE NR
Same PDCP PDU over
different RAT (DC)
Dual-Connectivity (EN-DC)
• Control channel duplication
• Polar coding for control information
• Optimized CQI/MCS tables

193
Channel Coding: Data Channel
Ø Enhanced LDPC Codes for eMBB Data Channel
• Advantages of LDPC codes
• Support high data rates (NR peak data rate: 20 Gbps DL/10 Gbps UL)
• High area and energy efficiency and good BLER performance
• Quasi-cyclic LDPC code enables high degree of parallelism
• Parity-check matrix is lifted from small base graph
• Two base graphs
• Optimized for different regions of code rates and code block sizes
• Length and rate flexibility
• Shortening and lifting to achieve 1-bit granularity on code block size
• Circular buffer with puncturing and repetition to achieve codeword
length flexibility;
• Support for both CC-HARQ and IR-HARQ
ØFuture Enhancements
• Potential modifications for URLLC data channel to address the error floor
and achieve high reliability
Channel coding was entirely redesigned for NR
LTE: Turbo NR: LDPC
Throughput Medium-low High
Area efficiency Low High
Energy efficiency Low High
Implementation complexity comparison (R1-167567)
Coverage regions of 2 LDPC base graphs
R=1/4
R=2/3
R=0.95
308 3840 Code
block size
Code
rate
8448
Base Graph 1
Base Graph 2

Channel Coding: LDPC
• Generator matrix
• Parity check matrix
A primer on linear block codes and Tanner graph representation
! =
#$$ #$% … #$'
⋮ ⋮ ⋮ ⋮
#)$ #)% … #)'
=
*$
⋮
*)
*$, *%, …, *) are linearly independent vectors of length ,
k information bits - = (/$, /%, …,/)) mapped to a ,-bit codeword:
1 = - ! = /$*$ + /%*%+ …+/)*)
3: (, − 6)×, matrix, each row is orthogonal to rows of !
Any codeword 1 satisEies: 13F = 0
• Tanner graph
3 =
1 0 1 0
0 1 1 1
x1+ x3 = 0
x2 + x3 + x4 = 0
x1
x2
x3
x4

• Binary Erasure Channel
An example of message passing decoding
x1+ x3 = 0
x2 + x3 + x4 = 0
1
x2=?
x3=?
0
1
0
1
1
1

• Parity check matrix H
• Base graph (matrix of 0’s and 1’s)
• Replace 1 with a cyclically shifted ZxZ identify matrix
• Replace 0 with a ZxZ all-zero matrix
Example:
1 1 0 1 1 0
0 1 1 0 1 0
1 0 1 0 0 1
3 2 N/A 2 1 N/A
N/A 3 1 N/A 0 N/A
1 N/A 3 N/A N/A 0
1 0 0 0
0 1 0 0
0 0 1 0
0 0 0 1
0 0 0 1
1 0 0 0
0 1 0 0
0 0 1 0
Shift 3 positions
0 0 0 1
1 0 0 0
0 1 0 0
0 0 1 0
0 0 1 0
0 0 0 1
1 0 0 0
0 1 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 1 0
0 0 0 1
1 0 0 0
0 1 0 0
0 1 0 0
0 0 1 0
0 0 0 1
1 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
0 0 0 0
First row of base graph results in the first 4 rows of H:
Source: 3GPP TR 38.802, Study on New Radio Access Technology -- Physical Layer Aspects V14.0.0 (2017-03)
This construction is sometimes called lifting.
Parity check matrix for high throughput

197
Channel Coding: Control Channel
Ø Polar Codes for Control Channel and Broadcast Channel
• Advantages of Polar codes
• Good BLER performance at small block sizes and no error floor
• Polar codes are used to encode DCI, UCI (>11 bits) and MIB
• A single nested polar sequence
• Short polar sequence is derived from long polar sequence
• Optimized Rate Matching (RM) scheme
• Select among three RM schemes: puncturing, shortening, repetition
• Unified circular buffer design for all three RM schemes
• Triangular channel interleaver (UCI only)
• Advanced polar code construction
• Distributed CRC for DCI and MIB to achieve early termination
• Parity check polar code for small size UCI for better performance
ØFuture Enhancements
• Extension of polar code for URLLC data channel
NR is the first commercial system adopting Polar codes
Unified circular buffer design for three RM schemes
Polar
Encoder
(Mother
code
length N)
Sub-
block
0
Sub-
block
1
Sub-
block
31
Sub-
block
0
Sub-
block
1
Sub-
block
31
Sub-block wise
interleaving
Circular
buffer
Shortening
Repetition
Puncturing
Selecting bits from the
unified circular buffer
Rate
matching

Channel Coding: Polar code
• Invented by Erdal Arikan in 2007
• Based on the phenomena of channel polarization
W
X Y
1. Binary Discrete Memoryless Channel (DMC)
2. Consider N copies of the DMC, and channel code X = GU
W
X1 Y1
W
X2 Y2
W
XN YN
...
G
U1
U2
UN
...
...
Information bits codeword ! "#, "%, … , "'; )#, )%, … , )'
= +
,-#
'
! ",; )#, )%, … , )' "#, … , ",.#
= +
,-#
'
!(",; )#, )%, … , )', "#, … , ",.#)
12,: ", → ()#, )%, … , )',"#, … , ",.#New channels
Transform scalar channels into equivalent vector channels

Channel Coding: Polar code
• N=8
Transform scalar channels into equivalent vector channels (Example)
• N=2
• N=4
W
W
Y1
Y2
U1
U2
W
W
W
W
Y1
Y2
Y3
Y4
U1
U2
U3
U4
W
W
W
W
W
W
W
W
U1'
U2'
U3'
U4'
U1"
U2"
U3"
U4"
Y1
Y2
Y3
Y4
Y5
Y6
Y7
Y8
U1
U2
U3
U4
U5
U6
U7
U8

QoS Provisioning
• 4G: there are 9 pre-defined QoS Class Identifier (QCI) values
By a combination of QCI, ARP, GBR/Non-GBR
Source: 3GPP TS 23.203, v11.2.0, “Policy and charging control architecture”, June, 2011.
QCI Resource
Type
Priority Packet Delay
Budget
Packet Error Loss
Rate
Example Services
1
GBR
2 100 ms 10-2 Conversational Voice
2 4 150 ms 10-3 Conversational Video (Live Streaming)
3 3 50 ms 10-3 Real Time Gaming
4 5 300 ms 10-6 Non-Conversational Video (Buffered Streaming)
5 1 100 ms 10-6 IMS Signalling
6
Non-GBR 6 300 ms 10-6
Video (Buffered Streaming)
TCP-based (e.g., www, e-mail, chat, ftp, p2p file
sharing, progressive video, etc.)
7
7 100 ms 10-3
Voice,
Video (Live Streaming)
Interactive Gaming
8
8 300 ms 10-6
Video (Buffered Streaming)
TCP-based (e.g., www, e-mail, chat, ftp, p2p file
9 9 sharing, progressive video, etc.)
GBR: Guaranteed Bit Rate
ARP: Allocation and Retention Priority

QoS Provisioning
• 5G: Added new QCI values
More flexible QoS differentiation and more efficient signaling
QCI Resource Type Priority
Level
Packet Delay
Budget
(NOTE 13)
Packet Error
Loss
Rate (NOTE 2)
Example Services
65 GBR 0.7 75 ms
10-2
Mission Critical user plane Push To Talk voice (e.g.,
MCPTT)
67 1.5 100 ms 10-3 Mission Critical Video user plane
75 2.5 50 ms 10-2 V2X messages
69 Non-GBR 0.5 60 ms 10-6 Mission Critical delay sensitive signalling (e.g., MC-
PTT signalling, MC Video signalling)
70 5.5 200 ms 10-6 Mission Critical Data (e.g. example services are the
same as QCI 6/8/9)
79 6.5 50 ms 10-2 V2X messages
80 6.8 10 ms 10-6 Low latency eMBB applications (TCP/UDP-based);
Augmented Reality

QoS Provisioning
• 5G: Added new QCI values
QCI Resource
Type
Priority
Level
Packet Delay
Budget
Packet Error
Loss
Rate
Maximum Burst
Size
Data Rate
Averaging
Window
Example Services
82 GBR 1.9 10 ms 10-4 255 bytes 2 s Discrete Automation
83 GBR 2.2 10 ms 10-4 1358 bytes 2 s Discrete Automation
84 GBR 2.4 30 ms 10-5 1358 bytes 2 s Intelligent Transport
Systems
85 GBR 2.1 5 ms 10-5 255 bytes 2 s
Electricity Distribution-
high voltage

QoS Provisioning
• 5G added SDAP (service data adaptation protocol) sublayer
• 5G introduced reflective QoS
• Derive uplink QoS from downlink QoS
RByRBx
IP Packet
H SDAP SDU
PDCP SDUH
RLC SDUH
MAC SDUH H
IP Packet
H SDAP SDU
PDCP SDUH
RLC SDUH
MAC SDU
IP Packet
H SDAP SDU
PDCP SDUH
SDU SegmentH
MAC SDU
SDU SegmentH
H H MAC SDU
...
...
...
...
MAC PDU – Transport Block
PDCP
RLC
SDAP
MAC
n n+1 m
Source: 3GPP TS 38.300, v15.2.0, “NR; NR and NG-RAN Overall Description”, June, 2018.
RB: Radio Bearer
PDCP: Packet Data Convergence Protocol

204
On-going and Future Work

Outlook for Release 16 Core Network
205
Release 16 aims to enable services intended to be supported by the 5G Core Network
205
5G Massive IoT (MIoT)
§ Enable MTC and Cellular IoT functionalities in the 5G Core Network
§ MTC related features from Rel-10 to Rel-14 (e.g. Congestion
Control, Monitoring, Small Data Transfer, CP/UP Optimizations,
Non-IP Data Delivery, etc…) shall be supported in 5GC
§ Enhancements the 5G system to address Massive IoT (MIoT) use
cases based on the initial 5G requirements
V2X services in 5G
§ System enhancements of EPS and 5G to support advanced
V2X Use cases:
§ Platooning, extended sensor sharing, enhanced
positioning accuracy, advanced driving and remote
driving
Access Traffic Steering Switch and Splitting
§ Multi-access (e.g. 3GPP and WLAN) PDU session management
§ Support traffic splitting between multiple accesses (e.g. 3GPP and
WLAN)
§ Switch traffic between multiple accesses
5G LAN Services
§ Enhancements to the 5G system to support 5GLAN service
§ Support for 3GPP network that is not for public use and
for which service continuity and roaming with a PLMN is
possible (a type network).
§ Support an isolated 3GPP network that does not interact
with a PLMN (b type network)

IEEE ICME Tutorial - Streaming and 5G Fundamentals

IEEE ICME Tutorial - Streaming and 5G Fundamentals

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to IEEE ICME Tutorial - Streaming and 5G Fundamentals

Similar to IEEE ICME Tutorial - Streaming and 5G Fundamentals (20)

More from Alpen-Adria-Universität

More from Alpen-Adria-Universität (20)

Recently uploaded

Recently uploaded (20)

IEEE ICME Tutorial - Streaming and 5G Fundamentals