Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Adaptive Streaming of Traditional and Omnidirectional Media
Ali C. Begen, Comcast/OzU
Christian Timmerer, AAU/Bitmovin
IEE...
Upon Attending This Tutorial, You Will Know About
• Principles of HTTP adaptive streaming for the Web/HTML5
• Principles o...
Ali C. Begen
• Electrical engineering degree from Bilkent
University (2001)
• Ph.D. degree from Georgia Tech (2006)
– Vide...
ACM MMSys 2018 CfP is Out!
IEEE ICME Tutorial - July 2017 4
• City: Amsterdam
• Dates: June 12-15, 2018
• Co-located with
...
Agenda
• Part I: Over-the-top (OTT) video and HTTP adaptive streaming
– HTML5 standard and media extensions
– HTTP adaptiv...
IPTV vs. IP (Over-the-Top) Video
First Things First
IPTV IP Video
Best-effort delivery
Quality not guaranteed
Mostly on de...
Part I: OTT Video and HTTP Adaptive Streaming
8
Internet (IP aka OTT) Video Essentials
• Reach all connected devicesReach
• Enable live and on-demand delivery to the ma...
9
Creating Revenue – Attracting Eye Balls
• High-end content
– Hollywood movies, TV shows
– Sports
• Excellent quality
– H...
One Request, One Response
Progressive Download
HTTP Request
HTTP Response
IEEE ICME Tutorial - July 2017 10
What is Streaming?
Streaming is transmission of a continuous content from a server
to a client and its simultaneous consum...
Stalls, Slow Start-Up, Plug-In and DRM Issues
IEEE ICME Tutorial - July 2017 12
Common Annoyances in Streaming
• Unsupport...
IEEE ICME Tutorial - July 2017 13
Video Delivery over HTTP
• Enables
playback while
still downloading
• Server sends the
f...
Adaptive Streaming over HTTP
14
Decoding and
Presentation
Streaming Client
Media Buffer
Content Ingest
(Live or Pre-captur...
Adapt Video to Web Rather than Changing the Web
15
Adaptive Streaming over HTTP
• Imitation of streaming via short downloa...
Example Representations
Encoding
Bitrate
Resolution
Rep. #1 3.45 Mbps 1280 x 720
Rep. #2 2.2 Mbps 960 x 540
Rep. #3 1.4 Mb...
List of Accessible Segments and Their Timings
An Example Manifest Format
MPD
Period id = 1
start = 0 s
Period id = 3
start...
Smart (aka Selfish and Greedy) Clients
18
Client manages
- Manifest(s)
- HTTP transport
- TCP connection(s)
Client monitor...
End-to-End Workflow for OTT
Production Preparation and Staging Distribution Consumption
News
Gathering
Sport Events
Premiu...
IEEE ICME Tutorial - July 2017 20
IEEE ICME Tutorial - July 2017 21
• HTML5 is a set of technologies that
allows more powerful Web sites and
applications
– ...
• These apps can
– leverage native APIs and offline capability
– provide cross-device compatibility (iOS,
Windows, Android...
IEEE ICME Tutorial - July 2017 23
Types of Browser-Based Playback
Source: CTA WAVE
Type 1: Minimum control architectural m...
IEEE ICME Tutorial - July 2017 24
MSE Support in Web Browsers
Source: http://caniuse.com/#search=mse
IEEE ICME Tutorial - July 2017 25
Content Protection – Famous Quotes
• “Digital files cannot be made uncopiable, any more ...
IEEE ICME Tutorial - July 2017 26
W3C Encrypted
Media Extensions
Source: https://www.w3.org/TR/encrypted-media/
IEEE ICME Tutorial - July 2017 27
EME Support in Web Browsers
Source: http://caniuse.com/#search=eme
Web Video Ecosystem
Encoding
Encryption
Rights Expression
Web Video App Framework
Decoding
Decryption
Rights Management
Th...
Contrary to a common misconception, with EME DRM
functionality is not in the HTML/JS app
There is no DRM in HTML5 with EME...
IEEE ICME Tutorial - July 2017
DRM Support on Desktop Browsers
30
Browser OS EME/CDM Flash Player NPAPI/ Silverlight 5
Chr...
DRM Support on Mobile Platforms
Platform Browser Native App / DRM
iOS No MSE/EME yet
HLS (AES-128 CBC) via <video>
Native ...
Ad Insertion Methods
• Server-based
– MPD is constructed or updated based on the ads
– For multi-period MPDs, we can use
•...
Ad Insertion – XLink Resolution
IEEE ICME Tutorial - July 2017 33
IEEE ICME Tutorial - July 2017 34
Streaming over HTTP – The Promise
• Leverage tried-and-true Web infrastructure for scali...
35
Does It Just Work?
• When streaming clients compete with other traffic, mostly yes
• When streaming clients compete wit...
A Single Microsoft Smooth Streaming Client under a Controlled Environment
36
Demystifying a Streaming Client
0
1
2
3
4
5
0...
10 (Commercial) Streaming Clients Sharing a 10 Mbps Link
IEEE ICME Tutorial - July 2017 37
Typical Example
0
200
400
600
8...
38
Viewer Experience Statistics
Selfishness Hurts Everyone
Source: Conviva Viewer Experience Report, 2015
IEEE ICME Tutori...
Inner and Outer Control Loops
39
HTTP Server
Manifest
Media
HTTP
Origin
Module
TCP Sender
Streaming Client
Manifest
Resour...
IEEE ICME Tutorial - July 2017 40
Tradeoffs in Adaptive Streaming
Overall quality
Quality stability
Proximity to live edge...
Two Competing Clients
41
Understanding the Root Cause
• Depending on the timing of the ON periods:
– Unfairness, underutil...
IEEE ICME Tutorial - July 2017 42
How to Solve the Issues?
Solution
Approaches
Fix the clients
Enable a control
plane
Get ...
IEEE ICME Tutorial - July 2017 43
One Slide on Software Defined Networking (SDN)
Control and data planes are decoupled, ne...
IEEE ICME Tutorial - July 2017 44
SDN-Based Bitrate Adaptation
Reading: “SDNDASH: improving QoE of HTTP adaptive streaming...
Control Points in the Ecosystem
• I want to make sure that my content is protected and looks awesome on any device
• I wan...
IEEE ICME Tutorial - July 2017 46
DASH Part 5: Server and Network Assisted DASH (SAND)
A Control Plane Approach
DANE: DASH...
Dead, Surviving and New-Born Technologies
• Move Adaptive Stream (Long gone)
– http://www.movenetworks.com
• Microsoft Smo...
• Fragmented architectures
– Advertising, DRM, metadata, blackouts, etc.
• Investing in more hardware and
software
– Incre...
49
DASH intends to be to
the Internet world …
what MPEG2-TS has been to
the broadcast world
IEEE ICME Tutorial - July 2017
50
DASH intendsed to be to
the Internet world …
what MPEG2-TS has been to
the broadcast world
IEEE ICME Tutorial - July 20...
What is HAPPENING in MPEG ?
DASH Design Principles
• DASH is not
– system, protocol, presentation, codec, interactivity, DRM, client specification
• D...
53
Shown in Red
Scope of MPEG DASH
HTTP Server DASH Client
Control Engine
MediaEngines
HTTP ClientHTTP 1.1
Segment
ParserM...
IEEE ICME Tutorial - July 2017 54
DASH (ISO/IEC 23009) Parts
• 23009-1: Media Presentation Description and Segment Formats...
IEEE ICME Tutorial - July 2017 55
DASH (ISO/IEC 23009) Parts
• 23009-5: Server and Network Assisted DASH (SAND)
– 1st edit...
IEEE ICME Tutorial - July 2017 56
Part 1 – 3rd Edition
• TV profile
– Broadcast TV content can be distributed over broadca...
IEEE ICME Tutorial - July 2017 57
Part 6 – DASH over Full Duplex HTTP-Based Protocols
• Part 6 describes how a client can ...
IEEE ICME Tutorial - July 2017 58
Ongoing Work as of MPEG 119 (July 2017)
• Core Experiments
– High quality VR delivery wi...
IEEE ICME Tutorial - July 2017 59
Common Media Application Format (CMAF)
• Media delivery has three main components:
– Med...
IEEE ICME Tutorial - July 2017 60
Common Media Application Format (CMAF)
• CMAF defines media profiles for
– Video
– Audio...
IEEE ICME Tutorial - July 2017 61
CMAF Development Timeline
Feb. 2016
Working Draft
June 2016
Committee
Draft
Nov. 2016
Dr...
IEEE ICME Tutorial - July 2017 62
CMAF ISO-BMFF Media Objects
Encoder
Encryption
Packaging
CMAF
Header
CMAF
Fragment
CMAF
...
CMAF Chunks
CMAF
Header
CMAF Content Consumption
IEEE ICME Tutorial - July 2017 63
Un-
packaging
MSE
CMAF
Track File
CMAF ...
• Content format issues
– Each asset is copied to multiple media
formats
• different video codecs
• different audio codecs...
CDN
delivery
ismc
Smooth
mpd
DASH
m3u8
HLS
Source
Encoder
f4m
HDS
m3u8
m3u8
mpd
mpd
ismc
ismc
f4m
f4m
mpd
CMAF
m3u8
mpd
m3...
CTA Web Application Video Ecosystem (WAVE)
Content specification based
upcoming CMAF, compatible
with DASH and HLS
Testabl...
Web Media API Community Group
• Developers want to deploy their content on a range of devices and platforms, e.g., TVs,
se...
Many Definitions Do Exist – It is Like an Elephant
What is Quality?
IEEE ICME Tutorial - July 2017 68
IEEE ICME Tutorial - July 2017 69
Segments Have Different Complexities
Bitrate
Quality Video
Segment #1
Equal Bitrate Allocation
among Segments
Consistent
Q...
Guidelines Limited Bitrate Variability to (Mostly) 10% So Far
Adaptation Feature Does Not Deliver Consistent Quality
Easy
...
What if We Encode in a More Subtle Fashion?
Easy
Moderate
Easy
Easy
Easy
Difficult
Difficult
Difficult
Moderate
Moderate
M...
Content Aware
Encoding
Content Aware
Streaming
Generating VBR-encoded segments is easy,
but streaming them is not!
IEEE IC...
Multiple Representations Naturally Enable “Cherry-Picking”
74
2.5 Mbps
Network
HTTP Server Streaming Client
…
…
…
…
…
3 4
...
75
Dimensions – In-Stream vs. Between-Streams
• Same principle applies to both:
– In-stream Case: Temporal bit shifting be...
Deployment Challenges
• Challenge 1: Development of quality metrics and temporal pooling models
– A common metric that is ...
Deployment Challenges
• Challenge 3: Development of metadata standards
– Computing the quality metric for each segment in ...
Deployment Challenges
• Challenge 4: Expansion to multi-client scenarios
– We need controlled unfairness (which is fairnes...
79
Need for Analytics
Service
Provider:
“Your video
or CDN
provider
must be
slow”
Video/CDN
Provider:
“Your home
network
m...
IEEE ICME Tutorial - July 2017 80
Four Major Areas of Further Exploration
• How to choose bitrate/resolution pairs to make...
Part II: Omnidirectional (360°) Media
MPEG’s Definition
What is Virtual Reality (VR)
VR is a rendered environment (visual and acoustic, pre-
dominantly real-wor...
IEEE ICME Tutorial - July 2017 83
Virtual Reality (VR) Puts Us in a Virtual World
Source: Phil Chou
IEEE ICME Tutorial - July 2017 84
Augmented Reality (AR) Puts Virtual Objects in Our World
Source: Phil Chou
Delivering High-Quality VR in Economic Scale is Extremely Challenging
The VR Challenge
30K x 24K x 36 x 60 x 2 / 600
~5.2 ...
Ultimate Level of Immersion
Interactions
So intuitive that they
become second nature
Sounds
So accurate that
they are true...
Visual
quality
Sound
quality
Intuitive
interactions
Immersive VR Has Extreme Requirements
Immersion
High resolution audio
...
Need for Higher Resolutions on Mobile Phones?
IEEE ICME Tutorial - July 2017 88
Source: Thomas Stockhammer
The Human Eye can Only See High Resolution Where the Fovea is Focused
Foveated Rendering Reduces Pixel Processing
High res...
Generate Video
• Simultaneously capture video with
multiple cameras from different views to
generate 360° spherical video
...
IEEE ICME Tutorial - July 2017 91
Adaptive Streaming of VR Content
• Required resolution of a panoramic video for achievin...
• 3-DoF
– “In which direction I look,” look around the
virtual world from a fixed point
– Detect rotational movement
• 6-D...
Lag Prevents Immersion and Causes Discomfort
Minimizing Motion-to Photon Latency is Important
Low latency Noticeable laten...
MPEG-I
• New MPEG project
– ISO/IEC 23090: Coded Representation of Immersive Media
• Five parts are currently in progress:...
OMAF
• Scope: 360° video and associated audio, 3-DoF only
• Focus on the media format that enables VR video production, de...
DASH/OMAF System Diagram
Multi-
camera
capture
Stitch Project
HEVC
encode
OMAF
composing
DASH server
DASH
client
network
H...
Adaptive Streaming of VR Content
IEEE ICME Tutorial - July 2017 97
Projection
format:
Equirectangular
projection
Coding fo...
2018 20202017 2019 2021
Internet Video
Coding
IoMT
Media Orchestration
CMAF
CDVA (Video Analysis)
Audio Wave
Field Coding
...
Tile-Based Streaming
IEEE ICME Tutorial - July 2017 99
Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6
Tile 7 Tile 8 Tile 9
Tile...
IEEE ICME Tutorial - July 2017 100
Tile-Based Streaming Options
IEEE ICME Tutorial - July 2017 101
ODV Quality Metrics: S-PSNR
IEEE ICME Tutorial - July 2017 102
ODV Quality Metrics: V-PSNR
Thank You
“Television! Teacher, mother, secret lover”
– Homer Simpson
IEEE ICME Tutorial - July 2017 103
IEEE ICME Tutorial - July 2017 104
Source Code for Client Implementations
• DASH Industry Forum
– http://dashif.org/softwa...
IEEE ICME Tutorial - July 2017 105
SAND Implementations on GitHub
• SANDLibrary: https://github.com/fraunhoferfokus/SANDLi...
Upcoming SlideShare
Loading in …5
×

Adaptive Streaming of Traditional and Omnidirectional Media

1,446 views

Published on

This tutorial consists of three main parts. In the first part, we provide a detailed overview of the HTML5 standard and show how it can be used for adaptive streaming deployments. In particular, we focus on the HTML5 video, media extensions, and multi-bitrate encoding, encapsulation and encryption workflows, and survey well-established streaming solutions. Furthermore, we present experiences from the existing deployments and the relevant de jure and de facto standards (DASH, HLS, CMAF) in this space. In the second part, we focus on omnidirectional (360) media from creation to consumption. We survey means for the acquisition, projection, coding and packaging of omnidirectional media as well as delivery, decoding and rendering methods. Emerging standards and industry practices are covered as well. The last part presents some of the current research trends, open issues that need further exploration and investigation, and various efforts that are underway in the streaming industry.

Published in: Technology

Adaptive Streaming of Traditional and Omnidirectional Media

  1. 1. Adaptive Streaming of Traditional and Omnidirectional Media Ali C. Begen, Comcast/OzU Christian Timmerer, AAU/Bitmovin IEEE ICME Tutorial – Hong Kong, July 2017
  2. 2. Upon Attending This Tutorial, You Will Know About • Principles of HTTP adaptive streaming for the Web/HTML5 • Principles of omnidirectional (360°) media delivery • Content generation/distribution/consumption workflows for traditional and omnidirectional media • Standards and emerging technologies in the adaptive streaming space • Current and future research on traditional and omnidirectional media delivery This tutorial is public, however, some material might be copyrighted Use proper citation when using content from this tutorial (Thanks to T. Stockhammer, J. Simmons, K. Hughes, C. Concolato, S. Pham, W. Law and many others for helping with the material) IEEE ICME Tutorial - July 2017 2
  3. 3. Ali C. Begen • Electrical engineering degree from Bilkent University (2001) • Ph.D. degree from Georgia Tech (2006) – Video delivery and multimedia communications • Research, development and standards at Cisco (2007-2015) – IPTV, content delivery, software clients – Transport and distribution over IP networks – Enterprise video • Consulting at Networked Media since 2016 • Assistant professor at OzU and IEEE Distinguished Lecturer since 2016 Christian Timmerer • Associate Professor at the Institute of Information Technology (ITEC), Multimedia Communication Group (MMC), Alpen-Adria- Universität Klagenfurt, Austria • Co-founder and CIO of Bitmovin • Research interests – Immersive multimedia communication – Streaming, adaptation, and – Quality of experience (QoE) • Blog: http://blog.timmerer.com; @timse7 Presenters Today IEEE ICME Tutorial - July 2017 3
  4. 4. ACM MMSys 2018 CfP is Out! IEEE ICME Tutorial - July 2017 4 • City: Amsterdam • Dates: June 12-15, 2018 • Co-located with – NOSSDAV – MoVid – MMVE • http://www.mmsys2018.org/
  5. 5. Agenda • Part I: Over-the-top (OTT) video and HTTP adaptive streaming – HTML5 standard and media extensions – HTTP adaptive streaming building blocks – Multi-bitrate encoding, encapsulation and encryption workflows – De jure and de facto standards (DASH, HLS, CMAF) – Experiences from the existing deployments • Part II: Omnidirectional (360°) media – Acquisition, projection, coding and packaging – Delivery, decoding and rendering – Emerging standards and industry practices – Current research trends, open issues, efforts that are underway in the streaming industry IEEE ICME Tutorial - July 2017 5
  6. 6. IPTV vs. IP (Over-the-Top) Video First Things First IPTV IP Video Best-effort delivery Quality not guaranteed Mostly on demand Paid or ad-based service Managed delivery Emphasis on quality Mostly linear TV Always a paid service IEEE ICME Tutorial - July 2017 6
  7. 7. Part I: OTT Video and HTTP Adaptive Streaming
  8. 8. 8 Internet (IP aka OTT) Video Essentials • Reach all connected devicesReach • Enable live and on-demand delivery to the mass marketScale • Provide TV-like or even better and richer viewer experienceQuality of Experience • Enable revenue generation thru paid content, subscriptions or targeted advertising Business • Satisfy regulations for captioning, ratings, parental control and advertising Regulatory IEEE ICME Tutorial - July 2017
  9. 9. 9 Creating Revenue – Attracting Eye Balls • High-end content – Hollywood movies, TV shows – Sports • Excellent quality – HD/3D/UHD audiovisual presentation w/o artifacts such as pixelization and rebuffering – Fast startup, fast zapping and low glass-to-glass delay • Usability – Navigation, content discovery, battery consumption, trick modes, social network integration • Service flexibility – Linear TV – Time-shifted and on-demand services • Reach – Any device, any time IEEE ICME Tutorial - July 2017
  10. 10. One Request, One Response Progressive Download HTTP Request HTTP Response IEEE ICME Tutorial - July 2017 10
  11. 11. What is Streaming? Streaming is transmission of a continuous content from a server to a client and its simultaneous consumption by the client Two Main Characteristics 1. Client consumption rate may be limited by real-time constraints as opposed to just bandwidth availability 2. Server transmission rate (loosely or tightly) matches to client consumption rate IEEE ICME Tutorial - July 2017 11
  12. 12. Stalls, Slow Start-Up, Plug-In and DRM Issues IEEE ICME Tutorial - July 2017 12 Common Annoyances in Streaming • Unsupported/wrong – protocol – plug-in – codec – format – DRM • Slow start-up • Poor quality, quality variation • Frequent freezes/glitches • Lack of seeking
  13. 13. IEEE ICME Tutorial - July 2017 13 Video Delivery over HTTP • Enables playback while still downloading • Server sends the file as fast as possible Progressive Download • Enables seeking via media indexing • Server paces transmission based on encoding rate Pseudo Streaming • Content is divided into short-duration chunks • Enables live streaming and ad insertion Chunked Streaming • Multiple versions of the content are created • Enables to adapt to network and device conditions Adaptive Streaming
  14. 14. Adaptive Streaming over HTTP 14 Decoding and Presentation Streaming Client Media Buffer Content Ingest (Live or Pre-captured) Multi-rate Encoder Packager Origin (HTTP) Server … … … … ServerStorage HTTP GET Request Response IEEE ICME Tutorial - July 2017
  15. 15. Adapt Video to Web Rather than Changing the Web 15 Adaptive Streaming over HTTP • Imitation of streaming via short downloads – Downloads small chunks to minimize bandwidth waste – Enables to monitor consumption and track the streaming clients • Adaptation to dynamic conditions and device capabilities – Adapts to dynamic conditions in the Internet and home network – Adapts to display resolution, CPU and memory resources of the streaming client à Facilitates “any device, anywhere, anytime” paradigm • Improved quality of experience (not always improved average quality) – Enables faster start-up and seeking, and quicker buffer fills – Reduces skips, freezes and stutters • Use of HTTP – Well-understood naming/addressing approach, and authentication/authorization infrastructure – Provides easy traversal for all kinds of middleboxes (e.g., NATs, firewalls) – Enables cloud access, leverages the existing and cheap HTTP caching infrastructure IEEE ICME Tutorial - July 2017
  16. 16. Example Representations Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 2.2 Mbps 960 x 540 Rep. #3 1.4 Mbps 960 x 540 Rep. #4 900 Kbps 512 x 288 Rep. #5 600 Kbps 512 x 288 Rep. #6 400 Kbps 340 x 192 Rep. #7 200 Kbps 340 x 192 Source: Vertigo MIX10, Alex Zambelli’s Streaming Media Blog, Akamai Vancouver 2010 Sochi 2014 Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 1.95 Mbps 848 x 480 Rep. #3 1.25 Mbps 640 x 360 Rep. #4 900 Kbps 512 x 288 Rep. #5 600 Kbps 400 x 224 Rep. #6 400 Kbps 312 x 176 Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 2.2 Mbps 1024 x 576 Rep. #3 1.4 Mbps 768 x 432 Rep. #4 950 Kbps 640 x 360 Rep. #5 600 Kbps 512 x 288 Rep. #6 400 Kbps 384 x 216 Rep. #7 250 Kbps 384 x 216 Rep. #8 150 Kbps 256 x 144 FIFA 2014 IEEE ICME Tutorial - July 2017 16
  17. 17. List of Accessible Segments and Their Timings An Example Manifest Format MPD Period id = 1 start = 0 s Period id = 3 start = 300 s Period id = 4 start = 850 s Period id = 2 start = 100 s Adaptation Set 0 subtitle turkish Adaptation Set 2 audio english Adaptation Set 1 BaseURL=http://abr.rocks.com/ Representation 2 Rate = 1 Mbps Representation 4 Rate = 3 Mbps Representation 1 Rate = 500 Kbps Representation 3 Rate = 2 Mbps Resolution = 720p Segment Info Duration = 10 s Template: 3/$Number$.mp4 Segment Access Initialization Segment http://abr.rocks.com/3/0.mp4 Media Segment 1 start = 0 s http://abr.rocks.com/3/1.mp4 Media Segment 2 start = 10 s http://abr.rocks.com/3/2.mp4 Adaptation Set 3 audio italian Adaptation Set 1 video Period id = 2 start = 100 s Representation 3 Rate = 2 Mbps Selection of components/tracks Well-defined media format Selection of representations Splicing of arbitrary content like ads Chunks with addresses and timing 17IEEE ICME Tutorial - July 2017
  18. 18. Smart (aka Selfish and Greedy) Clients 18 Client manages - Manifest(s) - HTTP transport - TCP connection(s) Client monitors/measures - Playout buffer - Download times and throughput - Local resources (CPU, memory, screen, etc.) - Dropped frames Client performs adaptation Request IEEE ICME Tutorial - July 2017 Response
  19. 19. End-to-End Workflow for OTT Production Preparation and Staging Distribution Consumption News Gathering Sport Events Premium Content Studio Multi-bitrate Encoding Encapsulation Protection Origin Servers VoD Content & Manifests Live Content & Manifests CDN IEEE ICME Tutorial - July 2017 19
  20. 20. IEEE ICME Tutorial - July 2017 20
  21. 21. IEEE ICME Tutorial - July 2017 21 • HTML5 is a set of technologies that allows more powerful Web sites and applications – Better semantics – Better connectivity – Offline and storage – Multimedia – 2D/3D graphics and effects – Performance and integration – Device access – Styling • Most interesting new elements – Semantic elements: <header>, <footer>, <article> and <section> – Graphic elements: <svg> and <canvas> – Multimedia elements: <audio> and <video> A Common Platform across Devices HTML5 Source: Mozilla MDN, w3schools.com
  22. 22. • These apps can – leverage native APIs and offline capability – provide cross-device compatibility (iOS, Windows, Android, etc.) – be packaged and published to app stores • For streaming: – To a user, it looks like a regular native app they have downloaded – The app is actually a container and loads loads dynamically from the web server – Server-side changes propagate automatically to installed apps Hosted and Progressive Web Apps IEEE ICME Tutorial - July 2017 22 Source: Google PWA, Microsoft HWA
  23. 23. IEEE ICME Tutorial - July 2017 23 Types of Browser-Based Playback Source: CTA WAVE Type 1: Minimum control architectural model for HTML5 support of adaptive streaming where manifest and heuristics are managed by the user agent Type 2: Adaptation control architectural model for HTML5 support of adaptive streaming providing script manageable features Type 3: Full media control architectural model for HTML5 support of adaptive streaming allowing script to explicitly send the media segments (MSE + EME)
  24. 24. IEEE ICME Tutorial - July 2017 24 MSE Support in Web Browsers Source: http://caniuse.com/#search=mse
  25. 25. IEEE ICME Tutorial - July 2017 25 Content Protection – Famous Quotes • “Digital files cannot be made uncopiable, any more than water can be made not wet” – Bruce Schneier (cryptographer), May 2001 • “We have Ph.D.'s here that know the stuff cold, and we don't believe it's possible to protect digital content” – Steve Jobs, December 2003 • “If we’re still talking about DRM in five years, please take me out and shoot me” – eMusic CEO David Pakman, February 2007 • “This is unethical” – Ian Hickson, HTML5 editor, upon learning of the Netflix-Google-Microsoft EME proposal - February 2012
  26. 26. IEEE ICME Tutorial - July 2017 26 W3C Encrypted Media Extensions Source: https://www.w3.org/TR/encrypted-media/
  27. 27. IEEE ICME Tutorial - July 2017 27 EME Support in Web Browsers Source: http://caniuse.com/#search=eme
  28. 28. Web Video Ecosystem Encoding Encryption Rights Expression Web Video App Framework Decoding Decryption Rights Management The fundamental Digital Rights Management problem derives from a lack of interoperability which prevents mobility of experience The solution is to combine interoperable commercial Web video content with a cross- platform Web video app framework IEEE ICME Tutorial - July 2017 28 Source: John Simmons
  29. 29. Contrary to a common misconception, with EME DRM functionality is not in the HTML/JS app There is no DRM in HTML5 with EME, and ECP requires that this be the case Digital Rights Management The HTML/JS app selects the DRM and controls key exchange between DRM client and server HTML/JavaScript Application Browser extends HTML5 media element to allow JavaScript handled key acquisition Browser A CDM exposes a key system to JavaScript; it is transparent whether the CDM is in the browser Content Decryption Module The Web, EME and Enhanced Content Protection (ECP) IEEE ICME Tutorial - July 2017 29 Source: John Simmons
  30. 30. IEEE ICME Tutorial - July 2017 DRM Support on Desktop Browsers 30 Browser OS EME/CDM Flash Player NPAPI/ Silverlight 5 Chrome Win Widevine (Yes) No OS X (Yes) No Linux (Yes) No Firefox Win Widevine Yes No OS X Yes No Linux Yes No Safari > OS X Yosemite (Fairplay) Yes Yes < OS X Yosemite No Yes Yes IE/Edge < Win 7 No Yes Yes Win 10 PlayReady Yes No
  31. 31. DRM Support on Mobile Platforms Platform Browser Native App / DRM iOS No MSE/EME yet HLS (AES-128 CBC) via <video> Native SDK and WebView: FairPlay Android MSE/EME Native + Android SDK (MediaDRM APIs) - (Widevine + OMA v2) ExoPlayer Native + WebView with Widevine Windows 10 MSE/EME WebViews/Hosted Web Apps with PlayReady IEEE ICME Tutorial - July 2017 31
  32. 32. Ad Insertion Methods • Server-based – MPD is constructed or updated based on the ads – For multi-period MPDs, we can use • XLink • MPD chaining – For single-period MPDs • Ads are spliced into a continuous stream • Splicing can be offline or just-in-time • App-based – Requires additional support in the application – Uses DASH-specified user-defined events IEEE ICME Tutorial - July 2017 32
  33. 33. Ad Insertion – XLink Resolution IEEE ICME Tutorial - July 2017 33
  34. 34. IEEE ICME Tutorial - July 2017 34 Streaming over HTTP – The Promise • Leverage tried-and-true Web infrastructure for scaling – Video is just ordinary Web content! • Leverage tried-and-true TCP – Congestion avoidance – Reliability – No special QoS for video It should all “just work” J
  35. 35. 35 Does It Just Work? • When streaming clients compete with other traffic, mostly yes • When streaming clients compete with each other, we begin to see problems: – The clients’ adaptation behaviors interact with each other – The competing clients form an “accidental” distributed control-feedback system • Unexpected behaviors will result in places like – Multiple screens within a household – ISP access and aggregation links – Small cells in stadiums and malls IEEE ICME Tutorial - July 2017
  36. 36. A Single Microsoft Smooth Streaming Client under a Controlled Environment 36 Demystifying a Streaming Client 0 1 2 3 4 5 0 50 100 150 200 250 300 350 400 450 500 Bitrate(Mbps) Time (s) Available Bandwidth Requests Chunk Tput Average Tput Reading: “An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP,” ACM MMSys 2011 Buffer-filling State Back-to-back requests (Not pipelined) Steady State Periodic requests IEEE ICME Tutorial - July 2017
  37. 37. 10 (Commercial) Streaming Clients Sharing a 10 Mbps Link IEEE ICME Tutorial - July 2017 37 Typical Example 0 200 400 600 800 1000 1200 1400 0 100 200 300 400 500 RequestedBitrate(Kbps) Time (s) Client1 Client2 Client3
  38. 38. 38 Viewer Experience Statistics Selfishness Hurts Everyone Source: Conviva Viewer Experience Report, 2015 IEEE ICME Tutorial - July 2017
  39. 39. Inner and Outer Control Loops 39 HTTP Server Manifest Media HTTP Origin Module TCP Sender Streaming Client Manifest Resource Monitors Streaming Application TCP ReceiverData / ACK Request Response There could be multiple TCPs destined to the same or different servers IEEE ICME Tutorial - July 2017
  40. 40. IEEE ICME Tutorial - July 2017 40 Tradeoffs in Adaptive Streaming Overall quality Quality stability Proximity to live edge Stalls Zapping/seeking time
  41. 41. Two Competing Clients 41 Understanding the Root Cause • Depending on the timing of the ON periods: – Unfairness, underutilization and/or instability may occur – Clients may grossly overestimate their fair share of the available bandwidth Clients cannot figure out how much bandwidth to use until they use too much Reading: “What happens when HTTP adaptive streaming players compete for bandwidth?,” ACM NOSSDAV 2012 IEEE ICME Tutorial - July 2017
  42. 42. IEEE ICME Tutorial - July 2017 42 How to Solve the Issues? Solution Approaches Fix the clients Enable a control plane Get support from the network
  43. 43. IEEE ICME Tutorial - July 2017 43 One Slide on Software Defined Networking (SDN) Control and data planes are decoupled, network intelligence and state are logically centralized, and the underlying network infrastructure is abstracted from the apps
  44. 44. IEEE ICME Tutorial - July 2017 44 SDN-Based Bitrate Adaptation Reading: “SDNDASH: improving QoE of HTTP adaptive streaming using software defined networking,” ACM MM 2016
  45. 45. Control Points in the Ecosystem • I want to make sure that my content is protected and looks awesome on any device • I want to make sure that my ads are viewed, trackable and measurable • I want to make sure that my servers are properly used and latency is low • I want to control the QoE of all my customers, differentiate my own services, make $$$ from OTT services • I want to make sure that my device/app provides the best possible video quality • I want the best quality for minimal $ ConsumerContent Provider ISP CDN ProviderAdvertiser Device/ App 45IEEE ICME Tutorial - July 2017
  46. 46. IEEE ICME Tutorial - July 2017 46 DASH Part 5: Server and Network Assisted DASH (SAND) A Control Plane Approach DANE: DASH-assisting network element PER: Parameters for enhancing reception PED: Parameters for enhancing delivery Media PER Messages Metrics and Status Messages PED Messages DANE (Origin) DANE Regular Network Element DASH Client DANE (Analytics)
  47. 47. Dead, Surviving and New-Born Technologies • Move Adaptive Stream (Long gone) – http://www.movenetworks.com • Microsoft Smooth Streaming (Legacy) – http://www.iis.net/expand/SmoothStreaming • Adobe Flash (Almost dead) – http://www.adobe.com/products/flashplayer.html • Adobe HTTP Dynamic Streaming (Legacy) – http://www.adobe.com/products/httpdynamicstreaming • Apple HTTP Live Streaming (The elephant in the room) – http://tools.ietf.org/html/draft-pantos-http-live-streaming (Soon to be an RFC) • MPEG DASH and CMAF (The standards) – http://mpeg.chiariglione.org/standards/mpeg-dash – http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format IEEE ICME Tutorial - July 2017 47
  48. 48. • Fragmented architectures – Advertising, DRM, metadata, blackouts, etc. • Investing in more hardware and software – Increased CapEx and OpEx • Lack of consistent analytics • Preparing and delivering each asset in several incompatible formats – Higher storage and transport costs • Confusion due to the lack of skills to troubleshoot problems • Lack of common experience across devices for the same service – Tricks, captions, subtitles, ads, etc. What does Fragmentation Mean? Higher Costs Less Scalability Smaller Reach Frustration Skepticism Slow Adoption IEEE ICME Tutorial - July 2017 48
  49. 49. 49 DASH intends to be to the Internet world … what MPEG2-TS has been to the broadcast world IEEE ICME Tutorial - July 2017
  50. 50. 50 DASH intendsed to be to the Internet world … what MPEG2-TS has been to the broadcast world IEEE ICME Tutorial - July 2017
  51. 51. What is HAPPENING in MPEG ?
  52. 52. DASH Design Principles • DASH is not – system, protocol, presentation, codec, interactivity, DRM, client specification • DASH is an enabler – It provides formats to enable efficient and high-quality delivery of streaming services over the Internet – System definition is left to other organizations • Design choices – Enable reuse of existing technologies (containers, codecs, DRM, etc.) – Enable deployment on top of CDNs – Enable live and on-demand experiences – Move intelligence from network to client, enable client differentiation – Provide simple interoperability points (profiles) IEEE ICME Tutorial - July 2017 52
  53. 53. 53 Shown in Red Scope of MPEG DASH HTTP Server DASH Client Control Engine MediaEngines HTTP ClientHTTP 1.1 Segment ParserMPD Transport MPD Parser MPD MPD MPD ... ... ... IEEE ICME Tutorial - July 2017
  54. 54. IEEE ICME Tutorial - July 2017 54 DASH (ISO/IEC 23009) Parts • 23009-1: Media Presentation Description and Segment Formats – Amd. 1: NTP sync, extended profiles – Amd. 2: SRD, URL parameter insertion, role extensions – Amd. 3: External MPD link, period continuity, generalized HTTP header extensions/queries – Amd. 4: TV profile, MPD chaining/resetting, data URLs in MPD, switching across adaptation sets – à 3rd edition (FDIS), to be published soon • 23009-2: Conformance and Reference Software – 2nd edition (FDIS), to be published in 2017 – Amd. 1: SAND conformance rules • 23009-3: Implementation Guidelines (Informative) – 2nd edition Amd. 1: Multi-track content guidelines and rules, fix on dependent representations • 23009-4: Segment Encryption and Authentication – 2nd edition is in progress
  55. 55. IEEE ICME Tutorial - July 2017 55 DASH (ISO/IEC 23009) Parts • 23009-5: Server and Network Assisted DASH (SAND) – 1st edition, published in Feb. 2017 • 23009-6: DASH over Full Duplex HTTP-based Protocols (FDH) – 1st edition (FDIS), to be published in 2017 • 23009-7: Delivery of CMAF Contents with DASH (Informative) – WD available, PDTR to be published after the July meeting
  56. 56. IEEE ICME Tutorial - July 2017 56 Part 1 – 3rd Edition • TV profile – Broadcast TV content can be distributed over broadcast and/or broadband networks using DASH – Segments do not need to be a switching access point • MPD chaining – Multiple MPDs can be chained to each other for easier ad insertion • Data URLs in MPD – Initialization segments can be included in the MPD (only in the TV profile) • Switching across adaptation sets – A capable client can switch between AVC and HEVC tracks
  57. 57. IEEE ICME Tutorial - July 2017 57 Part 6 – DASH over Full Duplex HTTP-Based Protocols • Part 6 describes how a client can signal various types of “push” behaviors to a server in HTTP/2 and for WebSockets • This is primarily targeted for low-latency live streaming • Similar activities – DASH over LTE broadcast (3GPP SA4) – DVB ABR multicast – ATSC 3.0 hybrid delivery
  58. 58. IEEE ICME Tutorial - July 2017 58 Ongoing Work as of MPEG 119 (July 2017) • Core Experiments – High quality VR delivery with DASH (DASH-VR) • Technologies under Consideration – Support for Controlled Playback in DASH – Playback control – Owner defined content identifiers within MPDs – Usage of HEVC tile tracks in DASH – Relation between presentation time and EPT – Service-level service protection using segment encryption
  59. 59. IEEE ICME Tutorial - July 2017 59 Common Media Application Format (CMAF) • Media delivery has three main components: – Media format – Manifest – Delivery • CMAF defines the media format only (Fragments, headers, segments, chunks, tracks) – CMAF does not specify a manifest format – CMAF does not specify a delivery method, either • CMAF uses ISO-BMFF and common encryption (CENC) – CENC means the media fragments can be decrypted/decoded by devices using different DRMs (aka multi-DRM support) – CMAF does not mandate CTR or CBC mode (which makes CMAF useful only for unencrypted content for now) • MPEG technologies (DASH and MMT) may be used for delivering CMAF content
  60. 60. IEEE ICME Tutorial - July 2017 60 Common Media Application Format (CMAF) • CMAF defines media profiles for – Video – Audio – Subtitle • CMAF defines presentation profiles by selection a media profile from each category • Current status – iOS 10 supports CMAF with HLS manifests – PDAM1 (w16821) • Discussions on SHVC support • AAC 7.1 support – Conformance and interop is WIP with DASH-IF – Non-MPEG codecs are under discussion
  61. 61. IEEE ICME Tutorial - July 2017 61 CMAF Development Timeline Feb. 2016 Working Draft June 2016 Committee Draft Nov. 2016 Draft International Standard Jan. 2017 Study of Draft International Standard May 2017 Final Draft International Standard July 2017 Approved International Standard Ballot Ballot Ballot
  62. 62. IEEE ICME Tutorial - July 2017 62 CMAF ISO-BMFF Media Objects Encoder Encryption Packaging CMAF Header CMAF Fragment CMAF Fragment CMAF Chunk CMAF Chunk CMAF Chunk CMAF Fragment R A P R A P R A P R A P CMAF Fragment CMAF Segment CMAF Segment CMAF Track File • Manifests typically provide URLs to – CMAF track files – CMAF header + CMAF segments • single/multiple fragment(s) – CMAF header + CMAF chunk
  63. 63. CMAF Chunks CMAF Header CMAF Content Consumption IEEE ICME Tutorial - July 2017 63 Un- packaging MSE CMAF Track File CMAF Fragments Switching can only happen seamlessly at CMAF Fragment boundaries CMAF Segments CMAF Header CMAF Header
  64. 64. • Content format issues – Each asset is copied to multiple media formats • different video codecs • different audio codecs • different (regional) frame rates – Cost to content creators and distributors – Inefficiencies in CDNs and higher storage costs • Platform issues – Lack of consistent app behavior across platforms – Varying video features, APIs and semantics across platforms • Playback issues – Codec incompatibility – Partial profile support – Switching bitrate glitches – Audio discontinuities – Ad splicing problems – Long-term playback instability – Request protocol deficiencies – Memory problems, CPU weaknesses – Scaling (display) issues – Variable HDR support – Unknown capabilities Commercial OTT Issues 64IEEE ICME Tutorial - July 2017
  65. 65. CDN delivery ismc Smooth mpd DASH m3u8 HLS Source Encoder f4m HDS m3u8 m3u8 mpd mpd ismc ismc f4m f4m mpd CMAF m3u8 mpd m3u8 mpd m3u8 mpd m3u8 mpd m3u8 mpd m3u8 More efficient edge caching Storage and packaging costs reduced Multi-Platform OTT Workflow with CMAF IEEE ICME Tutorial - July 2017 65
  66. 66. CTA Web Application Video Ecosystem (WAVE) Content specification based upcoming CMAF, compatible with DASH and HLS Testable requirements covering the most common device playback interoperability issues Reference application framework based on HTML5 providing functional guidelines for playback interop Content Specification HTML5 Reference Platform Device Playback Requirements Test Suite IEEE ICME Tutorial - July 2017 66
  67. 67. Web Media API Community Group • Developers want to deploy their content on a range of devices and platforms, e.g., TVs, set-tops and mobile devices • To ensure a smooth user experience across devices, these user agents need to support a minimum set of Web technologies • This Community Group plans to specify such a set of Web technologies and additionally plans to provide guidance for developers and implementers • Web Media APIs 2017 – Details the Web APIs that should be included in device implementations in 2017 – https://www.w3.org/community/webmediaapi/ • Web Media Application Developer Guidelines – Companion guide to outline best practices for implementing Web media apps – https://w3c.github.io/webmediaguidelines/ IEEE ICME Tutorial - July 2017 67
  68. 68. Many Definitions Do Exist – It is Like an Elephant What is Quality? IEEE ICME Tutorial - July 2017 68
  69. 69. IEEE ICME Tutorial - July 2017 69
  70. 70. Segments Have Different Complexities Bitrate Quality Video Segment #1 Equal Bitrate Allocation among Segments Consistent Quality Video Segment #2 70IEEE ICME Tutorial - July 2017
  71. 71. Guidelines Limited Bitrate Variability to (Mostly) 10% So Far Adaptation Feature Does Not Deliver Consistent Quality Easy Moderate Easy Easy Easy Difficult Difficult Difficult Moderate Moderate Moderate Moderate … S Time (s) Segment Size 0 2 4 6 8 10 12 14 16 18 20 22 24 Segment Quality QCBR Small variation in encoding bitrate Large variation in quality If there is something worse than having to watch a video at a lousy quality, it is to watch that video with varying quality 71IEEE ICME Tutorial - July 2017
  72. 72. What if We Encode in a More Subtle Fashion? Easy Moderate Easy Easy Easy Difficult Difficult Difficult Moderate Moderate Moderate Moderate … Time (s) Segment Size 0 2 4 6 8 10 12 14 16 18 20 22 24 Segment Quality QVBR While we spend the same total amount of bits, we not only increase average quality but also reduce quality variation Large variation in encoding bitrate Low variation in quality S HLS authoring spec for ATV allows 2x capping rate for VoD. For linear content, variability is limited to 10-25% range. 72IEEE ICME Tutorial - July 2017
  73. 73. Content Aware Encoding Content Aware Streaming Generating VBR-encoded segments is easy, but streaming them is not! IEEE ICME Tutorial - July 2017 73
  74. 74. Multiple Representations Naturally Enable “Cherry-Picking” 74 2.5 Mbps Network HTTP Server Streaming Client … … … … … 3 4 3 4 3 4 3 4 4 Mbps 3 Mbps 2 Mbps 1 Mbps 5 5 5 5 2 2 2 2 2 135 4 Time (s) SegmentSize … S SegmentQuality QVBR 0 2 4 6 8 10 IEEE ICME Tutorial - July 2017 Reading: “Streaming video over HTTP with consistent quality,” ACM MMSys 2014
  75. 75. 75 Dimensions – In-Stream vs. Between-Streams • Same principle applies to both: – In-stream Case: Temporal bit shifting between segments – Between-streams Case: Bit shifting between streams sharing a bottleneck link Bitrate Quality Video Segment 1 Video Segment 2 CBR CQ Bitrate Quality Stream 1 (News) Stream 2 (Sports) Equal Bandwidth Sharing CQ IEEE ICME Tutorial - July 2017
  76. 76. Deployment Challenges • Challenge 1: Development of quality metrics and temporal pooling models – A common metric that is suitable for a variety of content types – A temporal pooling model that will reliably work for different viewer profiles, devices and networks • Different viewers have different sensitivity levels to glitches for different content types, and they are also forgiving in different time scales – A young viewer (likely to have longer-term memory) watching sports on a big screen vs. an elder viewer (likely to have short-term memory) watching news on a smaller screen • Challenge 2: Integration into popular streaming client implementations – Many ecosystems are closed or proprietary, and one may not have access to the client algorithm to make the necessary changes (e.g., Apple HLS in iOS) – Standards bodies and industry consortiums may lead the way to develop certain guidelines 76IEEE ICME Tutorial - July 2017 Reading: “Quality-aware HTTP adaptive streaming,” IBC 2015
  77. 77. Deployment Challenges • Challenge 3: Development of metadata standards – Computing the quality metric for each segment in each representation for each content is a tedious task, which is the easiest to deal with at the encoder or packager – Packing the metric values and conveying this information to all the clients in a timely and scalable manner is an equally important task – The timed metadata spec in MPEG (ISO/IEC 23001-10) is a good candidate for this task 77IEEE ICME Tutorial - July 2017
  78. 78. Deployment Challenges • Challenge 4: Expansion to multi-client scenarios – We need controlled unfairness (which is fairness in quality not bitrate) among clients adaptively streaming the same or a different content over a network sharing resources (e.g., access network) • Easy scenario: One adult watching sports on a big screen vs. one adult watching a food show on a tablet • More complex scenario: One adult watching sports on a phone vs. three adults watching news on a big screen – The optimization across a number of streaming clients has to be done based on the utilities of the streamed videos, which depend on factors such as: • Spatial pooling model • Content types • Content features • Rendering devices • Audience profiles and sizes – SAND can help deploy controlled unfairness that we need in quality-aware streaming in multi- client scenarios 78IEEE ICME Tutorial - July 2017
  79. 79. 79 Need for Analytics Service Provider: “Your video or CDN provider must be slow” Video/CDN Provider: “Your home network must be slow” Consumer: “The device or the app is slow” Device/App Vendor: “It must be the OS” OS Vendor: “Your Internet connection must be bad” IEEE ICME Tutorial - July 2017
  80. 80. IEEE ICME Tutorial - July 2017 80 Four Major Areas of Further Exploration • How to choose bitrate/resolution pairs to make up/downshifts least visible • How to pick the segment durations • How to achieve content aware streaming and not just content aware encoding Content Preparation • What information could the network provide to streaming clients • How to achieve controlled unfairness • MPTCP or QUIC better than TCP? What about multicast and HTTP/2? Distribution and Delivery • How to model streaming dynamics for different genres • How to model the impact of faster zapping and trick modes on the QoE • Understanding the impact of QoE on viewer engagement QoE Modeling and Client Design • Understanding the interaction of adaptive streaming with caching in CDNs • Extracting actions based on real-time analytics • Fixing issues faster and remotely Analytics, Fault Isolation, Diagnostics
  81. 81. Part II: Omnidirectional (360°) Media
  82. 82. MPEG’s Definition What is Virtual Reality (VR) VR is a rendered environment (visual and acoustic, pre- dominantly real-world) providing an immersive experience to a user who can interact with it in a seemingly real or physical way using special electronic equipment IEEE ICME Tutorial - July 2017 82
  83. 83. IEEE ICME Tutorial - July 2017 83 Virtual Reality (VR) Puts Us in a Virtual World Source: Phil Chou
  84. 84. IEEE ICME Tutorial - July 2017 84 Augmented Reality (AR) Puts Virtual Objects in Our World Source: Phil Chou
  85. 85. Delivering High-Quality VR in Economic Scale is Extremely Challenging The VR Challenge 30K x 24K x 36 x 60 x 2 / 600 ~5.2 Gbps Resolution Bit depth Frame rate Stereoscopic Compression gain IEEE ICME Tutorial - July 2017 85
  86. 86. Ultimate Level of Immersion Interactions So intuitive that they become second nature Sounds So accurate that they are true to life Visuals So vibrant that they are eventually indistinguishable from the real world IEEE ICME Tutorial - July 2017 86
  87. 87. Visual quality Sound quality Intuitive interactions Immersive VR Has Extreme Requirements Immersion High resolution audio Up to human hearing capabilities 3D audio Realistic 3D, positional, surround audio that is accurate to the real world Precise motion tracking Accurate on-device motion tracking Minimal latency Minimized system latency to remove perceptible lag Natural user interfaces Seamlessly interact with VR using natural movements, free from wires Extreme pixel quantity and quality Screen is very close to the eyes Spherical view Look anywhere with a full 360° spherical view Stereoscopic display Humans see in 3D IEEE ICME Tutorial - July 2017 87 Source: Thomas Stockhammer
  88. 88. Need for Higher Resolutions on Mobile Phones? IEEE ICME Tutorial - July 2017 88 Source: Thomas Stockhammer
  89. 89. The Human Eye can Only See High Resolution Where the Fovea is Focused Foveated Rendering Reduces Pixel Processing High resolution Low resolution The GPU renders a small rectangle at a high resolution and the rest of the FoV at a lower resolution IEEE ICME Tutorial - July 2017 89
  90. 90. Generate Video • Simultaneously capture video with multiple cameras from different views to generate 360° spherical video • Use twice the cameras for stereoscopic video • Undistort, stitch together and map the discrete images to a equirectangular or cubemap format • Encode video Playback Video • Decode video • Apply an equirectangular or cubemap UV projection • Determine the pose and show appropriate view of 360° spherical video Generating and Consuming 360° Spherical Video Discrete unstitched camera images for 360° spherical view Equirectangular image Cubemap image Left eye VR headset view IEEE ICME Tutorial - July 2017 90
  91. 91. IEEE ICME Tutorial - July 2017 91 Adaptive Streaming of VR Content • Required resolution of a panoramic video for achieving 4K resolution for viewport with 120° field of view (FoV)
  92. 92. • 3-DoF – “In which direction I look,” look around the virtual world from a fixed point – Detect rotational movement • 6-DoF – “Where I am and in which direction I look,” move freely in the virtual world and look around corners – Detect rotational movement and translational movement Precise Motion Tracking of Head Movements and Gaze Z X Z Y Pitch Yaw Roll IEEE ICME Tutorial - July 2017 92
  93. 93. Lag Prevents Immersion and Causes Discomfort Minimizing Motion-to Photon Latency is Important Low latency Noticeable latency IEEE ICME Tutorial - July 2017 93
  94. 94. MPEG-I • New MPEG project – ISO/IEC 23090: Coded Representation of Immersive Media • Five parts are currently in progress: – Technical report: use cases, requirements and architectures – Omnidirectional media format (OMAF) – New and immersive video coding – New and immersive audio coding – Point cloud coding • Two more parts are considered – VR metadata – VR metrics IEEE ICME Tutorial - July 2017 94
  95. 95. OMAF • Scope: 360° video and associated audio, 3-DoF only • Focus on the media format that enables VR video production, delivery and consumption – Projection and region-wise packing that are used to generate 2D video from the sphere signal – File format encapsulation and metadata signaling – DASH encapsulation and metadata signaling – Video/audio codecs and profiles/brands – Informative examples of viewport-dependent approaches IEEE ICME Tutorial - July 2017 95
  96. 96. DASH/OMAF System Diagram Multi- camera capture Stitch Project HEVC encode OMAF composing DASH server DASH client network HEVC Decode Renderer Head mounted device (HMD) DASH encapsulation Head and eye tracking telemetry Request for tiles within FoV window Projection mapping metadata Projection mapping metadata authoring CDN VR player IEEE ICME Tutorial - July 2017 96
  97. 97. Adaptive Streaming of VR Content IEEE ICME Tutorial - July 2017 97 Projection format: Equirectangular projection Coding format: AVC File format: ISOBMFF, M2TS Delivery format: DASH, HLS Web clients: HTML5, MSE, WebVR
  98. 98. 2018 20202017 2019 2021 Internet Video Coding IoMT Media Orchestration CMAF CDVA (Video Analysis) Audio Wave Field Coding AR/VR Audio extensions New codec with Lightfield New & Immersive Video Coding Point Cloud Compression HDR TR HDR TR 2 Omnidirectional Media Format OMAF v2 Genome Compression Network-Distributed Video Coding Coding 2022 Systems and Tools Immersive Media AF Augmented Media AF Cross-platform media distribution VR360, on-demand and live (3-DoF) Immersive media with 6-DoF Combining natural and synthetic content MPEG’s Five-Year Roadmap IEEE ICME Tutorial - July 2017 98
  99. 99. Tile-Based Streaming IEEE ICME Tutorial - July 2017 99 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Encoding & Packaging Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Delivery MPEG-HEVC/H.265 Tiles in ISOBMFF Adaptive Player Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile-based streaming of VR/360° content with MPEG-DASH SRD … … … … Adaptive Streaming using MPEG-DASH SRD Head Mounted Displays Browsers, Smart (Mobile) Devices (Stereo) 2D, (Stereo) 3D
  100. 100. IEEE ICME Tutorial - July 2017 100 Tile-Based Streaming Options
  101. 101. IEEE ICME Tutorial - July 2017 101 ODV Quality Metrics: S-PSNR
  102. 102. IEEE ICME Tutorial - July 2017 102 ODV Quality Metrics: V-PSNR
  103. 103. Thank You “Television! Teacher, mother, secret lover” – Homer Simpson IEEE ICME Tutorial - July 2017 103
  104. 104. IEEE ICME Tutorial - July 2017 104 Source Code for Client Implementations • DASH Industry Forum – http://dashif.org/software/ • Open Source Implementations/Frameworks – http://dash.itec.aau.at/ – http://gpac.wp.mines-telecom.fr/ – https://github.com/google/shaka-player • JW Player – https://github.com/jwplayer/jwplayer
  105. 105. IEEE ICME Tutorial - July 2017 105 SAND Implementations on GitHub • SANDLibrary: https://github.com/fraunhoferfokus/SANDLibrary • SANDServer: https://github.com/fraunhoferfokus/SANDServer • TNO’s SAND demo: https://github.com/tnomedialab/sand ©MatthiasHeyde/FraunhoferFOKUS

×