Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Delivering Traditional and Omnidirectional Media

424 views

Published on

Universal media access as proposed in the late 90s is now closer to reality. Users can generate, distribute and consume almost any media content, anywhere, anytime and with/on any device. A major technical breakthrough was the adaptive streaming over HTTP resulting in the standardization of MPEG-DASH, which is now successfully deployed in most platforms. The next challenge in adaptive media streaming is virtual reality applications and, specifically, omnidirectional (360°) media streaming.

This tutorial first presents a detailed overview of adaptive streaming of both traditional and omnidirectional media, and focuses on the basic principles and paradigms for adaptive streaming. New ways to deliver such media are explored and industry practices are presented. The tutorial then continues with an introduction to the fundamentals of communications over 5G and looks into mobile multimedia applications that are newly enabled or dramatically enhanced by 5G.

A dedicated section in the tutorial covers the much-debated issues related to quality of experience. Additionally, the tutorial provides insights into the standards, open research problems and various efforts that are underway in the streaming industry.

Published in: Technology
  • Be the first to comment

Delivering Traditional and Omnidirectional Media

  1. 1. IEEE ICME Tutorial – San Diego, CA July 2018 Delivering Traditional and Omnidirectional Media Ali C. Begen, Liangping Ma and Christian Timmerer http://www.slideshare.net/christian.timmerer
  2. 2. Upon Attending This Tutorial, You Will Know About • Principles of HTTP adaptive streaming for the Web/HTML5 • Principles of omnidirectional (360°) media delivery • Content generation/distribution/consumption workflows for traditional and omnidirectional media • Standards and emerging technologies in the adaptive streaming space • 5G radio access and core networks, technologies and emerging multimedia use cases This tutorial is public, however, some material might be copyrighted Use proper citation when using content from this tutorial (Thanks to T. Stockhammer, J. Simmons, K. Hughes, C. Concolato, S. Pham, W. Law, N. Weil, P. Chou and many others for helping with the material) IEEE ICME Tutorial - July 2018 2
  3. 3. Ali C. Begen • Electrical engineering degree from Bilkent University (2001) • Ph.D. degree from Georgia Tech (2006) – Video delivery and multimedia communications • Research, development and standards at Cisco (2007-2015) – IPTV, content delivery, software clients – Transport and distribution over IP networks – Enterprise video • Consulting at Networked Media since 2016 • Assistant professor at OzU and IEEE Distinguished Lecturer since 2016 Christian Timmerer • Associate professor at the Institute of Information Technology (ITEC), Multimedia Communication Group (MMC), Alpen-Adria- Universität Klagenfurt, Austria • Co-founder and CIO of Bitmovin • Research interests – Immersive multimedia communication – Streaming, adaptation, and – Quality of experience (QoE) • Blog: http://blog.timmerer.com; @timse7 Presenters Today – Part I IEEE ICME Tutorial - July 2018 3
  4. 4. Liangping Ma • Member of Technical Staff, InterDigital (2009-) • Research Engineer, San Diego Research Center (2005-2007), Argon ST (2007-2009) • PhD, Electrical Engineering, University of Delaware (2004) • BS, Physics, Wuhan University, China (1998) • IEEE ComSoc Distinguished Lecturer since 2017 • Research interests – Video communication – 5G radio access network – Cognitive radios Presenters Today – Part II IEEE ICME Tutorial - July 2018 4
  5. 5. • Two tutorials, 30 talks on – HEVC, VVC and AV1 – Containers – HDR – Next-gen audio – The latest and greatest in streaming – Content protection and more • Cost: Free • Visit http://mile-high.video/ to register Next week in Denver, CO Mile High Video 2018 IEEE ICME Tutorial - July 2018 5
  6. 6. ACM MMSys 2019 Call for Papers and Special Sessions • City: Amherst, MA • Date: June 18-21, 2019 • Submission deadline: Oct. 30 • Co-located with – NOSSDAV – MMVE – Packet Video • http://www.mmsys2019.org/ IEEE ICME Tutorial - July 2018 6
  7. 7. Topics to Cover • Part I: Streaming (Morning) – HTTP adaptive streaming and workflows – HTML5 video and media extensions – Common issues in scaling and multi-screen/hybrid delivery – Overview of Streaming Standards (DASH, CMAF and Apple HLS) – Adaptive delivery of omnidirectional/360° video – The developing MPEG-OMAF and MPEG-I standards • Part II: Communications over 5G (Afternoon) – 5G fundamentals: core network, radio access network and QoS – Multimedia communication case studies – Improving QoE in streaming IEEE ICME Tutorial - July 2018 7
  8. 8. IPTV vs. IP (Over-the-Top) Video First Things First IPTV IP Video Best-effort delivery Quality not guaranteed Mostly on demand Paid or ad-based service Managed delivery Emphasis on quality Mostly linear TV Always a paid service IEEE ICME Tutorial - July 2018 8
  9. 9. Part I: Streaming HTTP Adaptive Streaming and Workflows
  10. 10. Internet (IP aka OTT) Video Essentials Reach all connected devicesReach Enable live and on-demand delivery to the mass marketScale Provide TV-like consistent rich viewer experienceQuality of Experience Enable revenue generation thru paid content, subscriptions, ads, etc.Business Satisfy regulations such as captioning, ratings and parental controlRegulatory IEEE ICME Tutorial - July 2018 10
  11. 11. Creating Revenue – Attracting Eye Balls • High-end content – Hollywood movies, TV shows – Sports • Excellent quality – HD/3D/UHD audiovisual presentation w/o artifacts such as pixelization and rebuffering – Fast startup, fast zapping and low glass-to-glass delay • Usability – Navigation, content discovery, battery consumption, trick modes, social network integration • Service flexibility – Linear TV – Time-shifted and on-demand services • Reach – Any device, any time IEEE ICME Tutorial - July 2018 11
  12. 12. One Request, One Response Progressive Download HTTP Request HTTP Response IEEE ICME Tutorial - July 2018 12
  13. 13. What is Streaming? Streaming is transmission of a continuous content from a server to a client and its simultaneous consumption by the client Two Main Characteristics 1. Client consumption rate may be limited by real-time constraints as opposed to just bandwidth availability 2. Server transmission rate (loosely or tightly) matches to client consumption rate IEEE ICME Tutorial - July 2018 13
  14. 14. Video Delivery over HTTP • Enables playback while still downloading • Server sends the file as fast as possible Progressive Download • Enables seeking via media indexing • Server paces transmission based on encoding rate Pseudo Streaming • Content is divided into short-duration chunks • Enables live streaming and ad insertion Chunked Streaming • Multiple versions of the content are created • Enables to adapt to network and device conditions Adaptive Streaming IEEE ICME Tutorial - July 2018 14
  15. 15. Adaptive Streaming over HTTP Decoding and Presentation Streaming Client Media Buffer Content Ingest (Live or Pre-captured) Multi-rate Encoder Packager Origin (HTTP) Server … … … … ServerStorage HTTP GET Request Response IEEE ICME Tutorial - July 2018 15
  16. 16. Adapt Video to Web Rather than Changing the Web Adaptive Streaming over HTTP • Imitation of streaming via short downloads – Downloads small chunks to minimize bandwidth waste – Enables to monitor consumption and track the streaming clients • Adaptation to dynamic conditions and device capabilities – Adapts to dynamic conditions in the Internet and home network – Adapts to display resolution, CPU and memory resources of the streaming client à Facilitates “any device, anywhere, anytime” paradigm • Improved quality of experience (not always improved average quality) – Enables faster start-up and seeking, and quicker buffer fills – Reduces skips, freezes and stutters • Use of HTTP – Well-understood naming/addressing approach, and authentication/authorization infrastructure – Provides easy traversal for all kinds of middleboxes (e.g., NATs, firewalls) – Enables cloud access, leverages the existing (cheap) HTTP caching infrastructure IEEE ICME Tutorial - July 2018 16
  17. 17. Stalls, Slow Start-Up, Plug-In and DRM Issues Common Annoyances in Streaming • Unsupported/wrong – protocol – plug-in – codec – format – DRM • Slow start-up • Poor quality, quality variation • Frequent freezes/glitches • Lack of seeking IEEE ICME Tutorial - July 2018 17
  18. 18. Dead, Surviving, Maturing and Newborn Technologies • Move Adaptive Stream (Long gone, but some components are in Slingbox) – http://www.movenetworks.com • Microsoft Smooth Streaming (Legacy) – http://www.iis.net/expand/SmoothStreaming • Adobe Flash (Almost dead) – http://www.adobe.com/products/flashplayer.html • Adobe HTTP Dynamic Streaming (Legacy) – http://www.adobe.com/products/httpdynamicstreaming • Apple HTTP Live Streaming (The elephant in the room) – https://tools.ietf.org/html/rfc8216 – https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis • MPEG DASH and CMAF (The standards) – http://mpeg.chiariglione.org/standards/mpeg-dash – http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format IEEE ICME Tutorial - July 2018 18
  19. 19. List of Accessible Segments and Their Timings An Example DASH Template-Based Manifest MPD Period id = 1 start = 0 s Period id = 3 start = 300 s Period id = 4 start = 850 s Period id = 2 start = 100 s Adaptation Set 0 subtitle turkish Adaptation Set 2 audio english Adaptation Set 1 BaseURL=http://abr.rocks.com/ Representation 2 Rate = 1 Mbps Representation 4 Rate = 3 Mbps Representation 1 Rate = 500 Kbps Representation 3 Rate = 2 Mbps Resolution = 720p Segment Info Duration = 10 s Template: 3/$Number$.mp4 Segment Access Initialization Segment http://abr.rocks.com/3/0.mp4 Media Segment 1 start = 0 s http://abr.rocks.com/3/1.mp4 Media Segment 2 start = 10 s http://abr.rocks.com/3/2.mp4 Adaptation Set 3 audio italian Adaptation Set 1 video Period id = 2 start = 100 s Representation 3 Rate = 2 Mbps Selection of components/tracks Well-defined media format Selection of representations Splicing of arbitrary content like ads Chunks with addresses and timing IEEE ICME Tutorial - July 2018 19
  20. 20. An Example HLS Playlist-Based Manifest #EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=232370,CODECS="mp4a.40.2, avc1.4d4015" gear1/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=649879,CODECS="mp4a.40.2, avc1.4d401e" gear2/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1,BANDWIDTH=41457,CODECS="mp4a.40.2" gear0/prog_index.m3u8 master.m3u8 Source: https://developer.apple.com/streaming/examples/ and https://www.gpac-licensing.com/2014/12/01/apple-hls-comparing-versions/ #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-VERSION:3 #EXT-X-MEDIA-SEQUENCE:0 #EXT-X-PLAYLIST-TYPE:VOD #EXTINF:9.97667, fileSequence0.ts #EXTINF:9.97667, fileSequence1.ts #EXTINF:9.97667, fileSequence2.ts . . . #EXT-X-ENDLIST gear1/prog_index.m3u8 IEEE ICME Tutorial - July 2018 20
  21. 21. Example Representations Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 2.2 Mbps 960 x 540 Rep. #3 1.4 Mbps 960 x 540 Rep. #4 900 Kbps 512 x 288 Rep. #5 600 Kbps 512 x 288 Rep. #6 400 Kbps 340 x 192 Rep. #7 200 Kbps 340 x 192 Source: Vertigo MIX10, Alex Zambelli’s Streaming Media Blog, Akamai, Comcast Vancouver 2010 Sochi 2014 Encoding Bitrate Resolution Rep. #1 3.45 Mbps 1280 x 720 Rep. #2 1.95 Mbps 848 x 480 Rep. #3 1.25 Mbps 640 x 360 Rep. #4 900 Kbps 512 x 288 Rep. #5 600 Kbps 400 x 224 Rep. #6 400 Kbps 312 x 176 Encoding Bitrate Resolution Rep. #1 18 Mbps 4K (60p) Rep. #2 12.2 Mbps 2560x1440 (60p) Rep. #3 4.7 Mbps 2K (60p) Rep. #4 3.5 Mbps 1280x720 (60p) Rep. #5 2 Mbps 1280 x 720 Rep. #6 1.2 Mbps 768 x 432 Rep. #7 750 Kbps 640 x 360 Rep. #8 500 Kbps 512 x 288 Rep. #9 300 Kbps 320 x 180 Rep. #10 200 Kbps 320 x 180 PyeongChang 2018 IEEE ICME Tutorial - July 2018 21
  22. 22. Smart and Selfish Clients HAS Working Principle - Client fetches and parses the manifest - Client uses the OS-provided HTTP stack (HTTP may run over TCP or QUIC) - Client uses the required decryption tools for the protected content Client monitors and measures - Size of the playout buffer (both in bytes and seconds) - Chunk download times and throughput - Local resources (CPU, memory, window size, etc.) - Dropped frames Client performs adaptation Request Response HTTPServer Client Client measures and reports metrics for analytics (One can also multicast media segments) IEEE ICME Tutorial - July 2018 22
  23. 23. Tradeoffs in Adaptive Streaming Overall quality Quality stability Proximity to live edge Stalls Zapping/seeking time IEEE ICME Tutorial - July 2018 23
  24. 24. End-to-End Workflow for OTT Production Preparation and Staging Distribution Consumption News Gathering Sport Events Premium Content Studio Multi-bitrate Encoding Encapsulation Protection Origin Servers VoD Content & Manifests Live Content & Manifests CDN IEEE ICME Tutorial - July 2018 24
  25. 25. Part I: Streaming HTML5 Video and Media Extensions
  26. 26. • HTML5 is a set of technologies that allows more powerful Web sites and applications – Better semantics – Better connectivity – Offline and storage – Multimedia – 2D/3D graphics and effects – Performance and integration – Device access – Styling • Most interesting new elements – Semantic elements: <header>, <footer>, <article> and <section> – Graphic elements: <svg> and <canvas> – Multimedia elements: <audio> and <video> A Common Platform across Devices HTML5 Source: Mozilla MDN, w3schools.com IEEE ICME Tutorial - July 2018 26
  27. 27. • These apps can – leverage native APIs and offline capability – provide cross-device compatibility (iOS, Windows, Android, etc.) – be packaged and published to app stores • For streaming: – To a user, it looks like a regular native app they have downloaded – The app is actually a container and loads loads dynamically from the web server – Server-side changes propagate automatically to installed apps Hosted and Progressive Web Apps Source: Google PWA, Microsoft HWA IEEE ICME Tutorial - July 2018 27
  28. 28. Types of Browser-Based Playback Source: CTA WAVE Type 1: Minimum control architectural model for HTML5 support of adaptive streaming where manifest and heuristics are managed by the user agent Type 2: Adaptation control architectural model for HTML5 support of adaptive streaming providing script manageable features Type 3: Full media control architectural model for HTML5 support of adaptive streaming allowing script to explicitly send the media segments (MSE + EME) IEEE ICME Tutorial - July 2018 28
  29. 29. MSE Support in Web Browsers (as of July’18) Source: http://caniuse.com/#search=mse IEEE ICME Tutorial - July 2018 29
  30. 30. Content Protection – Famous Quotes Digital files cannot be made uncopiable, any more than water can be made not wet Bruce Schneier (cryptographer), 2001 We have Ph.D.'s here that know the stuff cold, and we don't believe it's possible to protect digital content Steve Jobs, 2003 If we’re still talking about DRM in five years, please take me out and shoot me eMusic CEO David Pakman, 2007 IEEE ICME Tutorial - July 2018 30
  31. 31. W3C Encrypted Media Extensions Source: https://www.w3.org/TR/encrypted-media/ IEEE ICME Tutorial - July 2018 31
  32. 32. EME Support in Web Browsers (as of July’18) Source: http://caniuse.com/#search=eme IEEE ICME Tutorial - July 2018 32
  33. 33. Web Video Ecosystem Encoding Encryption Rights Expression Web Video App Framework Decoding Decryption Rights Management The fundamental Digital Rights Management problem derives from a lack of interoperability which prevents mobility of experience The solution is to combine interoperable commercial Web video content with a cross- platform Web video app framework Source: John Simmons IEEE ICME Tutorial - July 2018 33
  34. 34. Contrary to a common misconception, with EME DRM functionality is not in the HTML/JS app There is no DRM in HTML5 with EME, and ECP requires that this be the case The HTML/JS app selects the DRM and controls key exchange between DRM client and server Browser extends HTML5 media element to allow JavaScript handled key acquisition Browser A CDM exposes a key system to JavaScript; it is transparent whether the CDM is in the browser The Web, EME and Enhanced Content Protection (ECP) Source: John Simmons IEEE ICME Tutorial - July 2018 34
  35. 35. DRM Support on Desktop Browsers Browser OS EME/CDM Flash Player NPAPI/ Silverlight 5 Chrome Win Widevine (Yes) No OS X (Yes) No Linux (Yes) No Firefox Win Widevine Yes No OS X Yes No Linux Yes No Safari > OS X Yosemite (FairPlay) Yes Yes < OS X Yosemite No Yes Yes IE/Edge < Win 7 No Yes Yes Win 10 PlayReady Yes No IEEE ICME Tutorial - July 2018 35
  36. 36. DRM Support on Mobile Platforms Platform Browser Native App / DRM iOS No MSE/EME yet HLS (AES-128 CBC) via <video> Native SDK and WebView: FairPlay Android MSE/EME Native + Android SDK (MediaDRM APIs) - (Widevine + OMA v2) ExoPlayer Native + WebView with Widevine Windows 10 MSE/EME WebViews/Hosted Web Apps with PlayReady IEEE ICME Tutorial - July 2018 36
  37. 37. • Server-side ad insertion (SSAI) – Mostly used for linear/dynamic ad insertion (ads are still targeted) – Easier for less capable devices – Smoother transition between the content and ads – Uses XLink or MPD chaining in DASH for multi- period MPDs – Splices ads into a continuous stream (offline or just-in-time) for single-period MPDs • Client-side ad insertion – Requires additional support in the application – Uses events in DASH Ad Insertion Methods Preroll Idle Midroll Idle Postroll Preroll Content Midroll Content Postroll Pause Content Pause Content Pause Primary Stream Primary Stream Ad Stream IEEE ICME Tutorial - July 2018 37
  38. 38. Ad Insertion – XLink Resolution IEEE ICME Tutorial - July 2018 38
  39. 39. Part I: Streaming Common Issues in Scaling and Multi-Screen/Hybrid Delivery
  40. 40. Streaming over HTTP – The Promise • Leverage tried-and-true Web infrastructure for scaling – Video is just ordinary Web content! • Leverage tried-and-true TCP – Congestion avoidance – Reliability – No special QoS for video THERE IT SHOULD JUST WORK IEEE ICME Tutorial - July 2018 40
  41. 41. Doesitjustwork? Mostly yes, when streaming clients compete with other types of traffic Not really, when streaming clients compete with each other Streaming clients interact with each other forming an “accidental” distributed control-feedback system • Multiple screens within a household • ISP access/aggregation links • Small cells in stadiums and malls IEEE ICME Tutorial - July 2018 41
  42. 42. A Single Microsoft Smooth Streaming Client under a Controlled Environment Demystifying a Streaming Client 0 1 2 3 4 5 0 50 100 150 200 250 300 350 400 450 500 Bitrate(Mbps) Time (s) Available Bandwidth Requests Chunk Tput Average Tput Reading: “An experimental evaluation of rate-adaptation algorithms in adaptive streaming over HTTP,” ACM MMSys 2011 Buffer-filling State Back-to-back requests Steady State Periodic requests IEEE ICME Tutorial - July 2018 42
  43. 43. 10 (Commercial) Streaming Clients Sharing a 10 Mbps Link Selfishness Hurts Everyone 0 200 400 600 800 1000 1200 1400 0 100 200 300 400 500 RequestedBitrate(Kbps) Time (s) Client1 Client2 Client3 IEEE ICME Tutorial - July 2018 43
  44. 44. Viewer Experience Statistics Selfishness Hurts Everyone Source: Conviva Viewer Experience Report, 2015 IEEE ICME Tutorial - July 2018 44
  45. 45. Inner and Outer Control Loops HTTP Server Manifest Media HTTP Origin Module TCP Sender Streaming Client Manifest Resource Monitors Streaming Application TCP ReceiverData / ACK There could be multiple TCPs destined to the same or different servers Request / Response IEEE ICME Tutorial - July 2018 45
  46. 46. Streaming with Multiple TCP Connections • Using multiple concurrent TCPs – Can help mitigate head-of-line blocking – Allows fetching multiple (sub)segments in parallel – Allows to quickly abandon a non-working connection without having to slow-start a new one System performance deteriorates very quickly if many clients adopt this approach without limiting the aggregated bandwidth consumption IEEE ICME Tutorial - July 2018 46
  47. 47. Two Competing Clients Understanding the Root Cause • Depending on the timing of the ON periods: – Unfairness, underutilization and/or instability may occur – Clients may grossly overestimate their fair share of the available bandwidth Clients cannot figure out how much bandwidth to use until they use too much (Just like TCP) Reading: “What happens when HTTP adaptive streaming players compete for bandwidth?,” ACM NOSSDAV 2012 IEEE ICME Tutorial - July 2018 47
  48. 48. How to Solve the Issues? • Use a better adaptation algorithm like PANDA or BOLA • Use machine learning or deep learning like Pensieve • Improve the HTTP/TCP stack, try out the alternatives • Adopt ideas from game/consensus theory (GTA) Fix the clients and/or the transport • QoS in the core/edge • SDN Get support from the network • Assist the clients and network elements thru metrics and analytics Enable a control plane IEEE ICME Tutorial - July 2018 48
  49. 49. Bitrate Adaptation Schemes Bitrate Adaptation Schemes Client- based Adaptation Bandwidth- based Buffer- based Mixed adaptation Proprietary solutions MDP-based Server- based Adaptation Network- assisted Adaptation Hybrid Adaptation SDN-based Server and network- assisted Reading: “A survey on bitrate adaptation schemes for streaming media over HTTP,” IEEE Commun. Surveys Tuts., to appear IEEE ICME Tutorial - July 2018 49
  50. 50. High-Level Comparison between Different Schemes • The client-based adaptation schemes – Show a good performance in single and few-client scenarios – Largely fail in multi-client scenarios • The server-based adaptation schemes – Require custom servers – More effective in eliminating the bitrate oscillation problems – Less scalable due to increased complexity on the servers • The network-assisted adaptation and hybrid schemes – Show a good performance in both small and large populations – Require modifications on the clients, server and/or network devices – Pose practicality issues for deployment IEEE ICME Tutorial - July 2018 50
  51. 51. A Note on the Reproducibility • A performance comparison between different algorithms is often difficult due to – Lack of the source codes – Lack of a unified QoE framework and universally accepted metrics • Schemes have their own parameters and assumptions, and work well only under certain circumstances IEEE ICME Tutorial - July 2018 51
  52. 52. Game/Consensus Theory • Structuring a set of players in cooperative groups • Enables a strong level of cooperation among the players • Model and analyze fairness, stability and efficiency Coalition Formation • Players that are interested in reaching an agreement among themselves over a common objective (how to reach this agreement and on the terms of the agreement) • Concept solution: Nash Bargaining Solution (NBS) Bargaining Solution • Exploited together with NBS and coalition-formation game • Relied on a strong coordination among the players Consensus Theory IEEE ICME Tutorial - July 2018 52
  53. 53. GTA: A Game Theory Based ABR Scheme • Designed based on a cooperative game in the form of static formation-based coalitions • Formulates the ABR decision problem as a bargaining process and consensus mechanism • Outputs the optimal decision by reaching the Pareto optimal (PO) Nash bargaining solution (NBS) Game Theory Cooperative Coalition Formation Bargaining Solution Consensus Theory Strategic GTA GTA player Coalition Bargaining process IEEE ICME Tutorial - July 2018 53
  54. 54. • PANDA & CS2P Estimators – Throughput estimation • SAND Enabler – Implements various SAND-enabled communication interfaces • SSIMplus MAP Model – Maps content type, display resolution and service plan type into one common perceptual quality space in which a set of clusters are constructed • QoE Metrics – QoE model and various metrics • GT Agent – Develops the GTA decision rules for selecting the best bitrate GTA Components Reading: “Want to play DASH? a game theoretic approach for adaptive streaming over HTTP,” ACM MMSys 2018 IEEE ICME Tutorial - July 2018 54
  55. 55. Component #1: Cooperation Enabler GTA Design Non-Cooperative Mode • Mix of GTA and non-GTA players • Greedy interaction with non-GTA players • GTA clients may start in non- cooperative mode and then learn the accurate number of clients through potential and wonderful life utility (WLU) Cooperative Mode • Multiple GTA players • GTA players are aggregated into a set of clusters based on SSIMplus-based MAP IEEE ICME Tutorial - July 2018 55
  56. 56. Component #2: QoE Calculator GTA Design Avg. Quality Avg. Quality Switch Stalls Startup Delay IEEE ICME Tutorial - July 2018 56
  57. 57. Component #3 and #4: GT Strategy Calculator and GT (Dis)agreement Calculator GTA Design • GT Strategy Calculator – Compute the GT strategy (i.e., the action-utility R(A) relationship set) which is defined over the strategy space S as follows: • GT (Dis)agreement Calculator – Compute the GT disagreement decision • i.e., bargaining outcome disagreements (W) P = {p1,...,pN} is the set of N players R is the set of possible utilities when actions from A are taken • r is the player utility when action a is taken • k is the current step, K is total number of segments of video u R(A) is the set of possible actions with their achievable utilities R-(A-) is the set of pessimal actions with their resulting utilities S and W are defined by the function F which is defined over the space: F: (A → R) ∪ {W} PO (NBS): max. (S – W) AgreementpointDisagreementpoint MAX IEEE ICME Tutorial - July 2018 57
  58. 58. Component #5: QoE Optimizer GTA Design • GTA rule for ABR selection can be formulated as a network utility maximization – Strictly increasing concave – Solver: Dual decomposition + dynamic programming • Find the bargaining solution by solving the Problem(S,W) – Bargaining Process and consensus decision problem is defined as a pair Problem(S,W) – The bargaining solution is defined by a function F over the space F : (S, W ) → Rn – F specifies a unique Pareto optimal (PO) bargaining outcome (O* = F(S,W)) for every Problem(S,W) using Nash Bargaining Solution (NBS) – O* is the set that contains only the optimal bargaining outcomes C.1 Maintain the current buffer occupancy between the min and max buffer thresholds C.2 Satisfy the SSIMplus MAP model C.3 Lead to a unique (PO) NBS C.4 Chunk time constraint where max time needed to download the corresponding chunk ≦ chunk duration IEEE ICME Tutorial - July 2018 58
  59. 59. Source Code is Available at http://streaming.university/GTA/ dash.js Integration Gray boxes indicate the components that have been developed or modified IEEE ICME Tutorial - July 2018 59
  60. 60. Setup for Performance Evaluation • VoD experimental setup – Two machines (running Ubuntu 16.04 LTS), one for the dash.js player and one for the server – tc-NetEm to replicate the throughput profiles – PANDA parameters as described in the original paper – Throughput smoothing: the mean of the last three throughput estimations – The fastMPC horizon was set to the next three chunks – The min and max buffer occupancy thresholds were set to 8 and 32 seconds – CT = animation (BBB), chunk duration of 4 s, total duration of 600 s – 10 bitrate levels: [250, 300, 400, 700, 900, 1500, 2000, 3000, 3500, 4000] Kbps IEEE ICME Tutorial - July 2018 60
  61. 61. Results: Video Stability IEEE ICME Tutorial - July 2018 61
  62. 62. Results: QoE IEEE ICME Tutorial - July 2018 62
  63. 63. Numbers Indicate GTA Improvement over Other Schemes GTA Outperforms the Existing ABR Schemes BBA BOLA Rate-based Hybrid Quetra ELASTIC QDASH SARA FESTIVE QoE 55% 33.5% 35.5% 19.5% 39% 46.5% 47% 40.5% 30.5% Video Stability 60% 72% 37.5% 63.5% 66.5% 70% 66% 53% 63.5% Stall Events 13.6% 1.1% 6.1% 1.7% 3.87% 7.87% 5.87% 2.9% 3.6% IEEE ICME Tutorial - July 2018 63
  64. 64. http://streaming.university/GTA/ GTA Demo and Source Codes Offline Demo Online Demo IEEE ICME Tutorial - July 2018 64
  65. 65. Quickly Starting Media Streams Using QUIC QUIC: Multiplexed transport protocol over UDP • Reduced handshake latency • Improved congestion control • Improved loss recovery • Multiplexing streams Can QUIC help improve viewer experience in HTTP adaptive streaming? Reading: “Quickly starting media streams using QUIC,” PV 2018 IEEE ICME Tutorial - July 2018 65
  66. 66. • Start media streams more quickly • Reduce seeking latency • Cope better with frequent connection changes? Can QUIC help us IEEE ICME Tutorial - July 2018 66
  67. 67. Previous Research • Some reported – QUIC does not impact streaming performance – QUIC does not provide a boost to HAS • Others reported – QUIC’s 0-RTT performed better than the other protocols – QUIC provided better streaming, but only for high-quality video • Google said QUIC reduced YouTube rebuffer rates by 15% QUIC code evolves rapidly and 3rd-party implementations may not necessarily reflect protocol’s real performance IEEE ICME Tutorial - July 2018 67
  68. 68. Comparison between Our and Prior Work 1 WiFi, 4G/LTE and 3G, 2 Based on the third-party implementation version at the time of research, 3 Latest at the time of research. IEEE ICME Tutorial - July 2018 68
  69. 69. • Player Features – Python based – TCP and QUIC support integrated as subprocess – Scripted frame-seek (fast- forward) ability – BASIC, SARA, BBA-2 adaptation algorithms • Source Code – github.com/sevketarisu/quic- streaming github.com/sevketarisu/quic-streaming IEEE ICME Tutorial - July 2018 69
  70. 70. • Environment Setup – Public Internet – No traffic shaping – Each test repeated 10 times • Server – Apache HTTP server on Amazon EC2 – Located in Frankfurt, DE • Client – Runs Google’s proto-quic library – Located in Istanbul, TR – Connected via WiFi, LTE or 3G Chromium QUIC (proto-quic library) WiFi, LTE, 3G Amazon EC2 libcurl IEEE ICME Tutorial - July 2018 70
  71. 71. Evaluated Metrics • Average playback bitrate – Average of the bitrates of the downloaded segments • Average wait time after seeking – Time from the frame-seek request to the playback of the requested media – A rule of thumb is to keep this time under two seconds • Rebuffer rate IEEE ICME Tutorial - July 2018 71
  72. 72. Frame-Seek (Fast-Forward) Feature Implemented in the Player Frame-Seek Scenarios IEEE ICME Tutorial - July 2018 72
  73. 73. Frame-Seek Scenario Results 2.80 2.99 2.862.89 3.05 2.99 0.0 0.5 1.0 1.5 2.0 2.5 3.0 3.5 4.0 WiFi LTE 3G AveragePlaybackBitrate(Mbps) TCP QUIC Results are averaged for BASIC, SARA and BBA-2 algorithms. QUIC provided a higher (or at least an equal) average playback bitrate in all cases and for all algorithms Max Bitrate Available: 3.9 Mbps IEEE ICME Tutorial - July 2018 73
  74. 74. Frame-Seek Scenario Results 2.09 2.16 2.38 1.28 1.29 1.42 0.0 0.5 1.0 1.5 2.0 2.5 3.0 WiFi LTE 3G AverageWaitTimeafterSeeking(s) TCP QUIC Results are averaged for BASIC, SARA and BBA-2 algorithms. QUIC reduced wait times to less than 1.5 seconds IEEE ICME Tutorial - July 2018 74
  75. 75. Frame-Seek Scenario Results 3.67 3.67 5.67 1.00 1.33 2.00 0 1 2 3 4 5 6 WiFi LTE 3G RebufferRate(%) TCP QUIC QUIC’s benefits are bigger when delays are higher Results are averaged for BASIC, SARA and BBA-2 algorithms. IEEE ICME Tutorial - July 2018 75
  76. 76. Connection-Switch Simulation IEEE ICME Tutorial - July 2018 76
  77. 77. Connection-Switch Scenario IEEE ICME Tutorial - July 2018 77
  78. 78. WiFi - LTE Switches Connection-Switch Results 3.00 4.00 5.00 2.00 3.00 4.00 0 1 2 3 4 5 6 BASIC SARA BBA-2 RebufferRate(%) TCP QUIC QUIC provided a higher (or at least an equal) average playback bitrate for all algorithms IEEE ICME Tutorial - July 2018 78
  79. 79. WiFi - 3G Switches Connection-Switch Results 13.00 2.00 5.00 6.00 1.00 4.00 0 2 4 6 8 10 12 14 BASIC SARA BBA-2 RebufferRate(%) TCP QUIC QUIC provided a higher (or at least an equal) average playback bitrate for all algorithms IEEE ICME Tutorial - July 2018 79
  80. 80. But QUIC is still evolving in the IETF; should there be significant changes, the tests will need to be repeated – https://quicwg.org/ – https://github.com/quicwg Conclusions So Far • QUIC reduces wait times and rebuffer rates without reducing playback bitrate • QUIC outperforms TCP when frequent network changes occur • QUIC’s benefits are greater in networks with larger delay (e.g., early generation 3G networks) IEEE ICME Tutorial - July 2018 80
  81. 81. One Slide on Software Defined Networking (SDN) Control and data planes are decoupled, network intelligence and state are logically centralized, and the underlying network infrastructure is abstracted from the apps IEEE ICME Tutorial - July 2018 81
  82. 82. SDN-Based Bitrate Adaptation Reading: “SDNDASH: improving QoE of HTTP adaptive streaming using software defined networking,” ACM MM 2016 IEEE ICME Tutorial - July 2018 82
  83. 83. Server and Network-Assisted Adaptation • A control plane approach that offers asynchronous client-to-network, network-to-client and network-to-network communications – Collect metrics and status information from various entities in the system – Share this information to assist the clients and aware elements to improve media delivery • DASH part 5 (23009-5) outlines the server and network-assisted DASH (SAND) framework along with the necessary interfaces IEEE ICME Tutorial - July 2018 83
  84. 84. Conflicting Goals in the Media Delivery Chain • I want to make sure that my content is protected and looks awesome on any device • I want to make sure that my ads are viewed, trackable and measurable • I want to make sure that my servers are properly used and latency is low • I want to control the QoE of my customers, make $ from and differentiate my own video services • I want to make sure that my device/app provides the best possible video quality • I want the best quality for minimal $ ConsumerContent Provider ISP CDN ProviderAdvertiser Device/ App IEEE ICME Tutorial - July 2018 84
  85. 85. Introducing DASH-Assisting Network Elements (DANE) A Control Plane Approach Origin (HTTP) Server Encoder/ Transcoder Packager CDN RG Clients Media Parameters for enhancing reception (PER) Metrics and status messages Parameters for enhancing delivery (PED) Analytics Server IEEE ICME Tutorial - July 2018 85
  86. 86. Why is Latency Important? IEEE ICME Tutorial - July 2018 86
  87. 87. Contributors to the Latency • Video encoding pipeline duration • Ingest and packaging operations • Network propagation delays • CDN buffer delays • Media segment duration • Player behavior – Buffering – Playhead positioning – Resilience Current Implementations • Smooth Streaming: 2-second segments, usually 10 seconds of latency • DASH: 2-second segments supported by most players, usually 8 to 10 seconds of latency • HLS – Until mid-2016: 10-second segments, usually 30 seconds of latency – Since mid-2016: 6-second segments, usually 18 to 20 seconds of latency – Safari Mobile in iOS11: Autostart for live streams and support for short segments – App Store & iOS applications: 6-second segments are still mandatory Latency in One Slide IEEE ICME Tutorial - July 2018 87
  88. 88. Stream Start Time ≠ Latency Time Live encoder producing 2-second segments iOS (3 segments) Last available Lowest latency 1 2 3 4 Start Now 2 3 4 4 5 Latency: 7 s Latency: 3 s Latency: 2 s 6 seconds of buffer ~0 seconds of startup* 2 seconds of buffer ~0 seconds of startup* 2 seconds of buffer 1 second of startup* Segment fetching time is assumed to be negligible in this example 5 IEEE ICME Tutorial - July 2018 88
  89. 89. Latency Facts • Stream start time: Time it takes for the video to start playing after pressing ‘play’ • Hand–waving (glass-to-glass) latency: Time delay between an action occurring in front of the camera and the same action being visible on the display • Stream start time is not equal to latency – A stream may take 5 s to start and only be 2 s behind live – Conversely, a stream can start in 2 s and be 5 s behind live • A player can start quickly by choosing to start behind the live point (which increases latency) and build its buffer by fetching segments faster than real time till the live edge • Low latency is always a trade-off against robustness of the playback IEEE ICME Tutorial - July 2018 89
  90. 90. Part I: Streaming Overview of Streaming Standards (DASH, CMAF and Apple HLS)
  91. 91. Dead, Surviving, Maturing and Newborn Technologies • Move Adaptive Stream (Long gone, but some components are in Slingbox) – http://www.movenetworks.com • Microsoft Smooth Streaming (Legacy) – http://www.iis.net/expand/SmoothStreaming • Adobe Flash (Almost dead) – http://www.adobe.com/products/flashplayer.html • Adobe HTTP Dynamic Streaming (Legacy) – http://www.adobe.com/products/httpdynamicstreaming • Apple HTTP Live Streaming (The elephant in the room) – https://tools.ietf.org/html/rfc8216 – https://datatracker.ietf.org/doc/draft-pantos-hls-rfc8216bis • MPEG DASH and CMAF (The standards) – http://mpeg.chiariglione.org/standards/mpeg-dash – http://mpeg.chiariglione.org/standards/mpeg-a/common-media-application-format IEEE ICME Tutorial - July 2018 91
  92. 92. • Fragmented architectures – Advertising, DRM, metadata, blackouts, etc. • Investing in more hardware and software – Increased CapEx and OpEx • Lack of consistent analytics • Preparing and delivering each asset in several incompatible formats – Higher storage and transport costs • Confusion due to the lack of skills to troubleshoot problems • Lack of common experience across devices for the same service – Tricks, captions, subtitles, ads, etc. What does Fragmentation Mean? Higher Costs Less Scalability Smaller Reach Frustration Skepticism Slow Adoption IEEE ICME Tutorial - July 2018 92
  93. 93. DASH intends to be to the Internet world … what MPEG2-TS has been to the broadcast world IEEE ICME Tutorial - July 2018 93
  94. 94. DASH intendsed to be to the Internet world … what MPEG2-TS has been to the broadcast world IEEE ICME Tutorial - July 2018 94
  95. 95. Dynamic Adaptive Streaming over HTTP (DASH) • DASH is not – system, protocol, presentation, codec, interactivity, DRM, client specification • DASH is an enabler – It provides formats to enable efficient and high-quality delivery of streaming services – System definition is left to other organizations • Design choices – Reuse the existing technologies (containers, codecs, DRM, etc.) – Enable deployment on top of CDNs – Move intelligence from network to client, enable client differentiation – Provide simple interoperability points (profiles) IEEE ICME Tutorial - July 2018 95
  96. 96. Shown in Red Scope of MPEG DASH HTTP Server DASH Client Control Engine MediaEngines HTTP ClientHTTP/1.1 Segment ParserMPD Transport MPD Parser MPD MPD MPD ... ... ... IEEE ICME Tutorial - July 2018 96
  97. 97. Brief History of DASH at MPEG • Amd. 1: NTP sync, extended profiles • Amd. 2: SRD, URL parameter insertion, role extensions • Amd. 3: External MPD link, period continuity, generalized HTTP header extensions/queries • Amd. 4: TV profile, MPD chaining/resetting, data URLs in MPD, switching across adaptation sets • 3rd edition (FDIS) in ballot • Amd. 5 (WiP): Device information, quality equivalence descriptor, timed text roles, announcing popular content, flexible IOP signaling, early available periods, signaling missing/alternative segments 23009-1: Media Presentation Description and Segment Formats • 2nd edition was published in Oct. 2017 • Amd. 1 (WiP): SAND conformance rules 23009-2: Conformance and Reference Software • 3rd edition (WD) is in progress 23009-3: Implementation Guidelines (Informative) IEEE ICME Tutorial - July 2018 97
  98. 98. Brief History of DASH at MPEG • 2nd edition (FDIS) in ballot 23009-4: Segment Encryption and Authentication • 1st edition, published in Feb. 2017 23009-5: Server and Network Assisted DASH (SAND) • 1st edition, published in Dec. 2017 23009-6: DASH over Full Duplex HTTP-Based Protocols (FDH) • 1st edition (WD) is work in progress 23009-7: Delivery of CMAF Contents with DASH (Informative) IEEE ICME Tutorial - July 2018 98
  99. 99. • Describes various types of “push” behaviors in HTTP/2 and WebSockets • Primarily targeted for low-latency live streaming • Similar activities – DASH over LTE broadcast (3GPP SA4) – DVB ABR multicast – ATSC 3.0 hybrid delivery Part 6 – DASH over Full Duplex HTTP-Based Protocols IEEE ICME Tutorial - July 2018 99
  100. 100. Ongoing Work as of MPEG 123 (July 2018) • Technologies under Consideration (w17812) – Usage of HEVC tile tracks in DASH – Annotation and client model for content selection – Signaling for quality control – Announcing popular content in DASH – Using segment templates for forensic watermarking – Using DASH MPD chaining for mid-roll ads – DASH playlist description – Mixed MPD – Event processing model – Patch method for MPD updates IEEE ICME Tutorial - July 2018 100
  101. 101. Common Media Application Format (CMAF) • Media delivery has three main components: – Media format – Manifest – Delivery • CMAF defines the media format only (Fragments, headers, segments, chunks, tracks) – No manifest format or a delivery method is specified • CMAF uses ISO-BMFF and common encryption (CENC) – CENC means the media fragments can be decrypted/decoded by devices using different DRMs – CMAF does not mandate CTR or CBC mode • This makes CMAF useful only for unencrypted content for now, but industry is moving towards CBCS • Any delivery method may be used for delivering CMAF content – HTTP – RTP multicast/unicast – LTE broadcast IEEE ICME Tutorial - July 2018 101
  102. 102. CMAF (ISO/IEC 23000-19) • CMAF defines media profiles for – Video – Audio – Subtitle • CMAF defines presentation profiles by selection a media profile from each category • Current status – 1st edition was published in Jan. 2018 – Supported in iOS 10+ (with HLS playlists) – Amd. 1: SHVC media profile and additional audio media profiles – Amd. 2: xHE-AAC and other media profiles IEEE ICME Tutorial - July 2018 102
  103. 103. Each Frame can Be a CMAF Chunk CMAF ISO-BMFF Media Objects RAP … RAP … RAP … RAP … Fragment Fragment Fragment Fragment Segment Segment Track File … … … Encoding Packaging Encryption CMAF Header Seamless switching can only happen at fragment boundaries Manifests may provide URLs to • Track files • CMAF header + segments • CMAF header + chunks IEEE ICME Tutorial - July 2018 103
  104. 104. ismc Smooth mpd DASH m3u8 HLS Source Encoder f4m HDS m3u8 m3u8 mpd mpd ismc ismc f4m f4m Credit: Akamai IEEE ICME Tutorial - July 2018 104
  105. 105. ismc Smooth mpd DASH m3u8 HLS Source Encoder f4m HDS m3u8 m3u8 mpd mpd ismc ismc f4m f4m Credit: Akamai More packaging $ More storage $ Less efficient caching IEEE ICME Tutorial - July 2018 105
  106. 106. Multi-Platform OTT Workflow Efficient Distribution in the CDN CMAF Source Encoder m3u8 mpd Credit: Akamai m3u8 mpd m3u8 mpd m3u8 mpd m3u8 mpd m3u8 mpd IEEE ICME Tutorial - July 2018 106
  107. 107. Encode and Package Once, Deliver Efficiently Origin (HTTP) Server Encoder/ Transcoder LinearIngest On-demandContent Pick your favorite codec Pick your chunk and segment sizes CIF (SCTE) to client manifest translation withEBPmarkers Packager IEEE ICME Tutorial - July 2018 107
  108. 108. Non-HTTP Transport Options RTP Multicast - Define RTP payload format for CMAF chunks/fragments/segments - Use RFC 4588 and 6285 for loss recovery and rapid acquisition Broadcast - TS 26.346 and TS 26.347: Delivering DASH content via MBMS - A/331:2017: Carrying DASH content over broadcast and/or broadband networks Peer-to-Peer - WebRTC: Clients fetch DASH/CMAF pieces from each other in addition to CDN servers - This helps reduce the load on the CDN as well as the stream start times IEEE ICME Tutorial - July 2018 108
  109. 109. • Content format issues – Each asset is copied to multiple media formats • different video codecs • different audio codecs • different (regional) frame rates – Cost to content creators and distributors – Inefficiencies in CDNs and higher storage costs • Platform issues – Lack of consistent app behavior across platforms – Varying video features, APIs and semantics across platforms • Playback issues – Codec incompatibility – Partial profile support – Switching bitrate glitches – Audio discontinuities – Ad splicing problems – Long-term playback instability – Request protocol deficiencies – Memory problems, CPU weaknesses – Scaling (display) issues – Variable HDR support – Unknown capabilities Commercial OTT Issues IEEE ICME Tutorial - July 2018 109
  110. 110. CTA Web Application Video Ecosystem (WAVE) Steering Committee Technical Working Group Content Specification Task Force Content specification based upcoming CMAF, compatible with DASH and HLS Device Playback Capabilities Task Force Testable requirements covering the most common device playback interoperability issues HTML5 API Task Force Reference application framework based on HTML5 providing functional guidelines for playback interop IEEE ICME Tutorial - July 2018 110
  111. 111. Web Media API Snapshot 2017 • First annual API Snapshot published in Dec. 2017 – Lists key APIs supported in 2017 in all major HTML code bases – CTA-W3C agreement to co-publish this spec – Plan to propose Community Group spec as a W3C standards-track spec – Available at https://www.w3.org/2017/12/webmediaapi.html • CTA WAVE issued RFP to create a test suite for all listed APIs based on W3C API tests – Test suite will enable manufacturers to test that their HTML support is up-to-date • Web Media Application Developer Guidelines – Companion guide to outline best practices for implementing Web media apps – https://w3c.github.io/webmediaguidelines/ IEEE ICME Tutorial - July 2018 111
  112. 112. WAVE Current and Future Publications • Published – “Web Media API Snapshot 2017”, Final Community Group Report, Dec. 2017, https://www.w3.org/2017/12/webmediaapi.html – “Web Application Video Ecosystem – Content Specification”, Apr. 2018, https://members.cta.tech/ctaWAVE • Pending – Event Messages in WAVE (white paper) – Web Application Video Ecosystem (WAVE) Device Playback Capabilities – Web Media Application Developer Guidelines – Web Media Porting Specification IEEE ICME Tutorial - July 2018 112
  113. 113. Part I: Streaming Adaptive Delivery of Omnidirectional/360° Video
  114. 114. $44,000,000
  115. 115. Example Platform/Infrastructure https://bitmovin.com/
  116. 116. MPEG’s Definition What is Virtual Reality (VR) VR is a rendered environment (visual and acoustic, pre- dominantly real-world) providing an immersive experience to a user who can interact with it in a seemingly real or physical way using special electronic equipment IEEE ICME Tutorial - July 2018 116
  117. 117. Virtual Reality (VR) puts us in a Virtual World Source: Phil Chou IEEE ICME Tutorial - July 2018 117
  118. 118. Augmented Reality (AR) puts Virtual Objects in our World Source: Phil Chou IEEE ICME Tutorial - July 2018 118
  119. 119. Delivering High-Quality VR in Economic Scale is Extremely Challenging The VR Challenge 30K x 24K x 36 x 60 x 2 / 600 ~5.2 Gbps Resolution Bit depth Frame rate Stereoscopic Compression gain IEEE ICME Tutorial - July 2018 119
  120. 120. Ultimate Level of Immersion Interactions So intuitive that they become second nature Sounds So accurate that they are true to life Visuals So vibrant that they are eventually indistinguishable from the real world IEEE ICME Tutorial - July 2018 120
  121. 121. Visual quality Sound quality Intuitive interactions Immersive VR Has Extreme Requirements Immersion High resolution audio Up to human hearing capabilities 3D audio Realistic 3D, positional, surround audio that is accurate to the real world Precise motion tracking Accurate on-device motion tracking Minimal latency Minimized system latency to remove perceptible lag Natural user interfaces Seamlessly interact with VR using natural movements, free from wires Extreme pixel quantity and quality Screen is very close to the eyes Spherical view Look anywhere with a full 360° spherical view Stereoscopic display Humans see in 3D Source: Thomas Stockhammer IEEE ICME Tutorial - July 2018 121
  122. 122. Need for Higher Resolutions on Mobile Phones? Source: Thomas Stockhammer IEEE ICME Tutorial - July 2018 122
  123. 123. Omnidirectional/360° Video Capturing Devices • Stitching, projection formats • Encoding, encryption, encapsulation • Storage, content distribution, delivery • Processing, decoding, rendering Consumer Devices IEEE ICME Tutorial - July 2018 123
  124. 124. Adaptive Streaming of VR Content • Required resolution of a panoramic video for achieving 4K resolution for viewport with 120° field of view (FoV) IEEE ICME Tutorial - July 2018 124
  125. 125. Functional Architecture Encode, Encrypt, Encapsulate Decode, Decrypt, Decapsulate Project & Render Fuse, Stitch & Edit ① ② ③ ④ ⑤ Store & Deliver Capture Consume Content Creation NetworkServer Client Encoding Encryption Encapsulation Storage Delivery Decryption Decapsulation Decoding RenderingEditing Processing Processing Capture Acquisition Consumption Distribution Adaptive Delivery of Omnidirectional Video Interaction From ecosystem… To building blocks… IEEE ICME Tutorial - July 2018 125
  126. 126. Functional Architecture Encode, Encrypt, Encapsulate Decode, Decrypt, Decapsulate Project & Render Fuse, Stitch & Edit ① ② ③ ④ ⑤ Store & Deliver Capture Consume Content Creation NetworkServer Client Encoding Encryption Encapsulation Storage Delivery Decryption Decapsulation Decoding RenderingEditing Processing Processing Capture Acquisition Consumption Distribution Adaptive Delivery of Omnidirectional Video Interaction From ecosystem… To building blocks… IEEE ICME Tutorial - July 2018 126
  127. 127. Adaptive Streaming Options (1) • Traditional, viewport-agnostic streaming – ERP – handle like reg. video – Simple, easy, deployed today – Bandwidth waste, quality issues • Viewport-adaptive streaming – Multiple versions for predefined viewports – Various projection techniques (pyramid) – Bandwidth waste reduced, increased storage and CDN costs, limited flexibility X. Corbillon, et al., "Viewport-adaptive navigable 360-degree video delivery," 2017 IEEE International Conference on Communications (ICC), Paris, 2017, https://doi.org/10.1109/ICC.2017.7996611 IEEE ICME Tutorial - July 2018 127
  128. 128. Adaptive Streaming Options (2) • Tile-based streaming – Use tiling technique of modern video codecs – High complexity, full flexibility – Multiple challenges M. Graf, et al. 2017. Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation. Proc. ACM MMSys'17. https://doi.org/10.1145/3083187.3084016 C. Concolato, et al., "Adaptive Streaming of HEVC Tiled Videos using MPEG-DASH," IEEE TCSVT, 2017. https://doi.org/10.1109/TCSVT.2017.2688491 IEEE ICME Tutorial - July 2018 128
  129. 129. System Overview Tile-Based Adaptive Streaming Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Encoding & Packaging Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 DifferentQ uality Representations Delivery MPEG-HEVC/H.265 Tiles in ISOBMFF Adaptive Player Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile-based streaming of VR/360° content with MPEG-DASH SRD / OMAF … … … … Adaptive Streaming using MPEG-DASH SRD / OMAF Head Mounted Displays Browsers, Smart (Mobile) Devices (Stereo) 2D, (Stereo) 3D IEEE ICME Tutorial - July 2018 129
  130. 130. Encoding Options • AVC dominates the market • HEVC, VP9, AV1 support tiles – Divides a picture into independent, rectangular regions – Tradeoff: bitrate, quality, flexibility • Multiple tiling options available – Uniform vs. non-uniform tiling – Same vs. mixed resolutions • New quality metrics, mostly based on PSNR but subjective quality assessments/metrics increasing https://bitmovin.com/2017-video-developer-report/ I. D.D. Curcio, et al. 2017. Bandwidth Reduction of Omnidirectional Viewport-Dependent Video Streaming via Subjective Quality Assessment. Proc. AltMM'17. https://doi.org/10.1145/3132361.3132364 A. Zare, et al. 2016. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications. Proc. ACM MM'16. http://dx.doi.org/10.1145/2964284.2967292 IEEE ICME Tutorial - July 2018 130
  131. 131. Quality Metrics: S-PSNR M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 31–36. DOI:http://dx.doi.org/10.1109/ISMAR.2015.12 IEEE ICME Tutorial - July 2018 131
  132. 132. Quality Metrics: V-PSNR M. Yu, H. Lakshman, and B. Girod. 2015. A Framework to Evaluate Omnidirectional Video Coding Schemes. In 2015 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 31–36. DOI:http://dx.doi.org/10.1109/ISMAR.2015.12 IEEE ICME Tutorial - July 2018 132
  133. 133. Dataset • Segment length / intra period – 1s (tiled content) vs 1, 2, 4s (monolithic content) • Tiling pattern (columns × rows): 1×1, (i.e., tiles monolithic), 3×2, 5×3, 6×4, and 8×5 • Resolution: 1920×960, 3840×1920 and 7680×3840 • Map projection: equirectangular format • Quantization parameter: QP={22,27,32,37,42} • Head motion recordings for V-PSNR evaluation IEEE ICME Tutorial - July 2018 133
  134. 134. Bitrate Overhead due to Tiling 500 1000 1500 2000 2500 384042444648 Bitrate [kbps] Y−PSNR[dB] ● ● ● ● ● ● ● 1x1 tiles 3x2 tiles 5x3 tiles 6x4 tiles 8x5 tiles Tile Overhead for Resolution: 1920x960 Sequence: AssassinsCreed IEEE ICME Tutorial - July 2018 134
  135. 135. Bitrate Overhead due to Tiling 2000 4000 6000 8000 30323436384042 Bitrate [kbps] Y−PSNR[dB] ● ● ● ● ● ● ● 1x1 tiles 3x2 tiles 5x3 tiles 6x4 tiles 8x5 tiles Tile Overhead for Resolution: 1920x960 Sequence: ExploreTheWorld IEEE ICME Tutorial - July 2018 135
  136. 136. Bandwidth Requirements IEEE ICME Tutorial - July 2018 136
  137. 137. Adaptive Streaming Issues for (Tiled) 360° Video • Increasing number of segment/tile requests – HTTP/2 server push, query parameters, proprietary protocols – Additional functionality at server – breaks fundamental HAS requirements • Low latency streaming (see also next slide) – Reducing segment size impacts coding efficiency (1s vs. 4s) – CMAF chunks + other enhancements to enable sub-second latency – Remember: live internet video will grow 15-fold from 2016 to 2021 • Viewport prediction – Allows prefetching (caching) but cannot predict to much into future (1s) – Impact on segment size but situation will get better the more data is available – machine learning/AI will help • QoE (see also following slide) – Still in its infancy but situation much better than one year ago – Requires datasets, subjective studies, quality models, metrics M. Xu, et al., "A subjective visual quality assessment method of panoramic videos," Proc. ICME’17. https://doi.org/10.1109/ICME.2017.8019351 R. Schatz, et al., "Towards subjective quality of experience assessment for omnidirectional video streaming," Proc. QoMEX’17, https://doi.org/10.1109/QoMEX.2017.7965657 Y. Rai, at al. 2017. A Dataset of Head and Eye Movements for 360 Degree Images. Proc. ACM MMSys’17. https://doi.org/10.1145/3083187.3083218 S. Petrangeli, et al. 2017. An HTTP/2-Based Adaptive Streaming Framework for 360° Virtual Reality Videos. Proc. ACM MM'17. https://doi.org/10.1145/3123266.3123453 N. Bouzakaria, et al., "Overhead and performance of low latency live streaming using MPEG-DASH," Proc. IISA’14. https://doi.org/10.1109/IISA.2014.6878732 C.-L. Fan, et al. 2017. Fixation Prediction for 360° Video Streaming in Head- Mounted Virtual Reality. Proc. NOSSDAV'17. https://doi.org/10.1145/3083165.3083180 IEEE ICME Tutorial - July 2018 137
  138. 138. Lag Prevents Immersion and Causes Discomfort Minimizing Motion-to Photon Latency is Important Low latency Noticeable latency IEEE ICME Tutorial - July 2018 138
  139. 139. Motivation QoE of Omnidirectional Video • QoE of Omnidirectional Video (OV, alt.: 360º Movie) Streaming consumed via HMD – Immersive panorama video (or image) surrounding the user – User can turn head to changes gaze direction • OV streaming = content class with dramatically increasing popularity – Cisco VNI: globally, virtual–reality traffic will increase 61-fold between 2015 and 2020 Immersive Media VR Responsive VR OV MR AR R. Schatz, et al., "Towards subjective quality of experience assessment for omnidirectional video streaming," Proc. QoMEX’17, https://doi.org/10.1109/QoMEX.2017.7965657 IEEE ICME Tutorial - July 2018 139
  140. 140. Motivation QoE of Omnidirectional Video • Gaps in current state of the art – Lots of knowledge on 2D & 3D Video quality – Lots of knowledge on human factors in responsive AR/VR – BUT: lack of quantitative models and subj. testing methods for OV streaming Particularly as regards quality perception and the Impact of stalling on OV QoE QoE = m * c2 QoE = m * c2 ? IEEE ICME Tutorial - July 2018 140
  141. 141. IEEE ICME Tutorial - July 2018 141 Research Questions RQ1: How does stalling impact the QoE of HMD-based omnidirectional video (OV)? RQ2: What are QoE impact differences between HMD-based OV and traditional TV-based video consumption with respect to stalling? RQ3: What is the impact of different content motion levels on OV stalling perception?
  142. 142. IEEE ICME Tutorial - July 2018 142 Method: Subjective Lab Study • Test task: watch short video clips (duration 60s) impaired by stalling • Two Devices / Watching Modes: – 2D video on large TV screen (65”, Full HD) – Stereoscopic OV on an Oculus Rift DK2 HMD (with headphones and 3-axis orientation sensor) vs
  143. 143. IEEE ICME Tutorial - July 2018 143 Test Subjects • Recruited from Vienna City region, received monetary incentives • Recruited: 27 participants, but … … removed 5 subjects due to vision problems, lack of HMD fit and cybersickness à N=22 – 12 male, 10 female – 4 had some limited VR experience – Age: mean 37, median 31.5, min 22, max 65 years
  144. 144. IEEE ICME Tutorial - July 2018 144 Test Content (SRCs) • 2D / TV – Sintel movie (© Blender Foundation) – One 60s clip • OV / HMD – Vulkane – Terra X (© ZDF) – Two 60s clips extracted • Both feature: – Animation – Storyline
  145. 145. Test Setup IEEE ICME Tutorial - July 2018 145
  146. 146. Impairments (HRCs) and Test Conditions (PVSs) • PVS: – 2D/TV: 6 HRCs (Sintel movie) (+ 1 Training condition) – OV/HMD: 6 HRCs x 2 SRCs (static camera, moving camera) (1 Training condition) • Post-stimulus electronic questionnaire with 4/7 (TV/HMD) rating questions related to quality, immersion, etc. IEEE ICME Tutorial - July 2018 146
  147. 147. OV move: Four users could not complete the OV move conditions, i.e. N=18 How Annoying were Interruptions (If Any) of the Video Playback? Results: Impact of Stalling on User Annoyance Imperceptible Very annoying IEEE ICME Tutorial - July 2018 147
  148. 148. I Was Completely Captivated by the Virtual Environment Impact of Stalling on Perceived Involvement (IPQ INV4*) Fully agree Full disagree * from Igroup Presence Questionnaire (IPQ) IEEE ICME Tutorial - July 2018 148
  149. 149. Results / Findings • RQ1 (stalling QoE impact): – Strong impact of stalling number and duration (similar to previous results) – Existing stalling models can be reused • RQ2 (differences 2D/OV): – No significant differences between TV/2D and HMD/OV (à unexpected) • RQ3 (impact of OV content motion): – No significant differences between moving and static camera, just some trends (à unexpected) • In addition: – Splits yield some differences in rating variance and stalling detection Influence on variance: sex, prior vr experience Influence on stalling detection: commented on HMD resolution IEEE ICME Tutorial - July 2018 149
  150. 150. AdViSE • Adaptive Media Content [DASH, HLS, CMAF] • Players/Algorithms • Network Parameters Impaired Media Sequences Generate Impaired Media Sequences Templates [Startup Delay, Stalling, …] WESP QoE Evaluation Parameters [Questionnaire, Methodology, Crowdsourcing Platform, …] QoS/QoE Metrics Subjective Results + Other Data Reports Analysis AdViSE: Anatoliy Zabrovskiy, Evgeny Kuzmin, Evgeny Petrov, Christian Timmerer, and Christopher Mueller. 2017. AdViSE: Adaptive Video Streaming Evaluation Framework for the Automated Testing of Media Players. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM, New York, NY, USA, 217-220. DOI: https://doi.org/10.1145/3083187.3083221 WESP: Benjamin Rainer, Markus Waltl, Christian Timmerer, A Web based Subjective Evaluation Platform, In Proceedings of the 5th International Workshop on Quality of Multimedia Experience (QoMEX'13) (Christian Timmerer, Patrick Le Callet, Martin Varela, Stefan Winkler, Tiago H Falk, eds.), IEEE, Los Alamitos, CA, USA, pp. 24-25, 2013. https://doi.org/10.1109/QoMEX.2013.6603196 Objective and Subjective Evaluation Platform How to Evaluate HAS Systems? ① ② ③④ ⑤ Log of Segment Requests IEEE ICME Tutorial - July 2018 150
  151. 151. AdViSE: Adaptive Video Streaming Evaluation • Scalable, end-to-end HAS evaluation through emulation w/ a plethora of – content configurations – players/algorithms (including for player competition) – network parameters/traces • Real content and network settings with real dynamic, adaptive streaming including rendering! • Collection of various metrics from players: API or directly from the algorithms/HTML5 • Derived metrics and utilize QoE models proposed in the literature • Segment request log to generate impaired media sequence as perceived by end users for subjective quality testing IEEE ICME Tutorial - July 2018 151
  152. 152. WESP: Web-Based Subjective Evaluation Platform • Subjective quality assessments (SQA) are vital tool, reliable results, but cost-intensive • Utilize crowdsourcing to reduce costs • WESP – Enables easy and simple configuration of SQAs including possible integration of third-party tools for online surveys – Provides means to conduct SQAs using the existing crowdsourcing platforms taking into account best practice – Allows for the analysis of the results IEEE ICME Tutorial - July 2018 152
  153. 153. C. Timmerer, A. Zabrovskiy, and Ali C. Begen, Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems, Proceedings of the 1st IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR), Miami, FL, USA, April 2018. Example Evaluation Results • Test sequence encoded 15 different representation (Amazon Prime configuration: 400x224@100Kbps – 1920x1080@15Mbps) with 4s segment length • Bandwidth trajectory based on prior work proposed in literature; network delay 70 ms IEEE ICME Tutorial - July 2018 153
  154. 154. Bandwidth Index vs. QoE • Bandwidth index – Average bitrate – Efficiency – Stability • QoE model – Startup delay – Stalls C. Timmerer, A. Zabrovskiy, E. Kuzmin, E. Petrov, Quality of experience of commercially deployed adaptive media players, In 2017 21st Conference of Open Innovations Association (FRUCT) (Sergey Balandin, ed.), pp. 330-335, 2017. https://doi.org/10.23919/FRUCT.2017.8250200 IEEE ICME Tutorial - July 2018 154
  155. 155. Part I: Streaming The Developing MPEG-OMAF and MPEG-I Standards
  156. 156. Standardization Overview C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017. https://doi.org/10.1109/MCOMSTD.2017.1700038 Data Representations and Formats JPEG MPEG IEEE System Standards and APIs VR-IF 3GPP DVB Khronos W3C Quality of Experience QUALINET ITU-T VQEG Guidelines DASH-IF CTASVA SMPTE ETSI https://multimediacommunication.blogspot.com/2017/04/vr360-streaming-standardization-related.html IETF/IRTF IEEE ICME Tutorial - July 2018 156
  157. 157. Standardization Overview C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017. https://doi.org/10.1109/MCOMSTD.2017.1700038 Data Representations and Formats JPEG MPEG IEEE System Standards and APIs VR-IF 3GPP DVB Khronos W3C Quality of Experience QUALINET ITU-T VQEG Guidelines DASH-IF CTASVA SMPTE ETSI https://multimediacommunication.blogspot.com/2017/04/vr360-streaming-standardization-related.html IETF/IRTF IEEE ICME Tutorial - July 2018 157
  158. 158. Coded Representation of Immersive Media MPEG-I (ISO/IEC 23090) • Part 1: Architectures • Part 2: Omnidirectional media format (OMAF) • Part 3: Versatile Video Coding (VVC) • Part 4: Immersive audio • Part 5: Point cloud compression • Part 6: Immersive media metrics • Part 7: Immersive media metadata • Part 8: Network-based media processing (NBMP) IEEE ICME Tutorial - July 2018 158
  159. 159. Scope Omnidirectional Media Format (OMAF) • 360° video, images, audio and associated timed text – 3DoF only • Specifies – A coordinate system • Consists of a unit sphere and the x (back-to-front), the y (lateral, side-to-side), and the z (vertical, up) axes – Projection and rectangular region-wise packing methods • Used for conversion of a spherical video/image into a two-dimensional rectangular video/image – The sphere signal is the result of stitching of video signals captured by multiple cameras – A special case: fisheye video – Storage of omnidirectional media and the associated metadata using ISO-BMFF – Encapsulation, signaling and streaming of omnidirectional media in DASH and MMT – Media profiles and presentation profiles • Provide interoperable and conformance points for media codecs as well as media coding and encapsulation configurations • Provides some informative viewport-dependent 360° video processing approaches IEEE ICME Tutorial - July 2018 159
  160. 160. A: Real-world scene B: Multiple-sensors-captured video or audio D/D’: Projected/packed video E/E’: Coded video or audio bitstream F/F’: ISOBMFF file/segment OMAF player Acquisition Audio encoding Video encoding Image encoding File/segment encapsulation Delivery File/segment decapsulation Audio decoding Video decoding Image decoding Image rendering Audio rendering Loudspeakers/ headphones Display Head/eye tracking Fileplayback Orientation/ viewport metadata Orientation/viewport metadata Metadata Metadata A Ba Bi D Ea Ei Ev Fs F F’ F’s E’a E’v E’iD’ B’a A’i A’a Image stitching, rotation, projection, and region-wise packing https://mpeg.chiariglione.org/standards/mpeg-i/omnidirectional-media-format R. Skupin, et al., "Standardization Status of 360 degree Video Coding and Delivery,” Proc. IEEE VCIP’17 OMAFArchitecture
  161. 161. DASH/OMAF System Diagram Multi- camera capture Stitch Project HEVC encode OMAF composing DASH server DASH client HEVC Decode Renderer Head mounted device (HMD) DASH encapsulation Head and eye tracking telemetry Request for tiles within FoV window Projection mapping metadata Projection mapping metadata Authoring CDN VR player IEEE ICME Tutorial - July 2018 161
  162. 162. MPEG- 1,2,4,H,I MPEG-7 MPEG-21 MPEG-A MPEG- B,C,D, DASH MPEG- E,M MPEG- U,V Compression of video, audio and 3DG Description of video, audio and multimedia for content search Technologies for content e-commerce Multimedia application formats (combinations of content formats) Systems, video, audio and transport Multimedia platform technologies Device and application interfaces https://mpeg.chiariglione.org/meetings/123 MPEG’s Areas of Activity IEEE ICME Tutorial - July 2018 162
  163. 163. 1990 2000 20101995 2005 2015 MP3 MPEG-2 Video & Systems AVC HEVC MP4 FF MMTDASH AAC Internet Audio Digital Television Music Distribution Media Storage and Streaming UHD & Immersive Services New Forms of Digital Television Unified Media Streaming OFF HD DistributionMobile Video Custom Fonts on Web & DigiPub MPEG-4 Video CMAF 3D Audio 2020 MajorMPEG Standards
  164. 164. 6DoF Application Format? Genome Compression v2 PCC Extensions? OMAF v2 2018 20202017 2019 2021 IoMT Media Orchestration Descriptors for Video Analysis (CDVA) 6DoF Audio Point Cloud Compression OMAF v1 Genome Compression Network-based Media Processing Coding 2022 Scene Description for Immersive Media Versatile Video Coding 2023 MIAF Systems and ToolsWeb Resource Tracks Dense Representation of Light Field Video 3DoF+ Video MPEGRoadmap
  165. 165. Part II: Communications over 5G 5G Fundamentals: Core Network, Radio Access Network and QoS
  166. 166. 5G fundamentals: Core Network, Radio Access Network and QoS ICME 2018 Tutorial Liangping Ma July 23, 2018 © 2018 InterDigital, Inc. All rights reserved. 166
  167. 167. 167 Who is InterDigital? Invention, Collaboration, Contribution © 2018 InterDigital, Inc. All rights reserved. Berlin Montreal, QC (R&D) London (R&D) Seoul Buffalo, NY Melville, NY (R&D) Conshohocken, PA (R&D) Wilmington, DE (HQ) San Diego, CA (R&D) • Four decades of discovery and innovation in wireless • Pioneer in digital wireless technologies • Key contributions to global wireless standards • Inventing solutions for more efficient broadband networks • ~190 engineers (~80% of whom hold advanced degrees)
  168. 168. 168 What is 5G? © 2018 InterDigital, Inc. All rights reserved.
  169. 169. 169 5G Use Cases and Key Requirements Enhanced Mobile Broadband (eMBB) § Peak data rates: 20 Gbps (DL) and 10 Gbps (UL) § Peak spectral efficiency: 30 bps/Hz (DL) and 15 bps/Hz (UL) § 4 ms user plane latency § Indoor/hotspot and enhanced wide-area coverage Massive Machine Type Communications (mMTC) § Low data rates (1 to 100 kbps) § High device density (up to 1,000,000 /km2) § Latency: Seconds to hours § Low power: Up to 15 years battery life Ultra-Reliable and Low Latency Communications (URLLC) § Low to medium data rates (50 kbps to 10 Mbps) § 0.5 ms user plane latency § 99.999% reliability and availability within 1 ms § High mobility Key challenge for 5G design: support for different services having diverging requirements Source: ITU-R SG5 WP-5D © 2018 InterDigital, Inc. All rights reserved.
  170. 170. 170 What Differentiates 5G from Previous Generations? Uses Cases & Services Voice + SMS Voice + Small Data Mobile Broadband Enhanced Mobile Broadband Massive Machine Type Communications Ultra Reliable and Low Latency Communications Spectrum 200 KHz Channels Below 2 GHz 5 MHz Channels Below 3.6 GHz Up to 20 MHz Channels Below 3.8 GHz Up to 1 GHz Channels Below 100 GHz Radio Technology GSM/GPRS (Single RAT ) UMTS/HSPA (Single RAT) LTE/LTE-A (Single RAT) Multiple Radio Access Technologies Integrated in a 5G Network: 5G New Radio, LTE Advanced Pro, NB-IoT, Wi-Fi, … Network Topology Macro Cells Macro and Small Cells Heterogeneous Macro and Small Cells Challenging Traditional Cell and Network Concepts: Small Cells, Mobile Edge Computing, Network Slicing, … 2G 4G 5G3G © 2018 InterDigital, Inc. All rights reserved.
  171. 171. 2021 171 5G 3GPP Standardization Timeline 2017 2018 2019 2020 Rel-17 Work Items LTE Release 17LTE Release 15R14 NR SI Full NR RAN stand- alone and CT Specifications approved Rel-16 specification approved with some additional enhancements NR R15 first drop specification completed (NR non-standalone and CN standalone) Phased approach enables early commercial deployment of Phase 1 in 2019 and Phase 2 in 2022+ IMT-2020 Proposal Submission NR Rel-15 5G LTE + NB-IoT (required for mMTC use cases for IMT-2020) 5G New Radio (NR) … Allows for very early commercial implementation – No more hardware changes envisioned for R-15 Rel-16 SIs (RAN/SA) Rel-17 Study Items LTE Release 16 Additional Rel-16 SIs approved Rel-16 Work Items (R16) 3rd drop © 2018 InterDigital, Inc. All rights reserved.
  172. 172. Focus on completion of the protocol design § NAS protocol between the UE and the CN § Internal Core Network interfaces based on the service based architecture (SBA) § Interconnection of the Core Network with external networks First set of NR specifications completed § Ready for Hardware Implementation and for first phase of deployment - Non-Standalone NR only (i.e. connected and controlled by LTE) § RAN1 PHY and RAN2 User Plane completed for stand-alone NR § Support for Enhanced Mobile Broadband use cases only and low latency 172 5G New Radio Status – Rel-15 Core Network and Non Access Stratum Radio Access Network Next Gen Core(NGC) architecture completed § New fully flexible architecture based on Network Slicing concept and further separation of control and user planes § Key functions: Session Management, Mobility Management, QoS and access agnostic CN Focus on completion stand-alone NR § Support for stand-alone NR connected directly to a 5G CN (RAN2/RAN3/RAN4) § Add/complete additional enhancements to support ultra-reliable and low latency use cases (URLLC) (RAN1/RAN2) December 2017 June 2018 © 2018 InterDigital, Inc. All rights reserved.
  173. 173. 173 5G Design Principles © 2018 InterDigital, Inc. All rights reserved.
  174. 174. 174 5G NR Design Principles Architecture & Network • Flexible, modular and scalable architecture • Support high level of programmability and automation in network • Built in support of network customization via slicing • Easy integration of new services 5G is designed to be flexible and adaptable to meet wide range of current and future requirements Radio Access Network • Minimize always-on signaling • Flexible frame structure and numerology • Support wide range of frequency bands • Built-in support for beamforming • Easy integration of future features © 2018 InterDigital, Inc. All rights reserved.
  175. 175. 175 5G Technologies © 2018 InterDigital, Inc. All rights reserved.
  176. 176. 176 3GPP 5G Deployment Scenarios 5G RAN LTE/4G Network 5G Network LTE Base Station New Radio RAN Internet Non Standalone (NSA) 5G 5G New Radio 4G EPC 5G Core Network Slice #3 (URLLC) Slice #2 (IoT) Slice #1 (eMBB) SGW MME PGW HSS AMF Non-Standalone (NSA) 5G enables large-scale trials and deployments starting in 2019 3 Main Deployment Scenarios § NSA deployment - utilizes LTE radio and 4G EPC as an anchor and NR Radio as hotspot § Allows earlier implementability § Stand-alone NR – 5G RAN connecting to 5G CN only § LTE radio connected to 5G CN network 5G CN designed to interwork with the current EPC network to enable seamless migration for the operators © 2018 InterDigital, Inc. All rights reserved.
  177. 177. 177 Architecture options and specification timelines © 2018 InterDigital, Inc. All rights reserved. EPC NextGen Core LTE NR 3 Non-standalone Drop 1: Dec. 2017 NR 2 LTE 5 NextGen Core NextGen Core Standalone NR Drop 2: June 2018 EPC NextGen Core LTE NR 4 EPC NextGen Core LTE NR 7 LTE connected to 5G CN Drop 3: Dec 2018 – NR +
  178. 178. 178 Control and User Plane Separation © 2018 InterDigital, Inc. All rights reserved. Control and User plane separation modularizes the 5G Core Network Design § Reduces Latency on application service § By relocating UPFs which are closer to the RAN; § Or by selecting UPFs more appropriate for the intended UE usage type without increasing the number of control plane nodes § Supports increase of Data Traffic, by enabling to add user plane nodes without changing the number of CP resources in the network § Independent evolution of the CP and UP functions § Enables SDN routing to deliver user plane data more efficiently 5G Core Network Architecture
  179. 179. 179 Service Based Architecture (SBA) § 5G CN has been revolutionized with Service Based interfaces for core network interactions § CN interfaces have traditionally been Point-to-Point (P2P) e.g. MAP, Diameter § Network Functions (NFs) in 5G CN offer their services to other NFs via APIs based on web-based technology (e.g. HTTP). § Makes deployment easy as libraries, development tools, and other components are broadly available. § Flexible deployments of new services: possible to connect to other components without new specific interfaces § Network Repository Function (NRF) allows NFs to discover the services of other NFs. © 2018 InterDigital, Inc. All rights reserved. SBA shifts the paradigm from telecom-style protocol interfaces to web-based APIs UDMNSSF PCFNRF AUSF AF SMFAMF 5G CN Service Based Control Plane
  180. 180. Network Slice • Logical end-to-end network • Tailored based on service it offers (e.g.eMBB, IoT, URLCCC etc.) Slicing offers customized network resources for different verticals © 2018 InterDigital, Inc. All rights reserved. 180 Network Slicing Benefits of slicing • Flexible and modular network deployment • Isolation between dedicated network resources • New products and services can be brought to market rapidly and adapted to fast-changing demands
  181. 181. Virtual (Cloud) RAN Enablers ØStandards efforts to enable virtualization in multi- vendor environments • Central Unit / Distributed Unit split • CU/DU split below the PDCP • CU performs user management and user data processing • DU manages the radio and cell • Control Plane/User plane split • Central unit further split into virtual entities: • User data processing and user/mobility control • Support of lower layer split – no conclusion • Some of the radio control function performed by the central unit • Challenging due to ties with implementation Virtualization gives operators a flexible network design reducing cost 181© 2018 InterDigital, Inc. All rights reserved. CU DU Control Data gNB DU
  182. 182. 182 Network Function Virtualization (NFV) © 2018 InterDigital, Inc. All rights reserved. Sources: Network Functions Virtualisation -- White Paper on NFV priorities for 5G, ETSI, Feb. 2017. Network Functions Virtualisation – Introductory White Paper, ETSI, Oct. 2012. NFV is not limited to 5G Use cases for 5G: - Virtualization of the core network - Virtualization of the RAN Decouple network functions from hardware
  183. 183. 183 Numerology A numerology is defined by sub-carrier spacing and CP overhead ØOFDM waveform in both UL &DL • Optimized CP-OFDM with high spectrum utilization • Spectrum confinement techniques such as filtering and windowing • Simplifies design for D2D, Massive MIMO reciprocity & IAB. • DFT-s-FDM for coverage-limited scenarios in UL ØScalable Numerology • Subcarrier Spacing (SCS) scales with the BW (15#$% ∗ 2( ) • To support diverse spectrum bands and services • To avoid processing complexity associated with a large FFT size • Higher SCS -> Higher bands -> Shorter OFDM symbols • To address RF impairments in mmWave bands (e.g., Phase Noise), enable analog beamforming in mmWave bands and, lower latency • Lower SCS -> Larger cyclic prefix • More robust to the channel delay spread (e.g., Broadcast Service) • To support narrow band transmissions for mMTC services SCS [kHz] 15 30 60 120 240 OFDM Duration [µs] 66.67 33.33 16.67 8.33 4.17 CP Duration [µs] 4.69 2.34 1.17 0.58 0.29 Total Symbol [µs] 71.36 35.67 17.84 8.91 4.46 Slot Duration [µs] 1000 500 250 125 62.5 Band [GHz] < 6 < 6 < 6 < > 6 > 6 © 2018 InterDigital, Inc. All rights reserved. 15 kHz 30 kHz 60 kHz 120 kHz 50 MHz 400 MHz Max BW 100 MHz 200 MHz
  184. 184. 184 Frame Structure Flexible frame structure adapting to various bands, duplexing mode and service requirements UL ULCtrl DL UL ULCtrl DL ULCtrl Downlink-Centric Slot ØFlexible Frame Structure • Flexible TTI Duration § Scalable TTI enables efficient service multiplexing o Symbols across different SCSs are aligned at the symbol boundaries § Shorter TTI to support low latency application (e.g., URLLC) o User can also be schedule on a fraction of a slot (i.e., Mini-slot) § Longer TTI to achieve higher spectral efficiency (e.g., eMBB) o User can be scheduled on multiple-slots • Flexible Scheduling § Both dynamic TDD and FDD are supported § OFDM symbols in a slot can be DL, UL or flexible § Flexible symbols can be used as gap or for forward compatibility Uplink-Centric Slot 1 30 kHz 120 kHz 0 0 20 12 13 1 2 12 13 1 2 60 kHz 15 kHz 1 ms slot with 14 OFDM symbols 500 µs 250 µs 125 µs OFDM symbol Mini-Slot Slot with load balancing © 2018 InterDigital, Inc. All rights reserved. Dynamic TDD
  185. 185. 185 Bandwidth Adaptation Adapting UE’s receive and transmit bandwidth irrespective of gNB’s total supported bandwidth ØFlexible Bandwidth Allocation • NR supports a maximum BW of 400MHz per Component Carrier (CC) with up to 16 CC in Carrier Aggregation (CA) • Supporting bandwidth smaller than the CC bandwidth enabled by configuring a Bandwidth Part (BWP) • BWP Benefits include: § Enables UE power savings § Supports UE with lower maximum bandwidth capability § Single BPW is active at a time • Increases scheduling flexibility • Allow service multiplexing in frequency domain for use- cases with different numerology BWP1: 40 MHz and subcarrier spacing of 15 kHz BWP2: 10 MHz and subcarrier spacing of 15 kHz BWP3: 10 MHz and subcarrier spacing of 60 kHz © 2018 InterDigital, Inc. All rights reserved. Frequency Time BWP1 BWP2 BWP3 BWP2 BWP1 Carrier
  186. 186. 186 Antenna © 2018 InterDigital, Inc. All rights reserved. • More bandwidth leads to higher throughput • Need to go to higher bands • But higher bands have larger path loss than lower bands Free space path loss – Friis equation !" = !$%$ %" & 4()* + Sources: W. L. Stutzman and G. A. Thiele, Antenna Theory and Design, Wiley, 2013. Example: When f increases from 2.8GHz to 28 GHz, the path loss increases by 20 dB. frequencyReceived power R Pt Pr Why? The maximum effective area (effective aperture) or “collecting area” of an antenna is related to the frequency e.g. for an ideal (short) dipole antenna, maximum effective area = , -./0 Beamforming to close the link at higher frequencies
  187. 187. 187 Hybrid Beamforming © 2018 InterDigital, Inc. All rights reserved. • Analog beamforming: phase shifter • Digital beamforming: precoding matrix BB RF RF BB RF RF! = # $ + & where $ = +, = +-.+//, s +-. +// Phase shifter Tradeoff between cost and flexibility
  188. 188. Massive MIMO Ø Conjugate Beamforming • Coherent combining at receiver Ø For TDD, no performance loss when the number of antennas is much larger than the number of users (M/K → ∞) • Leverage channel reciprocity • Result of Law of Large Numbers Ø Pilot contamination • In mobile scenarios the channel changes over time • Contamination occurs when two users use the same pilot Original Work by Thomas L. Marzetta (2006) © 2018 InterDigital, Inc. All rights reserved. k 188
  189. 189. Massive MIMO Ø Enhanced CSI Feedback • Multi-resolution CSI • Low spatial resolution with a linear precoder targeting SU-MIMO operation (Type-I CSI) à low latency and overhead • High spatial resolution with a linear combination precoder targeting MU-MIMO operation (Type-II CSI) à better performance • Flexible CSI reporting timing • CSI reporting timing can be dynamically indicated to optimize latency based on CSI computational complexity • Hybrid beamforming support • Joint CSI reporting of beam index and precoding matrix index (PMI) Ø Enhanced UL Transmission • Non-Codebook based UL transmission • Introduced in NR to enable UL reciprocity functionality in TDD • UE determines its PUSCH precoder with some gNB assistance Work under 3GPP : DL control to trigger aperiodic CSI reporting : UL channel for aperiodic CSI reporting Low latency CSI High latency CSI Slot #n Slot #n+1 Slot #n+2 Slot #n+3 Slot #n+4 • gNB can dynamically indicate the CSI reporting timing © 2018 InterDigital, Inc. All rights reserved. 1) CSI-RS 3) DL Control 2) Sounding Signal 4) Precoded UL Data Precoder Calculation 189
  190. 190. 190 Beam Management Ø Beam management for high frequency • Support of high frequency spectrums (up to 52.6GHz) in NR which is different from LTE • Beam management is used to maintain optimal Tx/Rx beam pairing for control and data channels Ø Beamformed Initial Access • Fundamental difference with LTE RACH lies on beamforming • Beam association between RACH and SS/PBCH blocks • UE selects an associated UL RACH resource of a selected DL SS block based on L1-RSRP measurements Management of analog beams at Tx/Rx is newly introduced in NR © 2018 InterDigital, Inc. All rights reserved.
  191. 191. 191 Beam Failure Recovery Ø Recovery from beam failure • Beam-based system is susceptible to blockage, UE rotation, and mobility • Link fails when Tx-Rx beams for DL control channel are mismatched • RLF has been used in LTE when link fails but that requires a restart from initial access procedure à high latency • Beam recovery recovers beams for DL control channel • UE monitors quality of DL control channel beam and indicates new candidate beam using a dedicated PRACH resource when link failed • Beam recovery is used to recover the link before RLF à low latency A new mechanism introduced to handle frequent Tx/Rx beam mismatch Beam failure due to blockage New candidate beam PRACHfor beam1 PRACHfor beam2 PRACHfor beam3 UE sends PRACH resource associated with new candidate beam to trigger beam recovery © 2018 InterDigital, Inc. All rights reserved.
  192. 192. Ultra-Reliable and Low-Latency Communications NR introduced several mechanisms to enable URLLC 192© 2018 InterDigital, Inc. All rights reserved. Low Latency DL Data ULCtrl Gap HARQ-ACK Slot • Self-contained slot Transmitting HARQ-ACK in the self- contained slot immediately after receiving the DL data • Mini-Slot Slot Mini-Slot Smaller time units for data scheduling • Scalable subcarrier spacing • Grant Free UL transmission • Very low periodicity SR • Pre-emption High Reliability • Diversity via duplication f1 f2 Same PDCP PDU on two carriers Carrier-Aggregation LTE NR Same PDCP PDU over different RAT (DC) Dual-Connectivity (EN-DC) • Control channel duplication • Polar coding for control information • Optimized CQI/MCS tables
  193. 193. 193 Channel Coding: Data Channel Ø Enhanced LDPC Codes for eMBB Data Channel • Advantages of LDPC codes • Support high data rates (NR peak data rate: 20 Gbps DL/10 Gbps UL) • High area and energy efficiency and good BLER performance • Quasi-cyclic LDPC code enables high degree of parallelism • Parity-check matrix is lifted from small base graph • Two base graphs • Optimized for different regions of code rates and code block sizes • Length and rate flexibility • Shortening and lifting to achieve 1-bit granularity on code block size • Circular buffer with puncturing and repetition to achieve codeword length flexibility; • Support for both CC-HARQ and IR-HARQ ØFuture Enhancements • Potential modifications for URLLC data channel to address the error floor and achieve high reliability Channel coding was entirely redesigned for NR LTE: Turbo NR: LDPC Throughput Medium-low High Area efficiency Low High Energy efficiency Low High Implementation complexity comparison (R1-167567) Coverage regions of 2 LDPC base graphs © 2018 InterDigital, Inc. All rights reserved. R=1/4 R=2/3 R=0.95 308 3840 Code block size Code rate 8448 Base Graph 1 Base Graph 2
  194. 194. Channel Coding: LDPC • Generator matrix • Parity check matrix © 2018 InterDigital, Inc. All rights reserved. 194 A primer on linear block codes and Tanner graph representation ! = #$$ #$% … #$' ⋮ ⋮ ⋮ ⋮ #)$ #)% … #)' = *$ ⋮ *) *$, *%, …, *) are linearly independent vectors of length , k information bits - = (/$, /%, …,/)) mapped to a ,-bit codeword: 1 = - ! = /$*$ + /%*%+ …+/)*) 3: (, − 6)×, matrix, each row is orthogonal to rows of ! Any codeword 1 satisEies: 13F = 0 • Tanner graph 3 = 1 0 1 0 0 1 1 1 x1+ x3 = 0 x2 + x3 + x4 = 0 x1 x2 x3 x4
  195. 195. Channel Coding: LDPC • Binary Erasure Channel © 2018 InterDigital, Inc. All rights reserved. 195 An example of message passing decoding x1+ x3 = 0 x2 + x3 + x4 = 0 1 x2=? x3=? 0 1 0 1 1 1
  196. 196. Channel Coding: LDPC • Parity check matrix H • Base graph (matrix of 0’s and 1’s) • Replace 1 with a cyclically shifted ZxZ identify matrix • Replace 0 with a ZxZ all-zero matrix Example: © 2018 InterDigital, Inc. All rights reserved. 196 1 1 0 1 1 0 0 1 1 0 1 0 1 0 1 0 0 1 3 2 N/A 2 1 N/A N/A 3 1 N/A 0 N/A 1 N/A 3 N/A N/A 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 0 1 0 0 0 1 1 0 0 0 0 1 0 0 0 0 1 0 Shift 3 positions 0 0 0 1 1 0 0 0 0 1 0 0 0 0 1 0 0 0 1 0 0 0 0 1 1 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 1 0 0 0 1 0 0 0 0 1 0 0 0 0 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 First row of base graph results in the first 4 rows of H: Source: 3GPP TR 38.802, Study on New Radio Access Technology -- Physical Layer Aspects V14.0.0 (2017-03) This construction is sometimes called lifting. Parity check matrix for high throughput
  197. 197. 197 Channel Coding: Control Channel Ø Polar Codes for Control Channel and Broadcast Channel • Advantages of Polar codes • Good BLER performance at small block sizes and no error floor • Polar codes are used to encode DCI, UCI (>11 bits) and MIB • A single nested polar sequence • Short polar sequence is derived from long polar sequence • Optimized Rate Matching (RM) scheme • Select among three RM schemes: puncturing, shortening, repetition • Unified circular buffer design for all three RM schemes • Triangular channel interleaver (UCI only) • Advanced polar code construction • Distributed CRC for DCI and MIB to achieve early termination • Parity check polar code for small size UCI for better performance ØFuture Enhancements • Extension of polar code for URLLC data channel NR is the first commercial system adopting Polar codes Unified circular buffer design for three RM schemes Polar Encoder (Mother code length N) Sub- block 0 Sub- block 1 Sub- block 31 Sub- block 0 Sub- block 1 Sub- block 31 Sub-block wise interleaving Circular buffer Shortening Repetition Puncturing Selecting bits from the unified circular buffer Rate matching © 2018 InterDigital, Inc. All rights reserved.
  198. 198. Channel Coding: Polar code • Invented by Erdal Arikan in 2007 • Based on the phenomena of channel polarization © 2018 InterDigital, Inc. All rights reserved. 198 W X Y 1. Binary Discrete Memoryless Channel (DMC) 2. Consider N copies of the DMC, and channel code X = GU W X1 Y1 W X2 Y2 W XN YN ... G U1 U2 UN ... ... Information bits codeword ! "#, "%, … , "'; )#, )%, … , )' = + ,-# ' ! ",; )#, )%, … , )' "#, … , ",.# = + ,-# ' !(",; )#, )%, … , )', "#, … , ",.#) 12,: ", → ()#, )%, … , )',"#, … , ",.#New channels Transform scalar channels into equivalent vector channels
  199. 199. Channel Coding: Polar code • N=8 © 2018 InterDigital, Inc. All rights reserved. 199 Transform scalar channels into equivalent vector channels (Example) • N=2 • N=4 W W Y1 Y2 U1 U2 W W W W Y1 Y2 Y3 Y4 U1 U2 U3 U4 W W W W W W W W U1' U2' U3' U4' U1" U2" U3" U4" Y1 Y2 Y3 Y4 Y5 Y6 Y7 Y8 U1 U2 U3 U4 U5 U6 U7 U8
  200. 200. QoS Provisioning • 4G: there are 9 pre-defined QoS Class Identifier (QCI) values © 2018 InterDigital, Inc. All rights reserved. 200 By a combination of QCI, ARP, GBR/Non-GBR Source: 3GPP TS 23.203, v11.2.0, “Policy and charging control architecture”, June, 2011. QCI Resource Type Priority Packet Delay Budget Packet Error Loss Rate Example Services 1 GBR 2 100 ms 10-2 Conversational Voice 2 4 150 ms 10-3 Conversational Video (Live Streaming) 3 3 50 ms 10-3 Real Time Gaming 4 5 300 ms 10-6 Non-Conversational Video (Buffered Streaming) 5 1 100 ms 10-6 IMS Signalling 6 Non-GBR 6 300 ms 10-6 Video (Buffered Streaming) TCP-based (e.g., www, e-mail, chat, ftp, p2p file sharing, progressive video, etc.) 7 7 100 ms 10-3 Voice, Video (Live Streaming) Interactive Gaming 8 8 300 ms 10-6 Video (Buffered Streaming) TCP-based (e.g., www, e-mail, chat, ftp, p2p file 9 9 sharing, progressive video, etc.) GBR: Guaranteed Bit Rate ARP: Allocation and Retention Priority
  201. 201. QoS Provisioning • 5G: Added new QCI values © 2018 InterDigital, Inc. All rights reserved. 201 More flexible QoS differentiation and more efficient signaling Source: 3GPP TS 38.300, v15.3.0, “Policy and charging control architecture”, June, 2018. QCI Resource Type Priority Level Packet Delay Budget (NOTE 13) Packet Error Loss Rate (NOTE 2) Example Services 65 GBR 0.7 75 ms 10-2 Mission Critical user plane Push To Talk voice (e.g., MCPTT) 67 1.5 100 ms 10-3 Mission Critical Video user plane 75 2.5 50 ms 10-2 V2X messages 69 Non-GBR 0.5 60 ms 10-6 Mission Critical delay sensitive signalling (e.g., MC- PTT signalling, MC Video signalling) 70 5.5 200 ms 10-6 Mission Critical Data (e.g. example services are the same as QCI 6/8/9) 79 6.5 50 ms 10-2 V2X messages 80 6.8 10 ms 10-6 Low latency eMBB applications (TCP/UDP-based); Augmented Reality
  202. 202. QoS Provisioning • 5G: Added new QCI values © 2018 InterDigital, Inc. All rights reserved. 202 More flexible QoS differentiation and more efficient signaling Source: 3GPP TS 38.300, v15.3.0, “Policy and charging control architecture”, June, 2018. QCI Resource Type Priority Level Packet Delay Budget Packet Error Loss Rate Maximum Burst Size Data Rate Averaging Window Example Services 82 GBR 1.9 10 ms 10-4 255 bytes 2 s Discrete Automation 83 GBR 2.2 10 ms 10-4 1358 bytes 2 s Discrete Automation 84 GBR 2.4 30 ms 10-5 1358 bytes 2 s Intelligent Transport Systems 85 GBR 2.1 5 ms 10-5 255 bytes 2 s Electricity Distribution- high voltage
  203. 203. QoS Provisioning • 5G added SDAP (service data adaptation protocol) sublayer • 5G introduced reflective QoS • Derive uplink QoS from downlink QoS © 2018 InterDigital, Inc. All rights reserved. 203 More flexible QoS differentiation and more efficient signaling RByRBx IP Packet H SDAP SDU PDCP SDUH RLC SDUH MAC SDUH H IP Packet H SDAP SDU PDCP SDUH RLC SDUH MAC SDU IP Packet H SDAP SDU PDCP SDUH SDU SegmentH MAC SDU SDU SegmentH H H MAC SDU ... ... ... ... MAC PDU – Transport Block PDCP RLC SDAP MAC n n+1 m Source: 3GPP TS 38.300, v15.2.0, “NR; NR and NG-RAN Overall Description”, June, 2018. RB: Radio Bearer PDCP: Packet Data Convergence Protocol
  204. 204. 204 On-going and Future Work © 2018 InterDigital, Inc. All rights reserved.
  205. 205. Outlook for Release 16 Core Network © 2018 InterDigital, Inc. All rights reserved. 205 Release 16 aims to enable services intended to be supported by the 5G Core Network 205 5G Massive IoT (MIoT) § Enable MTC and Cellular IoT functionalities in the 5G Core Network § MTC related features from Rel-10 to Rel-14 (e.g. Congestion Control, Monitoring, Small Data Transfer, CP/UP Optimizations, Non-IP Data Delivery, etc…) shall be supported in 5GC § Enhancements the 5G system to address Massive IoT (MIoT) use cases based on the initial 5G requirements V2X services in 5G § System enhancements of EPS and 5G to support advanced V2X Use cases: § Platooning, extended sensor sharing, enhanced positioning accuracy, advanced driving and remote driving Access Traffic Steering Switch and Splitting § Multi-access (e.g. 3GPP and WLAN) PDU session management § Support traffic splitting between multiple accesses (e.g. 3GPP and WLAN) § Switch traffic between multiple accesses 5G LAN Services § Enhancements to the 5G system to support 5GLAN service § Support for 3GPP network that is not for public use and for which service continuity and roaming with a PLMN is possible (a type network). § Support an isolated 3GPP network that does not interact with a PLMN (b type network)

×