Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Framework for Adaptive Delivery of Omnidirectional Video

843 views

Published on

This paper addresses the need for a scientific framework enabling the adaptive delivery of omnidirectional video within heterogeneous environment. It considers state of the art techniques for the adaptive video streaming over HTTP and extends it towards omnidirectional/360-degree videos.

Human Vision and Electronic Imaging 2018, 28 January - 2 February, 2018 • Burlingame, California USA

Published in: Technology
  • Be the first to comment

A Framework for Adaptive Delivery of Omnidirectional Video

  1. 1. A Framework for Adaptive Delivery of Omnidirectional Video Priv.-Doz. Dr. Christian Timmerer Alpen-Adria-Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Information Technology (ITEC) w Multimedia Communication (MMC) w Sensory Experience Lab (SELab) http://blog.timmerer.com w http://selab.itec.aau.at/ w http://dash.itec.aau.at w christian.timmerer@itec.aau.at Chief Innovation Officer (CIO) at bitmovin GmbH http://www.bitmovin.com w christian.timmerer@bitmovin.com Human Vision and Electronic Imaging 2018, January 31, 2018 http://www.slideshare.net/christian.timmerer
  2. 2. Outline • Introduction • Framework • Encoding options and omnidirectional HAS • Standardization overview • Conclusion January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 2
  3. 3. Multimedia Traffic on the Internet • Real-time entertainment: Streaming video and audio; >70% of Internet traffic at peak periods • Forecast: Visual Networking Index (VNI) 2016-2021 – IP video traffic will be 82% of all consumer Internet traffic by 2021 (up from 73% in 2016); will grow threefold from 2016 to 2021 – Live Internet video will account for 13% of Internet video traffic by 2021; will grow 15-fold from 2016 to 2021 • Popular services – YouTube (17.53%), Netflix (35.15%), Amazon Video (4.26%), Hulu (2.68%); all delivered over-the-top (OTT) • More people now subscribe to Netflix (50.85M) than cable TV (48.61M) in the US (Q1 2017) January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 3 Global Internet Phenomena Report: 2016 https://www.businessinsider.com/netflix-more-popular-than-cable-us-chart-2017-6/
  4. 4. How does it work? In a nutshell… January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 4 Adaptation logic is within the client, not normatively specified by the standard, subject to research and development
  5. 5. Omnidirectional Video / 360° Video January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 5 Capturing Devices • Stitching, Projection Formats • Encoding, Encryption, Encapsulation • Storage, Content Distribution, Delivery • Processing, Decoding, Rendering, … Consumer Devices
  6. 6. Functional Architecture January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 6 Encode, Encrypt, Encapsulate Decode, Decrypt, Decapsulate Project & Render Fuse, Stitch & Edit ① ② ③ ④ ⑤ Store & Deliver Capture Consume Content Creation NetworkServer Client Encoding Encryption Encapsulation Storage Delivery Decryption Decapsulation Decoding RenderingEditing Processing Processing Capture Acquisition Consumption Distribution Adaptive Delivery of Omnidirectional Video Interaction From ecosystem… To buidling blocks…
  7. 7. Functional Architecture January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 7 Encode, Encrypt, Encapsulate Decode, Decrypt, Decapsulate Project & Render Fuse, Stitch & Edit ① ② ③ ④ ⑤ Store & Deliver Capture Consume Content Creation NetworkServer Client Encoding Encryption Encapsulation Storage Delivery Decryption Decapsulation Decoding RenderingEditing Processing Processing Capture Acquisition Consumption Distribution Adaptive Delivery of Omnidirectional Video Interaction From ecosystem… To buidling blocks…
  8. 8. Adaptive Streaming Options • Traditional, viewport-agnostic streaming – Simple, easy, deployed today – Bandwidth waste, quality issues • Viewport-adaptive streaming – Multiple versions for predefined viewports – Various projection techniques (pyramid) – Bandwidth waste reduced, increased storage and CDN costs, limited flexibility • Tile-based streaming – Use tiling technique of modern video codecs – High complexity, full flexibility – Multiple challenges January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 8 X. Corbillon, et al., "Viewport-adaptive navigable 360-degree video delivery," 2017 IEEE International Conference on Communications (ICC), Paris, 2017, https://doi.org/10.1109/ICC.2017.7996611 M. Graf, et al. 2017. Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation. Proc. ACM MMSys'17. https://doi.org/10.1145/3083187.3084016 C. Concolato, et al., "Adaptive Streaming of HEVC Tiled Videos using MPEG-DASH," IEEE TCSVT, 2017. https://doi.org/10.1109/TCSVT.2017.2688491
  9. 9. Tile-based Adaptive Streaming January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 9 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Encoding & Packaging Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 DifferentQuality Representations Delivery MPEG-HEVC/H.265 Tiles in ISOBMFF Adaptive Player Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile-based streaming of VR/360° content with MPEG-DASH SRD … … … … Adaptive Streaming using MPEG-DASH SRD Head Mounted Displays Browsers, Smart (Mobile) Devices (Stereo) 2D, (Stereo) 3D
  10. 10. Encoding Options • AVC dominates the market • HEVC, VP9, AV1 support tiles – Divides a picture into independent, rectangular regions – Tradeoff: bitrate, quality, flexibility • Multiple tiling options available – Uniform vs. non-uniform tiling – Same vs. mixed resolutions • New quality metrics, mostly based on PSNR but subjective quality assessments/metrics increasing January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 10 Bitmovin 2017 Video Developer Report https://bitmovin.com/2017-video-developer-report/ I. D.D. Curcio, et al. 2017. Bandwidth Reduction of Omnidirectional Viewport-Dependent Video Streaming via Subjective Quality Assessment. Proc. AltMM'17. https://doi.org/10.1145/3132361.3132364 A. Zare, et al. 2016. HEVC-compliant Tile-based Streaming of Panoramic Video for Virtual Reality Applications. Proc. ACM MM'16. http://dx.doi.org/10.1145/2964284.2967292
  11. 11. Dataset • Segment length / Intra period – 1s (tiled content) vs 1, 2, 4s (monolithic content) • Tiling pattern (columns × rows): 1×1, (i.e., tiles monolithic), 3×2, 5×3, 6×4, and 8×5 • Resolution: 1920×960, 3840×1920 and 7680×3840 • Map projection: equirectangular format • Quantization parameter: QP={22,27,32,37,42} • Head motion recordings for V-PSNR evaluation January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 11
  12. 12. Bitrate Overhead due to Tiling January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 12 500 1000 1500 2000 2500 384042444648 Bitrate [kbps] Y−PSNR[dB] ● ● ● ● ● ● ● 1x1 tiles 3x2 tiles 5x3 tiles 6x4 tiles 8x5 tiles Tile Overhead for Resolution: 1920x960 Sequence: AssassinsCreed 2000 4000 6000 8000 30323436384042 Bitrate [kbps] Y−PSNR[dB] ● ● ● ● ● ● ● 1x1 tiles 3x2 tiles 5x3 tiles 6x4 tiles 8x5 tiles Tile Overhead for Resolution: 1920x960 Sequence: ExploreTheWorld
  13. 13. Bandwidth Requirements January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 13
  14. 14. Adaptive Streaming Issues • Increasing number of segment/tile requests – HTTP/2 server push, query parameters, proprietary protocols – Additional functionality at server – breaks fundamental HAS requirements • Low latency streaming – Reducing segment size impacts coding efficiency (1s vs. 4s) – CMAF chunks + other enhancements to enable sub-second latency – Remember: live internet video will grow 15-fold from 2016 to 2021 • Viewport prediction – Allows prefetching (caching) but cannot predict to much into future (1s) – Impact on segment size but situation will get better the more data is available – machine learning/AI will help • Quality of Experience: QoE – Still in its infancy but situation much better than one year ago – Requires datasets, subjective studies, quality models, metrics January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 14 M. Xu, et al., "A subjective visual quality assessment method of panoramic videos," Proc. ICME’17. https://doi.org/10.1109/ICME.2017.8019351 R. Schatz, et al., "Towards subjective quality of experience assessment for omnidirectional video streaming," Proc. QoMEX’17, https://doi.org/10.1109/QoMEX.2017.7965657 Y. Rai, at al. 2017. A Dataset of Head and Eye Movements for 360 Degree Images. Proc. ACM MMSys’17. https://doi.org/10.1145/3083187.3083218 S. Petrangeli, et al. 2017. An HTTP/2-Based Adaptive Streaming Framework for 360° Virtual Reality Videos. Proc. ACM MM'17. https://doi.org/10.1145/3123266.3123453 N. Bouzakaria, et al., "Overhead and performance of low latency live streaming using MPEG-DASH," Proc. IISA’14. https://doi.org/10.1109/IISA.2014.6878732 C.-L. Fan, et al. 2017. Fixation Prediction for 360° Video Streaming in Head-Mounted Virtual Reality. Proc. NOSSDAV'17. https://doi.org/10.1145/3083165.3083180 Stay tuned for QoMEX'18, MMSys'18, PV'18, ICME'18, MM'18…
  15. 15. Standardization Overview January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 15 R. Skupin, et al., "Standardization Status of 360 degree Video Coding and Delivery,” Proc. IEEE VCIP’17. C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017. https://doi.org/10.1109/MCOMSTD.2017.1700038 B. Choi, et al., (editors) "Information Technology -- Coded Representation of Immersive Media (MPEG-I) -- Part 2: Omnidirectional Media Format," ISO/IEC 23090-2 Final Draft International Standard (FDIS), Dec. 2017.
  16. 16. Conclusions • In general, we know how it works and how to do it, but many challenges exist • Optimization required everywhere from creation to consumption • Need to reduce complexity to foster adoption and innovation • This framework… – Provides an overview; – Identifies building blocks; – Outlines state of the art; – Raises research questions and helps with respect to gap analysis. January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 16http://www.diamandis.com/peters-laws [10] When faced without a challenge, make one. [13] When in doubt: THINK. [26] If you can’t measure it, you can’t improve it. [22] The day before something is a breakthrough it’s a crazy idea.
  17. 17. Thank you for your attention January 31, 2018 Dr. Timmerer [AAU/Bitmovin] 17 ... questions, comments, etc. are welcome … Priv.-Doz. Dipl.-Ing. Dr. Christian Timmerer Associate Professor Alpen-Adria-Universität Klagenfurt, Department of Information Technology (ITEC) Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA christian.timmerer@itec.uni-klu.ac.at http://blog.timmerer.com/ Tel: +43/463/2700 3621 Fax: +43/463/2700 3699 © Copyright: Christian Timmerer

×