Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

HTTP Adaptive Streaming State of the Art and Challenges Ahead

899 views

Published on

Real-time entertainment services deployed over the open, unmanaged Internet – streaming audio and video – account now for more than 70% of the Internet traffic and it is assumed that this number will reach 80% by 2021. The technology used for such services is commonly referred to as HTTP Adaptive Streaming (HAS) and is widely adopted by various platforms such as YouTube, Netflix, Flimmit, etc. thanks to the standardization of MPEG-DASH and HLS. This talk will provide an overview of HAS, the state of the art of selected deployment options, and reviews work-in-progress as well challenges ahead. The main challenge can be characterized by the fact that (i) content complexity increases, (ii) delay or latency are vital application requirements, and (iii) Quality of Experience cannot be neglected anymore.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

HTTP Adaptive Streaming State of the Art and Challenges Ahead

  1. 1. HTTP Adaptive Streaming State of the Art and Challenges Ahead Priv.-Doz. Dr. Christian Timmerer [Ack: Ali C. Begen, Networked Media & Ozyegin University | Thomas Stockhammer, Qualcomm Inc. | Iraj Sodagar, Consultant] Alpen-Adria-Universität Klagenfurt (AAU)  Faculty of Technical Sciences (TEWI)  Department of Information Technology (ITEC)  Multimedia Communication (MMC) http://blog.timmerer.com  http://dash.itec.aau.at  christian.timmerer@itec.aau.at Chief Innovation Officer (CIO) at bitmovin GmbH http://www.bitmovin.com  christian.timmerer@bitmovin.com Internet QoE, July 2, 2018 http://www.slideshare.net/christian.timmerer
  2. 2. Importance of Multimedia Delivery • Real-time entertainment: Streaming video and audio; >70% of Internet traffic at peak periods • Popular services – YouTube (17.53%), Netflix (35.15%), Amazon Video (4.26%), Hulu (2.68%); all delivered over-the-top (OTT) • Forecast: Visual Networking Index (VNI) 2016-2021 (Sep’17) – IP video traffic will be 82% of all consumer Internet traffic by 2021 (up from 73% in 2016); will grow threefold from 2016 to 2021 – Live Internet video will account for 13% of Internet video traffic by 2021; will grow 15-fold from 2016 to 2021 • More people now subscribe to Netflix (50.85M) than cable TV (48.61M) in the US (Q1 2017) July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 2 Global Internet Phenomena Report: 2016 >70% Downstream Traffic
  3. 3. July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 3 $44,000,000
  4. 4. July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 4 Example Platform/Infrastructure https://bitmovin.com/
  5. 5. Common Annoyances in Streaming • Wrong format • Wrong protocol • Plugin requirements • DRM issues • Long start-up delay • Poor quality • Frequent stalls • Quality oscillations • No seeking features July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 5
  6. 6. Over-The-Top – Adaptive Media Streaming • In a nutshell… July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 6 Adaptation logic is within the client, not normatively specified by the standard, subject to research and development
  7. 7. Multi-Bitrate Encoding and Representation Switching July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 7 Contents on the Web Server Movie A – 200 Kbps Movie A – 400 Kbps Movie A – 1.2 Mbps Movie A – 2.2 Mbps . . . . . . Movie K – 200 Kbps Movie K – 500 Kbps Movie K – 1.1 Mbps Movie K – 1.8 Mbps . . . . . . Time (s) Start quickly Keep requesting Improve quality Loss/congestion detection Revamp quality ... . . . Segments
  8. 8. Adaptive Streaming over HTTP July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 8 … … … … HTTP GETs Client Buffer Media Player HTTP Server
  9. 9. Formats and Standards • Adobe – HTTP Dynamic Streaming (HDS) – Switched to DASH • Apple – HTTP Live Streaming (HLS) – Required for iOS • Microsoft – Smooth Streaming – Switched to DASH, almost.. • MPEG Dynamic Adaptive Streaming over HTTP (DASH) – Supported by Netflix, YouTube, Bitmovin, etc. • MPEG Common Media Application Format (MPEG-A Part 19) – The new kid on the block – support for “fragmented mp4 in HLS” – DASH/HLS convergence at segment level – some open issues with encryption format July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 9 Source: http://xkcd.com/927/
  10. 10. Scope of DASH: what is specified? July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 10 Media Presentation on HTTP Server DASH-enabled ClientMedia Presentation Description . . . Segment … . . .Segment … . . . Segment … . . .Segment … … Segments located by HTTP-URLs DASH Control Engine HTTP/1.1 HTTP Client MPD Parser Media Engine On-time HTTP requests to segments
  11. 11. Scope of DASH: what is specified? July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 11 Media Presentation on HTTP Server DASH-enabled ClientMedia Presentation Description . . . Segment … . . .Segment … . . . Segment … . . .Segment … … Segments located by HTTP-URLs DASH Control Engine HTTP/1.1 HTTP Client MPD Parser Media Engine On-time HTTP requests to segments
  12. 12. DASH Data Model July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 12 MPD Period id = 1 start = 0 s Period id = 3 start = 300 s Period id = 4 start = 850 s Period id = 2 start = 100 s Adaptation Set 0 subtitle turkish Adaptation Set 2 audio english Adaptation Set 1 BaseURL=http://abr.rocks.com/ Representation 2 Rate = 1 Mbps Representation 4 Rate = 3 Mbps Representation 1 Rate = 500 Kbps Representation 3 Rate = 2 Mbps Resolution = 720p Segment Info Duration = 10 s Template: 3/$Number$.mp4 Segment Access Initialization Segment http://abr.rocks.com/3/0.mp4 Media Segment 1 start = 0 s http://abr.rocks.com/3/1.mp4 Media Segment 2 start = 10 s http://abr.rocks.com/3/2.mp4 Adaptation Set 3 audio german Adaptation Set 1 video Period id = 2 start = 100 s Representation 3 Rate = 2 Mbps Selection of components/tracks Well-defined media format Selection of representations Splicing of arbitrary content like ads Chunks with addresses and timing
  13. 13. MPEG-DASH Features July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 13
  14. 14. https://bitmovin.com/demo/ July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 14
  15. 15. MPEG CMAF: Threat or Opportunity? + CMAF Media Object Model compatible with DASH Data Model + Segment formats based on ISOBMFF - Different manifest formats (MPD vs. m3u8) - CENC: AES-128 CBC (HLS) vs. AES-128 CTR (all others) modeJuly 2, 2018 Dr. Timmerer [AAU/Bitmovin] 15 https://bitmovin.com/what-is-cmaf-threat-opportunity/
  16. 16. CMAF ISO-BMFF Media Objects • Manifests typically provide URLs to – CMAF track files – CMAF header + CMAF segments • single/multiple fragment(s) – CMAF header + CMAF chunk Encoder Encryption Packaging CMAF Header CMAF Fragment CMAF Fragment CMAF Chunk CMAF Chunk CMAF Chunk CMAF Fragment R A P R A P R A P R A P CMAF Fragment CMAF Segment CMAF Segment CMAF Track File July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 16
  17. 17. Common Problems in DASH • Encoding | Packaging | Encryption – Guidelines: 3-20 different representations (mobile to UHD) – Segment length: 4s shows good tradeoff (2s vs. 9s) – Offline vs. on-the-fly • Delivery, distribution, CDN – MMSys’16 keynote by Neill Kipp: https://mmsys2016.itec.aau.at/ • Consumption and Quality of Experience (QoE) – Adaptation strategies: buffer- vs. throughput-based – Multi-client competition: on-off behavior – Quality-aware streaming: highest possible bitrate vs. highest quality – Inter-Destination Media Synchronization (IDMS): new applications – Virtual Reality / 360-degree video: tiled streamingJuly 2, 2018 Dr. Timmerer [AAU/Bitmovin] 17 S. Lederer, C. Müller and C. Timmerer, “Dynamic Adaptive Streaming over HTTP Dataset”, In Proceedings of the ACM Multimedia Systems Conference 2012, Chapel Hill, North Carolina, February 22-24, 2012.
  18. 18. Quality of Experience for DASH • Objective – Initial or startup delay (low) – Buffer underrun / stalls (zero) – Quality switches (low) – Media throughput (high) – [Other media-related configuration: encoding, representations, segment length, …] • Subjective – Mean Opinion Score (MOS) – various scales – Various methodologies (e.g., DSCQS, DSIS, ACR, PC, …) July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 18M. Seufert, et al., "A Survey on Quality of Experience of HTTP Adaptive Streaming,"IEEE Communications Surveys & Tutorials, vol.17, no.1, 2015. doi:10.1109/COMST.2014.2360940
  19. 19. July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 19 DASH QoE in Real-World B. Rainer, C. Timmerer, Quality of Experience of Web-based Adaptive HTTP Streaming Clients in Real-World Environments using Crowdsourcing, Proceedings of International Workshop on VideoNext: Design, Quality and Deployment of Adaptive Video Streaming, Sydney, Australia, Dec. 2014. https://doi.org/10.1145/2676652.2676656 Stalls and low quality are bad for QoE but not startup delay
  20. 20. July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 20 10 different adaptation algorithms C. Timmerer, M. Maiero, B. Rainer, Which Adaptation Logic? An Objective and Subjective Performance Evaluation of HTTP-based Adaptive Media Streaming Systems, In arXiv.org [cs.MM], N.N., vol. abs/1606.00341, N.N., pp. 11, 2016. https://arxiv.org/pdf/1606.00341
  21. 21. July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 21
  22. 22. AdViSE • Adaptive Media Content [DASH, HLS, CMAF] • Players/Algorithms • Network Parameters Impaired Media Sequences Generate Impaired Media Sequences Templates [Startup Delay, Stalling, …] WESP QoE Evaluation Parameters [Questionnaire, Methodology, Crowdsourcing Platform, …] QoS/QoE Metrics Subjective Results + Other Data Reports Analysis AdViSE: Anatoliy Zabrovskiy, Evgeny Kuzmin, Evgeny Petrov, Christian Timmerer, and Christopher Mueller. 2017. AdViSE: Adaptive Video Streaming Evaluation Framework for the Automated Testing of Media Players. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM, New York, NY, USA, 217-220. DOI: https://doi.org/10.1145/3083187.3083221 WESP: Benjamin Rainer, Markus Waltl, Christian Timmerer, A Web based Subjective Evaluation Platform, In Proceedings of the 5th International Workshop on Quality of Multimedia Experience (QoMEX'13) (Christian Timmerer, Patrick Le Callet, Martin Varela, Stefan Winkler, Tiago H Falk, eds.), IEEE, Los Alamitos, CA, USA, pp. 24-25, 2013. https://doi.org/10.1109/QoMEX.2013.6603196 How to evaluate HAS systems? ① ② ③④ ⑤ Log of Segment Requests July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 22
  23. 23. C. Timmerer, A. Zabrovskiy, and Ali C. Begen, Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems, Proceedings of the 1st IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR), Miami, FL, USA, April 2018. Example Evaluation Results • Test sequence encoded 15 different representation (Amazon Prime configuration: 400x224@100Kbps – 1920x1080@15Mbps) with 4s segment length • Bandwidth trajectory based on prior work proposed in literature; network delay 70ms July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 23
  24. 24. Bandwidth Index vs. QoE • Bandwidth Index – Avg. bitrate – Efficiency – Stability • QoE Model – Startup delay – Stalls July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 24 C. Timmerer, A. Zabrovskiy, E. Kuzmin, E. Petrov, Quality of experience of commercially deployed adaptive media players, In 2017 21st Conference of Open Innovations Association (FRUCT) (Sergey Balandin, ed.), pp. 330-335, 2017. https://doi.org/10.23919/FRUCT.2017.8250200
  25. 25. Omnidirectional Video / 360° Video July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 25 Capturing Devices • Stitching, Projection Formats • Encoding, Encryption, Encapsulation • Storage, Content Distribution, Delivery • Processing, Decoding, Rendering, … Consumer Devices C. Timmerer, A. C. Begen, A Framework for Adaptive Delivery of Omnidirectional Video, In Electronic Imaging – Human Vision and Electronic Imaging (HVEI), vol. 2018, no. 16, 2018.
  26. 26. Adaptive Streaming Options for 360 • Traditional, viewport-agnostic streaming – Simple, easy, deployed today – Bandwidth waste, quality issues • Viewport-adaptive streaming – Multiple versions for predefined viewports – Various projection techniques (pyramid) – Bandwidth waste reduced, increased storage and CDN costs, limited flexibility • Tile-based streaming – Use tiling technique of modern video codecs – High complexity, full flexibility – Multiple challenges July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 26 X. Corbillon, et al., "Viewport-adaptive navigable 360-degree video delivery," 2017 IEEE International Conference on Communications (ICC), Paris, 2017, https://doi.org/10.1109/ICC.2017.7996611 M. Graf, et al. 2017. Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation. Proc. ACM MMSys'17. https://doi.org/10.1145/3083187.3084016 C. Concolato, et al., "Adaptive Streaming of HEVC Tiled Videos using MPEG-DASH," IEEE TCSVT, 2017. https://doi.org/10.1109/TCSVT.2017.2688491
  27. 27. Tile-based Adaptive Streaming July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 27 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Encoding & Packaging Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Delivery MPEG-HEVC/H.265 Tiles in ISOBMFF Adaptive Player Tile 1 Tile 2 Tile 3 Tile 4 Tile 5 Tile 6 Tile 7 Tile 8 Tile 9 Tile 10 Tile 11 Tile 12 Tile 13 Tile 14 Tile 15 Tile 16 Tile 17 Tile 18 Tile 19 Tile 20 Tile 21 Tile 22 Tile 23 Tile 24 Tile-based streaming of VR/360° content with MPEG-DASH SRD / MPEG-I OMAF … … … … Adaptive Streaming using MPEG- DASH SRD / MPEG-I OMAF Head Mounted Displays Browsers, Smart (Mobile) Devices (Stereo) 2D, (Stereo) 3D
  28. 28. Bitrate Overhead due to Tiling July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 28
  29. 29. Bandwidth Requirements July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 29
  30. 30. Immersive Media Streaming • 3DoF, 3DoF+, windowed & omnidirectional 6DoF, 6DoF • Point cloud, light field, volumetric video, holograms, volograms, … – Content Creation for AR, VR, and Free Viewpoint Video, Aljosa Smolic https://www.qomex2018.org/keynotes/#AljT • Theses today… (Philip A. Chou, 8i) – Hologram compression today is like video compression in 1988 – Hologram streaming today is like video streaming in 1997 • MPEG-I – First standardization activity comprising now eight parts July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 30Holograms are the Next Video, Philip A. Chou, 8i, http://www.mmsys2018.org/program/keynotes/ C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017. https://doi.org/10.1109/MCOMSTD.2017.1700038 M. Hosseini, C. Timmerer, Dynamic Adaptive Point Cloud Streaming, 23rd Packet Video Workshop, Amsterdam, The Netherlands, June 2018. https://doi.org/10.1145/3210424.3210429 https://multimediacommunication.blogspot.com/2017/04/vr360-streaming-standardization-related.html
  31. 31. Multimedia Systems Tradeoff July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 31 Quality Content Time Quality of {Content, Service, Experience, Life, …} Content complexity: traditional AV, AR/VR/360, multi-modality/-sensory Based on Klara Nahrstedt at IEEE MIPR’18 Retreat End-to-end delay, startup delay, channel switching, synchronization, interaction
  32. 32. Deployment Thoughts • Proprietary ecosystems disappeared (Silverlight, Flash), no more plugins – HTML5! – MSE/EME available on all major browser platforms – Support for both DASH/HLS (+CMAF) and CENC • Rich feature set: multiple codecs, ads, DRM, multi-language/-audio, subtitles, VR/360, UHD, HFR, HDR, live, on-demand, analytics, … • Common implementation issues: start-up, buffering, high-quality, seamless switching, platform support, cost-effective, … • Solutions available for adaptive streaming, advertising, VR/360, live streaming, and DRM – Per-title/-scene/-shot adaptation, AV1, fast encoding, managed on-premise – Details available at https://bitmovin.com/ July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 32
  33. 33. Bitmovin kicks off NAB show 2018 with a bang ● Announcing $30M in Series B funding and exciting new product launches ● And yes, we are hiring… https://bitmovin.com/careers
  34. 34. References • B. Rainer, C. Timmerer, Quality of Experience of Web-based Adaptive HTTP Streaming Clients in Real-World Environments using Crowdsourcing, Proceedings of International Workshop on VideoNext: Design, Quality and Deployment of Adaptive Video Streaming, Sydney, Australia, Dec. 2014. https://doi.org/10.1145/2676652.2676656 • C. Timmerer, M. Maiero, B. Rainer, Which Adaptation Logic? An Objective and Subjective Performance Evaluation of HTTP-based Adaptive Media Streaming Systems, In arXiv.org [cs.MM], N.N., vol. abs/1606.00341, N.N., pp. 11, 2016. https://arxiv.org/pdf/1606.00341 • A. Zabrovskiy, E. Kuzmin, E. Petrov, C. Timmerer, and C. Mueller. 2017. AdViSE: Adaptive Video Streaming Evaluation Framework for the Automated Testing of Media Players. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM, New York, NY, USA, 217-220. DOI: https://doi.org/10.1145/3083187.3083221 • B. Rainer, M. Waltl, C. Timmerer, A Web based Subjective Evaluation Platform, In Proceedings of the 5th International Workshop on Quality of Multimedia Experience (QoMEX'13) (Christian Timmerer, Patrick Le Callet, Martin Varela, Stefan Winkler, Tiago H Falk, eds.), IEEE, Los Alamitos, CA, USA, pp. 24-25, 2013. https://doi.org/10.1109/QoMEX.2013.6603196 • C. Timmerer, A. Zabrovskiy, and Ali C. Begen, Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems, Proceedings of the 1st IEEE International Conference on Multimedia Information Processing and Retrieval (MIPR), Miami, FL, USA, April 2018. • C. Timmerer, A. Zabrovskiy, E. Kuzmin, E. Petrov, Quality of experience of commercially deployed adaptive media players, In 2017 21st Conference of Open Innovations Association (FRUCT) (Sergey Balandin, ed.), pp. 330-335, 2017. https://doi.org/10.23919/FRUCT.2017.8250200 • C. Timmerer, A. C. Begen, A Framework for Adaptive Delivery of Omnidirectional Video, In Electronic Imaging – Human Vision and Electronic Imaging (HVEI), vol. 2018, no. 16, 2018. • M. Graf, C. Timmerer, C. Mueller. Towards Bandwidth Efficient Adaptive Streaming of Omnidirectional Video over HTTP: Design, Implementation, and Evaluation. Proc. ACM MMSys'17. https://doi.org/10.1145/3083187.3084016 • C. Timmerer, "Immersive Media Delivery: Overview of Ongoing Standardization Activities," in IEEE Communications Standards Magazine, vol. 1, no. 4, pp. 71-74, Dec. 2017. https://doi.org/10.1109/MCOMSTD.2017.1700038 • M. Hosseini, C. Timmerer, Dynamic Adaptive Point Cloud Streaming, 23rd Packet Video Workshop, Amsterdam, The Netherlands, June 2018. https://doi.org/10.1145/3210424.3210429 July 2, 2018 Dr. Timmerer [AAU/Bitmovin] 34

×