HTTP Adaptive Streaming – Where Is It Heading?

HTTP Adaptive Streaming – Where Is It Heading?

• 1.
  HTTP Adaptive Streaming– Where Is It Heading? Christian Timmerer, Assoc.-Prof. at AAU, CIO at Bitmovin Hermann Hellwagner, Professor at AAU Klagenfurt, Austria November 30th , 2020 1
• 2.
  “By 2022, Internetvideo will represent 82% of all Internet traffic.” Cisco Visual Networking Index: Forecast and Trends, 2017–2022 (White Paper), Cisco, February 2019. 2
• 3.
  ● Introduction ● ATHENA ○Content Provisioning ○ Content Delivery ○ Content Consumption ○ End-to-End Aspects ○ Quality of Experience ● Conclusions: HAS - Where Is It Heading? Agenda 3
• 4.
  Presenters Christian Timmerer Assoc.-Prof at Alpen-Adria-UniversitätKlagenfurt CIO | Head of Research and Standardization at Bitmovin Hermann Hellwagner Professor at Alpen-Adria-Universität Klagenfurt 4 4
• 5.
  Video streaming isdominating today’s Internet traffic ● May 2020: 57.64%; YouTube is the undisputed king with 15.94% followed by Netflix with 11.42%* ● By 2022, video will account for 82% of global IP traffic and live video will increase 15-fold and reach 17% of Internet video traffic** Motivation Sources: * Sandvine Global Internet Phenomena (May 2020). ** Cisco Visual Networking Index (VNI), Complete Forecast Update, 2017–2022 (Dec. 2018) 5 5
• 6.
  HTTP Adaptive Streaming101 Adaptation logic is within the client, not normatively specified by a standard, subject to research and development 6 6 Client
• 7.
  “Application-oriented basic research”to address current and future research and deployment challenges of HAS and emerging streaming methods ATHENA - Adaptive Streaming over HTTP and Emerging Networked Multimedia Services Content Provisioning Content Delivery Content Consumption End-to-End Aspects ● Video encoding for HAS ● Quality-aware encoding ● Learning-based encoding ● Multi-codec HAS ● Edge computing ● Information CDN/SDN⇿clients ● Netw. assistance for/by clients ● Utility evaluation ● Bitrate adaptation schemes ● Playback improvements ● Context and user awareness ● Quality of Experience (QoE) studies ● Application/transport layer enhancements ● Quality of Experience (QoE) models ● Low-latency HAS ● Learning-based HAS https://athena.itec.aau.at/ 7 7 Funding:
• 8.
  Multimedia Systems Challengesand Tradeoffs 8 8 Basic figure by Klara Nahrstedt
• 9.
  Video coding forHTTP Adaptive Streaming ● Quality improvement Per-Title Encoding (PTE) et al.: per-title/scene/shot/segment, content-/context aware, content-adaptive, quality-aware encoding ● Runtime improvement Hardware-/software-based (cloud), parallel/distributed, information reuse from reference encodings (multi-rate/-resolution) ● Application scenarios Video on Demand (VoD incl. diff. flavors AVoD, SVoD), live (incl. diff. flavors), interactive, games, video conferencing Content Provisioning 9 9
• 10.
  Video Encoding Block Partitioning Motion Compensation Transformation & Quantization EntropyCoding Entropy Decoding Inverse Transformation & Inverse Quantization Inter or Intra Prediction Picture Buffer In-loop Filtering 10
• 11.
  Video Encoding withMachine Learning 11 11 Block Partitioning Motion Compensation Transformation & Quantization Entropy Coding Entropy Decoding Inverse Transformation & Inverse Quantization Inter or Intra Prediction Picture Buffer In-loop Filtering CTU Decision Prediction Optical Flow Detection Mode Prediction Angular Direction Prediction Deblocking with ML Denoising with ML Super-resolution
• 12.
  1) D. Schroeder etal. "Efficient multi-rate video encoding for HEVC-based adaptive HTTP streaming." IEEE Transactions on Circuits and systems for Video Technology 28.1 (2016): 143-157. 2) B. Guo, Y. Han, J. Wen, "Fast Block Structure Determination in AV1-based Multiple Resolutions Video Encoding," 2018 IEEE Int’l. Conf. on Multimedia and Expo (ICME), San Diego, CA, USA, July 2018. 3) H. Amirpour, E. Çetinkaya, C. Timmerer and M. Ghanbari, "Fast Multi-rate Encoding for Adaptive HTTP Streaming," Data Compression Conference (DCC), Snowbird, UT, USA, 2020, Encoding information can be used among different quality representations State-of-the-art: ● Encode the highest quality1) or the lowest quality2) as the reference first, then use this information Proposed method: ● Encode the highest quality first, then use its information to encode the lowest quality and then use information from both representations to encode the remaining representations3) ● Double bound for CTU search ranges Fast Multi-rate Encoding 12 12 QP1 QPN QPN-1QP3 QP2 ... Encodingruntime(norm.)
• 13.
  Parallel encoding isstill problematic State-of-the-art: ● Encode the highest quality1) or the lowest quality2) as the reference first, then use this information Proposed method: ● Try different quality levels as the reference representation to determine the best starting point for parallel encoding ● Encode the middle quality first, then use its information to reduce the time-complexity for higher qualities to eliminate possible bottlenecks3) ● Upper or lower bound depending on the quality level Fast Multi-rate Encoding 1) Schroeder et al., 2016 2) Guo et al., 2018 3) H. Amirpour, E. Çetinkaya, C. Timmerer and M. Ghanbari, "Towards Optimal Multirate Encoding for HTTP Adaptive Streaming," International MultiMedia Modeling Conference (MMM), Prague, Czech Republic, 2021 QPN/2 QPN QP2 QP1 ... 13 13 Encodingruntime(norm.)
• 14.
  Use ML toencode dependent representations State-of-the-art: ● Use a CNN to predict CTU depth decisions1) Proposed method: ● Train a CNN with encoding information obtained from the reference quality (the lowest quality) representation and use its decision to encode dependent representations2) ● Focus on parallel encoding, thus only apply for bottleneck situations ● Train different CNNs for different QP targets Fast Multi-rate Encoding with Machine Learning (FaME-ML) 1) Kim, Kyungah, and Won Woo Ro. "Fast CU depth decision for HEVC using neural networks." IEEE Transactions on Circuits and Systems for Video Technology 29.5 (2018): 1462-1473. 2) E. Çetinkaya, H. Amirpour, C. Timmerer and M. Ghanbari, “FaME-ML: Fast Multirate Encoding for HTTP Adaptive Streaming Using Machine Learning,” 2020 IEEE International Conference on Visual Communications and Image Processing (VCIP), Macau, 2020 QPN CNN QPN-1 QP1 QP2 ... HEVC HEVCHEVC CNN HEVC HEVC 14 14 Encodingruntime(norm.)
• 15.
  End Game ML Encoder ML Decoder 15 15 See also: ●CLIC: Workshop and Challenge on Learned Image Compression, https://www.compression.cc/ ● JPEG AI-based image coding, https://jpeg.org/ ● JVET (MPEG/VCEG) AI-based video coding, http://mpeg.org/
• 16.
  Network assistance forHTTP Adaptive Streaming ● Edge computing support (at CDN / cellular network edge) Functions at (or, assisted by) the edge: adaptation, analytics, (pre-)fetching, caching, transcoding, repackaging of content, request aggregation ● Server/network/CDN ↔ HAS client information exchange and collaboration IETF ALTO, MPEG SAND, MPEG NBMP, …; SDN-DASH, SDN-HAS, SABR, ... ● Use of modern network architecture features SW Defined Networking (SDN); Network Function Virtual. (NFV); MC-ABR ● Low-latency live streaming Use of MPEG CMAF, HTTP/1.1 Chunked Transfer Encoding (CTE), other protocol features (e.g., HTTP/2 Push); LL-HLS; specific network functions; CDN support Content Delivery 16 16
• 17.
  Approach: ● Deliver CMAFsegments only over the core/CDN ● Repackage into requested format at the edge Evaluation: ● Analytical model and simulation to assess bandwidth savings as compared to all-formats delivery (~ 20%) ● Measurements to get segment repackaging times (CMAF➝HLS: 45-67 ms, depending on seg. size) ● Real world-like testbed to assess “full” repackaging time (avg. 136 ms, CAdViSE on AWS cloud, 4-sec. seg./1080p) Dynamic Segment Repackaging at the Edge for HAS 17 17 Jesus Aguilar-Armijo, Babak Taraghi, Christian Timmerer, and Hermann Hellwagner. ”Dynamic Segment Repackaging at the Edge for HTTP Adaptive Streaming”. IEEE Int’l. Symposium on Multimedia (ISM'20). Dec. 2020.
• 18.
  Approach: ● Employ SDNand NFV concepts to mitigate Multicast ABR problems ● SDN: to set up and optimize multicast paths ● VRPs (Virtual Proxies): to aggregate clients’ requests ● VTFs (Virtual Transcoders): to transcode segments to quality levels requested by clients ● MILP optimization model to jointly construct optimal multicast tree and VTFs placement ● Heuristic algorithm (polynomial time) On Optimizing Resource Utilization in Live Video Streaming 18 18 Alireza Erfanian, Farzad Tashtarian, Reza Farahani, Christian Timmerer, and Hermann Hellwagner. ”On Optimizing Resource Utilization in AVC-based Real-time Video Streaming”. IEEE Conf. on Network Softwarization (NetSoft'20). June/July 2020.
• 19.
  Example: MC-ABR OSCAROSCAR (VTFs closer to edge) On Optimizing Resource Utilization in Live Video Streaming 19 19 37 Mbps 37 Mbps
• 20.
  Sample results: On OptimizingResource Utilization in Live Video Streaming 20 20 Bandwidth/#OFcommands(norm.) Comparison of proposed algorithm (heuristic) and other approaches in terms of bandwidth consumption and # OpenFlow commands generated for different network sizes Comparison of proposed algorithm and other approaches in terms of bandwidth consumption and # OpenFlow commands generated for different homogeneity levels of requests (small-scale network) Request sets (RS): homog. LQ / HQ heterogeneous
• 21.
  Player Adaptation Logicand Quality of Experience ● Bitrate adaptation schemes Client-based, server-based, network-assisted, hybrid, ML-based ● Application/transport layer enhancements HTTP/2 (TCP) and HTTP/3 (QUIC), proprietary formats (SRT, RIST, …), WebRTC, low-latency/delay ● Client playback improvements User-aware playback, content-enhancement filters, super-resolution ● Quality of Experience Objective and subjective quality assessment, models, analytics Content Consumption and End-to-End Aspects 21 21
• 22.
  Bitrate Adaptation Schemes Bitrate Adaptation Schemes Client-based Adaptation Bandwidth -based Buffer- based Mixed adaptation Proprietary solutions MDP-based Server-based Adaptation Network- assisted Adaptation Hybrid Adaptation SDN-based Serverand network- assisted A. Bentaleb, B. Taani, A.C. Begen, C. Timmerer and R. Zimmermann, "A Survey on Bitrate Adaptation Schemes for Streaming Media Over HTTP," IEEE Communications Surveys & Tutorials, vol. 21, no. 1, pp. 562-585, First Quarter 2019. 22 22 Adaptive bitrate (ABR) algorithms, adaptation logics, ...
• 23.
  Adaptive bitrate (ABR)algorithms choose the lowest-quality segments in the startup phase; quality switches due to throughput fluctuations. H2BR: complementary to existing ABR utilizing HTTP/2 features ● Server push: piggyback retrans. segments ● Stream priority: for concurrent streams ● Stream termination: for retrans. segments HTTP/2-Based Retransmission (H2BR) 23 23 Minh Nguyen, Christian Timmerer, and Hermann Hellwagner. “H2BR: An HTTP/2-based retransmission technique to improve the QoE of adaptive video streaming”. 25th ACM Workshop on Packet Video (PV'20), June 2020.
• 24.
  Client/server architecture withtwo computers ● HTTP/2 server and HTTP/2 client (both based on nghttp2) ● Dummynet emulates a state-of-the-art mobile network trace Video content ● Big Buck Bunny: 596 seconds ● Segment duration: 1s, 2s, 4s, 6s ● Quality: 20 versions ● Resolutions: 320x240, 480x360, 854x480, 1280x720, 1920x1080 Compared method ● SQUAD1) H2BR Evaluation Setup 24 24 Dummynet Throughput-based AGG Buffer-based BBA Hybrid SARA Last throughput 1) Cong Wang, Divyashri Bhat, Amr Rizk, Michael Zink.. “Design and Analysis of QoE-Aware Quality Adaptation for DASH: A Spectrum-Based Approach”. ACM Trans. Multimedia Comput. Commun. Appl. 13, 3s, Article 45, August 2017.
• 25.
  H2BR Experimental Results 25 25 OverallQoE score based on the ITU-T P.1203 QoE model mode 0 CDF of video quality in an experimental run (segment duration = 4s) H2BR can decrease lowest-quality playback by up to more than 70% QoE increase by up to 13%
• 26.
  Quality of Experience(QoE) ... ● “... is the degree of delight or annoyance of the user of an application or service. It results from the fulfillment of his or her expectations with respect to the utility and / or enjoyment of the application or service in the light of the user’s personality and current state.”1) ● … can be easily extended to various domains, e.g., immersive media experiences.2) 26 26 1) P. Le Callet, S. Möller, A. Perkis, et al. QUALINET White Paper on Definitions of Quality of Experience. European Network on Quality of Experience in Multimedia Systems and Services (COST Action IC 1003). 2012. 2) A. Perkis, C. Timmerer, et al. 2020. QUALINET White Paper on Definitions of Immersive Media Experience (IMEx). arXiv:2007.07032 [cs.MM] 3) Jeroen van der Hooft, Tim Wauters, Filip De Turck, Christian Timmerer, and Hermann Hellwagner. “Towards 6DoF HTTP Adaptive Streaming Through Point Cloud Compression”. 27th ACM Int’l. Conf. on Multimedia (MM'19). Oct. 2019. 3)
• 27.
  QoE Evaluation forAdaptive Point Cloud Streaming Volumetric media delivery with six degrees of freedom (6DoF) experience, but significant bandwidth consumption 27 27 Jeroen van der Hooft, Maria Torres Vega, Filip De Turck, Christian Timmerer, Raimund Schatz, and Ali C. Begen. “Subjective and Objective QoE Evaluation for Adaptive Point Cloud Streaming”. 12th Int’l. Conf. on Multimedia Quality of Experience (QoMEX’20). May 2020. Slides courtesy of Jeroen van der Hooft (adapted) 4.1 Gbps 3.8 Gbps 5.7 Gbps 5.6 Gbps
• 28.
  QoE Evaluation forAdaptive Point Cloud Streaming Volumetric media scene 28 28
• 29.
  QoE Evaluation forAdaptive Point Cloud Streaming Point cloud compression (PCC, with MPEG reference encoder) 29 29 5.7 Gbps 40.4 Mbps 4.5 Mbps
• 30.
  Research questions: 1. Whatis the impact of network and content characteristics on the perceived video quality? 2. How do objective metrics correlate with subjective ratings for perceived video quality? Methodology: ● 3 different video sequences were created ● Each one in 8 different configurations: ● Participants were asked to rate video quality ● Objective metrics were computed QoE Evaluation for Adaptive Point Cloud Streaming 30 30 Bandwidth [Mbps] Allocation Prediction 20 Visible objects 0 60 Visible objects 0 100 Visible objects 0 20 Visible objects 1 60 Visible objects 1 100 Visible objects 1 60 All objects 0 ∞ N/A N/A
• 31.
  Sample results (forRQ 1): QoE Evaluation for Adaptive Point Cloud Streaming 31 31 Subjects can distinguish between different bitrates Viewport prediction allows to improve the observed quality
• 32.
  Sample results (forRQ 2): QoE Evaluation for Adaptive Point Cloud Streaming 32 32 Clear correlation between objective metrics and MOS scores Subjective scores match best with SSIM
• 33.
  Content Delivery ● Edgecomputing support ● CDNs, SDN, NVF, … ⇿ clients ● Low-latency live streaming HTTP Adaptive Streaming – Where Is It Heading? 33 33 Content Consumption & E2E ● Client-side content improvement, SR ● Machine learning ● Appl. & transport layer enhancements https://athena.itec.aau.at/ Content Provisioning ● Content and context awareness ● Multi-codec (AVC, HEVC, VVC, VP9, AV1) ● Machine learning Quality of Experience ● Immersive content ● User-aware playback ● Better/new QoE models and analytics
• 34.
  ATHENA team Hadi Amirpour,Jesús Aguilar Armijo, Ekrem Çetinkaya, Alireza Erfanian, Reza Farahani, Mohammad Ghanbari, David Langmeier, Vignesh V Menon, Minh Nguyen, Babak Taraghi, Farzad Tashtarian; Hermann Hellwagner, Christian Timmerer (Inter-)National Collaborators ● Prof. Ali C. Begen, Ozyegin University, Turkey ● Prof. Filip De Turck, Ghent University – imec, Belgium ● Dr. Jeroen van der Hooft, Ghent University – imec, Belgium ● Prof. Mohamed-Chaker Larabi, University of Poitiers, France ● Dr. Raimund Schatz, AIT Austrian Institute of Technology, Austria ● Prof. Roger Zimmermann, NUS, Singapore ● Dr. Abdelhak Bentaleb, NUS, Singapore Acknowledgments 34 34 https://athena.itec.aau.at/
• 35.
  Thank you foryour attention 35