Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems


Published on

Streaming audio and video content currently accounts for the majority of the internet traffic and is typically deployed over the top of the existing infrastructure. We are facing the challenge of a plethora of media players and adaptation algorithms showing different behavior but lack a common framework for both objective and subjective evaluation of such systems. This paper aims to close this gap by (i) proposing such a framework, (ii) describing its architecture, (iii) providing an example evaluation, (iv) and discussing open issues.

Published in: Technology
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website!
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems

  1. 1. Automated Objective and Subjective Evaluation of HTTP Adaptive Streaming Systems Priv.-Doz. Dr. Christian Timmerer Alpen-Adria-Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Information Technology (ITEC) w Multimedia Communication (MMC) w Sensory Experience Lab (SELab) w w w Chief Innovation Officer (CIO) at bitmovin GmbH w 1st IEEE Multimedia Information Processing and Retrieval (MIPR) 2018, April 12, 2018
  2. 2. Outline • Introduction • Framework • Example Results • Discussion and Challenges April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 2
  3. 3. Multimedia Traffic on the Internet • Real-time entertainment: Streaming video and audio; >70% of Internet traffic at peak periods • Forecast: Visual Networking Index (VNI) 2016-2021 – IP video traffic will be 82% of all consumer Internet traffic by 2021 (up from 73% in 2016); will grow threefold from 2016 to 2021 – Live Internet video will account for 13% of Internet video traffic by 2021; will grow 15-fold from 2016 to 2021 • Popular services – YouTube (17.53%), Netflix (35.15%), Amazon Video (4.26%), Hulu (2.68%); all delivered over-the-top (OTT) • More people now subscribe to Netflix (50.85M) than cable TV (48.61M) in the US (Q1 2017) April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 3 Global Internet Phenomena Report: 2016
  4. 4. How does it work? In a nutshell… April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 4 Adaptation logic is within the client, not normatively specified by the standard, subject to research and development
  5. 5. Motivation and Features • Plethora of (commercial) media players and adaptation algorithms showing different behavior • We lack a common framework for both objective and subjective evaluation • Overview of features – End-to-end HAS evaluation of players deployed in industry and algorithms proposed in academia – Collection and analysis of objective streaming performance metrics – Subjective quality assessment utilizing crowdsourcing for QoE evaluation of HAS systems and QoE model testing/verification April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 5
  6. 6. AdViSE • Adaptive Media Content [DASH, HLS, CMAF] • Players/Algorithms • Network Parameters Impaired Media Sequences Generate Impaired Media Sequences Templates [Startup Delay, Stalling, …] WESP QoE Evaluation Parameters [Questionnaire, Methodology, Crowdsourcing Platform, …] QoS/QoE Metrics Subjective Results + Other Data Reports Analysis AdViSE: Anatoliy Zabrovskiy, Evgeny Kuzmin, Evgeny Petrov, Christian Timmerer, and Christopher Mueller. 2017. AdViSE: Adaptive Video Streaming Evaluation Framework for the Automated Testing of Media Players. In Proceedings of the 8th ACM on Multimedia Systems Conference (MMSys'17). ACM, New York, NY, USA, 217- 220. DOI: WESP: Benjamin Rainer, Markus Waltl, Christian Timmerer, A Web based Subjective Evaluation Platform, In Proceedings of the 5th International Workshop on Quality of Multimedia Experience (QoMEX'13) (Christian Timmerer, Patrick Le Callet, Martin Varela, Stefan Winkler, Tiago H Falk, eds.), IEEE, Los Alamitos, CA, USA, pp. 24-25, 2013. System Architecture ① ② ③④ ⑤ Log of Segment Requests April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 6
  7. 7. AdViSE • Scalable, end-to-end HAS evaluation through emulation w/ a plethora of – content configurations – players/algorithms (including for player competition) – network parameters/traces • Real content and network settings with real dynamic, adaptive streaming including rendering! • Collection of various metrics from players: API or directly from the algorithms/HTML5 • Derived metrics and utilize QoE models proposed in the literature • Segment request log to generate impaired media sequence as perceived by end users for subjective quality testing AdViSE: Adaptive Video Streaming Evaluation April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 7
  8. 8. WESP: Web-Based Subjective Evaluation Platform • Subjective quality assessments (SQA) are vital tool, reliable results, but cost-intensive • Utilize crowdsourcing to reduce costs • WESP – Enables easy and simple configuration of SQAs including possible integration of third-party tools for online surveys – Provides means to conduct SQAs using the existing crowdsourcing platforms taking into account best practice – Allows for the analysis of the results April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 8
  9. 9. Example Evaluation Results • Test sequence encoded 15 different representation (Amazon Prime configuration: 400x224@100Kbps – 1920x1080@15Mbps) with 4s segment length • Bandwidth trajectory based on prior work proposed in literature; network delay 70ms April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 9
  10. 10. Discussion Flexibility of the framework – high number of degrees of freedom – design the evaluation carefully • Content assets: content type, codec/coding parameters, bitrate ladder and segment length, formats (HLS, DASH, CMAF) • Network parameters: Bandwidth trajectory (i.e., predefined, network traces), delay, loss • End user device environment: device type, operating system, browser • Streaming performance metrics: average bitrate, startup time, stalls (frequency, duration), quality switches (frequency, amplitude) • Quantitative QoE models • General HAS evaluation setup: live vs. on-demand content, single player vs. multiple players competing for bandwidth • Templates for generating impaired media sequences: how to realize startup delay and stalls • Questionnaire for SQA including control questions (crowdsourcing) • SQA method: single stimulus, double stimulus, pair-wise comparison • Collection of results and further statistical analysis April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 10
  11. 11. Challenges • The reliability of results requires cross-validation – SQAs in controlled laboratory environments • Network emulation is a vital tool but with limitations – CDN, SDN, ICN, 5G • Reproducibility: providing containerized versions of the modules • Connect to large-scale research networks such as PlanetLab, Virtual Internet Routing Lab, or GENI April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 11
  12. 12. April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 12 Bitmovin kicks off NAB show 2018 with a bang! Announcing $30M in Series B funding and exciting new product launches. And yes, we are hiring…
  13. 13. Thank you for your attention April 12, 2018 Dr. Timmerer [AAU/Bitmovin] 13 ... questions, comments, etc. are welcome … Priv.-Doz. Dipl.-Ing. Dr. Christian Timmerer Associate Professor Alpen-Adria-Universität Klagenfurt, Department of Information Technology (ITEC) Universitätsstrasse 65-67, A-9020 Klagenfurt, AUSTRIA Tel: +43/463/2700 3621 Fax: +43/463/2700 3699 © Copyright: Christian Timmerer