With the recent surge in Internet multimedia traffic, the enhancement and improvement of media players, specifically Dynamic Adaptive Streaming over HTTP (DASH) media players happened at an incredible rate. DASH Media players take advantage of adapting a media stream to the network fluctuations by continuously monitoring the network and making decisions in near real-time. The performance of algorithms that are in charge of making such decisions was often difficult to be evaluated and objectively assessed from an End-to-end or holistic perspective [1]. CAdViSE provides a Cloud-based Adaptive Video Streaming Evaluation framework for the automated testing of adaptive media players [4]. We will introduce the CAdViSE framework, its application, and propose the benefits and advantages that it can bring to every web-based media player development pipeline. To demonstrate the power of CAdViSE in evaluating Adaptive Bitrate (ABR) algorithms we will exhibit its capabilities when combined with objective Quality of Experience (QoE) models. Our team at Bitmovin Inc. and ATHENA laboratory has selected the ITU-T P.1203 (mode 1) quality evaluation model in order to assess the experiments and calculate the Mean Opinion Score (MOS), and better understand the behavior of a set of well-known ABR algorithms in a real-life setting [2]. We will display how we tested and deployed our framework using a modular architecture into a cloud infrastructure. This method yields a massive growth to the number of concurrent experiments and the number of media players that can be evaluated and compared at the same time, thus enabling maximum potential scalability. In our team's most recent experiments, we used Amazon Web Services (AWS) for demonstration purposes. Another awesome feature of CAdViSE that will be discussed here is the ability to shape the test network with endless network profiles. To do so, we used a fluctuation network profile and a real LTE network trace based on the recorded internet usage of a bicycle commuter in Belgium. CAdViSE produces comprehensive logs for each experimental session. These logs can then be applied against different goals, such as objective evaluation or to stitch back media segments and conduct subjective evaluations. In addition, startup delays, stall events, and other media streaming defects can be imitated exactly as they happened during the experimental streaming sessions [3].

1. CAdViSE or how to find the Sweet Spots of ABR Systems
Babak Taraghi*, Abdelhak Bentaleb^, Christian Timmerer*, Roger Zimmermann^, and Hermann Hellwagner*
*Christian Doppler Laboratory ATHENA, Institute of Information Technology (ITEC), Alpen-Adria-Universität Klagenfurt
^Department of Computer Science, School of Computing (SoC), National University of Singapore
WHAT IS CAdViSE
Ø CAdViSE is a Cloud-based Adaptive Video Streaming Evaluation
framework for the automated testing of adaptive media players. We
will introduce the CAdViSE framework, its application, and propose
the benefits and advantages that it can bring to every web-based
media player development pipeline.
Ø To demonstrate the power of CAdViSE in evaluating Adaptive Bitrate
(ABR) algorithms we will exhibit its capabilities when combined with
objective Quality of Experience (QoE) models. Our team at Bitmovin
Inc. and ATHENA laboratory has selected the ITU-T P.1203 (mode 1)
quality evaluation model in order to assess the experiments and
calculate the Mean Opinion Score (MOS), and better understand the
behavior of a set of well-known ABR algorithms in a real-life setting.
Ø In our team’s most recent experiments, we used Amazon Web
Services (AWS) for demonstration purposes. Another awesome
feature of CAdViSE that will be discussed here is the ability to shape
the test network with endless network profiles. To do so, we used a
fluctuation network profile and a real LTE network trace based on the
recorded internet usage of a bicycle commuter in Belgium.
Ø CAdViSE produces comprehensive logs for each experimental
session. These logs can then be applied against different goals, such
as objective evaluation or to stitch back media segments and conduct
subjective evaluations. In addition, startup delays, stall events, and
other media streaming defects can be imitated exactly as they
happened during the experimental streaming sessions.
ACKNOWLEDGMENTS
The financial support of the Austrian Federal Ministry for Digital and
Economic Affairs, the National Foundation for Research, Technology and
Development, and the Christian Doppler Research Association, is
gratefully acknowledged. Christian Doppler Laboratory ATHENA:
https://athena.itec.aau.at/
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
<
0
.
0
5
1
<
0
.
1
0
1
<
0
.
1
5
1
<
0
.
2
0
1
<
0
.
2
5
1
<
0
.
3
0
1
<
0
.
3
5
1
<
0
.
4
0
1
<
0
.
4
5
1
<
0
.
5
0
1
<
0
.
5
5
1
<
0
.
6
0
1
<
0
.
6
5
1
<
0
.
7
0
1
<
0
.
7
5
1
<
0
.
8
0
1
<
0
.
8
5
1
<
0
.
9
0
1
<
0
.
9
5
1
<
1
.
0
0
1
Stall Event Duration (Second)
Noticed Missed
0
1
2
3
4
5
6
7
8
9
10
11
12
1
5
9
13
17
21
25
29
33
37
41
45
49
53
57
61
65
69
73
77
81
85
89
93
97
101
Number
of
Times
Stall
Event
Exposed
Stall Event Duration (Millisecond)
Missed Stall Events
Log. (Missed Stall Events)
ARCHITECTURE
Ø We tested and deployed our framework using a modular architecture
into a cloud infrastructure. This method yields a massive growth to the
number of concurrent experiments and the number of media players
that can be evaluated and compared at the same time, thus enabling
maximum potential scalability.
FastMPC Elastic BBA0 Quetra BOLA dash.js Shaka
Fluctuation 73.23 5.85 7.95 10.88 28.46 41.40 52.25
Ramp Down 30.63 8.35 6.18 10.33 11.29 21.29 34.90
Ramp Up 17.18 0.00 0.19 0.00 4.13 4.55 13.39
Stable 12.84 0.16 0.00 0.00 4.20 4.26 20.12
0
15
30
45
60
75
90
AVG.
STALL
(SECOND)
FastMPC Elastic BBA0 Quetra BOLA dash.js Shaka
Fluctuation 5.48 5.36 5.48 5.36 5.56 5.50 5.28
Ramp Down 5.56 5.29 5.41 5.43 5.57 5.54 5.40
Ramp Up 7.22 6.37 6.56 6.78 7.48 7.51 9.65
Stable 5.65 5.46 5.40 5.42 5.62 5.65 5.65
0
2
4
6
8
10
12
AVG.
STARTUP
(SECOND)
2.56
2.67 2.63
2.26
2.84
2.26
2.79
3.62 3.73 3.65
3.45
3.68
3.41
3.73
1.00
1.50
2.00
2.50
3.00
3.50
4.00
4.50
5.00
BBA0 BOLA dash.js Elastic FastMPC Quetra Shaka
Pearson's Correlation Coefficient 0.94
Objective MOS Subjective MOS
Figure 3: CAdViSE Architecture,
its Modules and the AWS Cloud
Deployment
Figure 4: QoE Models
Comparison with Obtained
Subjective Evaluation Results
Figure 1: Subjective Minimum Noticeable Stall event Duration (MNSD)
Evaluation Results. (left) The number of missed stall events: For stall events
with less than 4ms duration, all subjects miss the stall event, but with the
increasing stall duration this number is reduced.
Figure 2: Average stall duration (left) and average video startup delay (right) in comparison for seven well-known Adaptive Bitrate (ABR) algorithms.
Figure 5: Average QoE score of ABRs in Ramp Up network profile.
Scan me!
REFERENCES
[1] Babak Taraghi. 2021. End-to-End Quality of Experience Evaluation for HTTP Adaptive Streaming. In Proceedings of the 29th ACM International Conference on Multimedia (Chengdu, China) (MM ’21). Association for
Computing Machinery, New York, NY, USA, 2936–2939. https://doi.org/10.1145/3474085.3481025
[2] Babak Taraghi, Abdelhak Bentaleb, Christian Timmerer, Roger Zimmermann, and Hermann Hellwagner. 2021. Understanding Quality of Experience of Heuristic- Based HTTP Adaptive Bitrate Algorithms. In Proceedings
of the 12th ACM Mul- timedia Systems Conference (Istanbul, Turkey) (NOSSDAV ’21). Association for Computing Machinery, New York, NY, USA, 82–89. https://doi.org/10.1145/ 3458306.3458875
[3] Babak Taraghi, Minh Nguyen, Hadi Amirpour, and Christian Timmerer. 2021. Intense: In-Depth Studies on Stall Events and Quality Switches and Their Impact on the Quality of Experience in HTTP Adaptive Streaming.
IEEE Access 9 (2021), 118087–118098. https://doi.org/10.1109/ACCESS.2021.3107619
[4] Babak Taraghi, Anatoliy Zabrovskiy, Christian Timmerer, and Hermann Hell- wagner. 2020. CAdViSE: Cloud-Based Adaptive Video Streaming Evaluation Framework for the Automated Testing of Media Players. In
Proceedings of the 11th ACM Multimedia Systems Conference (Istanbul, Turkey) (MMSys ’20). As- sociation for Computing Machinery, New York, NY, USA, 349–352. https://doi.org/10.1145/3339825.3393581