The document discusses an automated Docker-based framework for evaluating video streaming quality of experience (QoE) in mobile broadband networks. It encompasses both client and server-side measurements and allows comprehensive monitoring across multiple layers while benchmarking various video players and algorithms. The framework integrates with analytics tools to capture extensive metadata, aiming to enhance video streaming research and applications.

1. Docker-Based Evaluation Framework for
Video Streaming QoE in Broadband Networks
Cise Midoglu1
, Anatoliy Zabrovskiy2
, Özgü Alay3
, Daniel Hölbling-Inzko4
, Carsten
Griwodz5
, Christian Timmerer2,4
1
Simula Research Laboratory, 2
University of Klagenfurt, 3
Simula Metropolitan Center for Digital
Engineering, 4
Bitmovin Inc, 5
University of Oslo
OS-05

2. Motivation
● Video streaming one of the top traffic contributors in the Internet
● Mobile broadband (MBB) networks becoming more and more prominent
● Metadata-rich measurement result
collection is challenging
● Over-the-top (OTT) video analytics
are not used widely in research
OS-05 2

3. Overview
● Automatised measurement and evaluation framework for video streaming
Quality of Experience (QoE)
OS-05 3

4. Overview
● Automatised measurement and evaluation framework for video streaming
Quality of Experience (QoE)
○ End-to-end with client and server side
○ Client-initiated active measurements
○ Docker virtualization → platform compatibility
○ Headless streaming → focus on objective QoE
○ Allows to monitor and collect data from multiple layers (application, transport,
network, physical)
○ Allows to benchmark video players, ABR algorithms, networks, operators, ...
OS-05 4

5. Client Side
● Designed to emulate real user initiating
a video streaming session
● Selenium instrumentation opens
headless browser and streams from
HTTP server
● Implemented as Docker container
● Compatible with mobile broadband
testbeds (e.g., MONROE)
● Configurable (network, player, ABR
algorithm, run duration, batch
experimentation, ...)
● Additional network monitoring (ping +
traceroute), and metadata if available
OS-05 5

6. Server Side
OS-05 6
● Server hosts landing pages with 3
different video players
○ Bitmovin Player v8
○ Shaka Player v2.5
○ DASH.js Player v2.9

7. Server Side
OS-05 7
● All test pages use the same asset
○ BigBuckBunny video
○ Encoded in 15 qualities (100 Kbps to
15 Mbps)
○ Retrieved from CDN

8. Server Side
OS-05 8
● All test pages are integrated with
Bitmovin Analytics web collector v2.3
○ JavaScript executed on client browser
in runtime

9. Server Side
OS-05 9
● All pages are integrated with Bitmovin
Analytics web collector v2.3
○ More than 70 parameters recorded for
each session (including startup time,
video quality, num. switches, stalls, ...)

10. Server Side
OS-05 10
● All pages are integrated with Bitmovin
Analytics web collector v2.3
○ Possible to view metrics in real-time
from the Bitmovin Dashboard, or export
in raw format using the Bitmovin
Analytics Export Service

11. MONROE Integration
OS-05 11

12. MONROE Integration
OS-05 12

13. MONROE Integration
OS-05 13

14. Use Cases and Intended Audience
OS-05 14

15. Demo
OS-05 15

16. 16
• Two grand challenges
– Improving open-source HEVC encoding
– Low-latency live streaming
• Focus areas in 2020
– Machine learning and statistical modeling for video streaming
– Volumetric media: from capture to consumption
– Fake media and tools for preventing illegal broadcasts
• A workshop (posters/demos) dedicated to middle and high-school students
• Two confirmed keynotes from Google and MIT
• Expecting reduced registration fees thanks to strong support
Important Dates Submit by
Research Track Jan. 10 (firm)
Demo Track Feb. 29
Open Source/Dataset Feb. 29
Workshops Mar. 27
Conference June 8-11
NEW
Visit http://acmmmsys.org today!
NEW

17. VBIM: Video QoE Dataset Generation
Docker-Based Evaluation Framework for Video Streaming QoE in Broadband Networks
Cise Midoglu, Anatoliy Zabrovskiy, Özgü Alay,
Daniel Hölbling-Inzko, Carsten Griwodz, Christian Timmerer
Simula Research Laboratory, University of Klagenfurt,
University of Oslo, Bitmovin Inc
Open Source Software Competition
Thursday 24 October
10:30 - 12:00
Risso 6