Ultra-High-Definition Quality of Experience with MPEG-DASH
1. Ultra-High-Definition Quality of
Experience with MPEG-DASH
Priv.-Doz. Dr. Christian Timmerer
Daniel Weinberger, Christopher Mueller and Stefan Lederer
Chief Innovation Officer (CIO) at bitmovin GmbH
http://www.bitmovin.com christian.timmerer@bitmovin.com
Alpen-Adria-Universität Klagenfurt (AAU) Faculty of Technical Sciences (TEWI) Department of Information
Technology (ITEC) Multimedia Communication (MMC) Sensory Experience Lab (SELab)
http://blog.timmerer.com http://selab.itec.aau.at/ http://dash.itec.aau.at christian.timmerer@itec.aau.at
http://www.slideshare.net/christian.timmerer
2. Outline
• Introduction
• Quality, Quality of Experience, and DASH
• Evaluation strategies
• Results
• Conclusions
• Acknowledgment [some slides]: Ali C. Begen, CISCO
April 16, 2015 NAB2015 - BEC - QoE-DASH 2
3. Introduction
• Real-time entertainment
– Streaming video and audio
– > 60% of Internet traffic
• All delivered over-the-top (OTT)
• MPEG Dynamic Adaptive
Streaming over HTTP (DASH)
– Coding format agnostic
• DASH Industry Forum
– Interoperability Points (IOPs) for
common codecs and others (v3.0)
– E.g., AVC/H.264, HEVC/H.265, and
AAC
April 16, 2015 NAB2015 - BEC - QoE-DASH 3
4. Over-The-Top – Adaptive Media Streaming
April 16, 2015 NAB2015 - BEC - QoE-DASH 4
Adaptation logic is within the
client, not normatively specified
by the standard, subject to
research and development
5. Open Digital Media Value Chain
April 16, 2015 NAB2015 - BEC - QoE-DASH 5
Create
Content
Aggregate
Monetize
Distribute
Content
Consume
Content
Any Content Any Storefront Any Network Any Device
CDNsMedia
Protocols
Internet
Transport
DRM
Encoding
Encapsulation
Dynamic
Ads
Clients
6. Simplified Example Workflow: bitcodin/bitdash
April 16, 2015 NAB2015 - BEC - QoE-DASH 6
Source: http://www.bitmovin.net/bitcodin-cloud-based-transcoding-streaming-platform/
7. Internet TV vs. Traditional TV in 2010
• Areas most important to
overall TV experience are
– Content
– Timing control
– Quality
– Ease of use
• While traditional TV surpasses
Internet TV only in quality, it
delivers better “overall
experience”
April 16, 2015 NAB2015 - BEC - QoE-DASH 7
When comparing traditional and Internet TV,
which option is better?
Traditional Internet
Content 7% 79%
Timing / Control 7% 83%
Quality 80% 16%
Ease of Use 23% 52%
Control (FF, etc.) 9% 77%
Portability 4% 92%
Interactivity 31% 52%
Sharing 33% 56%
Overall Experience 53% 33%
Source: Cisco IBSG Youth Survey, Cisco IBSG Youth Focus Group Sessions, 2010
8. Quality (of Experience)
• QoE as evolution of QoS [ITU-T
P.10/G.100]
• QoS: totality of characteristics
of a telecommunications
service that bear on its ability
to satisfy stated and implied
needs of the user of the service
• QoE: the overall acceptability
of an application or service, as
perceived subjectively by the
end-user
April 16, 2015 NAB2015 - BEC - QoE-DASH 8
Many definitions but in general, it’s like an elephant
9. Quality of Experience
• COST Action IC1003 – QUALINET (http://www.qualinet.eu/)
“the degree of delight or annoyance of the user of an application or service. It results
from the fulfillment of his or her expectations with respect to the utility and/or
enjoyment of the application or service in the light of the user’s personality and
current state”
• QoE influence factors
– Any characteristic of a user, system, service, application, or context
– Grouped into human, system, and context
• QoE features
– Perceivable, recognized and namable characteristic of the individual’s experience
– Depends on the level of direct perception, interaction, the usage situation
April 16, 2015 NAB2015 - BEC - QoE-DASH 9
10. QoE for DASH
• Different application domains have different QoE
requirements
– Need to provide specializations of the general QoE definition
– Take into account requirements formulated by means of
influence factors and features of QoE
• QoE influence factors for DASH
– Initial/start-up delay (low)
– Buffer underruns, stalls, freezes (zero)
– Quality switches (low)
– Media throughput (high)
– …
April 16, 2015 NAB2015 - BEC - QoE-DASH 10
11. QoE Evaluation for DASH-based Services
• Test sequence
– Many datasets available
– Adopted Big Buck Bunny & DASHed it with bitcodin
• Players
– bitdash
– …and compare it with ten different adaptation algorithms
• Objective evaluation
– Test setup
– Predefined bandwidth trajectory (or real network traces)
• Subjective evaluation
– Lab vs. crowdsourcing
April 16, 2015 NAB2015 - BEC - QoE-DASH 11
http://www.bitcodin.com/
http://www.dash-player.com/
15. Subjective Evaluation
• Microworker platform
– Limited to Europe, USA/Canada, India
• DASH clients
– DASH-JS (dash.itec.aau.at)
– dash.js (DASH-IF)
– YouTube
• Tears of Steal trailer according to YouTube
configuration
• Screening techniques
– Browser fingerprinting
– Presentation time
– QoE ratings and Pre-Questionnaire
April 16, 2015 NAB2015 - BEC - QoE-DASH 15
16. What about 4K and 8K?
April 16, 2015 NAB2015 - BEC - QoE-DASH 16
• Why? – because we can!
• Supported on the Web
– HTML5, MSE
– AVC/H.264
– [HEVC/H.265 needed
to lower bitrate]
• See demo @
http://www.dash-
player.com/
• UHD-QoE evaluation
17. Conclusions
• QoE for DASH-based services (a rule of thumb)
– Startup delay (low [but live vs. on-demand & short vs. long-tail
content])
– Buffer underrun / stalls (zero)
– Quality switches (low) and media throughput (high)
– Energy- and cost-awareness (data plan)
• No general applicable QoE model for DASH
– (Too) many factors influencing / features of QoE for DASH-based
services
– Methodology for reproducible research is in place and well established
– Ample research opportunities
April 16, 2015 NAB2015 - BEC - QoE-DASH 17
Main QoE
factors for DASH