QoE++: Shifting from Ego- to Eco-System? QoE research has advanced significantly in recent years with a focus on the QoE ego-system. Thereby, different facets have been addressed by the research community like subjective user studies to identify QoE influence factors for particular applications like video streaming, QoE models to capture the effects of those influence factors on concrete applications, QoE monitoring approaches at the end user site but also within the network to assess QoE during service consumption and to provide means for QoE management for improved QoE. However, in order to progress in the area of QoE, new research directions have to be taken. There is a need for QoE++. The application of QoE in practice needs to consider the entire QoE eco-system and the stakeholders along the service delivery chain to the end user. In comparison to the traditional QoE ego-system thinking, the QoE eco-system addresses among others the following research topics: in-session vs. global system perspective, short- vs. long-time scales when considering QoE, single vs. multi-user QoE, single vs. concurrent usage of applications and services, user vs. business perspective by addressing all key stakeholder goals. QoE++ requires (a) to extend current QoE models by the different perspectives of the QoE eco-system including the service provider perspective, (b) to incorporate user behavior as part of the model, (c) and to identify and include relevant internal and external context factors including physical, cultural, social, economic context. QoE++ faces several major challenges. (1) Can we utilize QoE for network & service management? Or is it more appropriate to consider user engagement or user behavior? Which context factors are relevant or are such context-factors even more important for network & service management, e.g. in order to foresee and react on flash crowds? (2) How to realize cross-layer optimization between applications and their demands and the network capabilities, and thus a network-wide QoE optimization? Is SDN the right technology to cope with those challenges? (3) Can we transform QoE into business models, SLAs, etc.? Or is it possible to 'trade' QoE? For example, offering WiFi sharing at home, a user may get improved service delivery and QoE by its ISP. (4) Do we understand fundamental models and natural relationships of QoE++? How can we extend existing QoE models to take into account the service provider's perspective? How to quantify QoE fairness? What is the relationship between QoE and user behavior? Following QoE++ will shift from ego- to eco-systems and give answers to those questions. In this talk, we will discuss QoE++ and highlight some of the challenges above.

1. Prof. Dr. Tobias Hoßfeld
Chair of Modeling of
Adaptive Systems (MAS)
Institute for Computer
Science and Business
Information Systems (ICB)
University of Duisburg-Essen
www.mas.wiwi.uni-due.de
QoE++: Shifting from
Ego- to Eco-System?
IFIP/IEEE QCMan 2015
Ottawa, 11 May 2015

2. 1. Current Status: Managing the QoE Ego-System
2. Some Observations on QoE
3. QoE++: The QoE Eco-System

3. Interest in QoE over the last 10 years
• Number of publications per year when searching for „QoE“
• Academic interest is increasing! Industrial Employment?
mas.wiwi.uni-due.de 3
2 5 8 18
53
89
149
194
298 286
393
0
50
100
150
200
250
300
350
400
450
#Publications/Year
Year
IEEE Xplore (Metadata)
547 513
709 709
1100
1370
1700
2190
2730
3410
3570
0
500
1000
1500
2000
2500
3000
3500
4000
#Publications/Year
Year
Google Scholar

4. Research Communities
mas.wiwi.uni-due.de 4
0 1 2 3 4 5 6
1
2
3
4
5
number of stallings
MOS
crowdsourcing
laboratoryQoE
Multimedia
Encoder
Decoder
0 20 40 60 80 100
0
5
10
15
20
25
30
ratio of buffering events
playtime(min)
Engagement
Application
Control Plane Application
Controller
Network
Control Plane
Data Plane
Application
Networking

5. Current Status: QoE Management
mas.wiwi.uni-due.de 5
• Application level,
end user site
• Within network, …
• Cross-layer
approaches
• Realization, e.g.
SDN, …
• Parametric models
• Machine learning
• …
• Subjective &
objective tests
• Crowdsourcing
• …
Key
Influence
Factors
QoE
Model
QoE
Monitor-
ing
QoE
Manage-
ment

6. Concept of QoE Management
Cloud / DC
Access
Network
Core
Network
Access
Network
Cloud service providerEnd user
Cloud / DC
QoE Management requires
1. QoE Model
2. QoE Monitoring
3. QoE Control
Network provider
mas.wiwi.uni-due.de 6

7. The QoE Ego-System
• Main focus
– in-session
– short-time scale
– single user QoE
– single apps
– user perspective
• Typical research questions
– What are the key QoE influence factors?
– How and where to monitor QoE and its influence factors?
– How to deliver contents and control traffic management?
– How to adapt contents and media to current network situation?
– How to exchange information between network and application to
overcome QoE issues?
mas.wiwi.uni-due.de 7

8. SOME OBSERVATIONS

9. QoE Models: Complexity and Generic Relationships
mas.wiwi.uni-due.de 9
• Model is intended to fulfill a
certain goal
$$$
• Generic relationships need
to be considered, e.g. IQX
𝑄𝑜𝐸 𝑥 = 𝛼 ⋅ 𝑒−𝛽 + 𝛾

10. Subjective Testing
• Subjective Experiments
– Quantifying QoE of improved system
– Challenging: proper test design,
implementation, analysis
– Limited by pool of test subjects
• Crowdsourcing
– Access to large pool of humans
– Challenging: remote conduction
of tests, statistical analysis
mas.wiwi.uni-due.de 10
What is 𝜶?

11. Crowdsourced QoE: Best Practices
Conceptual aspects
Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B.,
Habigt, J., Diepold, K., & Tran-Gia, P. (2014).
Best practices for QoE crowdtesting: QoE
assessment with crowdsourcing. Multimedia,
IEEE Transactions on, 16(2), 541-558.Pseudo reliable crowd
Lab
Tester
Filtering
- Demographics
- Hardware requirements
- Reliability
- …
Training
Phase 1
QoE - Test - Software based screening
mechanisms
- Content questions,
reliability checks
- Incentive design, variable
payments
- …
Post
processing
Phase 2
- Statistical analysis
- …
Practical aspects
Tobias Hoßfeld, Matthias Hirth, Judith Redi,
Filippo Mazza, Pavel Korshunov, et al.. Best
Practices and Recommendations for
Crowdsourced QoE - Lessons learned from the
Qualinet Task Force "Crowdsourcing, 2014.
https://hal.archives-ouvertes.fr/hal-01078761/
mas.wiwi.uni-due.de 11

12. Do we need QoE?
Can we utilize QoE for network & service management?
Is it more appropriate to consider other means?

13. Measurement Studies for HTTP Video Streaming
mas.wiwi.uni-due.de 13
0 1 2 3 4 5 6
1
2
3
4
5
number of stallings
MOS
crowdsourcing
laboratory
QoE
0 20 40 60 80 100
0
5
10
15
20
25
30
ratio of buffering events
playtime(min)
Engagement
Engagement data:
Dobrian, F., Sekar, V., Awan, A., Stoica, I., Joseph,
D., Ganjam, A., Zhan, J. & Zhang, H. (2011).
Understanding the impact of video quality on user
engagement. ACM SIGCOMM Computer
Communication Review, 41(4), 362-373.
System
Model
QoE data:
Hoßfeld, T., Schatz, R., Biersack, E., & Plissonneau,
L. (2013). Internet video delivery in YouTube: from
traffic measurements to quality of experience.
InData Traffic Monitoring and Analysis (pp. 264-
301). Springer Berlin Heidelberg.
Output: stalling
pattern
Input: network
and video
characteristics

14. User Behavior and QoE
• Example: QoE and User Engagement in
HTTP Video Streaming
• Different video buffer
durations 𝑑∗ investigated
• Stakeholder
interested in
watch time, e.g. selling
advertisements
• Strong relationship,
but complementary
approach
mas.wiwi.uni-due.de 14

15. What are proper QoE models?
How can we extend existing QoE models to take into
account the service provider's perspective, individual user
perceptions?

16. Beyond Mean Opinion Scores (MOS)
• MOS is one measure for QoE!
• Confidence intervals
show statistical significance,
but not reliability!
• Reliability metrics quantify how
reliable your data is.
• Standard deviation quantifies
the user diversity.
• Quantiles are of interest
for service providers.
mas.wiwi.uni-due.de 16
Excellent!
Bad!
Fair!Good!
Poor!

Fair = 3

17. Limitations of MOS
• Results from subjective experiments on video QoE
• Service providers
defines a threshold 𝜃
of acceptable quality
• Probability of
dissatisfied users:
𝑃 𝑅 < 𝜃 .
• But: Service provider
wants to satisfy majority of users  e.g. quantiles
mas.wiwi.uni-due.de 17

18. Individual QoE Profiles per User?
mas.wiwi.uni-due.de 18
QoE Model for MOS
System Model
Do we need user profiles?
Do we need usage
scenarios?
Parameterization of
QoE
Impact of
user profile
Consequences for
QoE Management
Parameterized
wrt. user profile Impact of buffer size, video
bitrate, network conditions
Talk later by Christian Moldovan:
To Each According to his Needs: Dimensioning
Video Buffer for Specific User Profiles and Behavior
by C. Moldovan, C. Schwartz, T. Hossfeld
Users more or
less sensitive to
delays and stalling

19. Is context more important than QoE?
Which context factors are relevant or are such context-
factors even more important for network & service
management, e.g. in order to foresee and react on flash
crowds?

20. Example: HTTP Adaptive Streaming with Context
• Use context and context predictors in adaptive streaming strategies
• Predict bandwidth and buffer state based on location, connectivity state
(3G, WiFi, upcoming vertical/horizontal handovers), social (e.g. flash
crowds), mobility (tunnel)
• Include context
information
– for buffering and quality
level selection
strategy
– for caching decisions
mas.wiwi.uni-due.de 20
User performs QoE
management?!

21. QOE++: THE QOE ECO-SYSTEM

22. Transition to QoE Eco-System
• QoE eco-system
– in-session vs. global
– short- vs. long-time scale
– single vs. multi-user QoE
– single vs. concurrent apps
– user vs. business perspective
– all key stakeholder goals
• Requirements
– Extend current QoE models by the
different stakeholder perspectives of the QoE eco-system
– Incorporate user behavior as part of the model
– Identify and include relevant internal and external context factors
including physical, cultural, social, economic context.
Content
ProviderISPs
CDNs
$$$
$$$
$$$
$$$
Ads
Data
analysis
…
mas.wiwi.uni-due.de 22

23. Comprehensive Framework: QoE and User Behavior
mas.wiwi.uni-due.de 23
Reichl, P.; Egger, S.; Möller, S.; Kilkki, K.; Fiedler, M.; Hossfeld, T.; Tsiaras, C.; Asrese, A.:
Towards a comprehensive framework for QoE and user behavior modelling. QoMEX 2015

24. An abstract view
mas.wiwi.uni-due.de 24
Quality of Experience Network Layers
Management
Application /
Service
Network
QoE++
Technical
realization,
e.g. SDNMonitoring
Model
Cross-layer
approach, interaction
of control loops,
economic
traffic
management
Viewpoint
Top down:
theoretical
framework
Methodology
Bottom up:
use-case &
technology
driven
Intermediate
players, e.g.
cloud
……

25. QoE++ Research Directions
• Can we utilize QoE for network & service management?
– User engagement and user behavior
– Context factors
• How to realize QoE management?
– Cross-layer optimization: application demands vs. network capabilities
– SDN as technology path
• Can we transform QoE into business models, SLAs, etc.?
– Or is it possible to 'trade' QoE? For example, offering WiFi sharing at
home, a user may get improved service delivery and QoE by its ISP.
• Do we understand QoE as well as fundamental models and
natural relationships?
– Extend existing QoE models 𝑓 System, User state, Content, Context
– Relationship between QoE and user behavior?
• Theoretical user-centric performance evaluation approaches
mas.wiwi.uni-due.de 25

26. THANKS

27. Additional Pointers (and references therein…) for
HTTP Streaming QoE
Overview on HTTP Adaptive Streaming and HAS QoE. Seufert, M.; Egger, S.; Slanina, M.; Zinner, T.; Ho0feld, T.; Tran-Gia, P., "A
Survey on Quality of Experience of HTTP Adaptive Streaming," Communications Surveys & Tutorials, IEEE , vol.17, no.1, pp.469,492,
2015
doi: 10.1109/COMST.2014.2360940
HTTP Streaming QoE Model: Total Stalling, Stalling frequency. Hoßfeld, T., Schatz, R., Biersack, E., & Plissonneau, L. (2013).
Internet video delivery in YouTube: from traffic measurements to quality of experience. In Data Traffic Monitoring and Analysis (pp. 264-
301). Springer Berlin Heidelberg.
HTTP Streaming model: initial delay, : Total Stalling, Stalling frequency. Tobias Hoßfeld, Christian Moldovan, Christian Schwartz:
To Each According to his Needs: Dimensioning Video Buffer for Specific User Profiles and Behavior. In: QCMAN 2015. Ottawa, Canada
2015.
Time on high layer in HAS: Subjective Study. Hoßfeld, T., Seufert, M., Sieber, C., & Zinner, T. (2014). Assessing Effect Sizes of
Influence Factors Towards a QoE Model for HTTP Adaptive Streaming. In Proceedings of the 6th International Workshop on Quality of
Multimedia Experience (QoMEX 2014), Singapore.
HTTP Adaptive Streaming model: Total Stalling, Stalling frequency and quality adaptation. Hossfeld, Tobias; Skorin-Kapov, Lea;
Haddad, Yoram; Pocta, Peter; Siris, Vasilios A. ;Zgank, Andrej; Melvin, Hugh;: Can context monitoring improve QoE? A case study of
video flash crowds in the Internet of Services. In: QCMAN 2015 - Third IFIP/IEEE International Workshop on Quality of Experience
Centric Management. Ottawa, Canada 2015.
Concrete HAS Implementation. Sieber, C.; Hossfeld, T.; Zinner, T.; Tran-Gia, P.; Timmerer, C., "Implementation and user-centric
comparison of a novel adaptation logic for DASH with SVC," Integrated Network Management (IM 2013), 2013 IFIP/IEEE International
Symposium on , vol., no., pp.1318,1323, 27-31 May 2013
Benchmarking Framework: Optimial HAS QoE. Hoßfeld, T., Seufert, M., Sieber, C., Zinner, T., & Tran-Gia, P. (2015). Identifying QoE
optimal adaptation of HTTP adaptive streaming based on subjective studies. Computer Networks, 81, 320-332.
mas.wiwi.uni-due.de 27

28. Literature References from the Keynote
Conceptual aspects: Crowdsourced QoE. Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia,
P. (2014). Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. Multimedia, IEEE Transactions
on, 16(2), 541-558.
Practical aspects: Crowdsourced QoE. Tobias Hoßfeld, Matthias Hirth, Judith Redi, Filippo Mazza, Pavel Korshunov, et
al.. Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force
"Crowdsourcing, 2014. https://hal.archives-ouvertes.fr/hal-01078761/
HTTP Streaming model: Total Stalling, Stalling frequency. Hoßfeld, T., Schatz, R., Biersack, E., & Plissonneau, L. (2013).
Internet video delivery in YouTube: from traffic measurements to quality of experience. InData Traffic Monitoring and
Analysis (pp. 264-301). Springer Berlin Heidelberg.
Beyond MOS: Quantiles and SOS for Service Providers. Hoßfeld, Tobias; Heegard, Poul; Varela, Martin: QoE beyond the
MOS: Added Value Using Quantiles and Distributions. QoMEX 2015, Costa Navarino, Greece 2015.
QoE and User Behavior Model - Conceptual approach. Reichl, Peter; Egger, Sebastian; Möller, Sebastian; Kilkki, Kalevi;
Fiedler, Markus; Hossfeld, Tobias; Tsiaras, Christos;Asrese, Alemnew: Towards a comprehensive framework for QoE and
user behavior modelling. In: QoMEX 2015. Costa Navarino, Greece 2015.
User profiles and QoE / HTTP Streaming model for initial delay and stalling. Tobias Hoßfeld, Christian Moldovan,
Christian Schwartz: To Each According to his Needs: Dimensioning Video Buffer for Specific User Profiles and Behavior. In:
QCMAN 2015. Ottawa, Canada 2015.
mas.wiwi.uni-due.de 28