Quality of Experience (QoE) and QoE models are of an increasing importance to networked systems. The traditional QoE modeling for video streaming applications builds a one-size-fits-all QoE model that underserves atypical viewers who perceive QoE differently. To address the problem of atypical viewers, this paper proposes iQoE (individualized QoE), a method that employs explicit, expressible, and actionable feedback from a viewer to construct a personalized QoE model for this viewer. The iterative iQoE design exercises active learning and combines a novel sampler with a modeler. The chief emphasis of our paper is on making iQoE sample-efficient and accurate. By leveraging the Microworkers crowdsourcing platform, we conduct studies with 120 subjects who provide 14,400 individual scores. According to the subjective studies, a session of about 22 minutes empowers a viewer to construct a personalized QoE model that, compared to the best of the 10 baseline models, delivers the average accuracy improvement of at least 42% for all viewers and at least 85% for the atypical viewers. The large-scale simulations based on a new technique of synthetic profiling expand the evaluation scope by exploring iQoE design choices, parameter sensitivity, and generalizability.

1. Leonardo Peroni1,2 Sergey Gorinsky1 Farzad Tashtarian 3 Christian Timmerer 3
Empowerment of Atypical Viewers
via Low-Effort Personalized Modeling
of Video Streaming Quality
1IMDEA Networks Institute
2Universidad Carlos III de Madrid
Spain
1
3Alpen-Adria Universität Klagenfurt
Austria
CoNEXT, Paris, France, 7 December 2023

2. Adaptive Bitrate (ABR) Video Streaming
L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Video chunks
Internet Client User
Server
3
3
3
2
2
2
1
1
1
4
4
4
Image
quality
Variable bandwidth
4 3 2 1 4 3 2 1
Stall
2
Background Motivation Design Evaluation Conclusion

3. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
 Satisfaction with the experience as perceived by the user
 Directly measurable through subjective assessment
 Experience characterized by objective influence factors (IFs)
• Stall duration
• Image quality
• …
4 3 2 1
Stall
Experience User Slider
Quality of Experience (QoE)
3
Background Motivation Design Evaluation Conclusion

4. experience
L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
score
score
score
MOS
influence factors
raters
1
𝑛
1
𝑛
.
4 3 2 1
Stall
Subjective Test with a Group of Raters
4
Background Motivation Design Evaluation Conclusion
 Mean opinion score (MOS)
• opinion of the average rater

5. score
modeler
score
score MOSes
experiences
sampler
experiences
experience MOS
influence factors
raters
1
𝑛
1
𝑛
. QoE model
…
L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Traditional QoE Modeling
5
Background Motivation Design Evaluation Conclusion

6. score
modeler
score
score MOSes
experiences
sampler
experiences
experience MOS
influence factors
raters
1
𝑛
1
𝑛
.
model
viewer regular streaming
QoE-based ABR streaming system
viewer regular streaming
traditional QoE modeling
QoE model
…
L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Usage of the QoE Model by ABR Streaming
6
Background Motivation Design Evaluation Conclusion

7. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
 Atypical: 10% furthest from the median QoE perception
The one-size-fits-all QoE model is inaccurate for atypical viewers
Is the QoE Model Accurate for Everyone?
7
Background Motivation Design Evaluation Conclusion
Atypical raters
Average rater
All raters
(Waterloo-IV dataset)

8. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Can the personalization be done with low effort for the viewer?
Our Approach
8
Background Motivation Design Evaluation Conclusion
 The viewer is the sole rater for the personalized model
 Personalization of QoE models
 Potential options
• Indirect feedback
• Multiple reference groups
• Tuning of a generic QoE model
 Feedback requirements
• From this viewer
• Explicit
• Expressible
• Actionable

9. viewer
experiences
experience
RIGS sampler
scores
score
models and scores influence factors
model
XSVR modeler
iQoE
experiences
…
L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
 Limiting the subjective tests to a small number of experiences
 New sampler and modeler in active learning
• Random Improved Greedy Sampling (RIGS)
• eXtended Support Vector Regression (XSVR)
Individualized QoE (iQoE)
9
Background Motivation Design Evaluation Conclusion

10. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
 Input space: IF values of experiences
 Output space: 1-20 (bad) 21-40 (poor) 41-60 (fair)
61-80 (good) 81-100 (excellent)
315 experiences
(Waterloo-IV dataset)
How to Sample the Experience Set Representatively?
10
Background Motivation Design Evaluation Conclusion

11. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Quick but with poor coverage of the input space
Random Sampling (RS)
11
Background Motivation Design Evaluation Conclusion
 Random selection of 50 out of the 315 experiences
1-20 (bad) 21-40 (poor) 41-60 (fair) 61-80 (good) 81-100(excellent)
All RS

12. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
 GS accounts for Euclidean distances in the space of IF values
Poor coverage of the output space
Greedy Sampling (GS)
12
Background Motivation Design Evaluation Conclusion
RS GS
1-20 (bad) 21-40 (poor) 41-60 (fair) 61-80 (good) 81-100(excellent)

13. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Highest complexity with initally small gain
Improved Greedy Sampling (IGS)
13
Background Motivation Design Evaluation Conclusion
 IGS accounts for Euclidean distances in both spaces of IF values and scores
GS IGS
1-20 (bad) 21-40 (poor) 41-60 (fair) 61-80 (good) 81-100(excellent)

14. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Our RIGS Sampler
14
Background Motivation Design Evaluation Conclusion
 RS for the first h (10) experiences and IGS for the experiences h+1 to H (50)
RIGS
1-20 (bad) 21-40 (poor) 41-60 (fair) 61-80 (good) 81-100(excellent)

15.  Set of 10 influence factors as features
• Representation identifier
• Stall duration
• Bitrate
• Chunk size
• Frame width
• Frame height
• Highest bitrate flag
• Peak Signal-to-Noise Ratio (PSNR)
• Structural Similarity Index Measure (SSIM)
• Video Multi-method Assessment Fusion (VMAF)
L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
 Support Vector Regression (SVR)
Our XSVR Modeler
15
Background Motivation Design Evaluation Conclusion
All are important according
to permutation feature
importance

16. viewer regular streaming
QoE-based ABR streaming system
viewer regular streaming
QoE models
viewer
experiences
experience
RIGS sampler
scores
score
models and scores influence factors
model
XSVR modeler
iQoE
experiences
…
L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE’s Role in ABR Streaming
16
Background Motivation Design Evaluation Conclusion

17. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Browser Interface
 Direct recruitment of 34 viewers
 Extension with Microworkers to a total of 120 viewers
 120 experiences for training and testing
 Playback time of 8 seconds without stalls for each experience
Online Subjective Study with Real Viewers
17
Background Motivation Design Evaluation Conclusion

18. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Average MAE decreases at least 42% and 85% for
all raters and atypical raters, respectively
Prominent existing QoE models
Accuracy of iQoE vs. Traditional QoE Modeling
18
Background Motivation Design Evaluation Conclusion
Better

19. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
 Playback time
 Extra time
7 minutes
22 Minutes
The viewer exerts reasonably low effort
Viewer’s Effort
19
Background Motivation Design Evaluation Conclusion
 Expected effort of the viewer

20. Scores of
32 real raters
8 QoE models
(profiles)
L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
 Extension of real-world experiments to evaluate:
• Design choices
• Parameter sensitivity
• iQoE overhead
 New technique of synthetic profiling
• 1000 experiences assessed by 256 synthetic raters
Simulations
20
Background Motivation Design Evaluation Conclusion
256 synthetic raters

21. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE consistently outperforms design alternatives
iQoE RIGS + Extreme Gradient Boosting
RIGS + Random Forest RIGS + Gaussian Process
Sensitivity to the Modeler Choice
21
Background Motivation Design Evaluation Conclusion
Better

22. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Processing and memory overhead of iQoE is negligible
iQoE Per-Iteration Time
22
Background Motivation Design Evaluation Conclusion
Average Standard deviation
Switch from RS to IGS

23.  iQoE, a new method for personalization of QoE models
• Active learning, RIGS sampler, XSVR modeler
 Results
• Average accuracy improvement
• 42% for all viewers
• 85% for atypical viewers
• Viewer’s effort of 50 assessments with total training time of 22 minutes
L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
 Evaluation
• Online subjective studies with 120 real viewers
• Simulations
Dataset and code available at:
https://github.com/Leo-rojo/iQoE_Dataset_and_Code
Conclusion
23
Background Motivation Design Evaluation Conclusion

24. Backup Slides
24

25. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Inaccuracy of Traditional QoE Modeling
25
Background Motivation Design Evaluation Conclusion

26. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Results of the Subjective Studies
26
Background Motivation Design Evaluation Conclusion

27. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Permutation Feature Importance
27
Background Motivation Design Evaluation Conclusion

28. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE vs. Traditional QoE Modeling
28
Background Motivation Design Evaluation Conclusion

29. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE vs. Traditional QoE Modeling
29
Background Motivation Design Evaluation Conclusion

30. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE vs. Personalized QoE Modeling
30
Background Motivation Design Evaluation Conclusion

31. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE vs. Multiple Reference Groups
31
Background Motivation Design Evaluation Conclusion

32. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Synthetic Raters vs. Real Raters
32
Background Motivation Design Evaluation Conclusion

33. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Sensitivity to the Sampler Choice
33
Background Motivation Design Evaluation Conclusion

34. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Sensitivity to the Modeler Choice
34
Background Motivation Design Evaluation Conclusion
iQoE RIGS + Extreme Gradient Boosting
RIGS + Random Forest RIGS + Gaussian Process

35. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Sensitivity to the Modeler and Sampler Choices
35
Background Motivation Design Evaluation Conclusion

36. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
Sensitivity to the h Parameter and Training-Set Share
36
Background Motivation Design Evaluation Conclusion

37. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE Generalizability
37
Background Motivation Design Evaluation Conclusion

38. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE Dataset
38
Background Motivation Design Evaluation Conclusion

39. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE vs Multiple Reference Groups
39
Background Motivation Design Evaluation Conclusion

40. L. Peroni, S. Gorinsky, F. Tashtarian, and C. Timmerer, “…Low-Effort Personalized Modeling of Video Streaming Quality”
iQoE Processing and Memory Overhead
40
Background Motivation Design Evaluation Conclusion