Can networks deliver quality of experience?

1,870 views

Published on

Video is by far the predominant consumer of network capacity. Yet, the Internet is a ‘video repellent’ machine, one that can transfer data but has no notion of deadlines. So what are we getting from modern video services? How can we measure quality of experience? And can we predict the quality perceived by the user, starting from simple network measurements? In this talk I give a critical perspective on conventional QoE assessment, ending up with a controversial proposition.
This talk is based on the following material:

V. Menkovski, A. Liotta
'Adaptive Psychometric Scaling for Video Quality Assessment'
Journal of Signal Processing: Image Communication (Elsevier, 2012)
http://bit.ly/JSP-2012

V. Menkovski, G. Exarchakos, A. Liotta
'The Value of Relative Quality in Video Delivery'
Journal of Mobile Multimedia. Vol.7(3), pp. 151-162 (Sept. 2011)
http://bit.ly/JMM-2011

V. Menkovski, G. Exarchakos, A. Liotta
'Online Learning for Quality of Experience Management'
The annual machine learning conference of Belgium and The Netherlands, Leuven, Belgium, 2010
http://bit.ly/BENELEARN-2010

A. Liotta, G. Exarchakos
'Networks for Pervasive Services: Six Ways to Upgrade the Internet'
Springer (2011)
http://bit.ly/pervasive-networks

Published in: Technology
1 Comment
3 Likes
Statistics
Notes
No Downloads
Views
Total views
1,870
On SlideShare
0
From Embeds
0
Number of Embeds
15
Actions
Shares
0
Downloads
0
Comments
1
Likes
3
Embeds 0
No embeds

No notes for slide

Can networks deliver quality of experience?

  1. 1. Can networks deliverQuality of Experience? Prof. Antonio Liotta Eindhoven University of Technology, NL http://bit.ly/autonomic_networks Twitter: a_liotta
  2. 2. Four questions about Quality of Experience• Why is QoE so important?• How good are we at scoring QoE?• How can we use Machine Learning to manage QoE?• Can networks learn about QoE too?Prof. A. Liotta 2
  3. 3. Why we should NEVER ignore QoE in streaming servicesSending rate: 2048 Kbps HIGHER 39.48 dB - 23.85 dBReceived quality LOWER 15.65 dB Prof. A. Liotta 3
  4. 4. Why we should NEVER ignore QoE in streaming servicesSending rate: 768 Kbps LOWER 35.15 dB - 16.03 dBReceived quality HIGHER 19.12 dB Prof. A. Liotta 4
  5. 5. QoE ≠ Σ QoS Video stream Overprovisioning !!Prof. A. Liotta 5
  6. 6. QoE results from a diversity of factors Transport Environmental EncodingProf. A. Liotta 6
  7. 7. Understanding QoE is essential for video delivery (Internet not designed for QoS nor QoE) 4 3 2 1 2 1 Server A 1 4 3 4 3 2 4 3 Client B No guarantees about: • Delivery of all packets • Delivery order • Delivery time Further details: A.Liotta, G.Exarchakos, “The network as we know it”Prof. A. Liotta (Springer, 2011) – http://bit.ly/pervasive-networks 7
  8. 8. Without ‘edge tricks’ the Internet wouldn’t operate (MPEG-4 video, 1% packet loss, no buffering) J. Okyere-Benya, M. Aldiabat, V. Menkovski, G. Exarchakos, A. Liotta Video Quality Degradation on IPTV Networks, IEEE ICNC‟12, USA, Feb. 2012 http://bit.ly/ICNC-12Prof. A. Liotta 8
  9. 9. Trends: 1. Traffic explosion Anything on video – more producers than watchers? (*) Source: Cisco dataEstimated IP traffic in 2015: 1.3 Zettabytes (1021 bytes) (*) forecast, Feb 2011 Prof. A. Liotta 9
  10. 10. Trends: 2. extremely mobile video(mobile video responsible for majority of mobile traffic)Prof. A. Liotta 10
  11. 11. Trends: 3. M2M video (new communication patterns and ‘volumes’) Building ICT autom. Security Energy & Safety Consu- mer & Retail Home Health- Transpor care Indust. tation Autom.(*) Source: Cisco dataforecast, Feb 2011 Prof. A. Liotta 11
  12. 12. Trends: 4. opportunistic communications (even more ‘video repellent’ than ordinary IP network) A.Liotta, G.Exarchakos, “Spontaneous networks”Prof. A. Liotta (Springer, 2011) – http://bit.ly/pervasive-networks 12
  13. 13. Trends: 4. opportunistic communications (even more ‘video repellent’ than ordinary IP network) A.Liotta, G.Exarchakos, “Spontaneous networks”Prof. A. Liotta (Springer, 2011) – http://bit.ly/pervasive-networks 13
  14. 14. QoE: the missing link towards closed-loop delivery Video stream N-QoS A-QoS Objective Subjective inference QoE QoE Some Some control Expensive process actuators techniquesProf. A. Liotta 14
  15. 15. Subjective QoE: so far a helpless effort Annoying Expensive Inaccurate Impossible to capture complexity and variability of the delivery contextProf. A. Liotta 15
  16. 16. Absolute rating comes with huge variability and bias (mean opinion score) 35% VQEG HD5 30% 25% STDEV[%scale] 20% 17,94% 1 15% 10% 5% 0% 0% 50% 100% 0 MOS [%scale]Further details:“Report on the Validation of Video Quality Models for High Definition Video Content”by the Video Quality Experts Group, Jun. 2010. Prof. A. Liotta 16
  17. 17. Human perception is biased (the same video format gives different perceptions) K. Seshadrinathan et al. “Study of subjective and objective quality assessment of video”Prof. A. Liotta 17 IEEE Trans. Image Processing, Vol.29(6). June 2010.
  18. 18. Both videos are encoded exactly in the same way but the “pedestrian” one is “perceived” as worse• Difference from unimpaired reference is greater in ‘pedestrian’ video• But the overlapping Gaussians indicate that many subjects had opposite perceptions Differential scoring is more accurate than absolute MOS BUT IT’S CLEAR THATProf. A. Liotta humans are not good at ranking 18
  19. 19. Can we use ‘machine learning’ to model QoE on aQoE domain continuous scale? Cumulative Gaussian (2 parameters) QoS domain Psychophysics quantitatively investigates the relationship Prof. A. Liotta between physical stimuli and the sensation of perception 19
  20. 20. A more effective question (Maximum Likelihood Different Scaling)Which one of these two pairs has bigger difference?Prof. A. Liotta 20
  21. 21. Responses fit a psychometric curve (human perception) DEVIATION OF RESPONSES BETWEEN 1 AND 10% DEPENDING ON VIDEO TYPE V. Menkovski, G. Exarchakos, A. Liotta, The Value of Relative Quality in Video Delivery, Journal of Mobile Multimedia. Vol.7(3), pp. 151-162 (Sept. 2011) http://bit.ly/JMM-2011Prof. A. Liotta 21
  22. 22. Use of psychometric curves for QoE management (resource optimization zone) ZONE 1 QoS deltas don’t produce delta QoEs 364 KbpsProf. A. Liotta 512 Kbps 22
  23. 23. Use of psychometric curves for QoE management (quality optimization zone) ZONE 2 strong non linearity 64 KbpsProf. A. Liotta 256 Kbps 23
  24. 24. MLDS is more accurate than conventional QoE rating but still unscalable • Must consider all combinations of samples • A full round of tests including 10 levels of stimuli requires 10 210 tests 4 • The test matrix explodes as we consider more parameters Can we speed up the prediction-model learning process?Prof. A. Liotta 24
  25. 25. Active learning helps eliminating the redundant tests • After the first few test we can start estimating the answers of the remaining tests • The estimation of the unanswered test uses the characteristics of the psychometric curve to reduce the problem domain River bed Tractor Blue skyProf. A. Liotta 25
  26. 26. Learning convergence varies for different videos but always leads to improved scalabilityV. Menkovski, A. Liotta, Adaptive Psychometric Scaling for Video Quality AssessmentJournal of Signal Processing: Image Communication (Elsevier, 2012)http://bit.ly/JSP-2012 Prof. A. Liotta 26
  27. 27. Reinforcement Learning to realize ‘trial & error’ network loops „Sport over mobile phone‟ QoS probe actuators Optimizing QoE Machine QoE measure QoS prediction Learning or inferenceProf. A. Liotta 27
  28. 28. Networks quickly learn how to deal with new conditions(problem domain is constrained to psychometric function) 100 95 90 85 Accuracy 80 75 70 Old conditions New conditions 65 60 55 50 1030 1090 1150 1020 1080 1140 10 70 130 190 250 310 370 430 490 550 610 670 730 790 850 910 970 60 120 180 240 300 360 420 480 540 600 660 720 780 840 900 960 New ‘trial & error’ samples V. Menkovski, G. Exarchakos, A. Liotta, Online Learning for Quality of Experience Management The annual machine learning conference of Belgium and The Netherlands, Leuven, Belgium, 2010 http://bit.ly/BENELEARN-2010 Prof. A. Liotta 28
  29. 29. Take-home messages • Existing QoE methods are – annoying, expensive, inaccurate, ineffective • What is the ‘right’ question? – we are good at spotting difference of differences • Learning how to deal with new perturbations is more promising than brute-force subjective studies – human perception is a moving target – ML works with incomplete information, extrapolates non- obvious patterns and handles the unknown via trial&errorProf. A. Liotta 29
  30. 30. Thank you ! More about my work http://bit.ly/autonomic_networks In the press http://bit.ly/press_articles “All of YouTube through “Cognitive “Networks for a 40-year-old funnel” Interconnections” pervasive services”http://bit.ly/Volkskrant-EN http://bit.ly/booklet-antonio http://bit.ly/pervasive-networks Prof. A. Liotta 30

×