OneClick: A Framework for Measuring Network Quality of Experience


Published on

As the service requirements of network applications shift from high throughput to high media quality, interactivity, and responsiveness, the definition of QoE (Quality of Experience) has become multidimensional. Although it may not be difficult to measure individual dimensions of the QoE, how to capture users’ overall perceptions when they are using network applications remains an open question.

In this paper, we propose a framework called OneClick to capture users’ perceptions when they are using network applications. The framework only requires a subject to click a dedicated key whenever he/she feels dissatisfied with the quality of the application in use. OneClick is particularly effective because it is intuitive, lightweight, efficient, time-aware, and application-independent. We use two objective quality assessment methods, PESQ and VQM, to validate OneClick’s ability to evaluate the quality of audio and video clips. To demonstrate the proposed framework’s efficiency and effectiveness in assessing user experiences, we implement it on two applications, one for instant messaging applications, and the other for firstperson shooter games. A Flash implementation of the proposed framework is also presented.

Published in: Technology, Design
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

OneClick: A Framework for Measuring Network Quality of Experience

  1. 1. OneClick ‐‐ A Framework for Measuring  Network Quality of Experience Kuan‐Ta Chen, Cheng‐Chu Tu, Wei‐Cheng Xiao  Institute of Information Science, Academia Sinica (Presenter on INFOCOM’09: Polly Huang from  National Taiwan University) IEEE INFOCOM 2009
  2. 2. QoS and QoE  QoS (Quality of service) The quality level of system performance metric Communication networks: delay, loss rate DBMS: query completion time QoE (Quality of experience) The quality of how users “feel” about a service Subjective: Mean Opinion Score (MOS) Objective: PSNR (picture), PESQ (voice), VQM (video) IEEE INFOCOM 2009
  3. 3. Relationship between QoS and QoE Comfort range Too bad to perceive QoE Marginal benefit is small QoS,  e.g., network bandwidth IEEE INFOCOM 2009
  4. 4. Knowing the Relationship is Important! So we know  How to adapt voice/video/game data rate (QoS) for user  satisfaction (QoE) So we really know  How to send multimedia data over the Internet OneClick: A Framework for Measuring Network Quality of Experience 4
  5. 5. Measuring QoS and QoE QoS (A great body of work) Measure  network loss, delay, available bandwidth Inference  topology Estimate network capacity etc Still not quite the human experience which is multi-dimensional QoE (Some work) Objective: PSNR (picture), PESQ (voice), VQM (video) Subjective: MOS (general) What’s left! OneClick: A Framework for Measuring Network Quality of Experience 5
  6. 6. MOS (Mean Opinion Score) Problems 1. Slow in scoring (think/interpretation time) 2. People are limited by finite memory 3. Cannot capture users’ perceptions over time 4. MOS is coarse in scale granularity 5. Dissimilar interpretations of  the scale among users IEEE INFOCOM 2009
  7. 7. Our Ambition Identify a simple and yet efficient way  to measure users’ satisfaction IEEE INFOCOM 2009
  8. 8. The Idea: Click, Click, Click  Web surfing Click on a link  You wait, and you refresh the link You wait, and you refresh the link again, and again, and … Knocking at someone’s door Knock on the door You wait, and you knock on the door again You wait, and you knock on the door again and again, and … OneClick: A Framework for Measuring Network Quality of Experience 8
  9. 9. Introducing OneClick Time Application Quality User Satisfaction Click Click ClickClick Click Click Click Click ClickClick Click Click ClickClick Click Click Click Click Click Click Click Click Click Click Click ClickClick Click User Feedback Simple instruction to users: Click when you feel dissatisfied Click multiple times when you feel even less satisfied Clicking rate as the QoE OneClick: A Framework for Measuring Network Quality of Experience 9
  10. 10. Nice Things about OneClick Natural We are already doing it to show lost of patience all the time Bad‐memory proof Real‐time decisions No need to “remember” past experience Time‐aware Capture users’ responses at the time of the problems Useful to study recency, memory access, and habituation  effect
  11. 11. Easy to Implement As a plug‐in to your network applications Flash version done! Co‐measurement of QoS and QoE Time Application Quality User Satisfaction Click Click ClickClick Click Click Click Click ClickClick Click Click ClickClick Click Click Click Click Click Click Click Click Click Click Click ClickClick Click User Feedback Application Quality OneClick: A Framework for Measuring Network Quality of Experience 11
  12. 12. Talk Progress Overview Methodology Pilot Study Validation Case Studies Conclusion OneClick: A Framework for Measuring Network Quality of Experience 12
  13. 13. Human as a QoE Rating System vary this: affect Application QoS Network Setting User observe this: reflect Application QoE Click Events IEEE INFOCOM 2009
  14. 14. QoE QoS Modeling Click events as a counting process Poisson regression C(t): QoE Clicking rate at time t N1(t), N2(t), … : QoS Network conditions at time t  αi : Regression coefficients Derived from the maximum likelihood method IEEE INFOCOM 2009
  15. 15. Wait a Minute… Response delays? Users may not be able to click immediately after they are  aware of the degraded quality Clicking rate of a user consistent? Does a subject give similar ratings in repeated experiments? Clicking rate consistent across users? Different subjects may have different preference on click  decisions. IEEE INFOCOM 2009
  16. 16. Pilot Study An 5‐minute English song Audio quality of AIM Messenger with various  network settings IEEE INFOCOM 2009
  17. 17. Test Material Compilation For each network setting Play the song Record the song K settings  K recordings A test material =  Random non‐overlapping segments  from K different recordings IEEE INFOCOM 2009
  18. 18. Response Delays Try Poisson regression on C(t+x) to N1(t), N2(t), … Varying x Show the goodness of fit per x IEEE INFOCOM 2009
  19. 19. 1‐2 Seconds Delay Response delay calibration needed! IEEE INFOCOM 2009
  20. 20. Our Solution Shift the click event process by time d d is decided by model fitting  Let d be the x such that the goodness of fit is the best Let d be the x such that the residual deviance is the min OneClick: A Framework for Measuring Network Quality of Experience 20
  21. 21. Consistency of C(t+d) from Same User IEEE INFOCOM 2009
  22. 22. Consistency of C(t+d) from Different Users Cross-user normalization needed!
  23. 23. Calibration and Normalization Added OneClick Response Delay Regression Modeling Measurement Calibration With Normalization User #1 User #2
  24. 24. Talk Progress Overview Methodology Pilot Study Validation Case Studies Conclusion OneClick: A Framework for Measuring Network Quality of Experience 24
  25. 25. Rationale Direct: get people to do OneClick and MOS Exact problem we are trying to solve Click Rate MOS PESQ/VQM Indirect: get people to do OneClick and PESQ/VQM OneClick: A Framework for Measuring Network Quality of Experience 25
  26. 26. [Validation] PESQ‐based Validation PESQ: Perceptual Evaluation of Speech Quality OneClick vs. PESQ to evaluate the audio quality of three  VoIP applications AIM  MSN Messenger Skype Network factors Loss rates (0% – 30%) Bandwidth (10 Kbps – 100 Kbps) IEEE INFOCOM 2009
  27. 27. [Validation] Qualitative Comparison Network Loss Rate Bandwidth
  28. 28. [Validation] VQM‐based Validation VQM: Video Quality Measurement OneClick vs. VQM to evaluate video quality of two video  codecs H.264 WMV9 (Windows Media Video) Factors Compression bit rate (200 Kbps – 1000 Kbps) IEEE INFOCOM 2009
  29. 29. [Validation] Qualitative Comparison IEEE INFOCOM 2009
  30. 30. Talk Progress Overview Methodology Pilot Study Validation Case Studies Conclusion OneClick: A Framework for Measuring Network Quality of Experience 30
  31. 31. Case Studies Evaluation of applications’ QoE VoIP applications AIM  MSN Messenger Skype First‐person shooter games Halo Unreal Tournament IEEE INFOCOM 2009
  32. 32. [Case Study] Varying Bandwidth MSN Messenger is generally the worst Skype is the best if bw < 80 Kbps, otherwise AIM is the best IEEE INFOCOM 2009
  33. 33. [Case Study] Contour Lines of Click Rates Slope of  contour line Application’s sensitivity to loss vs. bandwidth shortage AIM is relatively more sensitive to network losses
  34. 34. [Case Study] Comfort Region Comfort Region: a set of network configurations that leads to  satisfactory QoE Skype is the best in bw‐restricted scenarios (< 60 Kbps) when loss  rate is < 10%
  35. 35. Talk Progress Overview Methodology Pilot Study Validation Case Studies Conclusion OneClick: A Framework for Measuring Network Quality of Experience 35
  36. 36. Nice about OneClick Natural & fast We are already doing it to show lost of patience all the time Bad‐memory proof No need to “remember” past experience Time‐aware Capture users’ responses at the time of the problems Fine‐grain The score can be 0.2, 3.5, or even 12.345 Normalized user interpretation  Different interpretations are normalized Easy to implement IEEE INFOCOM 2009
  37. 37. OneClick Online IEEE INFOCOM 2009
  38. 38. On‐Going Work Large‐scale experiments (by crowdsourcing) Click rate vs. MOS QoE‐centric multimedia networking As an example, Tuning the Redundancy Control Algorithm of  Skype for User Satisfaction, IEEE INFOCOM 2009.  OneClick: A Framework for Measuring Network Quality of Experience 38
  39. 39. Thank You! Kuan‐Ta Chen IEEE INFOCOM 2009