Online Game QoE EvaluationOnline Game QoE Evaluationusingusing Paired ComparisonsPaired ComparisonsSheng-Wei Chen ( 陳昇瑋 )I...
MotivationOnline gaming has been shown a profitable killer application ofthe Internet.To provide better gaming service to ...
QoS and QoEQoS (Quality of service)The quality level of “native” performance metric Communication networks: delay, loss r...
A typical relationship between QoS and QoEQoS, e.g., network bandwidthQoEHard to tell “very bad”from “extremely bad”Margin...
Our ObjectiveA methodology which can efficiently and reliably measure agame’s realtimeliness and interactivity in certain ...
MOS (Mean Opinion Score)Difficulty in scoringDissimilar interpretations about the scalesMOS scale is NOT an interval scale...
What We ProposeUse paired comparisons for evaluating online games’ QoE invarious network scenarios.In a paired comparison ...
Features of Paired ComparisonGeneralizable across games and game genresBurden of experiment participants is lowSubjects do...
Our ContributionProposed to use paired comparisons and probabilistic choice models toquantify games’ QoE under various net...
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 10Talk ProgressOverviewPaired ComparisonProbabilistic...
Paired Comparison: An ExampleAbsolute choice frequencies of 234 subjects choosing between pairs of ninecelebrities.L. B. J...
Probabilistic Choice ModelingBradley-Terry-Luce (BTL) modelTi : stimulus in the experimentPi,j : probability of choosing T...
Verification of Raw InputsStochastic transitivity propertiesthree variants of transitivity property: weak (WST), moderate ...
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 14Talk ProgressOverviewPaired ComparisonProbabilistic...
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 15Experiment MethodologyThree FPS gamesAlien Arena 20...
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 16Experiment Results: Data Summary
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 17Talk ProgressOverviewPaired ComparisonProbabilistic...
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 19Effect of Network Delay
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 20Effect of Network Delay: ObservationsThe effect of ...
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 21Effect of Network Loss
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 22Effect of Delay Jitter
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 23Effect of Delay Jitter: ObservationsEach of the gam...
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 24Talk ProgressOverviewPaired ComparisonProbabilistic...
Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 25Conclusion and Future WorkLooks Halo has a better d...
Thank You!Sheng-Wei Chen ( 陳昇瑋 )http://www.iis.sinica.edu.tw/~swc
Upcoming SlideShare
Loading in …5
×

Online Game QoE Evaluation using Paired Comparisons

688 views

Published on

To satisfy players' gaming experience, there is a strong need for a technique that can measure a game's quality systemically, efficiently, and reliably. In this paper, we propose to use paired comparisons and probabilistic choice models to quantify online games's QoE under various network situations. The advantages of our methodology over the traditional MOS ratings are 1) the rating procedure is simpler thus less burden is on experiment participants, 2) it derives ratio-scale scores, and 3) it enables systematic verification of participants' inputs.

As a demonstration, we apply our methodology to evaluate three popular FPS (first-person-shooter) games, namely, Alien Arena (Alien), Halo, and Unreal Tournament (UT), and investigate their network robustness. The results indicate that Halo performs the best in terms of their network robustness against packet delay and loss. However, if we take the degree of the games' sophistication into account, we consider that the robustness of UT against downlink delays should able be improved. We also show that our methodology can be a helpful tool for making decisions about design alternatives, such how dead reckoning algorithms and time synchronization mechanisms should be implemented.

Published in: Entertainment & Humor, Sports
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
688
On SlideShare
0
From Embeds
0
Number of Embeds
148
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Online Game QoE Evaluation using Paired Comparisons

  1. 1. Online Game QoE EvaluationOnline Game QoE Evaluationusingusing Paired ComparisonsPaired ComparisonsSheng-Wei Chen ( 陳昇瑋 )Institute of Information ScienceAcademia Sinica
  2. 2. MotivationOnline gaming has been shown a profitable killer application ofthe Internet.To provide better gaming service to playersgame developers: improve the quality of game softwarenetwork engineers: improve the quality of network infrastructure There is a strong need for a technique that can measure agame’s quality systematically, efficiently, and reliably
  3. 3. QoS and QoEQoS (Quality of service)The quality level of “native” performance metric Communication networks: delay, loss rate Voice/audio codec: fidelity DBMS: query completion timeQoE (Quality of experience)How users “feel” about a serviceUsually multi-dimensional, and tradeoffs exist between differentdimensions (download time vs. video quality, responsiveness vs.smoothness)However, a unified (scalar) index is normally desired!
  4. 4. A typical relationship between QoS and QoEQoS, e.g., network bandwidthQoEHard to tell “very bad”from “extremely bad”Marginal benefit is small
  5. 5. Our ObjectiveA methodology which can efficiently and reliably measure agame’s realtimeliness and interactivity in certain network scenarioApplications, to name a few, …Game developmentcomparison with competitorsdesign alternative comparisonGame server deploymentserver deployment and network planningGame playaccess network evaluation
  6. 6. MOS (Mean Opinion Score)Difficulty in scoringDissimilar interpretations about the scalesMOS scale is NOT an interval scaleNot verifiable
  7. 7. What We ProposeUse paired comparisons for evaluating online games’ QoE invarious network scenarios.In a paired comparison test,a participant is simply asked to compare two stimuli at a timevote (decide) whose quality is better based on his/her perceptionThe decision is simpler than the MOS method as the five-scalerating has been reduced to a dichotomous choice.
  8. 8. Features of Paired ComparisonGeneralizable across games and game genresBurden of experiment participants is lowSubjects do not have to bother mapping their sensation magnitude onto ascale.Interval scale scores can be computed by probabilistic choicemodelsThe comparison results can be verified
  9. 9. Our ContributionProposed to use paired comparisons and probabilistic choice models toquantify games’ QoE under various network situations.the rating procedure is simpler thus less burden is on experiment participants,it derives interval-scale scores,it enables systematic verification of participants’ inputs.Applied the proposed methodology to evaluate the network robustness ofthree popular FPS gamesthree games exhibit very different behavior in reaction to network impairmentwe can even infer how the time synchronization mechanism is implemented in eachgame from the QoE measurements
  10. 10. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 10Talk ProgressOverviewPaired ComparisonProbabilistic Choice ModelingExperiment MethodologyExperiment ResultsConclusion & Future work
  11. 11. Paired Comparison: An ExampleAbsolute choice frequencies of 234 subjects choosing between pairs of ninecelebrities.L. B. Johnson (LBJ), Harold Wilson (HW), and Charles De Gaulle (CDG) arepoliticians; Johnny Unitas (JU), Carl Yastrzemski (CY), and A. J. Foyt (AJF) areathletes; Brigitte Bardot (BB), Elizabeth Taylor (ET), and Sophia Loren (SL) arefemale movie stars.
  12. 12. Probabilistic Choice ModelingBradley-Terry-Luce (BTL) modelTi : stimulus in the experimentPi,j : probability of choosing Ti over Tju(Ti) : the estimated score of the Ti
  13. 13. Verification of Raw InputsStochastic transitivity propertiesthree variants of transitivity property: weak (WST), moderate (MST), andstrong (SST) stochastic transitivityThe three versions of transitivity imply that if Pij ≥ 0.5 and Pjk ≥ 0.5, thenKendall’s u-coefficientGood-of-fit of the BTL model
  14. 14. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 14Talk ProgressOverviewPaired ComparisonProbabilistic Choice ModelingExperiment MethodologyExperiment ResultsConclusion & Future work
  15. 15. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 15Experiment MethodologyThree FPS gamesAlien Arena 2008 (Alien)HaloUnreal Tournament 2004 (UT)Automatic stimulus switching strategyPlayers make votes when they are able to decide whether setting A or B isbetterA test will continue until both participants make their choicestimesetting A setting A setting Bsetting B setting A setting B
  16. 16. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 16Experiment Results: Data Summary
  17. 17. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 17Talk ProgressOverviewPaired ComparisonProbabilistic Choice ModelingExperiment MethodologyExperiment ResultsConclusion & Future work
  18. 18. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 19Effect of Network Delay
  19. 19. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 20Effect of Network Delay: ObservationsThe effect of delay is different on the uplink and downlinkUplink packetsusers’ commandsgame clients can provide immediate feedback for some actions, but stillneed to wait for the environmental changes due to players’ actionsDownlink packets1) environmental changes, 2) actions of other playersSome proposals to remedy screen freeze, e.g., dead reckoningWe believe that Halo implements dead reckoning
  20. 20. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 21Effect of Network Loss
  21. 21. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 22Effect of Delay Jitter
  22. 22. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 23Effect of Delay Jitter: ObservationsEach of the games is robust to delay jitters in one direction, and susceptible todelay jitters in another directionWe believe that this behavior is due to the different locations where a gameimplements its time synchronization mechanismReasons for time synchronization:1) keeps game states in each client consistent;2) maintain fairness to make a faster client slower;3) prevents time-based cheating attacks. Alien implements time synchronization in its client; Halo and UT implement thesynchronization at the serverConsidering uplink bandwidth is usually more scarce, Alien’s design is better
  23. 23. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 24Talk ProgressOverviewPaired ComparisonProbabilistic Choice ModelingExperiment MethodologyExperiment ResultsConclusion & Future work
  24. 24. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 25Conclusion and Future WorkLooks Halo has a better design, however, …The variety in game design, scene complexity, game pace, game rules, playingstrategies were not considered Scene and special effect complexity in Alien is far more sophisticated UT also has splendid special effects Halo provide only primitive effects reasonable for Halo to have higher network robustnessFuture goalssummarize the similarity and difference of games of the same typecomparison of different game genresunderstand the requirement of network support from each genre of games25
  25. 25. Thank You!Sheng-Wei Chen ( 陳昇瑋 )http://www.iis.sinica.edu.tw/~swc

×