To satisfy players' gaming experience, there is a strong need for a technique that can measure a game's quality systemically, efficiently, and reliably. In this paper, we propose to use paired comparisons and probabilistic choice models to quantify online games's QoE under various network situations. The advantages of our methodology over the traditional MOS ratings are 1) the rating procedure is simpler thus less burden is on experiment participants, 2) it derives ratio-scale scores, and 3) it enables systematic verification of participants' inputs. As a demonstration, we apply our methodology to evaluate three popular FPS (first-person-shooter) games, namely, Alien Arena (Alien), Halo, and Unreal Tournament (UT), and investigate their network robustness. The results indicate that Halo performs the best in terms of their network robustness against packet delay and loss. However, if we take the degree of the games' sophistication into account, we consider that the robustness of UT against downlink delays should able be improved. We also show that our methodology can be a helpful tool for making decisions about design alternatives, such how dead reckoning algorithms and time synchronization mechanisms should be implemented.

1. Online Game QoE EvaluationOnline Game QoE Evaluation
usingusing Paired ComparisonsPaired Comparisons
Sheng-Wei Chen ( 陳昇瑋 )
Institute of Information Science
Academia Sinica

2. Motivation
Online gaming has been shown a profitable killer application of
the Internet.
To provide better gaming service to players
game developers: improve the quality of game software
network engineers: improve the quality of network infrastructure
 There is a strong need for a technique that can measure a
game’s quality systematically, efficiently, and reliably

3. QoS and QoE
QoS (Quality of service)
The quality level of “native” performance metric
 Communication networks: delay, loss rate
 Voice/audio codec: fidelity
 DBMS: query completion time
QoE (Quality of experience)
How users “feel” about a service
Usually multi-dimensional, and tradeoffs exist between different
dimensions (download time vs. video quality, responsiveness vs.
smoothness)
However, a unified (scalar) index is normally desired!

4. A typical relationship between QoS and QoE
QoS, e.g., network bandwidth
QoE
Hard to tell “very bad”
from “extremely bad”
Marginal benefit is small

5. Our Objective
A methodology which can efficiently and reliably measure a
game’s realtimeliness and interactivity in certain network scenario
Applications, to name a few, …
Game development
comparison with competitors
design alternative comparison
Game server deployment
server deployment and network planning
Game play
access network evaluation

6. MOS (Mean Opinion Score)
Difficulty in scoring
Dissimilar interpretations about the scales
MOS scale is NOT an interval scale
Not verifiable

7. What We Propose
Use paired comparisons for evaluating online games’ QoE in
various network scenarios.
In a paired comparison test,
a participant is simply asked to compare two stimuli at a time
vote (decide) whose quality is better based on his/her perception
The decision is simpler than the MOS method as the five-scale
rating has been reduced to a dichotomous choice.

8. Features of Paired Comparison
Generalizable across games and game genres
Burden of experiment participants is low
Subjects do not have to bother mapping their sensation magnitude onto a
scale.
Interval scale scores can be computed by probabilistic choice
models
The comparison results can be verified

9. Our Contribution
Proposed to use paired comparisons and probabilistic choice models to
quantify games’ QoE under various network situations.
the rating procedure is simpler thus less burden is on experiment participants,
it derives interval-scale scores,
it enables systematic verification of participants’ inputs.
Applied the proposed methodology to evaluate the network robustness of
three popular FPS games
three games exhibit very different behavior in reaction to network impairment
we can even infer how the time synchronization mechanism is implemented in each
game from the QoE measurements

10. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 10
Talk Progress
Overview
Paired Comparison
Probabilistic Choice Modeling
Experiment Methodology
Experiment Results
Conclusion & Future work

11. Paired Comparison: An Example
Absolute choice frequencies of 234 subjects choosing between pairs of nine
celebrities.
L. B. Johnson (LBJ), Harold Wilson (HW), and Charles De Gaulle (CDG) are
politicians; Johnny Unitas (JU), Carl Yastrzemski (CY), and A. J. Foyt (AJF) are
athletes; Brigitte Bardot (BB), Elizabeth Taylor (ET), and Sophia Loren (SL) are
female movie stars.

12. Probabilistic Choice Modeling
Bradley-Terry-Luce (BTL) model
Ti : stimulus in the experiment
Pi,j : probability of choosing Ti over Tj
u(Ti) : the estimated score of the Ti

13. Verification of Raw Inputs
Stochastic transitivity properties
three variants of transitivity property: weak (WST), moderate (MST), and
strong (SST) stochastic transitivity
The three versions of transitivity imply that if Pij ≥ 0.5 and Pjk ≥ 0.5, then
Kendall’s u-coefficient
Good-of-fit of the BTL model

14. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 14
Talk Progress
Overview
Paired Comparison
Probabilistic Choice Modeling
Experiment Methodology
Experiment Results
Conclusion & Future work

15. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 15
Experiment Methodology
Three FPS games
Alien Arena 2008 (Alien)
Halo
Unreal Tournament 2004 (UT)
Automatic stimulus switching strategy
Players make votes when they are able to decide whether setting A or B is
better
A test will continue until both participants make their choices
timesetting A setting A setting Bsetting B setting A setting B

16. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 16
Experiment Results: Data Summary

17. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 17
Talk Progress
Overview
Paired Comparison
Probabilistic Choice Modeling
Experiment Methodology
Experiment Results
Conclusion & Future work

18. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 19
Effect of Network Delay

19. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 20
Effect of Network Delay: Observations
The effect of delay is different on the uplink and downlink
Uplink packets
users’ commands
game clients can provide immediate feedback for some actions, but still
need to wait for the environmental changes due to players’ actions
Downlink packets
1) environmental changes, 2) actions of other players
Some proposals to remedy screen freeze, e.g., dead reckoning
We believe that Halo implements dead reckoning

20. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 21
Effect of Network Loss

21. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 22
Effect of Delay Jitter

22. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 23
Effect of Delay Jitter: Observations
Each of the games is robust to delay jitters in one direction, and susceptible to
delay jitters in another direction
We believe that this behavior is due to the different locations where a game
implements its time synchronization mechanism
Reasons for time synchronization:
1) keeps game states in each client consistent;
2) maintain fairness to make a faster client slower;
3) prevents time-based cheating attacks.
 Alien implements time synchronization in its client; Halo and UT implement the
synchronization at the server
Considering uplink bandwidth is usually more scarce, Alien’s design is better

23. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 24
Talk Progress
Overview
Paired Comparison
Probabilistic Choice Modeling
Experiment Methodology
Experiment Results
Conclusion & Future work

24. Sheng-Wei Chen / Online Game QoE Evaluation using Paired Comparisons 25
Conclusion and Future Work
Looks Halo has a better design, however, …
The variety in game design, scene complexity, game pace, game rules, playing
strategies were not considered
 Scene and special effect complexity in Alien is far more sophisticated
 UT also has splendid special effects
 Halo provide only primitive effects
 reasonable for Halo to have higher network robustness
Future goals
summarize the similarity and difference of games of the same type
comparison of different game genres
understand the requirement of network support from each genre of games
25

25. Thank You!
Sheng-Wei Chen ( 陳昇瑋 )
http://www.iis.sinica.edu.tw/~swc