Goals of this presentation
• Global trends in online games
• Impact of Latency
• Genres and characteristics
• Architectures
• NetworkTrafficCharacteristics
• Estimating Quality of Experience
• Qoe-Enhancing Mechanisms
• Conclusions
2

Video
3
Youtube link: https://www.youtube.com/watch?v=QAnW9HTkbsI

GLOBALTRENDS IN ONLINE
GAMES
28.10.2015. 4

Gamer population
10.01.2014. 6

Shift towards online
• Multiplayer games
• Social games
• Mobile games
• Content distribution
• DRM
7

Types of multiplayer
• Earliest ways – hot seat and split screen
• Playing over local networks (e.g. Internet Café)
• Playing through Internet (online)

Who are the consumers?
Today’s video games provide rich, engaging entertainment for all
players across all platforms. Our “2015 Essential Facts About
the Computer and Video Game Industry” report underscores
how video games have evolved into a mass medium: more than
150 million Americans play video games and 42 percent play
video games regularly, or at least three hours per week. The
average gamer is 35 years old, and 74 percent are
age 18 or older. Nearly half (44 percent) of gamers are
female and women over the age of 18 represent a significantly
greater portion of the game-playing population (33 percent) than
boys age 18 or younger (15 percent). A majority of parents
(63 percent) say video games are a positive part of their child’s
life: 85 percent think that game play is fun for the whole family
and 75 percent believe playing games offers a good
opportunity to connect with their child. (…).
Source: Entertainment Software Association (ESA) http://www.theesa.com/about-esa/industry-facts/
10

Personal Computer (PC)
• Multi purpose device – not dedicated purely
for purpose of playing games
• “Natural” place for networked games
• Almost all PCs equipped with a network
interface
• “Core players” – perceived as a device for core
gaming audience
12

Consoles
• Consoles of the newest generation are
equipped with network interfaces
• Supported by cloud server infrastructure
• XboX One supported by 300 000 servers
• Additional payment for multiplayer
– PlayStation Plus account
– Microsoft Xbox Live account
10.01.2014. 13

Mobile
• Mobile phones, tablets, handheld consoles
• Relatively new but very large market
• Biggest problems for Quality of Experience of online games –
variability of network parameters (latency, latency variation,
and packet loss) Source: Ericsson Mobility Report
http://www.ericsson.com/res/docs/2013/ericsson-mobility-report-june-2013.pdf
14

Business models
• Pay to play
– Game client/account
– Subscription
– Additions to existing games
• Free to play (F2P)
– Micro transactions
– Additional content
– Premium accounts
– Cosmetic/usability improvements
• Combinations
15

IMPACT OF LATENCY
28.10.2015. 16

Impact of the network: example
At t=0.0, client 2 shoots the duck
10.01.2014. 17
3
50ms
100 ms
150 ms
2
1
SHOOTS!!!
t = 0ms

3
50ms
100 ms
150 ms
2
1
t = 100ms
At t=100.0, the server receives the shot
10.01.2014. 18

3
50ms
100 ms
150 ms
2
1
t = 150ms
At t=150.0, client 1 sees the shot (and the
retriever appears in the server)
10.01.2014. 19

3
50ms
100 ms
150 ms
2
1
t = 200ms
retriever appears in the client 1)
10.01.2014. 20

3
50ms
100 ms
150 ms
2
1
t = 200ms
retriever appears in the client 1)
10.01.2014. 21

GENRES AND CHARACTERISTICS
28.10.2015. 22

Game genres
• Game categorization:
– Action
– Adventure (e.g., Broken Sword)
– Arcade (e.g., Pinball)
– Children’s Entertainment (e.g., Bob the Builder)
– Family Entertainment (e.g., Mahjongg)
– Fighting (e.g., Mortal Combat)
– Flight (e.g.,Wing Commander)
– Racing (e.g., Need For Speed)
– Role Playing (e.g., World ofWarcraft)
– Shooter (e.g., Quake)
– RealTime Strategy (e.g., Starcraft)
– MOBA (e.g. League of Legends)
– Other Games
NPD Group Inc., NDP Software Category Definitions, 2008,
https://www5.npd.com/tech/pdf/swcategories.pdf.
23

Types of multiplayer
• The most popular online
multiplayer games:
– MMORPG (Massively Multiplayer
Online RPG)
– FPS (First Person Shooter)
– RTS (RealTime Strategy)
– MOBA (Multiplayer Online Battle
Arena) – combination RTS and RPG
genres
– Sports

Delay sensitivity
– MMORPG 120ms
– FPS 80 ms
– RTS 200-500 ms
The experience of the player increases his/her delay
sensitivity, i.e. an experienced player gets annoyed
more easily than a novel one.
The behavior and the skill level of the other players in
the party also have an influence on the subjective
quality.

Architecture
• Increasing dominance of client – server (C-S)
– Cheating avoidance
– Easier synchronization
– Billing
• Peer to Peer (P2P)
– Very few true Peer to Peer games (e.g., Demigod)
– Great scalability for large scale virtual worlds
– A lot of research activity (e.g., scalability for
Minecraft projects)
27

Server organization in C-S model
• Server included in the game and one client acts
as the server (e.g., Counter Strike)
• Server application included in the game and
players create their own servers without knowing
it (e.g., Call of Duty for PS2)
• Server fully controlled by the
developer/publisher (e.g., World ofWarcraft)
Multiplayer match organization may be
orchestrated by the game provider, third parties, or
by players themselves
28

Server organization in C-S model
• Server application included in the game and
players create their own servers without
knowing it. Problem: Server migration
29
0
10
20
30
40
50
60
70
80
400 420 440 460 480 500 520 540 560 580 600
Bandwidth[kbps]
Game time [s]
Bandwidth sent and received
From server #1
From server #2
0
10
20
30
40
50
60
70
80
400 420 440 460 480 500 520 540 560 580 600
Bandwidth[kbps]
Game time [s]
Bandwidth sent and received
To server #1
To server #2

NETWORKTRAFFIC
CHARACTERISTICS
28.10.2015. 30

Information transferred
• What information does the traffic comprise?
– Player commands/inputs
– Virtual world state refreshes
– Chat
– Audio flows for player communication
• Some games have in-builtVoIP systems
• Many players use stand alone applications (Teamspeak,
Ventrilo, Skype…)
– 3D data describing virtual world (Second Life)
– Video
• Send by cloud based games
• Streaming of gaming sessions
31

Network Characteristics
Quake 2 (1997)
World ofTanks (2014)
32
0
200
400
600
800
1000
1200
1400
40 50 60 70 80 90 100 110
Numberofpackets
Packet size [bytes]
Quake 2. Packet Size Histogram
0
200
400
600
800
1000
1200
40 50 60 70 80 90 100 110
Numberofpackets
Packet size [bytes]
World of Tanks. Packet Size Histogram
0
10
20
30
40
50
60
70
10 30 50 70 90 110 130 150Numberofpackets
Inter-Packet Time [ms]
Quake 2. Inter-Packet Time Histogram
0
5
10
15
20
25
10 30 50 70 90 110 130 150
Numberofpackets
Inter-Packet Time [ms]
World of Tanks. Inter-Packet Time Histogram

Bottlenecks in Client-server model
Three potential bottlenecks:
- uplink: gamers send their actions
- server: calculation of the next state
- downlink: send the state to players
Server processing
capacity limit
33

Bottlenecks in Client-server model
Two of the bottlenecks are network-related
The traffic to be exchanged grows with N2
N(N-1)
Server processing
capacity limit
34

Why so small?
• Market penetration!
• World ofWarcraft was released in 2004 – in
order to reach as much users as possible it
needed to work on 33,6k modem
• UnrealTournament on 14,4k
• High broadband penetration –will games use
more and more bandwidth?
35

Traffic characterization
• Game flows:
– Long lived
– High packet rate
– Small payload sizes
– Low bandwidth usage
– Using both UDP andTCP
– Dependant on the game genre
36

TCP for a real-time game?
It sounds counterintuitive, but many MMORPGs
do useTCP
NormalTCP (FTP) MMORPG
37
0
5
10
15
20
25
0 10 20 30 40 50 60 70 80
TCPwindow
simulation time [s]
TCP Window, FTP
0
5
10
15
20
25
0 10 20 30 40 50 60 70 80
TCPwindow
simulation time [s]
TCP Window, MMORPG

Game traffic evolution? – Not really
1-5kbps
(2-8 players)
2-3 kbps
(independent of
number of players)
M. Claypool, D. LaPoint, and J. Winslow, “Network Analysis of Counter-strike and Starcraft,” in Proceedings of the
22nd IEEE International Performance, Computing, and Communications Conference (IPCCC), USA, April 2003.
C-S. Lee, “The Revolution of StarCraft Network Traffic” in Proceedings of the 11th Annual Workshop on Network and
Systems Support for Games NetGames 2012
38

Client versions
• Specific application per game
– Full clients (all the information stored in the client on
player’s device – single player games)
– Hybrid clients (need to communicate with the server)
• Clients encompassing multiple games
– Browser based games
– Cloud based games (thin clients)
• Client version is dependant of where game logic
and rendering is executed which heavily affects
traffic characteristics
39

Cloud-based games
• Cloud gaming traffic
– Very high bandwidth usage
– High quality video
– Very delay sensitive (no client side optimization)
– * no high market penetration: Onlive shut down last 30th April
– New services: PlayStation Now
40

OnLive downstream traffic
41
M. Claypool, D. Finkel, A. Grant, and M. Solano: “Thin to win? Network performance analysis of the OnLive thin client
game system”. 11th Annual Workshop on Network and Systems Support for Games (NetGames), 2012 (pp. 1-6). IEEE.

1
10
100
1000
10000
Warcraft III
(RTS)
World of
Warcraft
(MMORPG)
Madden NFL
(Sports)
Unreal
Tournament
(FPS)
Second Life
(CVE)
Crazy Taxi
(Cloud)
Bandwidth[kbit/s]
Bandwidth usage across genres
42

ESTIMATING QUALITY OF
EXPERIENCE
43

Adapting QoE estimators fromVoIP
• ITU’s E-Model: 150 ms (OWD)
– ITU G.107:
• R factor, ranging from 0 to 100
• Directly translated into a Mean Opinion Score (MOS)
from 1 (bad) to 5 (excellent)
• This can be adapted to online games:
– Using the “lag” or “ping”
– Subjective tests with real users
– Objective tests with bots
44

Network Impairments
• Latency
– LAN parties avoid this, but we are considering the
Internet
– Parts
• Generation delay (capturing key stroke) is negligible
• Packet transfer delay (our problem)
• Processing delay. It is only a problem in cloud games:
– server has to render the scene, code it in a video and send it
to the client application, which has to decode and display it to
the user (100 to 300 ms)
45

Network Impairments
• Jitter
– We cannot use a de-jitter buffer, as inVoIP
– It is used as an input for QoE estimation
• It can be translated into packet loss if packets are
reordered
46
Delay 1 Delay 2 Delay 3 Delay i
Transmission
Reception
1
1



 
n
delaydelay
IPDV
ii
...

Network Impairments
• Packet loss
– InTCP-based games, it is only translated into an additional
delay (retransmission)
– QoE-enhanced mechanisms for hiding the effect of packet loss.
– The client and the server may be able to do a prediction if a
packet is lost, based on the current movement of the avatar.
– If the forecasted position and the one arrived in the next packet
are not the same, sudden and abrupt movement.
– different degrees of robustness against packet loss.
• Halo stopped working when packet loss was roughly 4 %,
• The users of Quake III did not experience any degradation in the
quality even with a packet loss of about 35 %.
47

QoE models
• “Impairment factor” proposed in [*]:
Impairment factor = ( WL × L + WJ × J ) ( 1 + E )
– WL and WJ : weighting factors for latency (L) and
jitter (J) respectively
– E is the packet loss rate.
48
*Ubicom White Paper (2005) OPScore, or Online Playability Score: A
Metric for Playability of Online Games with Network Impairments.

QoE models
• In [*] only delay and jitter are considered (the
considered FPS game has a very good
method for concealing packet loss):
Impairment factor = 0.104 · ping + jitter
49
*Wattimena AF, Kooij RE, van Vugt JM, Ahmed OK (2006) Predicting the perceived
quality of a first person shooter: the Quake IV G-model. In Proceedings of 5th ACM
SIGCOMM workshop on Network and system support for games (NetGames '06). ACM,
New York, NY, USA, Article 42. doi: 10.1145/1230040.1230052

QoE models
• In [*] packet loss is not considered, taking
into account that aTCP-based game is being
tested, so lost packets are retransmitted:
MOS = 5.17 − 0.012 · delay − 0.018 · jitter
50
*Ries M, Svoboda P, Rupp M (2008) Empirical Study of Subjective Quality for Massive
Multiplayer Games. Proceedings of the 15th International Conference on Systems,
Signals and Image Processing: 181 - 184. doi: 10.1109/IWSSIP.2008.4604397

Limitations of QoE models
• One model is only valid for a single title. Some generic models have
been proposed, but the weighting factors have to be tuned according to
each game → new subjective tests are required.
• The models involve a simplification: the influence of a network
impairment is modified depending on the activity performed by the
player [*].
• Most models do not take context and users parameters into account.
The subjective quality estimation for online games is a much more
complex problem than in the case of Voice over IP.
51
*Suznjevic M, Skorin-Kapov L, Matijasevic M (2013) The impact of user, system, and
context factors on gaming QoE: A case study involving MMORPGs. Network and
Systems Support for Games (NetGames), 12th Annual Workshop on. IEEE. doi:
10.1109/NetGames.2013.6820606

QOE-ENHANCING MECHANISMS
28.10.2015. 52

Delay related methods
• Client side prediction
• Dead reckoning
• Server side delay compensation
• Geographical server distribution
53

Client side prediction
• The client generates a player command
• The next status of the game is calculated by the server
and sent to all the clients, and each of them renders
the scene for its player.
• During this interval, something has to be shown to the
player:
– Performing the movement of the client locally, assuming
that the server will accept the command.
• Drawback: if the server’s response does not fit with
the client prediction, a perceptible shift in the position
of the avatar will be appreciated by the player.
54

Dead reckoning
• Algorithm for estimating the position of an
entity in the virtual world based on its previous
position, orientation, speed, acceleration, etc.
• If a spike of delay or packet loss appears, each
copy of the virtual world will continue to
estimate movement of those entities.
• When an inconsistency is found, some games are
able to make the transition less abrupt, using
smoothing algorithms.
55

Server side delay compensation
• The server “rewinds time” according to the
particular client’s latency, calculating execution of a
particular command (e.g. whether the player’s shot
has managed to hit the target).
56

57
Jack shoots

58
Wang is
here when
the packet
arrives to
the server

59
Wang is
here when
the packet
arrives to
the server
But Wang was
here when
Jack shot

Shooting around the corner problem
(virtual world inconsistency)
10.01.2014. 60
Jack Game server Wang
Position 1
Position 2
Shot towards
Position 1
Wang DEAD
Network delay scheme
time
Position 1
Wang hit!
Wang DEAD
Position 2
Shot
Jack’s
avatar
Wang’s
avatar
Position 1
Wang’s
avatar
Position 2
Virtual world scheme

Geographical server distribution
• A natural method for reducing latency.
• Optimization of server locations.The closer the
server is to the player (physically) the transfer
part of the latency is lower.
• Many games report the geographical place
where their servers are.
• In other cases, players are allowed to select the
server where they want to play.
• They may be able to select a server in another
zone, but in that case they are aware of the
possibility of having a higher latency.
61

Scalability related methods
• Area of Interest
• Sharding
• Zoning
• Mirroring
• Instancing
62

Area of Interest
• All of the players in a massive virtual world receive real
time information regarding all of the other players →
impossible network requirements.
• An Area of Interest (AOI) is defined around the avatar,
also including its field of view.
• Only the information generated by other avatars in
player’s AOI is sent to him/her.
• Keep the processing load and the traffic to be sent to
the player in reasonable limits.
• The statistical model of the traffic which depends on
the number of players in the server.
63

Sharding
• A virtual world can be distributed across game servers in two
ways:
– one logical instance of the virtual world is created for all players and is
spanned across all the machines of the game server farm (e.g. EvE
Online)
– the virtual world is replicated on more than one logical instance,
called shard.
64

Sharding
• Shards partition the player base across
several logical instances of the virtual world
(players in different shards cannot interact
with each other)
• In WoW it is now possible for certain zones to
be shared across different shards, so players
from different shards are now able to play
together.
65

TheAsakai incident
What if a lot of users decides to fight at one
solar system??
Youtube link: http://www.youtube.com/watch?v=_iQw3YcLoQU
Around 2850 people in the same fight at the same time

Integration of multiple games into one
virtual world (Dust 514 and EvE Online)
67
Youtube link: http://www.youtube.com/watch?v=eS4rAYrRHWc, air support

War without the warchief
Youtube link: http://www.youtube.com/watch?v=ZzsIiSTnQfI

Zoning
• Scalability on a single shard.
• Partitions the virtual world into geographical areas
called zones, which can be handled independently by
separate machines.
• In the past, zones had borders (e.g. invisible walls) with
transition spots between them, such as portals,
because transition between zones required certain
time. Bad influence on the players’ immersion.
• Technique called seamless zoning: zones divided by
some geographic markers (e.g. mountains), but there
are no loading screens when players cross.
69

Mirroring
• “Hot spots” appear in virtual worlds, and they typically include
major cities or zones in which gathering of players is common.
• Parallelization of game sessions with a large density of players
located and interacting within each other’s AOI.
• Mirroring: distributing the load by replicating the same game zone
on several servers.
– In each replicated server, the state for a subset of entities (i.e. active
entities) is calculated, while the remaining entities (i.e. shadowed
entities) states are calculated in the other participating servers, and
are synchronized across servers
• The overhead of synchronizing shadowed entities is much lower
than the overhead of computing all entities as active entities [*].
70
*Muller J, Metzen H, Ploss A, Schellmann M, Gorlatch S (2006) Rokkatan: Scaling an RTS
game design to the massively multiplayer realm. Comput. Entertain. 4, 3, Article 11. doi:
10.1145/1146816.1146833

Instancing
• a simplification of mirroring, or even sharding on
a smaller scale.
• Distributes the session load onto multiple
parallel instances for the highly populated zones.
• The instances are independent of each other,
which means that two avatars from different
instances will not be able to interact with each
other, even if they are located at coordinates
within their AOI.
• It is common in many MMORPGs.
71

Scalability related methods
72
Sharding
Zoning
Virtual word 2
MirroringInstancing
Virtual world
Virtual word 2Virtual world 1
Zone
Zoning
Zone
Active entity Shadowed entity Interaction border
Single logical
instance
Sharding

Conclusions #1
• Companies employ different techniques to provide games
in a scalable, reliable and profitable way, with the main
objective of achieving a good Quality of Experience level.
• There is a special difficulty with this kind of service, since
gamers show a very demanding profile.
• The synchronization and the maintaining of a coherent
game state to be shared by the applications of all the
players is not a trivial problem.
• Different genres of online games present specific latency
requirements, depending on the game dynamics, their
characteristics and the level of interaction between the
players.
73

Conclusions #2
• Online games have employed different
architectures, being client-server the one
employed in the vast majority of the cases.
• Online games (except “cloud games”) generate
low-bandwidth traffic flows.
• Different protocols (UDP orTCP) are employed
atTransport level.
• Different techniques used for estimating the
user’s QoE from network parameters.
• Latency is the most important parameter.
74

Conclusions #3
• DifferentQoE-enhancing mechanisms are
used by game providers, as client-side
prediction, server delay compensation.
• Scalability-related techniques are also used
for improving user’s experience.
75

Acknowledgments
• Thanks to Mirko Suznjevic for his help with
the presentation.
• Projects that supported this work:
– Wi-5 H2020 Project (g.a. 644262)
– European Social Fund in collaboration with the
Government of Aragon.
76
This work has been partially
ﬁnanced by the EU H2020 Wi-5
project, http://www.wi5.eu/
(Grant Agreement no: 644262)

Source: mmodata.org
Subscription based model in decline
(MMORPGs)
A player in the USA:
$15 a month
12 months
$180 a year

Game network traffic - global trends
• Global game traffic
– Very small share of the global volume
– 22% CAGR (Compounded Annual Growth Rate)
10.01.2014. 80

Server organization
• Small virtual worlds
• Usually less then 100 players per map
• Servers hosted by players (lower complexity)
• Very densely geographically distributed (to maximally reduce
network delay)
81

CDF’s of different FPS games
X. Che and B. Ip, “Review: Packet-level traffic analysis of online games from the genre
characteristics perspective”, Journal of Network Computing Appl. 35, 240–252 (2012) 82

World ofTanks
• Effect of player’s death on downlink
83
0
5
10
15
20
25
30
35
0 100 200 300 400 500 600 700
Bandwidth[kbps]
game time [s]
Server-to-client bandwidth
Bandwidth
Tank destroyed

World ofTanks
• No effect of death on uplink
84
0
5
10
15
20
25
0 100 200 300 400 500 600 700
Bandwidth[kbps]
game time [s]
Client-to-server bandwidth
Bandwidth
Tank destroyed

Massively Multiplayer Role-Playing
Games (MMORPGs)
• Gameplay characteristics
– Wide range of possible activities
– Very large virtual worlds
– Virtual economies
– Large number of players in same virtual world (up to tens of
thousands)
• Traffic characteristics
– Much more variable traffic characteristics
– Less fault tolerance
– TCP and UDP
– Looser latency constraints
– Lower packet rate
– Lower bandwidth usage
85

MMORPG - examples
RuneScape
World of WarcraftEverQuest
EVE Online

MMORPG architecture: challenges
• Massively Multiplayer Online Role-Playing Games
– A large number of players which share one virtual world
– WoW – 12 million players (at the peak of popularity)
• Main issues:
– Calculation of the virtual world state
– Consistency
– Cheating avoidance
– Scalability (all servers need to be under control of the provider)
• Two solutions:
– Single space worlds (using huge server farms e.g., EvE Online)
– “Sharding” of virtual word into multiple replicas of the virtual world
across which the players are distributed
• In recent years overlay systems are created over shards (e.g.,World of
Warcraft)

Transport protocols
• Which protocolTCP or UDP?
– Depending on the game genre and mechanic
Protocol MMORPGs
TCP World of Warcraft, Lineage I/II, Guild Wars,
Ragnarok Online, Anarchy Online, Mabinogi
UDP EverQuest, City of Heroes, Star Wars Galaxies,
Ultima Online, Asherons Call, Final Fantasy XI
TCP/
UDP
Dark Age of Camelot

MMORPGs andTCP
• TCP not designed for a real time interactive
application!!! (yet it works)
• Application limited not network limited flows
• Multiple thinTCP flows behave unlike one fatTCP flow
• Mechanisms inTCP directly deteriorate the experience
of the players (delayed ACK, Nagle algorithm)
• Mechanisms ofTCP do not work efficiently for
MMORPG (cwnd reduced due to application not
having something to send)
• High signaling overhead due to small packets
• High number of “pure” ACKS
89

CDF’s of different MMORPGs
X. Che and B. Ip, “Review: Packet-level traffic analysis of online games from the genre
characteristics perspective”, Journal of Network Computing Appl. 35, 240–252 (2012)
90

Impact of MMORPG actions on
network traffic
• Use caseWorld ofWarcraft
• Bandwidth difference – up to 5 times
92
M. Suznjevic, O. Dobrijevic, and M. Matijasevic, "MMORPG Player Actions: Network Performance, Session Patterns and Latency
Requirements Analysis," Multimedia Tools and Applications, vol. 45 no. 1-3, pp. 191-214, 2009.

RealTime Strategies (RTS)
• Usually omnipresent perspective
• Two major components
– Development
– Fighting
• Smaller scale in multiplayer commonly < 10
players in a match
• Recently a sub-genre of RTS games has
increased in popularity – Action RTS (or
MultiplayerOnline BattleArena – MOBA)
93

Game network engines
• Synchronization of the game state between participating
players
• Starcraft 2 uses the “simulation model”
– P2P in a client server model!
– No central authority (server does NOT hold the game state)
– Game completely deterministic – same inputs should yield the
same results
– Every player’s command is queued up to be done at some point
in the future (typically, around 12 frames i. e. 200ms).
– Every player sends the inputs to other players (through the
server)
– Once all inputs are received game tick is calculated on client
side
94

Pros and cons
• Pros:
–Synchronization of only a few commands
instead of positions of thousands of units
– Very low bandwidth usage
• Cons
– Observable input delay (i.e., “lag”) – units do not
respond immediately
– Slowest player slowing down the game for all
– Possible desynchronization – end of a match
95

Online games: a real-time problem for the network