The Effect of Router Buffer Size on Subjective Gaming Quality Estimators Based on Delay and Jitter

THE EFFECT OF ROUTER
BUFFER SIZE ON SUBJECTIVE
GAMING QUALITY ESTIMATORS
BASED ON DELAY AND JITTER
GTC

Communication
Technologies Group

Jose Saldana
Julián Fernández-Navajas
José Ruiz-Mas
Eduardo Viruete Navarro
Luis Casadesus
University of Zaragoza, Spain

Index
-

I. Introduction
II. Related Works
III. Tests and Results
IV. Conclusions

Introduction
- The Internet was not designed
for real-time services.
- First deployed ones: e-mail, file
transfer

CCNC 2012. DENVECT Workshop. Las Vegas Jan 14, 2012

Introduction
- But real-time services are being
widely used: VoIP, video
conference, online gaming


Introduction
- Problem: Using a best effort
network for a real-time service.
- Users demand a quality similar to
the one they are used to.
- Research: Find the relationship
between network impairments and
perceived quality.
-

Delay, packet loss, bandwidth, jitter.


Introduction
- E-Model: ITU’s solution that
estimates perceived quality of voice,
as a function of delay, packet loss,
codec, etc.
- Battery of surveys in order to obtain
a MOS (Mean Opinion Score) model


FPS online games
- Very stringent Real-time
requirements:
- Interactivity (video)
- Players: Difficult to satisfy


FPS online games
- Traffic characteristics
-

-

UDP
Small packets (100 bytes maximum)
High frequency (25 to 85 pps)

A universal MOS does not exist
-

Some games are more sensitive to delay, or
packet loss, or jitter, etc.


II. Related Works
- Buffer sizing problem
- Measuring subjective quality for
online games

Buffer sizing problem
- Players mainly use access networks
- We are considering low-end routers,
i.e. the ones we can find in access
networks
-

Drop-tail FIFO tiny buffers, of some tens of
kilobytes


Buffer sizing problem
- Different proposals:
- Rule of the thumb: C x RTT
-

Maximize buffer occupancy
Too much delay

- Stanford model: C x RTT / sqrt(N)
- Tiny buffer: 20 to 50 packets


Buffer sizing trade-off
Bigger
delays

Bigger
Buffer

Bigger
jitter

Smaller
delays

Smaller
Buffer

Smaller
packet
loss


Smaller
jitter

Bigger
packet
loss

Subjective quality for FPS
- MOS as a function of SRT (linear dependence)
C. Schaefer, T. Enderes, H. Ritter, M. Zitterbart. “Subjective quality assessment
for multiplayer real-time games”. In Proc 1st workshop on Network and system
support for games (NetGames '02). ACM, New York, NY, USA, 74-78. 2002.

- Influence of delay and packet loss (not
developing a MOS)
S. Zander, G. Armitage, “Empirically Measuring the QoS Sensitivity of Interactive
Online Game Players”. In Proc. Australian Telecommunications Networks &
Applications Conference (ATNAC 2004), Sydney, Australia, Dec. 2004.

-

Packet loss: different behavior:
-

Halo: Does not work with 4% loss
Quake III: Works with 35% loss


Subjective quality for FPS
- Only delay and jitter, but separately
M. Dick, O. Wellnitz, L. Wolf. “Analysis of factors affecting players' performance
and perception in multiplayer games”. In Proc. 4th ACM SIGCOMM workshop on
Network and system support for games (NetGames '05). ACM, New York, NY,
USA, 1-7, 2005.

- First developed MOS, adapted from E-model
A. F. Wattimena, R. E. Kooij, J. M. van Vugt, O. K. Ahmed, “Predicting the
perceived quality of a first person shooter: the Quake IV G-model”. In Proc. 5th
SIGCOMM workshop Network and system support for games (NetGames '06),
ACM, New York, NY, USA, 2006.

-

Only delay and jitter were considered
Game: Quake IV
G-model: We will use it to study quality


Previous results with VoIP
- E-Model uses delay and packet
loss.
- Jitter is not considered, as a dejitter buffer is used


Previous results for VoIP
-

MOS presents a monotonically decreasing
behavior with background traffic
E-Model MOS
5
4.5
4

MOS

3.5
3

2.5
1 call
5 calls
10 calls
15 calls
20 calls

2
1.5
1
400

450

500

550

600

650

700

750

800

background traffic (kbps)


850

900

950

1000

Scenarios of interest
- A number of players connected
to the server sharing the same
router
buffer

Internet
.
.
.

Router
Game &
background
traffic

Users


Game Server

Test methodology
- The same scenario as in VoIP
- FPS game with a MOS: Quake IV
- Traffic traces available from CAIA
project
- Only client-to-server (the most
restrictive one)


Test methodology
- Inter-packet time and packet
size histograms
-

40.7 kbps / user

40 50 60 70 80 90 100 110
bytes

0

10 20 30 40 50 60 70
ms

79.5 bytes avg

64 pps

Test methodology
- 20 players sharing an Internet
connection
- Buffer: Drop-tail, fixed kB size
Bandwidth
Buffer size
2 Mbps
3 Mbps

10 kB
20 kB
50 kB
100 kB


Test methodology
-

Background traffic sharing the
connection
-

-

50% packets 40 bytes
10% packets 576 bytes
40% packets 1,500 bytes

Network RTT (avg 30 ms) added offline
buffer
Internet
.
.
.

Router
Game &
background
traffic

Users


Game Server

Test results: delay
RTT, 2 Mbps
450
10 kB
20 kB

400

50 kB
100 kB

350
300

ms

250
200
150
100
50
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: delay
RTT, 2 Mbps
450
10 kB
20 kB

400

50 kB
100 kB

350
300

ms

250
200

Bandwidth limit
40.7 x 20 = 814 kbps

150
100
50
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: delay
RTT, 2 Mbps
450
10 kB
20 kB

400

50 kB

Unacceptable delay

100 kB

350
300

ms

250
200
150
100
50
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: delay
RTT, 3 Mbps
450
10 kB
20 kB

400

50 kB
100 kB

350
300

ms

250
200
150
100
50
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: delay
RTT, 3 Mbps
450
10 kB
20 kB

400

50 kB
100 kB

350
300

ms

250
200

Bandwidth limit
40.7 x 20 = 814 kbps

150
100
50
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: delay
RTT, 3 Mbps
450
10 kB
20 kB

400

50 kB

Unacceptable delay

100 kB

350
300

ms

250
200
150
100
50
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: jitter
Jitter (delay standard deviation), 2 Mbps
110
10kB
100

20 kB
50 kB

90

100kB

80

ms

70

60
50
40
30
20
10
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


110
10kB
100

20 kB
50 kB

90

Above the bandwidth
limit, the jitter
decreases, as the
buffer is always full

100kB

80

ms

70

60
50

Bandwidth limit
40.7 x 20 = 814 kbps

40
30
20
10
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


110
10kB
100

20 kB
50 kB

90

100kB

80

ms

70

60
50
40
30
20
10
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


110
10kB
100

20 kB

The peak is reduced
when bandwidth grows

50 kB

90

100kB

80

ms

70

60
50
40
30
20
10
0
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: packet loss
Packet Loss, buffer 10kB, 2 Mbps
45%
game (79 bytes avg)
40 bytes

40%

576 bytes
1500 bytes

35%
30%
25%
20%
15%
10%
5%
0%
0

200

400

600

800

1000

1200

1400

1600

1800



2000

2200

2400

2600

2800

3000

45%
game (79 bytes avg)
40 bytes

40%

576 bytes
1500 bytes

35%
30%

Small packets have a clear
advantage in a drop-tail
size limited buffer

25%
20%
15%
10%
5%
0%
0

200

400

600

800

1000

1200

1400

1600

1800



2000

2200

2400

2600

2800

3000

45%
game (79 bytes avg)
40 bytes

40%

576 bytes
1500 bytes

35%
30%
25%
20%
15%
10%
5%
0%
0

200

400

600

800

1000

1200

1400

1600

1800



2000

2200

2400

2600

2800

3000

Test results: MOS
-

G-Model MOS formula:
x = 0.104*ping_average + jitter_average

MOS = -0.00000587 x 3 + 0.00139 x2- 0.114 x + 4.37

-

Packet loss is not considered, unless it is
above 35%


Test results: MOS
G-Model MOS, 2 Mbps
5

4.5
4

MOS

3.5
3
2.5
2

10 kB
20 kB

1.5

50 kB
100 kB

1
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: MOS
G-Model MOS, 2 Mbps
5

4.5

Good quality
4

MOS

3.5

Medium quality
3
2.5
2

Bad quality

10 kB
20 kB

1.5

50 kB
100 kB

1
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: MOS
G-Model MOS, 2 Mbps
5

Delay and jitter
grow

4.5

Delay grows, but
jitter decreases
Not monotonically
decreasing

4

MOS

3.5
3
2.5
2

10 kB
20 kB

1.5

50 kB
100 kB

1
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: MOS
G-Model MOS, 2 Mbps
5

Small buffers
present the best
behavior

4.5
4

MOS

3.5
3
2.5
2

10 kB
20 kB

1.5

50 kB
100 kB

1
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: MOS
G-Model MOS, 3 Mbps
5
4.5
4

MOS

3.5
3
2.5
2

10 kB

20 kB
1.5

50 kB
100 kB

1
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: MOS
G-Model MOS, 3 Mbps
5
4.5

Good quality
4

MOS

3.5

Medium quality
3
2.5
2

Bad quality

10 kB

20 kB
1.5

50 kB
100 kB

1
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Test results: MOS
G-Model MOS, 3 Mbps
5

Acceptable MOS
values above
bandwidth limit

4.5
4

MOS

3.5
3
2.5
2

10 kB

20 kB
1.5

50 kB
100 kB

1
0

200

400

600

800

1000 1200 1400 1600 1800 2000 2200 2400 2600 2800 3000


Conclusions
-

Access networks with low-end routers
Importance of the buffer size
Small buffers are better for real-time apps
Buffer implementation can penalyze big
packets
We cannot separately study each network
impairment
Need for subjective quality estimators to
calculate MOS
If delay and jitter are the considered
impairments, the jitter peak produces a MOS
valley


THANK YOU

GTC

Communication
Technologies Group

jsaldana@unizar.es

Online games: genres

Real-time strategy

Sports

MMORPG

FPS


FPS online games
- Delay: Very important


The Effect of Router Buffer Size on Subjective Gaming Quality Estimators Based on Delay and Jitter

Recommended

Recommended

More Related Content

Similar to The Effect of Router Buffer Size on Subjective Gaming Quality Estimators Based on Delay and Jitter

Similar to The Effect of Router Buffer Size on Subjective Gaming Quality Estimators Based on Delay and Jitter (20)

More from Jose Saldana

More from Jose Saldana (20)

Recently uploaded

Recently uploaded (20)

The Effect of Router Buffer Size on Subjective Gaming Quality Estimators Based on Delay and Jitter