1. Akka for real-time Multiplayer
mobile games

2. Yan Cui
http://theburningmonk.com
@theburningmonk
Principal Engineer @

5. Real-Time games in Top 100 Grossing (2017)
2014
2015
2016
2017
(3)
(6)
(8)
(13)
2018 ???

6. Enabling Factors
Source: PC Mag

7. Enabling Factors
Source: OpenSignal

8. Enabling Factors

9. > $400M Monthly Revenue
Source: Bloomberg
> 80M DAU
Source: Tencent

11. 10-20 inputs/s, sensitive to lags (> 300ms)

12. unpredictable network, limited bandwidth

14. Decisions, decisions...
Build vs Buy?
Global deployment vs Centralized?
TCP vs UDP?
Server Authoritative vs Lock-Step?

15. Constraints/Trade-offs
Latency (RTT)
Cost
Complexity
Scalability
Operational overhead

16. Global Deployment
vs
Centralised

19. 10-20 inputs/s, sensitive to lags (> 300ms)

22. optimize for this

23. Global Deployment
! Players are geo-routed to closest multiplayer server.
! Matched with other players in the same geo-region for best UX.
! No need for players to “choose server”, it should just work.

24. Global Deployment
! Should leaderboards be global or regional?
! Should guilds/alliances be global or regional?
! Should chatrooms be global or regional?
! Should liveops events be global or regional?
! Should players be allowed to play with others in another region? ie.
play with distant relatives/friends.
! Should players be allowed to switch default region?
eg. moved to Europe after Brexit

25. Server Authoritative
vs
Lock-Step

26. Server Authoritative
! Server decides game logic.
! Client sends all inputs to server.
! Client receives game state (either full, or delta) from server.

27. Server Authoritative
! Server decides game logic.
! Client sends all inputs to server.
! Client receives game state (either full, or delta) from server.
! Client keeps internal state for game world, which mirrors server state.
! Client doesn’t modify world state directly, only display with some prediction to
mask network latency.

28. Client 1 Client 2Server
C1 control 1 C2 control 1
game state 1

29. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
game state 1
game state 2

30. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
game state 1
game state 2

31. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
game state 1
game state 2
game state 3
C1 control 1
C2 control 1
C2 control 2
game state 3
C1 control 1
C2 control 1
C2 control 2
C2 control 3

32. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
C2 control 3
game state 1
game state 2
game state 3
C1 control 1
C2 control 1
C2 control 2
game state 3
C1 control 1
C2 control 1
C2 control 2
game state 4

33. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
C2 control 3
game state 1
game state 2
game state 3
C1 control 1
C2 control 1
C2 control 2
game state 3
C1 control 1
C2 control 1
C2 control 2
game state 4

34. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
C2 control 3
game state 1
game state 2
game state 3
C1 control 1
C2 control 1
C2 control 2
game state 3
C1 control 1
C2 control 1
C2 control 2
game state 5
C2 control 3
game state 4

35. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
C2 control 3
game state 1
game state 2
game state 3
C1 control 1
C2 control 1
C2 control 2
game state 3
C1 control 1
C2 control 1
C2 control 2
game state 5
C2 control 3
game state 4

37. Pros
! Always in-sync.
! Hard to cheat - no memory hacks, etc.
! Easy (and quick) to join mid-match.
! Server can detect lagged/DC’d client and take over with AI.

38. Cons
! High server load.
! High bandwidth usage.
! Synchronization on the client is complicated.
! Little experience in the company with server-side .Net stack.
(bus factor of 1)
! .NetCore was/is still a moving target.

39. high server load and
bandwidth needs
client has to receive
more data

40. Lock-Step*
! Client sends all inputs to server.
! Server collects all inputs, and buffers them.
! Server sends all buffered inputs to all clients X times a second.
* traditional RTS games tend to use peer-to-peer model

41. Lock-Step*
! Client sends all inputs to server.
! Server collects all inputs, and buffers them.
! Server sends all buffered inputs to all clients X times a second.
! Client executes all inputs in the same order.
! Because everyone is 'guaranteed' to have executed the same input at the
same frame in the same order, we get synchronicity.
! Use prediction to mask network latency.
* traditional RTS games tend to use peer-to-peer model

42. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
C2 control 3
C1 control 1
C2 control 1
C2 control 2
C1 control 1
C2 control 1
C2 control 2
C2 control 3
inputs, instead
of game state

43. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
C2 control 3
C1 control 1
C2 control 1
C2 control 2
C1 control 1
C2 control 1
C2 control 2
C2 control 3
RTT: time between sending an input to
receiving it back from server

44. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
C2 control 3
C1 control 1
C2 control 1
C2 control 2
C1 control 1
C2 control 1
C2 control 2
C2 control 3

45. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
C2 control 3
C1 control 1
C2 control 1
C2 control 2
C1 control 1
C2 control 1
C2 control 2
C2 control 3
RTT
frame time
latency

46. Client 1 Client 2Server
C1 control 1 C2 control 1
C2 control 2
C2 control 3
C1 control 1
C2 control 1
C2 control 2
C1 control 1
C2 control 1
C2 control 2
C2 control 3
RTT
frame time
RTT = latency x 2 + X
Xmin = 0, Xmax = frame time
latency

48. Pros
! Light server load.
! Lower bandwidth usage.
! Simpler server implementation.

49. Cons
! Needs deterministic game engine.
! Unity has long-standing determinism problem with floating point.
! Hackable, requires some form of server-side validation.
! All clients must take over lagged/DC’d client with AI.
! Slower to join mid-match, need to process all inputs.
! Need to ensure all clients in a match are compatible.

50. fix-point math, server
validation, ...

53. bandwidth

54. Build vs Buy

56. ApeSync

57. +

58. MATCH 1
C1 input
C2 input
current frame history
frame 1
frame 2
frame 3
buffering

59. connection open
MATCH 1
C1 input
C2 input
current frame history
frame 1
frame 2
frame 3
buffering

60. connection open
authenticate
MATCH 1
C1 input
C2 input
current frame history
frame 1
frame 2
frame 3C3 joined
buffering

61. connection open
authenticate
send/receive
MATCH 1
C1 input
C2 input
current frame history
frame 1
frame 2
frame 3C3 joined
buffering

62. MATCH 1
C1 input
C2 input
current frame history
frame 1
frame 2
frame 3C3 joined
C3 input
connection open
authenticate
send/receive
buffering

63. MATCH 1
C1 input
C2 input
current frame history
frame 1
frame 2
frame 3C3 joined
C3 input
connection open
authenticate
send/receive
buffering
broadcast!

64. MATCH 1
current frame history
frame 1
frame 2
frame 3
C1 input
C2 input
C3 joined
C3 input
connection open
authenticate
send/receive
buffering
broadcast!

65. MATCH 1
current frame history
frame 1
frame 2
frame 3
frame 4
connection open
authenticate
send/receive
buffering
broadcast!

66. MATCH 1
current frame history
frame 1
frame 2
frame 3
frame 4
connection open
authenticate
send/receive
buffering
broadcast!
C3 input

68. concurrency

69. MATCH 1
current frame history
frame 1
frame 2
frame 3
...
C1 input
C2 input
C3 joined
C3 input
connection open
authenticate
send/receive
C1 input
buffering
broadcast!

70. MATCH 1
current frame history
frame 1
frame 2
frame 3
...
C1 input
C2 input
C3 joined
C3 input
C1 input
C2 input
buffering
broadcast!

71. MATCH MATCH MATCH MATCH MATCH

72. MATCH MATCH MATCH MATCH MATCH
MATCH MATCH MATCH MATCH MATCH

73. MATCH MATCH MATCH MATCH MATCH
MATCH MATCH MATCH MATCH MATCH
MATCH MATCH MATCH MATCH MATCH

74. MATCH
C1 input
C2 input
current frame history
frame 1
frame 2
frame 3C3 joined
connection open
authenticate
send/receive

75. MATCH
C1 input
C2 input
current frame history
frame 1
frame 2
frame 3C3 joined
connection open
authenticate
send/receive

76. MATCH
current frame history
frame 1
frame 2
frame 3
C1 input
C2 input
C3 joined
Socket
actor
Match
actor

77. MATCH
current frame history
frame 1
frame 2
frame 3
C1 input
C2 input
C3 joined
Root Aggregate
Socket
actor
Match
actor

78. MATCH
current frame history
frame 1
frame 2
frame 3
C1 input
C2 input
C3 joined
Root Aggregate
Socket
actor
Match
actor

79. MATCH
current frame history
frame 1
frame 2
frame 3
C1 input
C2 input
C3 joined

80. MATCH
current frame history
frame 1
frame 2
frame 3
C1 input
C2 input
C3 joined
C3 joined
act locally
think globally
how actors interact with each other
aka, the “protocol”

82. the secret to building high
performance systems is simplicity
complexity kills performance

83. Higher CCU per server
Fewer servers
Lower cost
Less operational overhead
Performance Matters

84. We should forget about small
efficiencies, say about 97% of the
time: premature optimization is the
root of all evil. Yet we should not
pass up our opportunities in that
critical 3%.
Performance Matters

85. We should forget about small
efficiencies, say about 97% of the
time: premature optimization is the
root of all evil. Yet we should not
pass up our opportunities in that
critical 3%.
Performance Matters

86. Threads are heavy OS constructs.
Each thread is allocated 1MB stack space by default.
Context Switching is expensive at scale.
Actors are cheap.
Actor system can optimise use of threads to minimise context switching.
Actor Model
>

87. Non-blocking I/O framework for JVM.
Very performant.
Simplifies implementation of socket servers (TCP/ UDP).
UDP support is “meh”...
Netty

88. Custom network protocol (bandwidth).
Minimise Netty object creations (GC pressure).
Minimise the no. of message hops inside Akka (latency).
Performance Tuning

89. Buffer pooling (GC pressure).
Using direct buffers (GC pressure).
Performance Tuning

90. Disable Nagle's algorithm (latency).
TCP tuning (including BBR, etc. with differing results).
Epoll.
ENA-enabled EC2 instances.
Performance Tuning

91. Custom Reliable UDP protocol, optimized for countries with poor networking
conditions.
Performance Tuning

92. AWS Lambda functions to run bot clients (written with Akka):
! Cheaper
! Faster to boot up
! Easy to update
Each Lambda invocation could simulate up to 100 bots.
Automated Load Testing

93. http://bit.ly/2xgGHXZ

95. from US-EAST (Lambda)
to EU-WEST (game server)

96. optimize for tail latencies
from US-EAST (Lambda)
to EU-WEST (game server)

97. Monitoring

98. Monitoring

99. Performance in the Wild(in poor networking condition)
95 percentile
max playable RTT
outperforms Photon
on performance

100. Performance in the Wild(in poor networking condition)
fewer disconnects

101. Performance in the Wild
! Improved KPIs - D1 retention, session time, etc.
! 14% cheaper vs. Photon, based on current cost projection