You are not alone - Scaling multiplayer games

6,652 views

Published on

At Wooga we are creating the next generation of social games. To be truly social, a game needs to offer real interaction between players in different forms.

Taking one of our upcoming games as an example we will present the state synchronization used in client and server, how we do real time communication without websockets and how we are making a backend scalable enough to support millions of users every day. We will also share secrets about our prototypes for first person and real time strategy games.

The Erlang process model with evented IO, asynchronous messaging, links and isolation fits very well to real time multiplayer games. Come to our talk to see how it's used to build a very successful business.

Published in: Technology
0 Comments
8 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,652
On SlideShare
0
From Embeds
0
Number of Embeds
2,706
Actions
Shares
0
Downloads
66
Comments
0
Likes
8
Embeds 0
No embeds

No notes for slide

You are not alone - Scaling multiplayer games

  1. 1. You are not aloneScaling Multiplayer GamesJesper Richter-Reichhelm - @jrireiKnut Nesheim - @knutin
  2. 2. > 1,000,000 daily users
  3. 3. > 1,000,000 daily users> 50,000 concurrent users
  4. 4. > 1,000,000 daily users> 50,000 concurrent users> 10,000 requests / second
  5. 5. Ok, you canhandle that.
  6. 6. Ok, you canhandle that.But why onlysingle player?
  7. 7. The Games The Past The Present The FutureThe Foundation
  8. 8. Server Database
  9. 9. Server Database
  10. 10. Server Database
  11. 11. Server Database
  12. 12. Server Database
  13. 13. Server Database
  14. 14. DB is the bottleneck
  15. 15. Server Database
  16. 16. Server Database
  17. 17. Server Database
  18. 18. Server Database One  Game  Session
  19. 19. Server Database One  Game  Session
  20. 20. session
  21. 21. session session sessionsession
  22. 22. Server session session sessionsession
  23. 23. Server session session sessionsession
  24. 24. Server session session sessionsession
  25. 25. Server session session sessionsession
  26. 26. Server Server Server session session session session session session sessionsession session session session session
  27. 27. The Games The Past The Present The FutureThe Foundation
  28. 28. es Thoughts t Clientent Server re Operationation
  29. 29. es Thoughts t Clientent Server re Operationation
  30. 30. You are not alone
  31. 31. Pessimistic
  32. 32. PessimisticSchizophrenic
  33. 33. PessimisticSchizophrenic Optimistic
  34. 34. Pessimistic
  35. 35. Client Server A AClient Server B B
  36. 36. Client ServerA  @  B AClient Server B B
  37. 37. Client ServerA  @  B AClient Server B B
  38. 38. Client ServerA  @  B AClient Server B B
  39. 39. Client ServerA  @  B AClient Server B B’
  40. 40. Client ServerA  @  B AClient Server B B’
  41. 41. Client ServerA  @  B AClient Server B B’
  42. 42. Client ServerA  @  B’ AClient Server B B’
  43. 43. Client ServerA  @  B’ AClient Server B B’
  44. 44. Client ServerA  @  B’ AClient Server B’ B’
  45. 45. Waiting on server responses?
  46. 46. Waiting on server responses? Are you out of your mind?
  47. 47. Schizophrenic
  48. 48. Client Server A AClient Server B B
  49. 49. Client ServerA  @  B AClient Server B B
  50. 50. Client ServerA  @  B A  +  BClient Server B B
  51. 51. Client ServerA  @  B’ A  +  BClient Server B B
  52. 52. Client ServerA  @  B’ A  +  BClient Server B B
  53. 53. Client ServerA  @  B’ A  +  B’Client Server B B
  54. 54. Client ServerA  @  B’ A  +  B’Client Server B B
  55. 55. Client ServerA  @  B’ A  +  B’Client Server B B
  56. 56. Client ServerA  @  B’ A  +  B’Client Server B’ B
  57. 57. Client ServerA  @  B’ A  +  B’Client Server B’ B
  58. 58. Client ServerA  @  B’ A  +  B’Client Server B’ B’
  59. 59. What about real interactions?
  60. 60. What about real interactions?And what about complexityin the backend?
  61. 61. Optimistic
  62. 62. Client Server A AClient Server B B
  63. 63. Client ServerA  @  B AClient Server B B
  64. 64. Client ServerA  @  B’ AClient Server B B
  65. 65. Client ServerA  @  B’ AClient Server B B
  66. 66. Client ServerA  @  B’ AClient Server B B
  67. 67. Client ServerA  @  B’ AClient Server B B’
  68. 68. Client ServerA  @  B’ AClient Server B B’
  69. 69. Client ServerA  @  B’ AClient Server B B’
  70. 70. Client ServerA  @  B’ AClient Server B’’ B’
  71. 71. Client ServerA  @  B’ AClient Server B’’ B’
  72. 72. Client ServerA  @  B’ AClient Server B’’ B’
  73. 73. Client ServerA  @  B’ AClient Server B’ B’
  74. 74. Hmm, client rollbacks?
  75. 75. Hmm, client rollbacks?Are you sure you can build that?
  76. 76. es tent reation
  77. 77. es Thoughts t Clientent Server re Operationation
  78. 78. Client Server A AClient Server B B
  79. 79. Client  A
  80. 80. statecount  0 max  3 Client  A
  81. 81. user  A move     statecount  0 max  3 Client  A
  82. 82. user  A move pick   statecount  0 max  3 Client  A
  83. 83. user  A move pick   statecount  1 max  3 Client  A
  84. 84. user  A move pick   statecount  1 max  3 Client  A Server
  85. 85. user  A user  A move pick   statecount  1 max  3 Client  A Server
  86. 86. user  A user  A move pick loca,on   state count  0count  1 max  3 Client  A Server
  87. 87. user  A user  A move pick loca,on   state count  1count  1 max  3 Client  A Server
  88. 88. user  A user  A move pick loca,on   state count  1count  1 max  3 Client  A Server
  89. 89. user  A user  A move pick loca,on   state count  1count  1 max  3 history move  A pick  A   Client  A Server
  90. 90. user  A user  A move pick   loca,on   state count  1count  1 max  3 history move  A pick  A     Client  A Server
  91. 91. user  A user  A move pick   loca,on   state count  1count  1 max  3 history move  A pick  A     Client  A Server
  92. 92. user  A user  A move pick   loca,on   state count  1count  1 max  3 user  B history move  A pick  A     Client  A Server
  93. 93. user  A user  A move pick   loca,on   state count  1count  1 max  3 user  B history move  A pick  A   loca,on   Client  A Server
  94. 94. user  A user  A move pick   loca,on   state count  2count  1 max  3 user  B history move  A pick  A   loca,on   Client  A Server
  95. 95. user  A user  A move pick   loca,on   state count  2count  1 max  3 user  B history move  A pick  A   loca,on   Client  A Server
  96. 96. user  A user  A move pick   loca,on   state count  2count  1 max  3 user  B history move  A pick  A pick  B loca,on   Client  A Server
  97. 97. user  A user  A move pick   loca,on   state count  2count  1 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  98. 98. user  A user  A move pick   loca,on   state count  2count  2 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  99. 99. user  A user  A move pick   loca,on   state count  2count  2 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  100. 100. user  A user  A move pick pick loca,on   state count  2count  2 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  101. 101. user  A user  A move pick pick loca,on   state count  2count  3 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  102. 102. user  A user  A move pick pick loca,on   state count  2count  3 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  103. 103. user  A user  A move pick pick loca,on   state count  2count  3 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  104. 104. user  A user  A move pick pick loca,on   state count  3count  3 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  105. 105. user  A user  A move pick pick loca,on   state count  3count  3 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  106. 106. user  A user  A move pick pick loca,on   state count  3count  3 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on     Client  A Server
  107. 107. user  A user  A move pick pick loca,on   state count  3count  3 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on   pick  B   Client  A Server
  108. 108. user  A user  A move pick pick loca,on   state count  3count  3 max  3 user  B user  B history pick move  A pick   pick  A pick  B loca,on   pick  B   Client  A Server
  109. 109. user  A user  A move pick pick loca,on   state count  3count  3 max  3 user  B user  B history pick move  A pick   pick  A pick  B loca,on   pick  B   Client  A Server
  110. 110. user  A user  A move pick pick loca,on   state count  3count  3 max  3 user  B user  B history pick move  A pick   pick  A pick  B loca,on   pick  B   Client  A Server
  111. 111. user  A user  A move pick pick loca,on   state count  3count  3 max  3 user  B user  B history pick move  A pick   pick  A pick  B loca,on   pick  B   Client  A Server
  112. 112. user  A user  A       loca,on   state count  3count  0 max  3 user  B user  B history   move  A   pick  A pick  B loca,on   pick  B Client  A Server
  113. 113. user  A user  A move     loca,on   state count  3count  0 max  3 user  B user  B history   move  A   pick  A pick  B loca,on   pick  B Client  A Server
  114. 114. user  A user  A move pick   loca,on   state count  3count  1 max  3 user  B user  B history   move  A   pick  A pick  B loca,on   pick  B Client  A Server
  115. 115. user  A user  A move pick   loca,on   state count  3count  2 max  3 user  B user  B history pick move  A   pick  A pick  B loca,on   pick  B Client  A Server
  116. 116. user  A user  A move pick   loca,on   state count  3count  3 max  3 user  B user  B history pick move  A pick pick  A pick  B loca,on   pick  B Client  A Server
  117. 117. ...(Let’s look inside)
  118. 118. es tent reation
  119. 119. es Thoughts t Clientent Server re Operationation
  120. 120. Erlang
  121. 121. ErlangProcess isolation
  122. 122. ErlangProcess isolation Networking
  123. 123. ErlangProcess isolation Networking Distribution
  124. 124. User process
  125. 125. User processStores user state
  126. 126. User process Stores user stateReal-time push to client
  127. 127. World process
  128. 128. World processSerializes updates
  129. 129. World process Serializes updatesConcurrency “control”
  130. 130. World process Serializes updatesConcurrency “control” Runs game logic
  131. 131. Game logic
  132. 132. Game logic Pure
  133. 133. Game logic Puref(State, Cmd) -> {NewState, Reply}
  134. 134. Game logic Puref(State, Cmd) -> {NewState, Reply} Transactions
  135. 135. Real-time push
  136. 136. Real-time pushBrowser support
  137. 137. Real-time push Browser support“Transfer-Encoding: chunked”
  138. 138. Real-time push Browser support“Transfer-Encoding: chunked”State updates as JSON
  139. 139. es tent reation
  140. 140. es Thoughts t Clientent Server re Operationation
  141. 141. Elli
  142. 142. ElliEfficient, robust
  143. 143. Elli Efficient, robustHigh-throughput, low latency
  144. 144. Elli Efficient, robustHigh-throughput, low latency Request-response model
  145. 145. Elli Efficient, robustHigh-throughput, low latency Request-response model github.com/knutin/elli
  146. 146. Distribution
  147. 147. DistributionIdea: Identical nodes, Riak
  148. 148. Distribution Idea: Identical nodes, RiakNo centralization/serialization
  149. 149. Distribution Idea: Identical nodes, RiakNo centralization/serialization No SPOF
  150. 150. Distribution needs
  151. 151. Distribution needs Locking
  152. 152. Distribution needs Locking Dynamic cluster
  153. 153. Meet “locker”
  154. 154. Meet “locker”Distributed lock table
  155. 155. Meet “locker”Distributed lock table CAS
  156. 156. Meet “locker”Distributed lock table CASWrite quorum, 2PC
  157. 157. Meet “locker” Distributed lock table CAS Write quorum, 2PCAsynchronous replication
  158. 158. Meet “locker” Distributed lock table CAS Write quorum, 2PCAsynchronous replication330 SLOC, one gen_server
  159. 159. The Games The Past The Present The FutureThe Foundation
  160. 160. Research
  161. 161. ResearchAka “hacking”
  162. 162. Everything is possible
  163. 163. Everything is possible First person
  164. 164. Everything is possible First person Real-time strategy
  165. 165. Everything is possible First person Real-time strategy Turn-based
  166. 166. Everything is possible First person Real-time strategy Turn-based Etc
  167. 167. Nothing is possible
  168. 168. Nothing is possibleNetwork latency kills gameplay
  169. 169. Lag compensation
  170. 170. Lag compensation Many solutions
  171. 171. Lag compensation Many solutions Depends on game
  172. 172. FPS
  173. 173. FPSSpeed is king
  174. 174. FPS Speed is kingDivergent simulations
  175. 175. FPS Speed is kingDivergent simulationsServer keeps history
  176. 176. FPS Speed is king Divergent simulations Server keeps historyValidates delayed commands
  177. 177. RTS
  178. 178. RTSSynchronization is king
  179. 179. RTSSynchronization is king Identical simulations
  180. 180. RTS Synchronization is king Identical simulationsServer buffers commands
  181. 181. RTS Synchronization is king Identical simulationsServer buffers commands Broadcasts, lockstep
  182. 182. The Games The Past The Present The FutureThe Foundation
  183. 183. Small independent teams
  184. 184. Each team adds something
  185. 185. Based on previous learnings
  186. 186. Standing on theshoulders of giants
  187. 187. Slides: http://woo.ga/backend
  188. 188. Slides: http://woo.ga/backendCode: http://github.com/wooga
  189. 189. Questions?
  190. 190. Questions? @jrirei @knutinhttp://wooga.com/jobs

×