Painful success - lessons learned while scaling up

4,181 views
4,119 views

Published on

Full of hope we started developing our biggest games so far in 2009. In 2010 we released it and gradually scaled up to 1 million users a day. But we were forced to rewrite our whole persistence layer and migrate most of our data from MySQL to Redis to make it work. In 2011 we even got 2 million daily users, but we also had to operate 200 servers which was painful - especially when the whole data center went down. So in 2012 we wanted to make everything better and started large refactoring projects - and made everything worse.
But in 2013 everything will run smoothly and painless - promised! At least we hope so.

Published in: Technology
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,181
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
45
Comments
0
Likes
10
Embeds 0
No embeds

No notes for slide

Painful success - lessons learned while scaling up

  1. 1. PAINFUL SUCCESSJesper Richter-Reichhelm (@jrirei) / session 4853
  2. 2. Pain
  3. 3. Flash client Backend
  4. 4. 7M players / month
  5. 5. 7M players / month10K API calls / second
  6. 6. 7M players / month10K API calls / second200K DB operations / second
  7. 7. 7M players / month10K API calls / second200K DB operations / second100K DB writes / second
  8. 8. 7M players / month10K API calls / second200K DB operations / second100K DB writes / second2 devops since 2009
  9. 9. Success
  10. 10. Painful Success
  11. 11. Painful Success
  12. 12. 2009Decisions
  13. 13. What is the rightsoftware stack?
  14. 14. Browser lbapp app app SQL SQL sql sql
  15. 15. 2010Scaling Up
  16. 16. 100%# 80%# 60%# 40%# 20%# 0%# 0# 5# 10# 15# 20# 25# 30# 35# 40#
  17. 17. 100%# 80%# 60%# 40%# 20%# 0%# 0# 5# 10# 15# 20# 25# 30# 35# 40#
  18. 18. 100%# 80%# 60%# 40%# 20%# 0%# 0# 5# 10# 15# 20# 25# 30# 35# 40#
  19. 19. Small things...
  20. 20. AMF responses
  21. 21. Checking connection ‘status’
  22. 22. ActiveRecord caching
  23. 23. Lesson 1:Always check back on reality!
  24. 24. Big Things...
  25. 25. Lot’s of tuning
  26. 26. Lot’s of tuningand more memory
  27. 27. appmasterslave
  28. 28. app 1. create new DBsmaster masterslave slave
  29. 29. app 1. create new DBsmaster master 2. setup replicationslave slave
  30. 30. app 1. create new DBsmaster master 2. setup replicationslave slave
  31. 31. app 1. create new DBsmaster master 2. setup replication 3. start using mastersslave slave
  32. 32. app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. cut replicationslave slave
  33. 33. app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. cut replicationslave slave 5. remove old DBs
  34. 34. app 1. create new DBs master 2. setup replication 3. start using masters 4. cut replication slave 5. remove old DBs
  35. 35. app 1. create new DBsmaster master 2. setup replication 3. start using mastersslave slave
  36. 36. app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. remove old DBsslave slave
  37. 37. app 1. create new DBs master ? 2. setup replication 3. start using masters 4. remove old DBs slave
  38. 38. app 1. create new DBs master 2. setup replication 3. start using masters 4. remove old DBs slave 5. cut replication
  39. 39. app 1. create new DBsmaster master 2. setup replication 3. start using mastersslave slave
  40. 40. app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
  41. 41. app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
  42. 42. app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
  43. 43. app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
  44. 44. app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
  45. 45. app 1. create new DBs 2. setup replication 3. start using masters 4. drop database;
  46. 46. app 1. create new DBs 2. setup replication 3. start using masters 4. drop database; 5. Nooooooooo!
  47. 47. app 1. create new DBs 2. setup replication 3. start using masters 4. drop database; 5. Nooooooooo!
  48. 48. Lesson 2:You will make mistakes!
  49. 49. More things...
  50. 50. app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. cut replicationslave slave 5. truncate data
  51. 51. app 1. create new DBsmaster master AB AB 2. setup replication 3. start using masters 4. cut replicationslave slave AB AB 5. truncate data
  52. 52. app A B 1. create new DBsmaster master AB AB 2. setup replication 3. start using mastersslave slave AB AB
  53. 53. app A B 1. create new DBsmaster master AB AB 2. setup replication 3. start using masters 4. cut replicationslave slave AB AB
  54. 54. app A B 1. create new DBsmaster master AB AB 2. setup replication 3. start using masters 4. cut replicationslave slave AB AB 5. truncate data
  55. 55. app A B 1. create new DBsmaster master AB AB 2. setup replication 3. start using masters 4. cut replicationslave slave AB AB 5. truncate data
  56. 56. appmaster master master master AC BDslave slave slave slave AC BD
  57. 57. appmaster master master master AC BD AC BDslave slave slave slave AC BD AC BD
  58. 58. appmaster master master master AC BD AC BDslave slave slave slave AC BD AC BD
  59. 59. appmaster master master master AC BD AC BDslave slave slave slave AC BD AC BD
  60. 60. The right thing...
  61. 61. Lesson 3:Software is easy, data is hard!
  62. 62. 44 GB in 8 days24 GB
  63. 63. Not enough memory
  64. 64. Not enough memory=> no backups
  65. 65. Not enough memory=> no backups=> no new slaves
  66. 66. Not enough memory=> no backups=> no new slaves=> big problem
  67. 67. 38 GB in 3 days24 GB
  68. 68. v2.2 in ixedF 38 GB in 3 days 24 GB
  69. 69. 2011Operation Hell
  70. 70. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
  71. 71. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
  72. 72. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app SQL app app app app app EBS EBS EBS EBSSQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
  73. 73. lb app app lb app appBrowser SQL EBS lb app app lb app app
  74. 74. lb app app lb app appBrowser SQL EBS lb app app lb app app
  75. 75. lb app app lb app appBrowser SQL EBS lb app app lb app app
  76. 76. lb app app lb app appBrowser SQL EBS lb app app lb app app
  77. 77. lb app app lb app appBrowser SQL EBS lb app app lb app app
  78. 78. lb app app lb app appBrowser SQL EBS lb app app lb app app
  79. 79. lb app app lb app appBrowser SQL EBS lb app app lb app app
  80. 80. lb app app lb app appBrowser SQL EBS lb app app lb app app
  81. 81. 2,000,000"1,500,000"1,000,000" 500,000" 0" Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"
  82. 82. 2,000,000"1,500,000"1,000,000" AWS  outage 500,000" in  Ireland 0" Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"
  83. 83. http://en.wikipedia.org/wiki/File:St%C3%B6wer_Titanic.jpg
  84. 84. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
  85. 85. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
  86. 86. Installed in 3 years:
  87. 87. Installed in 3 years:98 MySQL machines
  88. 88. Installed in 3 years:98 MySQL machines195 Redis machines
  89. 89. 2012/2013Refactoring
  90. 90. Lessons learned
  91. 91. Server Database
  92. 92. Server Database
  93. 93. Server Database
  94. 94. Server Database
  95. 95. Server Database
  96. 96. Server Database
  97. 97. Server Database
  98. 98. Server Database
  99. 99. Server Database
  100. 100. Server Database One Game Session
  101. 101. Server Database One Game Session
  102. 102. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
  103. 103. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app Rd rd
  104. 104. lb lb lb lb app app app Rd rd
  105. 105. 1. Separate user and world DBs
  106. 106. 1. Separate user and world DBs2. Migrate to JRuby
  107. 107. 1. Separate user and world DBs2. Migrate to JRuby3. Introduce life cycle
  108. 108. 1. Separate user and world DBs2. Migrate to JRuby3. Introduce life cycle4. Remove (user) DBs
  109. 109. 1. Separate user and world DBs2. Migrate to JRuby3. Introduce life cycle4. Remove (user) DBs5. Enjoy!
  110. 110. Separate user DBs
  111. 111. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd
  112. 112. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd
  113. 113. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd
  114. 114. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd rd rd
  115. 115. JRuby
  116. 116. http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/
  117. 117. JRuby v2
  118. 118. Throughput200150100 50 0 Rails 2.3.11 MRI 1.8.7
  119. 119. Throughput200150100 50 0 Rails 2.3.11 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7
  120. 120. Throughput200150100 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3
  121. 121. Throughput200150100 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
  122. 122. Throughput200 a rk150 c hm ben n o100 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
  123. 123. Throughput200 a rk150 c hm ben n o100 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
  124. 124. Throughput200 a rk150 c hm ben n o100 multi threaded 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
  125. 125. Throughput200 a rk150 c hm ben tuning n o100 multi threaded 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
  126. 126. Throughput200 cpu a rk saturation150 c hm ben tuning n o100 multi threaded 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
  127. 127. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
  128. 128. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
  129. 129. NEW OLD
  130. 130. NEW OLD
  131. 131. NEW OLD
  132. 132. NEW OLD
  133. 133. Life cycle
  134. 134. app
  135. 135. app
  136. 136. servlet app
  137. 137. servlet archiver app
  138. 138. servlet archiver User app Registry
  139. 139. servlet archiver User app Registry S3
  140. 140. servlet archiver User app RegistrySQL Rd S3
  141. 141. servlet archiver User app RegistrySQL Rd S3
  142. 142. servlet archiver User app Registry WorldSQL Rd Data S3
  143. 143. Load Balancer servlet archiver User app Registry WorldSQL Rd Data S3
  144. 144. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd rd rd
  145. 145. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd rd rd rd
  146. 146. Removing DBs
  147. 147. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd rd rd rd
  148. 148. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app SQL SQL SQL SQL sql sql Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
  149. 149. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app SQL SQL SQL sql sql Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
  150. 150. Enjoy
  151. 151. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app SQL SQL SQL sql sql Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
  152. 152. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app SQL SQL sql Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
  153. 153. lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
  154. 154. http://www.flickr.com/photos/aigle_dore/
  155. 155. http://www.flickr.com/photos/aigle_dore/
  156. 156. Looking Back
  157. 157. What is the rightsoftware stack?
  158. 158. What is the rightsoftware stack?
  159. 159. What is the rightsoftware stack?
  160. 160. What kind of application are we building?
  161. 161. User
  162. 162. UserAvatar
  163. 163. User Cus-Avatar tomer
  164. 164. User Cus-Avatar Garden tomer
  165. 165. User Cus-Avatar Garden tomer Deco- Tiles rations
  166. 166. User Cus-Avatar Garden tomer Deco- Tiles rations
  167. 167. User User Cus- Cus-Avatar Garden Avatar Garden tomer tomer Deco- Deco- Tiles Tiles rations rations
  168. 168. User User Cus- Cus-Avatar Garden Avatar Garden tomer tomer Deco- Deco- Tiles Tiles rations rations
  169. 169. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
  170. 170. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
  171. 171. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
  172. 172. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
  173. 173. http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
  174. 174. What kind of application are we building?
  175. 175. Plain File System
  176. 176. Handle statethe right way!
  177. 177. Lesson 3:Software is easy, data is hard!
  178. 178. Lesson 2:You will make mistakes!
  179. 179. Lesson 1:Always check back on reality!
  180. 180. Lesson 1: Always check backon your assumptions!
  181. 181. Thank you
  182. 182. Jesper Richter-Reichhelm @jrirei woo.ga/backend wooga.com/jobs
  183. 183. Jesper Richter-Reichhelm @jrirei wooga.com/jobs woo.ga/backend

×