Downloaded 59 times
Uploaded byWooga
Games for the Masses (Jax)
The document discusses the evolution of architecture at the gaming company Wooga. It began by using Ruby on Rails with a MySQL database, which worked for low user numbers but struggled as users grew. A second team then used Redis as the main database instead of MySQL for its speed. The teams combined approaches, migrating static data to MySQL and keeping dynamic data in Redis. This allowed scaling to millions of daily users. The architecture evolved further to make servers and databases stateless to improve reliability.
More Related Content
Similar to Games for the Masses (Jax)
Games for the Masses (Jax)
- 1. GAMES FOR THE MASSES Wie DevOps die Entwicklung von Architektur verändert Jesper Richter-‐Reichhelm, @jrirei
- 2.
- 3. DEV -‐ OPS Some say: Admins learn to be ‘Agile’ • Scrum and Kanban
- 4. DEV -‐ OPS Some say: Admins learn to be ‘Agile’ • Scrum and Kanban Others say: Admins learn to program their setup • Chef, Puppet
- 5. DEV -‐ OPS Some say: Admins learn to be ‘Agile’ • Scrum and Kanban Others say: Admins learn to program their setup • Chef, Puppet I say: This is all nice, but not enough • Developers need to learn a lot about opera=on
- 6. Classic roles in IT Arch Dev Op
- 7. Classic roles in IT Arch Dev Op
- 8. Classic roles in IT Arch Dev Op
- 9. Classic roles in IT Arch Dev Op
- 10. Classic roles in IT Arch Dev Op
- 11. Classic roles in IT Arch Dev Op
- 12. Classic roles in IT Arch Dev Op
- 13. Classic roles in IT Arch Dev Op
- 14. Classic roles in IT Arch Dev Op
- 15. Classic roles in IT Arch Dev Op ???
- 17. Our games all look the same Flash client Backend
- 18. Our games all look the same Flash client Game Session Asynch. CommunicaMon
- 19. Our games all look the same Backend State Changes ValidaMon Persistence
- 20. But the scale is interesMng 14 billion requests / month
- 21. But the scale is interesMng 14 billion requests / month
- 22. But the scale is interesMng 14 billion requests / month >100,000 DB operaMons / second
- 23. But the scale is interesMng 14 billion requests / month >100,000 DB operaMons / second >50,000 DB updates / second
- 24. Classic roles in IT Arch Dev Op ???
- 25.
- 26. Roles in Wooga’s IT Arch
- 27. Roles in Wooga’s IT Arch Dev
- 28. Roles in Wooga’s IT Arch Dev Op
- 29. Roles in Wooga’s IT Arch Arch Dev Dev Op Op
- 30. 2 Developers to do it all Typical team setup 4 product managers 4 ar=sts 4 frontend engineers 2 backend engineers -‐ design, implementa7on, opera7on
- 31. Wooga has dedicated game teams
- 34.
- 35. Oct 2009 Jan 2010
- 36. Oct 2009 Jan 2010 Oct 2010
- 37. Oct 2009 Jan 2010 Oct 2010 Aug 2011
- 38. Architecture EvoluMon at Wooga The Start The Next Step Best of Two Worlds Company Values
- 39. Oct 2009: 1st team wanted good code quality Good code quality Easy to understand Easy to test Easy to refactor
- 40. Oct 2009: 1st team wanted good code quality Good code quality Easy to understand Easy to test Easy to refactor
- 41. EvoluMon I: Use Ruby (on Rails) Oct 2009
- 42. A basic setup using sharding worked fine lb app app app app app app app app app My My SQL SQL slave slave
- 43. 250K daily users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life was good !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
- 44. 250K daily users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life was good NO MORE !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
- 45. Welcome to 6 weeks of pain! Heavy opMmizaMons were necessary
- 46. Welcome to 6 weeks of pain! Heavy opMmizaMons were necessary Numerous small fixes regarding DB config
- 47. Welcome to 6 weeks of pain! Heavy opMmizaMons were necessary Numerous small fixes regarding DB config More shards
- 48. Welcome to 6 weeks of pain! Heavy opMmizaMons were necessary Numerous small fixes regarding DB config More shards Even more shards
- 49. Welcome to 6 weeks of pain! Heavy opMmizaMons were necessary Numerous small fixes regarding DB config More shards Even more shards SpliNng the model to get more shards
- 50. Early sharding hell: 8 master and 8 slaves lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave
- 51. At 500K daily users we were at a dead end &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
- 52.
- 53. Jan 2010: Meanwhile at the 2nd team Don’t break the bank Make it faster Make it cheaper Make it simpler
- 54. Jan 2010: Meanwhile at the 2nd team Don’t break the bank Make it faster Make it cheaper Make it simpler
- 55. EvoluMon II: Use Redis as main database Jan 2010 Oct 2009
- 56. If MySQL is a truck Fast enough Disk based Robust Fast enough disk based robust
- 57. If MySQL is a truck, Redis is a race car Super fast RAM based Fragile Super fast RAM based fragile
- 58. Bare metal for low latency! lb app app app app app app app Re-‐ Re-‐ dis dis disk (S3)
- 59. How could we apply that knowledge? &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
- 60. Migrate data from MySQL to Redis
- 61. Migrate data from MySQL to Redis No downMmes allowed
- 62. Migrate data from MySQL to Redis No downMmes allowed Migrate data on demand
- 63. Migrate data from MySQL to Redis No downMmes allowed Migrate data on demand On every access check if data needs migra=on
- 64. Migrate data from MySQL to Redis No downMmes allowed Migrate data on demand On every access check if data needs migra=on Migrate if needed
- 65. Migrate data from MySQL to Redis No downMmes allowed Migrate data on demand On every access check if data needs migra=on Migrate if needed Con=nue using Redis
- 66. Typical migraMon throughput over 3 days
- 67. Big and staMc data in MySQL, rest goes to Redis 256 GB data 60 GB data 10% writes 50% writes hOp://www.flickr.com/photos/erix/245657047/
- 68. One team saved the other one &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
- 69. One team saved the other one &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
- 70. We now have more than 2 million users / day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
- 71. We now have more than 2 million users / day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" AWS outage #!!$!!!" in Ireland !" '()*%!" +,-*%!" ./0*%!" +12*%%" '()*%%" +,-*%%" ./0*%%"
- 72. 10 single-‐points-‐of-‐failure -‐ no fun at all! lb lb app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app My My My My My redis redis redis redis redis SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave slave slave
- 73.
- 74. Architecture EvoluMon at Wooga The Start: Ruby The Next Step Best of Two Worlds Company Values
- 75. Stateless servers and DBs Server Database
- 76. Stateless servers and DBs Server Database
- 77. Stateless servers and DBs Server Database
- 78. Stateless servers and DBs Server Database
- 79. Stateless servers and DBs Server Database
- 80. Stateless servers and DBs Server Database
- 81. Stateful servers and DBs Server Database
- 82. Stateful servers and DBs Server Database
- 83. Stateful servers and DBs Server Database
- 84. Stateful servers and DBs Server Database One Game Session
- 85. Stateful servers and DBs Server Database One Game Session
- 86. Oct 2010: 3rd team used a stateful server If DBs are the problem Don’t use them Store state in server Need to be robust
- 87. Oct 2010: 3rd team used a stateful server If DBs are the problem Don’t use them Store state in server Need to be robust
- 88. EvoluMon III: Use Erlang for a stateful server Oct 2010 Jan 2010 Oct 2009
- 89. Stateful servers are not as hard as you think session
- 90. Stateful servers are not as hard as you think session session session session
- 91. Stateful servers are not as hard as you think Server session session session session
- 92. Stateful servers are not as hard as you think Server session session session session S3
- 93. Stateful servers are not as hard as you think Server session session session session S3
- 94. Stateful servers are not as hard as you think Server session session session session S3
- 95. Stateful servers are not as hard as you think Server session session session session S3
- 96. Stateful servers are not as hard as you think Server session session session session S3
- 97. Stateful servers are not as hard as you think Server session session session session S3
- 98. Stateful servers are not as hard as you think Server Server Server session session session session session session session session session session session session S3
- 99. With stateful server the DB is less used Ruby Stateless Erlang Stateful 30,000 22,500 15,000 7,500 0 database operations / sec
- 100. With stateful server the DB is less used Ruby Stateless Erlang Stateful 30,000 22,500 15,000 700 7,500 0 database operations / sec
- 101. Deploying with a stateful server In order to bring up a new version
- 102. Deploying with a stateful server In order to bring up a new version Just deploy it Hot code replacement is great!
- 103. There are even more advantages Faster than Ruby (5,000 rps / node) -‐ CPU bound
- 104. There are even more advantages Faster than Ruby (5,000 rps / node) -‐ CPU bound Very few SPOFs -‐ ... and those are easy to recover
- 105. There are even more advantages Faster than Ruby (5,000 rps / node) -‐ CPU bound Very few SPOFs -‐ ... and those are easy to recover TransacMonal logic -‐ Invariants instead of explicit error handling
- 106. Example “controller” in Erlang
- 107. Example “controller” in Erlang
- 108. Example “controller” in Erlang
- 109. Example “controller” in Erlang
- 110. Example “controller” in Erlang TransacMonal behavior
- 111. Example “controller” in Erlang TransacMonal behavior Erlang code is not that hard to read, isn’t it?
- 112.
- 113. Architecture EvoluMon at Wooga The Start: Ruby The Next Step: Erlang Best of Two Worlds Company Values
- 114. Aug 2011: 4th team wanted both Erlang is great Concurrency, robustness Great for opera=on
- 115. Aug 2011: 4th team wanted both Erlang is great Concurrency, robustness Great for opera=on Ruby is great Concise, expressive, testable Great for development
- 116. Aug 2011: 4th team wanted both Erlang is great Concurrency, robustness Great for opera=on Ruby is great Concise, expressive, testable Great for development
- 117. Aug 2011: 4th team wanted both Erlang is great Concurrency, robustness Great for opera=on Ruby is great Concise, expressive, testable Great for development
- 118. EvoluMon IV: The best out of two worlds Aug 2011 Oct 2010 Jan 2010 Oct 2009
- 119. The basic setup looks exactly like before Server Server Server session session session session session session session session session session session session S3
- 120. Bringing 2 worlds together Server session session ... session
- 121. Bringing 2 worlds together Server session sender session ... session
- 122. Bringing 2 worlds together Server Worker session sender Worker session Worker ... Worker session Worker
- 123. Bringing 2 worlds together Server Worker session sender Worker session Worker ... Worker receiver session Worker
- 124. Bringing 2 worlds together Server Worker session sender Worker session Worker ... Worker receiver session Worker
- 125. Bringing 2 worlds together Server Worker session sender Worker session Worker ... Worker receiver session Worker
- 126. Game state Game state is split in mulMple parts user, map, fruit_trees etc.
- 127. Game state Game state is split in mulMple parts user, map, fruit_trees etc. Erlang does not care about content Serialized Ruby objects
- 128. Game state Game state is split in mulMple parts user, map, fruit_trees etc. Erlang does not care about content Serialized Ruby objects Erlang does know mapping of state parts to URLs Mapping provided by Ruby on startup
- 129. Looking back at the game acMon
- 130. Looking back at the game acMon
- 131. Looking back at the game acMon Mapping of state parts to game acMons
- 132. Looking back at the game acMon Mapping of state parts to game acMons Worker knows mapping
- 133. Looking back at the game acMon Mapping of state parts to game acMons Worker knows mapping Worker pushes mapping to Erlang on startup
- 134. Looking back at the game acMon Mapping of state parts to game acMons Worker knows mapping Worker pushes mapping to Erlang on startup Erlang can query mapping if needed
- 135. NICE! http://www.flickr.com/photos/aigle_dore/
- 136. Architecture EvoluMon at Wooga The Start: Ruby The Next Step: Erlang Best of Two Worlds Company Values
- 137. Each new game brought us innovaMon Aug 2011 Oct 2010 Jan 2010 Oct 2009
- 138. We’ve learned to value Small teams over big teams
- 139. We’ve learned to value Collaboration over competition
- 140. We’ve learned to value Generalists over specialists
- 141. We’ve learned to value Effort reduction over cost reduction
- 142. We’ve learned to value Innovation over risk mitigation
- 143. A good value system We’ve learned to value Small teams over Big teams CollaboraMon over Compe==on Generalists over Specialists Effort reducMon over Cost reduc=on InnovaMon over Risk mi=ga=on
- 144.
- 145. It works! Be fast,be bold!
- 146. QuesMons? Jesper Richter-‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs