Games for the Masses: Scaling Rails to the Extreme

5,224 views

Published on

This presentation explains and compares the work of two engineering teams that build Facebook game backends at wooga: Cloud vs. dedicated servers, SQL vs. NoSQL and in-memory database vs. a database hierarchy. He will highlight their respective advantages and disadvantages and discuss some common patterns both teams came up with to solve their problems.

Published in: Technology
0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,224
On SlideShare
0
From Embeds
0
Number of Embeds
65
Actions
Shares
0
Downloads
138
Comments
0
Likes
16
Embeds 0
No embeds

No notes for slide

Games for the Masses: Scaling Rails to the Extreme

  1. 1. GAMES  FOR  THE  MASSES SCALING  RAILS  TO  THE  EXTREME EuRuKo  2011  in  Berlin
  2. 2. Who  is  that  guy? Jesper  Richter-­‐Reichhelm @jrirei Berlin,  Germany Head  of  Engineering  @  wooga
  3. 3. Wooga  develops  social  games
  4. 4. Wooga  has  dedicated  game  teams PHP Ruby Ruby SQL SQL  +  Redis Redis Cloud Cloud Bare  Metal
  5. 5. Wooga  has  dedicated  game  teams PHP Ruby Ruby SQL SQL  +  Redis Redis Cloud Cloud Bare  Metal
  6. 6. Both  games  are  similar  in  their  setup Flash  client Ruby  backend
  7. 7. Both  games  have  similar  problems High  traffic • 200+  million  h=p  requests  /  day High  DB  throughput • 100.000  DB  operaDons  /  second Hardest  problems  to  solve • DB  migraDons,  DB  failover,  DB  performance,  ...
  8. 8. What’s  best? Big  cloud  vs.  bare  metal Hybrid  (SQL  +  Redis)  vs.  Redis  only In-­‐memory  vs.  DB  hierarchy Conclusion:  So  what’s  best?
  9. 9. Big  cloud  vs.  bare  metal Big  cloud:  Amazon  EC2 • new  server  in  5  minutes • automated  deployment • load  based  setup
  10. 10. Big  cloud  vs.  bare  metal Big  cloud:  Amazon  EC2 Bare  metal:  Hetzner • new  server  in  5  minutes • new  server  in  2-­‐14  days • automated  deployment • automated  deployment • load  based  setup
  11. 11. What  kind  of  app  server  are  available? XL  CPU  instance • 8  virtual  cores • 8  GB  RAM
  12. 12. Bare  metal  servers  are  more  powerful XL  CPU  instance EQ10  server • 8  virtual  cores • 12  “processing  threads” • 8  GB  RAM • 24  GB  RAM
  13. 13. Bare  metal  servers  are  more  powerful XL  CPU  instance EQ10  server • 8  virtual  cores • 12  “processing  threads” • 8  GB  RAM • 24  GB  RAM =>  390,-­‐  €  /  month =>  200,-­‐  €  /  month
  14. 14. And  how  is  the  network? min  ping:  0.2  -­‐  0.3  ms min  ping:  0.2  ms
  15. 15. Want  a  fast  network?  Get  your  own! min  ping:  0.2  -­‐  0.3  ms min  ping:  0.2  ms max  ping:  7.8  -­‐  59  ms max  ping:  0.3  ms avg  ping:  0.8  -­‐  4.4  ms avg  ping:  0.2  ms
  16. 16. Want  a  fast  network?  Get  your  own! min  ping:  0.2  -­‐  0.3  ms min  ping:  0.2  ms max  ping:  7.8  -­‐  59  ms max  ping:  0.3  ms avg  ping:  0.8  -­‐  4.4  ms avg  ping:  0.2  ms 22  DB  calls  per  hfp  calls saved  13  -­‐  92  ms  latency
  17. 17. What’s  befer?  Depends  on  your  needs! Big  cloud  is  great  if  you • don’t  know  your  load • scale  up  /  down  quickly
  18. 18. What’s  befer?  Depends  on  your  needs! Big  cloud  is  great  if  you Bare  metal  is  great  if  you • don’t  know  your  load • want  to  save  money • scale  up  /  down  quickly • need  stable  performance
  19. 19. What’s  befer?  Depends  on  your  needs! Big  cloud  is  great  if  you Bare  metal  is  great  if  you • don’t  know  your  load • want  to  save  money • scale  up  /  down  quickly • need  stable  performance Say:  “Don’t  worry” Say:  “Think  ahead”
  20. 20. What’s  best? Big  cloud  vs.  bare  metal Hybrid  (SQL  +  Redis)  vs.  Redis  only In-­‐memory  vs.  DB  hierarchy Conclusion:  So  what’s  best?
  21. 21. Hybrid  (SQL  +  Redis)  vs.  Redis Hybrid  (SQL  +  Redis) Redis  only • started  with  MySQL • hit  a  wall  due  to  IO  limits • introduced  Redis  later
  22. 22. If  MySQL  is  a  truck,  ... Fast  enough  for  reads Can  store  on  disk Robust  replicakon h=p://www.flickr.com/photos/erix/245657047/
  23. 23. If  MySQL  is  a  truck,  Redis  is  a  Ferrari Fast  enough  for  reads Super  fast  reads/writes Can  store  on  disk Out  of  memory  =>  dead Robust  replicakon Fragile  replicakon h=p://www.flickr.com/photos/erix/245657047/
  24. 24. Hybrid  (SQL  +  Redis)  vs.  Redis Hybrid  (SQL  +  Redis) Redis  only • started  with  MySQL • hit  a  wall  due  to  IO  limits • introduced  Redis  later
  25. 25. Hybrid  (SQL  +  Redis)  vs.  Redis Hybrid  (SQL  +  Redis) Redis  only • started  with  MySQL • hit  a  wall  due  to  IO  limits • introduced  Redis  later MySQL  has  its  hard  limits Works  fine  if  done  right
  26. 26. How  many  tables  to  store  a  user  in? 22  MySQL  tables 7  Redis  hashes
  27. 27. How  many  tables  to  store  a  user  in? 22  MySQL  tables 1  Redis  Hash 7  Redis  hashes
  28. 28. How  many  tables  to  store  a  user  in? 22  MySQL  tables 1  Redis  Hash 7  Redis  hashes Easy  to  implement Less  top  level  keys Faster  for  small  changes Export  and  import  is  easy
  29. 29. What’s  best? Big  cloud  vs.  bare  metal Hybrid  (SQL  +  Redis)  vs.  Redis  only In-­‐memory  vs.  DB  hierarchy Conclusion:  So  what’s  best?
  30. 30. Redis  storage  type Redis  works  in-­‐memory Dumps  slow  down  master • Dumping  on  slaves  only
  31. 31. Redis  storage  type Redis  works  in-­‐memory Redis  used  vm Dumps  slow  down  master Dumps  to  disk  impossible • Dumping  on  slaves  only • Own  system  to  store   JSON  files  to  disk
  32. 32. Redis  storage  type Redis  works  in-­‐memory Redis  used  vm Dumps  slow  down  master Dumps  to  disk  impossible • Dumping  on  slaves  only • Own  system  to  store   JSON  files  to  disk Redis  became  a  cache  only
  33. 33. Redis  storage  type Redis  works  in-­‐memory Redis  now  uses  diskstore Dumps  slow  down  master • Dumping  on  slaves  only
  34. 34. Redis  storage  type Redis  works  in-­‐memory Redis  now  uses  diskstore Dumps  slow  down  master • Experimental  branch • Dumping  on  slaves  only • Stores  async.  to  disk  
  35. 35. Redis  storage  type Redis  works  in-­‐memory Redis  now  uses  diskstore Dumps  slow  down  master • Experimental  branch • Dumping  on  slaves  only • Stores  async.  to  disk   Redis  became  a  DB  again
  36. 36. There  are  many  inackve  users   hourly 0,3M hourly 0,1M daily 1,3M daily 0,4Mmonthly 8,0M monthly 3,0Mlifetime 20,8M lifetime 6,0M 0M 8M 16M 24M 0M 2M 4M 6M
  37. 37. Not  all  users  play  at  the  same  day In-­‐memory DB  hierarchy • Redis  in-­‐memory • Redis  diskstore  on  disk 100%  of  users  in  RAM 3%  of  users  in  RAM
  38. 38. What’s  best? Big  cloud  vs.  bare  metal Hybrid  (SQL  +  Redis)  vs.  Redis  only In-­‐memory  vs.  DB  hierarchy Conclusion:  So  what’s  best?
  39. 39. So,  what  didn’t  work? Big  cloud Bare  metal Hybrid  (SQL  +  Redis) Redis  only In-­‐memory  “only” DB  hierarchy
  40. 40. So,  what  didn’t  work? Big  cloud Bare  metal Hybrid  (SQL  +  Redis) Redis  only In-­‐memory  “only” DB  hierarchy
  41. 41. Let’s  dig  a  bit  deeper 1,300,000  daily  users 400,000  daily  users   5000  hfp  calls/s  peak 2500  hfp  calls/s  peak 100  ms  response  kme 10  ms  response  kme 35-­‐70  servers 11  servers
  42. 42. Hoskng  costs  /  DAU  compared HosDng 5x  higher
  43. 43. There  are  2  main  reasons  for  higher  cost Databases Other Other:  3x  higher DB:  20x  higher
  44. 44. There  are  2  main  reasons  for  higher  cost Databases Other Other:  3x  higher No  DB  hierarchy! DB:  20x  higher (both  RAM  and  cost)
  45. 45. So,  what  didn’t  work? Big  cloud Bare  metal Hybrid  (SQL  +  Redis) Redis  only In-­‐memory  “only” DB  hierarchy
  46. 46. So,  what  didn’t  work? ds  ... Big  cloud dep en Bare  metal Hybrid  (SQL  +  Redis) Redis  only In-­‐memory  “only” DB  hierarchy
  47. 47. So,  what  didn’t  work? ds  ... Big  cloud dep en Bare  metal Hybrid  (SQL  +  Redis) Redis  only nev er In-­‐memory  “only” in! DB  hierarchy a ga
  48. 48. So,  what  didn’t  work? ds  ... Big  cloud dep en Bare  metal Hybrid  (SQL  +  Redis) er, Redis  only er,  ev In-­‐memory  “only” n ev n! DB  hierarchy  ag ai ev er
  49. 49. Next  game  will  do  things  befer Start  in  cloud  and  then  go  metal? Redis  only DB  hierarchy
  50. 50. BTW,  wooga  is  hiring! Q  &  A Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs
  51. 51. BTW,  wooga  is  hiring! Thank  you Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs

×