Your SlideShare is downloading. ×
Games for the Masses: Scaling Rails to the Extreme
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Games for the Masses: Scaling Rails to the Extreme

4,656
views

Published on

This presentation explains and compares the work of two engineering teams that build Facebook game backends at wooga: Cloud vs. dedicated servers, SQL vs. NoSQL and in-memory database vs. a database …

This presentation explains and compares the work of two engineering teams that build Facebook game backends at wooga: Cloud vs. dedicated servers, SQL vs. NoSQL and in-memory database vs. a database hierarchy. He will highlight their respective advantages and disadvantages and discuss some common patterns both teams came up with to solve their problems.

Published in: Technology

0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
4,656
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
135
Comments
0
Likes
16
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. GAMES  FOR  THE  MASSES SCALING  RAILS  TO  THE  EXTREME EuRuKo  2011  in  Berlin
  • 2. Who  is  that  guy? Jesper  Richter-­‐Reichhelm @jrirei Berlin,  Germany Head  of  Engineering  @  wooga
  • 3. Wooga  develops  social  games
  • 4. Wooga  has  dedicated  game  teams PHP Ruby Ruby SQL SQL  +  Redis Redis Cloud Cloud Bare  Metal
  • 5. Wooga  has  dedicated  game  teams PHP Ruby Ruby SQL SQL  +  Redis Redis Cloud Cloud Bare  Metal
  • 6. Both  games  are  similar  in  their  setup Flash  client Ruby  backend
  • 7. Both  games  have  similar  problems High  traffic • 200+  million  h=p  requests  /  day High  DB  throughput • 100.000  DB  operaDons  /  second Hardest  problems  to  solve • DB  migraDons,  DB  failover,  DB  performance,  ...
  • 8. What’s  best? Big  cloud  vs.  bare  metal Hybrid  (SQL  +  Redis)  vs.  Redis  only In-­‐memory  vs.  DB  hierarchy Conclusion:  So  what’s  best?
  • 9. Big  cloud  vs.  bare  metal Big  cloud:  Amazon  EC2 • new  server  in  5  minutes • automated  deployment • load  based  setup
  • 10. Big  cloud  vs.  bare  metal Big  cloud:  Amazon  EC2 Bare  metal:  Hetzner • new  server  in  5  minutes • new  server  in  2-­‐14  days • automated  deployment • automated  deployment • load  based  setup
  • 11. What  kind  of  app  server  are  available? XL  CPU  instance • 8  virtual  cores • 8  GB  RAM
  • 12. Bare  metal  servers  are  more  powerful XL  CPU  instance EQ10  server • 8  virtual  cores • 12  “processing  threads” • 8  GB  RAM • 24  GB  RAM
  • 13. Bare  metal  servers  are  more  powerful XL  CPU  instance EQ10  server • 8  virtual  cores • 12  “processing  threads” • 8  GB  RAM • 24  GB  RAM =>  390,-­‐  €  /  month =>  200,-­‐  €  /  month
  • 14. And  how  is  the  network? min  ping:  0.2  -­‐  0.3  ms min  ping:  0.2  ms
  • 15. Want  a  fast  network?  Get  your  own! min  ping:  0.2  -­‐  0.3  ms min  ping:  0.2  ms max  ping:  7.8  -­‐  59  ms max  ping:  0.3  ms avg  ping:  0.8  -­‐  4.4  ms avg  ping:  0.2  ms
  • 16. Want  a  fast  network?  Get  your  own! min  ping:  0.2  -­‐  0.3  ms min  ping:  0.2  ms max  ping:  7.8  -­‐  59  ms max  ping:  0.3  ms avg  ping:  0.8  -­‐  4.4  ms avg  ping:  0.2  ms 22  DB  calls  per  hfp  calls saved  13  -­‐  92  ms  latency
  • 17. What’s  befer?  Depends  on  your  needs! Big  cloud  is  great  if  you • don’t  know  your  load • scale  up  /  down  quickly
  • 18. What’s  befer?  Depends  on  your  needs! Big  cloud  is  great  if  you Bare  metal  is  great  if  you • don’t  know  your  load • want  to  save  money • scale  up  /  down  quickly • need  stable  performance
  • 19. What’s  befer?  Depends  on  your  needs! Big  cloud  is  great  if  you Bare  metal  is  great  if  you • don’t  know  your  load • want  to  save  money • scale  up  /  down  quickly • need  stable  performance Say:  “Don’t  worry” Say:  “Think  ahead”
  • 20. What’s  best? Big  cloud  vs.  bare  metal Hybrid  (SQL  +  Redis)  vs.  Redis  only In-­‐memory  vs.  DB  hierarchy Conclusion:  So  what’s  best?
  • 21. Hybrid  (SQL  +  Redis)  vs.  Redis Hybrid  (SQL  +  Redis) Redis  only • started  with  MySQL • hit  a  wall  due  to  IO  limits • introduced  Redis  later
  • 22. If  MySQL  is  a  truck,  ... Fast  enough  for  reads Can  store  on  disk Robust  replicakon h=p://www.flickr.com/photos/erix/245657047/
  • 23. If  MySQL  is  a  truck,  Redis  is  a  Ferrari Fast  enough  for  reads Super  fast  reads/writes Can  store  on  disk Out  of  memory  =>  dead Robust  replicakon Fragile  replicakon h=p://www.flickr.com/photos/erix/245657047/
  • 24. Hybrid  (SQL  +  Redis)  vs.  Redis Hybrid  (SQL  +  Redis) Redis  only • started  with  MySQL • hit  a  wall  due  to  IO  limits • introduced  Redis  later
  • 25. Hybrid  (SQL  +  Redis)  vs.  Redis Hybrid  (SQL  +  Redis) Redis  only • started  with  MySQL • hit  a  wall  due  to  IO  limits • introduced  Redis  later MySQL  has  its  hard  limits Works  fine  if  done  right
  • 26. How  many  tables  to  store  a  user  in? 22  MySQL  tables 7  Redis  hashes
  • 27. How  many  tables  to  store  a  user  in? 22  MySQL  tables 1  Redis  Hash 7  Redis  hashes
  • 28. How  many  tables  to  store  a  user  in? 22  MySQL  tables 1  Redis  Hash 7  Redis  hashes Easy  to  implement Less  top  level  keys Faster  for  small  changes Export  and  import  is  easy
  • 29. What’s  best? Big  cloud  vs.  bare  metal Hybrid  (SQL  +  Redis)  vs.  Redis  only In-­‐memory  vs.  DB  hierarchy Conclusion:  So  what’s  best?
  • 30. Redis  storage  type Redis  works  in-­‐memory Dumps  slow  down  master • Dumping  on  slaves  only
  • 31. Redis  storage  type Redis  works  in-­‐memory Redis  used  vm Dumps  slow  down  master Dumps  to  disk  impossible • Dumping  on  slaves  only • Own  system  to  store   JSON  files  to  disk
  • 32. Redis  storage  type Redis  works  in-­‐memory Redis  used  vm Dumps  slow  down  master Dumps  to  disk  impossible • Dumping  on  slaves  only • Own  system  to  store   JSON  files  to  disk Redis  became  a  cache  only
  • 33. Redis  storage  type Redis  works  in-­‐memory Redis  now  uses  diskstore Dumps  slow  down  master • Dumping  on  slaves  only
  • 34. Redis  storage  type Redis  works  in-­‐memory Redis  now  uses  diskstore Dumps  slow  down  master • Experimental  branch • Dumping  on  slaves  only • Stores  async.  to  disk  
  • 35. Redis  storage  type Redis  works  in-­‐memory Redis  now  uses  diskstore Dumps  slow  down  master • Experimental  branch • Dumping  on  slaves  only • Stores  async.  to  disk   Redis  became  a  DB  again
  • 36. There  are  many  inackve  users   hourly 0,3M hourly 0,1M daily 1,3M daily 0,4Mmonthly 8,0M monthly 3,0Mlifetime 20,8M lifetime 6,0M 0M 8M 16M 24M 0M 2M 4M 6M
  • 37. Not  all  users  play  at  the  same  day In-­‐memory DB  hierarchy • Redis  in-­‐memory • Redis  diskstore  on  disk 100%  of  users  in  RAM 3%  of  users  in  RAM
  • 38. What’s  best? Big  cloud  vs.  bare  metal Hybrid  (SQL  +  Redis)  vs.  Redis  only In-­‐memory  vs.  DB  hierarchy Conclusion:  So  what’s  best?
  • 39. So,  what  didn’t  work? Big  cloud Bare  metal Hybrid  (SQL  +  Redis) Redis  only In-­‐memory  “only” DB  hierarchy
  • 40. So,  what  didn’t  work? Big  cloud Bare  metal Hybrid  (SQL  +  Redis) Redis  only In-­‐memory  “only” DB  hierarchy
  • 41. Let’s  dig  a  bit  deeper 1,300,000  daily  users 400,000  daily  users   5000  hfp  calls/s  peak 2500  hfp  calls/s  peak 100  ms  response  kme 10  ms  response  kme 35-­‐70  servers 11  servers
  • 42. Hoskng  costs  /  DAU  compared HosDng 5x  higher
  • 43. There  are  2  main  reasons  for  higher  cost Databases Other Other:  3x  higher DB:  20x  higher
  • 44. There  are  2  main  reasons  for  higher  cost Databases Other Other:  3x  higher No  DB  hierarchy! DB:  20x  higher (both  RAM  and  cost)
  • 45. So,  what  didn’t  work? Big  cloud Bare  metal Hybrid  (SQL  +  Redis) Redis  only In-­‐memory  “only” DB  hierarchy
  • 46. So,  what  didn’t  work? ds  ... Big  cloud dep en Bare  metal Hybrid  (SQL  +  Redis) Redis  only In-­‐memory  “only” DB  hierarchy
  • 47. So,  what  didn’t  work? ds  ... Big  cloud dep en Bare  metal Hybrid  (SQL  +  Redis) Redis  only nev er In-­‐memory  “only” in! DB  hierarchy a ga
  • 48. So,  what  didn’t  work? ds  ... Big  cloud dep en Bare  metal Hybrid  (SQL  +  Redis) er, Redis  only er,  ev In-­‐memory  “only” n ev n! DB  hierarchy  ag ai ev er
  • 49. Next  game  will  do  things  befer Start  in  cloud  and  then  go  metal? Redis  only DB  hierarchy
  • 50. BTW,  wooga  is  hiring! Q  &  A Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs
  • 51. BTW,  wooga  is  hiring! Thank  you Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs

×