Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

4,256 views

Published on

Short talk about what Wooga learned by implementing and operating two social games at AWS using EC2 and S3.

Published in: Technology

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

  1. 1. ARCHITECTURE  EVOLUTION AT  WOOGA How  does  your  company  learn  new  things?Jesper  Richter-­‐Reichhelm,  @jrirei
  2. 2. Our  games  all  look  the  same Flash  client Backend
  3. 3. But  the  scale  is  interesFng 14  billion  requests  /  month
  4. 4. But  the  scale  is  interesFng 14  billion  requests  /  month
  5. 5. But  the  scale  is  interesFng 14  billion  requests  /  month >100,000  DB  operaFons  /  second
  6. 6. But  the  scale  is  interesFng 14  billion  requests  /  month >100,000  DB  operaFons  /  second >50,000  DB  updates  /  second
  7. 7. Architecture  EvoluFon  at  Wooga The  Start The  Next  Step Best  of  Two  Worlds
  8. 8. Oct  2009:  Monster  World  based  on  Ruby Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactor
  9. 9. Oct  2009:  Monster  World  based  on  Ruby Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactor
  10. 10. Oct  2009:  Monster  World  based  on  Ruby Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactor
  11. 11. A  basic  setup  using  sharding  worked  fine lb app app app My My SQL SQL slave slave
  12. 12. A  basic  setup  using  sharding  worked  fine lb app app app app app My My SQL SQL slave slave
  13. 13. A  basic  setup  using  sharding  worked  fine lb app app app app app app app app app My My SQL SQL slave slave
  14. 14. 250K  daily  users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" Life  was  good !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  15. 15. 250K  daily  users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" Life  was  good NO  MORE !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  16. 16. Early  sharding  hell:  8  master  and  8  slaves lb app app app app app app app app app app app app app app app app app app My My SQL SQL slave slave
  17. 17. Early  sharding  hell:  8  master  and  8  slaves lb app app app app app app app app app app app app app app app app app app My My My My SQL SQL SQL SQL slave slave slave slave
  18. 18. Early  sharding  hell:  8  master  and  8  slaves lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave
  19. 19. At  500K  daily  users  we  were  at  a  dead  end&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  20. 20. OUCH!http://www.flickr.com/photos/billue_the_bear/
  21. 21. Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis MySQL
  22. 22. Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis MySQL Redis
  23. 23. Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis MySQL Redis 256  GB  data 10%  writes
  24. 24. Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis MySQL Redis 256  GB  data 60  GB  data 10%  writes 50%  writes
  25. 25. Redis  saved  the  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  26. 26. Redis  saved  the  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  27. 27. We  now  have  more  than  2  million  users  /  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  28. 28. We  now  have  more  than  2  million  users  /  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" AWS  service #!!$!!!" disrupFon in  Ireland !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  29. 29. 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app app My My redis redis SQL SQL slave slave slave slave
  30. 30. 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app appMy My My My My redis redisSQL SQL SQL SQL SQLslave slave slave slave slave slave slave
  31. 31. 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app appMy My My My My redis redis redis redis redisSQL SQL SQL SQL SQLslave slave slave slave slave slave slave slave slave slave
  32. 32. http://www.flickr.com/photos/wolfsavard/OUCH!
  33. 33. Architecture  EvoluFon  at  Wooga The  Start:  Ruby  +  AutomaFon The  Next  Step Best  of  Two  Worlds
  34. 34. Stateless  servers  and  DBs Server Database
  35. 35. Stateless  servers  and  DBs Server Database
  36. 36. Stateless  servers  and  DBs Server Database
  37. 37. Stateless  servers  and  DBs Server Database
  38. 38. Stateless  servers  and  DBs Server Database
  39. 39. Stateless  servers  and  DBs Server Database
  40. 40. “Stateless  applica,on  servers   guarantee  one  thing:
  41. 41. “Stateless  applica,on  servers   guarantee  one  thing: The  data  is  never where  you  need  it!” Paolo  Negri,  Developer  @  Wooga
  42. 42. Stateful  servers  and  DBs Server Database
  43. 43. Stateful  servers  and  DBs Server Database
  44. 44. Stateful  servers  and  DBs Server Database
  45. 45. Stateful  servers  and  DBs Server Database One  Game  Session
  46. 46. Stateful  servers  and  DBs Server Database One  Game  Session
  47. 47. Oct  2010:  Magic  Land  based  on  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robust
  48. 48. Oct  2010:  Magic  Land  based  on  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robust
  49. 49. Oct  2010:  Magic  Land  based  on  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robust
  50. 50. Stateful  servers  are  not  as  hard  as  you  think session
  51. 51. Stateful  servers  are  not  as  hard  as  you  think session session session session
  52. 52. Stateful  servers  are  not  as  hard  as  you  think Server session session session session
  53. 53. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  54. 54. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  55. 55. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  56. 56. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  57. 57. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  58. 58. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  59. 59. Stateful  servers  are  not  as  hard  as  you  think Server Server Server session session session session session session session session session session session session S3
  60. 60. With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 7,500 0 database  operations  /  sec
  61. 61. With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 700 7,500 0 database  operations  /  sec
  62. 62. Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version
  63. 63. Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version Just  deploy  it Hot  code  replacement  is  great!
  64. 64. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound
  65. 65. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recover
  66. 66. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recover TransacFonal  logic -­‐ Invariants  instead  of  explicit  error  handling
  67. 67. NICE! http://www.flickr.com/photos/aigle_dore/
  68. 68. Architecture  EvoluFon  at  Wooga The  Start:  Ruby  +  AutomaFon The  Next  Step:  Erlang  +  S3 Company  Values
  69. 69. A  good  value  system We’ve  learned  to  value
  70. 70. A  good  value  system We’ve  learned  to  value Small  teams                              over   Big  teams
  71. 71. A  good  value  system We’ve  learned  to  value     Small  teams                              over   Big  teams CollaboraFon                        over Compe??on
  72. 72. A  good  value  system We’ve  learned  to  value     Small  teams                              over   Big  teams CollaboraFon                        over Compe??on Generalists                                over Specialists
  73. 73. A  good  value  system We’ve  learned  to  value     Small  teams                              over   Big  teams CollaboraFon                        over Compe??on Generalists                                over Specialists Effort  reducFon                over Cost  reduc?on
  74. 74. A  good  value  system We’ve  learned  to  value     Small  teams                              over   Big  teams CollaboraFon                        over Compe??on Generalists                                over Specialists Effort  reducFon                over Cost  reduc?on InnovaFon                                  over Risk  mi?ga?on
  75. 75. It works!
  76. 76. It works!Be fast, be bold!
  77. 77. QuesFons?Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs

×