Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
ARCHITECTURE  EVOLUTION       AT  WOOGA               How  does  your  company  learn  new  things?Jesper  Richter-­‐Reich...
Our  games  all  look  the  same       Flash  client               Backend
But  the  scale  is  interesFng             14  billion  requests  /  month
But  the  scale  is  interesFng             14  billion  requests  /  month
But  the  scale  is  interesFng             14  billion  requests  /  month          >100,000  DB  operaFons  /  second
But  the  scale  is  interesFng             14  billion  requests  /  month          >100,000  DB  operaFons  /  second   ...
Architecture  EvoluFon  at  Wooga  The  Start  The  Next  Step  Best  of  Two  Worlds
Oct  2009:  Monster  World  based  on  Ruby Good  code  quality   Easy  to  understand   Easy  to  test   Easy  to  refactor
Oct  2009:  Monster  World  based  on  Ruby Good  code  quality   Easy  to  understand   Easy  to  test   Easy  to  refactor
Oct  2009:  Monster  World  based  on  Ruby Good  code  quality   Easy  to  understand   Easy  to  test   Easy  to  refactor
A  basic  setup  using  sharding  worked  fine                            lb                    app     app    app         ...
A  basic  setup  using  sharding  worked  fine                              lb                app   app     app    app    a...
A  basic  setup  using  sharding  worked  fine                                  lb        app   app   app   app     app    ...
250K  daily  users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"                            Life  was  good        !"        ()*%...
250K  daily  users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"                            Life  was  good NO  MORE        !"   ...
Early  sharding  hell:  8  master  and  8  slaves                                   lb         app   app   app   app     a...
Early  sharding  hell:  8  master  and  8  slaves                                     lb         app   app     app   app  ...
Early  sharding  hell:  8  master  and  8  slaves                                               lb             app        ...
At  500K  daily  users  we  were  at  a  dead  end&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"        !"        ()*%!"   +,-*%!...
OUCH!http://www.flickr.com/photos/billue_the_bear/
Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis          MySQL
Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis          MySQL                        Redis
Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis          MySQL                        Redis       256  GB  data  ...
Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis          MySQL                        Redis       256  GB  data  ...
Redis  saved  the  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"        !"        ()*%!"   +,-*%!"   ./0*%!"   +12*%%"   ()*%...
Redis  saved  the  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"        !"        ()*%!"   +,-*%!"   ./0*%!"   +12*%%"   ()*%...
We  now  have  more  than  2  million  users  /  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!"        !"        ()*%!"   +,-*...
We  now  have  more  than  2  million  users  /  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!"                                        ...
10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!                                        lb         lbapp   app  ...
10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!                                               lb         lbapp ...
10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!                                               lb         lbapp ...
http://www.flickr.com/photos/wolfsavard/OUCH!
Architecture  EvoluFon  at  Wooga  The  Start:  Ruby  +  AutomaFon  The  Next  Step  Best  of  Two  Worlds
Stateless  servers  and  DBs         Server                Database
Stateless  servers  and  DBs         Server                Database
Stateless  servers  and  DBs         Server                Database
Stateless  servers  and  DBs         Server                Database
Stateless  servers  and  DBs         Server                Database
Stateless  servers  and  DBs         Server                Database
“Stateless  applica,on  servers       guarantee  one  thing:
“Stateless  applica,on  servers       guarantee  one  thing:      The  data  is  never     where  you  need  it!”    Paolo...
Stateful  servers  and  DBs          Server              Database
Stateful  servers  and  DBs          Server              Database
Stateful  servers  and  DBs          Server              Database
Stateful  servers  and  DBs          Server                        Database                   One  Game  Session
Stateful  servers  and  DBs          Server                        Database                   One  Game  Session
Oct  2010:  Magic  Land  based  on  stateful  server If  DBs  are  the  problem      Don’t  use  them      Store  state  i...
Oct  2010:  Magic  Land  based  on  stateful  server If  DBs  are  the  problem      Don’t  use  them      Store  state  i...
Oct  2010:  Magic  Land  based  on  stateful  server If  DBs  are  the  problem      Don’t  use  them      Store  state  i...
Stateful  servers  are  not  as  hard  as  you  think                        session
Stateful  servers  are  not  as  hard  as  you  think                              session                            sess...
Stateful  servers  are  not  as  hard  as  you  think                           Server                              sessio...
Stateful  servers  are  not  as  hard  as  you  think                           Server                              sessio...
Stateful  servers  are  not  as  hard  as  you  think                           Server                              sessio...
Stateful  servers  are  not  as  hard  as  you  think                           Server                              sessio...
Stateful  servers  are  not  as  hard  as  you  think                           Server                              sessio...
Stateful  servers  are  not  as  hard  as  you  think                           Server                              sessio...
Stateful  servers  are  not  as  hard  as  you  think                           Server                              sessio...
Stateful  servers  are  not  as  hard  as  you  think         Server            Server             Server            sessi...
With  stateful  server  the  DB  is  less  used             Ruby  Stateless            Erlang  Stateful  30,000  22,500  1...
With  stateful  server  the  DB  is  less  used             Ruby  Stateless            Erlang  Stateful  30,000  22,500  1...
Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version
Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version Just  deploy  it    Hot  code  replacement ...
There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node)     -­‐ CPU  bound
There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node)     -­‐ CPU  bound Very  few  SPOFs     -­‐ ....
There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node)     -­‐ CPU  bound Very  few  SPOFs     -­‐ ....
NICE!        http://www.flickr.com/photos/aigle_dore/
Architecture  EvoluFon  at  Wooga  The  Start:  Ruby  +  AutomaFon  The  Next  Step:  Erlang  +  S3  Company  Values
A  good  value  system We’ve  learned  to  value
A  good  value  system We’ve  learned  to  value Small  teams                              over     Big  teams
A  good  value  system We’ve  learned  to  value      Small  teams                              over     Big  teams Collab...
A  good  value  system We’ve  learned  to  value      Small  teams                              over     Big  teams Collab...
A  good  value  system We’ve  learned  to  value      Small  teams                              over     Big  teams Collab...
A  good  value  system We’ve  learned  to  value      Small  teams                              over     Big  teams Collab...
It works!
It works!Be fast, be bold!
QuesFons?Jesper  Richter-­‐Reichhelm          @jrirei slideshare.net/wooga    wooga.com/jobs
Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)
Upcoming SlideShare
Loading in …5
×

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

4,235 views

Published on

Short talk about what Wooga learned by implementing and operating two social games at AWS using EC2 and S3.

Published in: Technology

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

  1. 1. ARCHITECTURE  EVOLUTION AT  WOOGA How  does  your  company  learn  new  things?Jesper  Richter-­‐Reichhelm,  @jrirei
  2. 2. Our  games  all  look  the  same Flash  client Backend
  3. 3. But  the  scale  is  interesFng 14  billion  requests  /  month
  4. 4. But  the  scale  is  interesFng 14  billion  requests  /  month
  5. 5. But  the  scale  is  interesFng 14  billion  requests  /  month >100,000  DB  operaFons  /  second
  6. 6. But  the  scale  is  interesFng 14  billion  requests  /  month >100,000  DB  operaFons  /  second >50,000  DB  updates  /  second
  7. 7. Architecture  EvoluFon  at  Wooga The  Start The  Next  Step Best  of  Two  Worlds
  8. 8. Oct  2009:  Monster  World  based  on  Ruby Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactor
  9. 9. Oct  2009:  Monster  World  based  on  Ruby Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactor
  10. 10. Oct  2009:  Monster  World  based  on  Ruby Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactor
  11. 11. A  basic  setup  using  sharding  worked  fine lb app app app My My SQL SQL slave slave
  12. 12. A  basic  setup  using  sharding  worked  fine lb app app app app app My My SQL SQL slave slave
  13. 13. A  basic  setup  using  sharding  worked  fine lb app app app app app app app app app My My SQL SQL slave slave
  14. 14. 250K  daily  users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" Life  was  good !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  15. 15. 250K  daily  users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" Life  was  good NO  MORE !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  16. 16. Early  sharding  hell:  8  master  and  8  slaves lb app app app app app app app app app app app app app app app app app app My My SQL SQL slave slave
  17. 17. Early  sharding  hell:  8  master  and  8  slaves lb app app app app app app app app app app app app app app app app app app My My My My SQL SQL SQL SQL slave slave slave slave
  18. 18. Early  sharding  hell:  8  master  and  8  slaves lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave
  19. 19. At  500K  daily  users  we  were  at  a  dead  end&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  20. 20. OUCH!http://www.flickr.com/photos/billue_the_bear/
  21. 21. Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis MySQL
  22. 22. Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis MySQL Redis
  23. 23. Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis MySQL Redis 256  GB  data 10%  writes
  24. 24. Big  and  staFc  data  in  MySQL,  rest  goes  to  Redis MySQL Redis 256  GB  data 60  GB  data 10%  writes 50%  writes
  25. 25. Redis  saved  the  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  26. 26. Redis  saved  the  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  27. 27. We  now  have  more  than  2  million  users  /  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  28. 28. We  now  have  more  than  2  million  users  /  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" AWS  service #!!$!!!" disrupFon in  Ireland !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  29. 29. 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app app My My redis redis SQL SQL slave slave slave slave
  30. 30. 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app appMy My My My My redis redisSQL SQL SQL SQL SQLslave slave slave slave slave slave slave
  31. 31. 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app appMy My My My My redis redis redis redis redisSQL SQL SQL SQL SQLslave slave slave slave slave slave slave slave slave slave
  32. 32. http://www.flickr.com/photos/wolfsavard/OUCH!
  33. 33. Architecture  EvoluFon  at  Wooga The  Start:  Ruby  +  AutomaFon The  Next  Step Best  of  Two  Worlds
  34. 34. Stateless  servers  and  DBs Server Database
  35. 35. Stateless  servers  and  DBs Server Database
  36. 36. Stateless  servers  and  DBs Server Database
  37. 37. Stateless  servers  and  DBs Server Database
  38. 38. Stateless  servers  and  DBs Server Database
  39. 39. Stateless  servers  and  DBs Server Database
  40. 40. “Stateless  applica,on  servers   guarantee  one  thing:
  41. 41. “Stateless  applica,on  servers   guarantee  one  thing: The  data  is  never where  you  need  it!” Paolo  Negri,  Developer  @  Wooga
  42. 42. Stateful  servers  and  DBs Server Database
  43. 43. Stateful  servers  and  DBs Server Database
  44. 44. Stateful  servers  and  DBs Server Database
  45. 45. Stateful  servers  and  DBs Server Database One  Game  Session
  46. 46. Stateful  servers  and  DBs Server Database One  Game  Session
  47. 47. Oct  2010:  Magic  Land  based  on  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robust
  48. 48. Oct  2010:  Magic  Land  based  on  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robust
  49. 49. Oct  2010:  Magic  Land  based  on  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robust
  50. 50. Stateful  servers  are  not  as  hard  as  you  think session
  51. 51. Stateful  servers  are  not  as  hard  as  you  think session session session session
  52. 52. Stateful  servers  are  not  as  hard  as  you  think Server session session session session
  53. 53. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  54. 54. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  55. 55. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  56. 56. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  57. 57. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  58. 58. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  59. 59. Stateful  servers  are  not  as  hard  as  you  think Server Server Server session session session session session session session session session session session session S3
  60. 60. With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 7,500 0 database  operations  /  sec
  61. 61. With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 700 7,500 0 database  operations  /  sec
  62. 62. Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version
  63. 63. Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version Just  deploy  it Hot  code  replacement  is  great!
  64. 64. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound
  65. 65. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recover
  66. 66. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recover TransacFonal  logic -­‐ Invariants  instead  of  explicit  error  handling
  67. 67. NICE! http://www.flickr.com/photos/aigle_dore/
  68. 68. Architecture  EvoluFon  at  Wooga The  Start:  Ruby  +  AutomaFon The  Next  Step:  Erlang  +  S3 Company  Values
  69. 69. A  good  value  system We’ve  learned  to  value
  70. 70. A  good  value  system We’ve  learned  to  value Small  teams                              over   Big  teams
  71. 71. A  good  value  system We’ve  learned  to  value     Small  teams                              over   Big  teams CollaboraFon                        over Compe??on
  72. 72. A  good  value  system We’ve  learned  to  value     Small  teams                              over   Big  teams CollaboraFon                        over Compe??on Generalists                                over Specialists
  73. 73. A  good  value  system We’ve  learned  to  value     Small  teams                              over   Big  teams CollaboraFon                        over Compe??on Generalists                                over Specialists Effort  reducFon                over Cost  reduc?on
  74. 74. A  good  value  system We’ve  learned  to  value     Small  teams                              over   Big  teams CollaboraFon                        over Compe??on Generalists                                over Specialists Effort  reducFon                over Cost  reduc?on InnovaFon                                  over Risk  mi?ga?on
  75. 75. It works!
  76. 76. It works!Be fast, be bold!
  77. 77. QuesFons?Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs

×