GAMES  FOR  THE  MASSES                           How  DevOps  Affects  Architecture Jesper  Richter-­‐Reichhelm,  @jrireiW...
Wednesday, March 7, 2012
Our  games  all  look  the  same                           Flash  client   BackendWednesday, March 7, 2012
Our  games  all  look  the  same                           Flash  client                                               Gam...
Our  games  all  look  the  same                                         Backend                     State  Changes       ...
But  the  scale  is  interesEng                           14  billion  requests  /  monthWednesday, March 7, 2012
But  the  scale  is  interesEng                           14  billion  requests  /  monthWednesday, March 7, 2012
But  the  scale  is  interesEng                              14  billion  requests  /  month                           >10...
But  the  scale  is  interesEng                              14  billion  requests  /  month                           >10...
2  Developers  to  do  it  all      Typical  team  setup        4  product  managers        4  ar0sts        4  frontend  ...
Wooga  has  dedicated  game  teamsWednesday, March 7, 2012
Wednesday, March 7, 2012
Wednesday, March 7, 2012
Oct  2009Wednesday, March 7, 2012
Oct  2009   Jan  2010Wednesday, March 7, 2012
Oct  2009   Jan  2010                           Oct  2010Wednesday, March 7, 2012
Oct  2009   Jan  2010                           Oct  2010   Aug  2012Wednesday, March 7, 2012
Architecture  EvoluEon  at  Wooga          The  Start          The  Next  Step          Best  of  Two  Worlds          Com...
Oct  2009:  1st  team  wanted  good  code  quality      Good  code  quality        Easy  to  understand        Easy  to  t...
Oct  2009:  1st  team  wanted  good  code  quality      Good  code  quality        Easy  to  understand        Easy  to  t...
EvoluEon  I:  Use  Ruby  (on  Rails)                           Oct  2009Wednesday, March 7, 2012
A  basic  setup  using  sharding  worked  fine                                                     lb                      ...
250K  daily  users    &$!!!$!!!"    %$#!!$!!!"    %$!!!$!!!"       #!!$!!!"                                     Life  was ...
250K  daily  users    &$!!!$!!!"    %$#!!$!!!"    %$!!!$!!!"       #!!$!!!"                                     Life  was ...
Welcome  to  6  weeks  of  pain!          Heavy  opEmizaEons  were  necessaryWednesday, March 7, 2012
Welcome  to  6  weeks  of  pain!          Heavy  opEmizaEons  were  necessary            Numerous  small  fixes  regarding ...
Welcome  to  6  weeks  of  pain!          Heavy  opEmizaEons  were  necessary            Numerous  small  fixes  regarding ...
Welcome  to  6  weeks  of  pain!          Heavy  opEmizaEons  were  necessary            Numerous  small  fixes  regarding ...
Welcome  to  6  weeks  of  pain!          Heavy  opEmizaEons  were  necessary            Numerous  small  fixes  regarding ...
Early  sharding  hell:  8  master  and  8  slaves                                                             lb          ...
At  500K  daily  users  we  were  at  a  dead  end    &$!!!$!!!"    %$#!!$!!!"    %$!!!$!!!"       #!!$!!!"               ...
OUCH!http://www.flickr.com/photos/billue_the_bear/Wednesday, March 7, 2012
Jan  2010:  Meanwhile  at  the  2nd  team      Don’t  break  the  bank        Make  it  faster        Make  it  cheaper   ...
Jan  2010:  Meanwhile  at  the  2nd  team      Don’t  break  the  bank        Make  it  faster        Make  it  cheaper   ...
EvoluEon  II:  Use  Redis  as  main  database                                       Jan  2010                           Oc...
If  MySQL  is  a  truck                                                                               Fast  enough        ...
If  MySQL  is  a  truck,  Redis  is  a  race  car                                                                         ...
Bare  metal  for  low  latency!                                               lb                           app   app   app...
How  could  we  apply  that  knowledge?    &$!!!$!!!"    %$#!!$!!!"    %$!!!$!!!"       #!!$!!!"                 !"       ...
On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
Typical  migraEon  throughput  over  3  daysWednesday, March 7, 2012
Big  and  staEc  data  in  MySQL,  rest  goes  to  Redis                       256  GB  data                              ...
One  team  saved  the  other  one    &$!!!$!!!"    %$#!!$!!!"    %$!!!$!!!"       #!!$!!!"                 !"             ...
One  team  saved  the  other  one    &$!!!$!!!"    %$#!!$!!!"    %$!!!$!!!"       #!!$!!!"                 !"             ...
We  now  have  more  than  2  million  users  /  day    &$!!!$!!!"    %$#!!$!!!"    %$!!!$!!!"       #!!$!!!"             ...
We  now  have  more  than  2  million  users  /  day    &$!!!$!!!"    %$#!!$!!!"    %$!!!$!!!"                            ...
10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all!                                                         lb     ...
http://www.flickr.com/photos/wolfsavard/OUCH!Wednesday, March 7, 2012
Architecture  EvoluEon  at  Wooga          The  Start:  Ruby          The  Next  Step          Best  of  Two  Worlds      ...
Stateless  servers  and  DBs                           Server   DatabaseWednesday, March 7, 2012
Stateless  servers  and  DBs                           Server   DatabaseWednesday, March 7, 2012
Stateless  servers  and  DBs                           Server   DatabaseWednesday, March 7, 2012
Stateless  servers  and  DBs                           Server   DatabaseWednesday, March 7, 2012
Stateless  servers  and  DBs                           Server   DatabaseWednesday, March 7, 2012
Stateless  servers  and  DBs                           Server   DatabaseWednesday, March 7, 2012
Stateful  servers  and  DBs                           Server   DatabaseWednesday, March 7, 2012
Stateful  servers  and  DBs                           Server   DatabaseWednesday, March 7, 2012
Stateful  servers  and  DBs                           Server   DatabaseWednesday, March 7, 2012
Stateful  servers  and  DBs                           Server                        Database                              ...
Stateful  servers  and  DBs                           Server                        Database                              ...
Oct  2010:  3rd  team  used  a  stateful  server      If  DBs  are  the  problem           Don’t  use  them           Stor...
Oct  2010:  3rd  team  used  a  stateful  server      If  DBs  are  the  problem           Don’t  use  them           Stor...
EvoluEon  III:  Use  Erlang  for  a  stateful  server                                                   Oct  2010         ...
Stateful  servers  are  not  as  hard  as  you  think                           sessionWednesday, March 7, 2012
Stateful  servers  are  not  as  hard  as  you  think                                 session                             ...
Stateful  servers  are  not  as  hard  as  you  think                              Server                                 ...
Stateful  servers  are  not  as  hard  as  you  think                              Server                                 ...
Stateful  servers  are  not  as  hard  as  you  think                              Server                                 ...
Stateful  servers  are  not  as  hard  as  you  think                              Server                                 ...
Stateful  servers  are  not  as  hard  as  you  think                              Server                                 ...
Stateful  servers  are  not  as  hard  as  you  think                              Server                                 ...
Stateful  servers  are  not  as  hard  as  you  think                              Server                                 ...
Stateful  servers  are  not  as  hard  as  you  think                           Server      Server          Server        ...
With  stateful  server  the  DB  is  less  used                           Ruby  Stateless            Erlang  Stateful     ...
With  stateful  server  the  DB  is  less  used                           Ruby  Stateless            Erlang  Stateful     ...
Deploying  with  a  stateful  server      In  order  to  bring  up  a  new  versionWednesday, March 7, 2012
Deploying  with  a  stateful  server      In  order  to  bring  up  a  new  version      Just  deploy  it         Hot  cod...
There  are  even  more  advantages      Faster  than  Ruby  (5,000  rps  /  node)                  -­‐ CPU  boundWednesday...
There  are  even  more  advantages      Faster  than  Ruby  (5,000  rps  /  node)                  -­‐ CPU  bound      Ver...
There  are  even  more  advantages      Faster  than  Ruby  (5,000  rps  /  node)                  -­‐ CPU  bound      Ver...
Example  “controller”  in  ErlangWednesday, March 7, 2012
Example  “controller”  in  ErlangWednesday, March 7, 2012
Example  “controller”  in  ErlangWednesday, March 7, 2012
Example  “controller”  in  ErlangWednesday, March 7, 2012
Example  “controller”  in  Erlang      Central  handling  of  effectsWednesday, March 7, 2012
Example  “controller”  in  Erlang      Central  handling  of  effects      TransacEonal  behaviorWednesday, March 7, 2012
Example  model  in  ErlangWednesday, March 7, 2012
Example  model  in  ErlangWednesday, March 7, 2012
Example  model  in  ErlangWednesday, March 7, 2012
Example  model  in  ErlangWednesday, March 7, 2012
Example  model  in  ErlangWednesday, March 7, 2012
Example  model  in  Erlang      Erlang  code  is  not  that  hard  to  read,  isn’t  it?Wednesday, March 7, 2012
http://www.flickr.com/photos/hotreactor/Wednesday, March 7, 2012
Architecture  EvoluEon  at  Wooga          The  Start:  Ruby          The  Next  Step:  Erlang          Best  of  Two  Wor...
Aug  2011:  4th  team  wanted  both      Erlang  is  great         Concurrency,  robustness         Great  for  opera0onWe...
Aug  2011:  4th  team  wanted  both      Erlang  is  great         Concurrency,  robustness         Great  for  opera0on  ...
Aug  2011:  4th  team  wanted  both      Erlang  is  great         Concurrency,  robustness         Great  for  opera0on  ...
Aug  2011:  4th  team  wanted  both      Erlang  is  great         Concurrency,  robustness         Great  for  opera0on  ...
EvoluEon  IV:  The  best  out  of  two  worlds                                                               Aug  2011    ...
The  basic  setup  looks  exactly  like  before                           Server      Server          Server              ...
Example  controller  in  RubyWednesday, March 7, 2012
Example  controller  in  RubyWednesday, March 7, 2012
Example  controller  in  RubyWednesday, March 7, 2012
Example  controller  in  RubyWednesday, March 7, 2012
Example  controller  in  Ruby      DSL-­‐like  definiEon  of  game  acEonWednesday, March 7, 2012
Example  controller  in  Ruby      DSL-­‐like  definiEon  of  game  acEon      Skinny  as  controllers  should  beWednesday...
Example  model  in  RubyWednesday, March 7, 2012
Example  model  in  RubyWednesday, March 7, 2012
Example  model  in  RubyWednesday, March 7, 2012
Example  model  in  RubyWednesday, March 7, 2012
Example  model  in  Ruby      Easily  unit  testableWednesday, March 7, 2012
Example  model  in  Ruby      Easily  unit  testable      Minimal  amount  of  codeWednesday, March 7, 2012
Bringing  2  worlds  together                                     Server                           session                ...
Bringing  2  worlds  together                                     Server                           session                ...
Bringing  2  worlds  together                                     Server            Worker                           sessi...
Bringing  2  worlds  together                                     Server              Worker                           ses...
Bringing  2  worlds  together                                     Server              Worker                           ses...
Bringing  2  worlds  together                                     Server              Worker                           ses...
Game  state      Game  state  is  split  in  mulEple  parts        user,  map,  fruit_trees  etc.Wednesday, March 7, 2012
Game  state      Game  state  is  split  in  mulEple  parts        user,  map,  fruit_trees  etc.      Erlang  does  not  ...
Game  state      Game  state  is  split  in  mulEple  parts        user,  map,  fruit_trees  etc.      Erlang  does  not  ...
Looking  back  at  the  game  acEonWednesday, March 7, 2012
Looking  back  at  the  game  acEonWednesday, March 7, 2012
Looking  back  at  the  game  acEon      Mapping  of  state  parts  to  game  acEonsWednesday, March 7, 2012
Looking  back  at  the  game  acEon      Mapping  of  state  parts  to  game  acEons        Worker  knows  mappingWednesda...
Looking  back  at  the  game  acEon      Mapping  of  state  parts  to  game  acEons        Worker  knows  mapping        ...
Looking  back  at  the  game  acEon      Mapping  of  state  parts  to  game  acEons        Worker  knows  mapping        ...
NICE!                           http://www.flickr.com/photos/aigle_dore/Wednesday, March 7, 2012
Architecture  EvoluEon  at  Wooga          The  Start:  Ruby          The  Next  Step:  Erlang          Best  of  Two  Wor...
Each  new  game  brought  us  innovaEon                                                               Aug  2011           ...
We’ve  learned  to    value                           Small teams                               over                      ...
We’ve  learned  to    value                           Collaboration                                over                   ...
We’ve  learned  to    value                           Generalists                              over                       ...
We’ve  learned  to    value                  Effort reduction                        over                   cost reduction...
We’ve  learned  to    value                             Innovation                                over                    ...
A  good  value  system      We’ve  learned  to  value      Small  teams                              over     Big  teams  ...
It works!Wednesday, March 7, 2012
It works!                 Be fast, be bold!Wednesday, March 7, 2012
QuesEons?                           Jesper  Richter-­‐Reichhelm                                     @jrirei               ...
Upcoming SlideShare
Loading in...5
×

Games for the Masses (QCon London 2012)

3,848

Published on

Building a backend for a successful social game is always challenging: It needs to service over 1 million users per day that generate 10.000 http requests per second or more whereas the vast majority of those requests are changing persistent state. Using a conventional technology stack that leads to over 50,000 database writes per second. Throughout the last two years half a dozen teams at Wooga have set out to build a backends for social games, each trying to improve on previous solutions. Each team was able to leverage experiences made by other teams but was free to choose their own technology stack and hosting environment. They also operated the game themselves in a DevOps way. This talk will trace back that evolution of backends: Starting out with a simple LAMP stack, first replacing PHP by Ruby, then replacing relational by NoSQL databases and ending up in maintaining stateful application servers utilizing Erlang OTP - and more. We will discuss limitations and problems we faced in live operation and show how later teams improved on the overall design.

Published in: Technology

Games for the Masses (QCon London 2012)

  1. 1. GAMES  FOR  THE  MASSES How  DevOps  Affects  Architecture Jesper  Richter-­‐Reichhelm,  @jrireiWednesday, March 7, 2012
  2. 2. Wednesday, March 7, 2012
  3. 3. Our  games  all  look  the  same Flash  client BackendWednesday, March 7, 2012
  4. 4. Our  games  all  look  the  same Flash  client Game  Session Asynch.  CommunicaEonWednesday, March 7, 2012
  5. 5. Our  games  all  look  the  same Backend State  Changes ValidaEon PersistenceWednesday, March 7, 2012
  6. 6. But  the  scale  is  interesEng 14  billion  requests  /  monthWednesday, March 7, 2012
  7. 7. But  the  scale  is  interesEng 14  billion  requests  /  monthWednesday, March 7, 2012
  8. 8. But  the  scale  is  interesEng 14  billion  requests  /  month >100,000  DB  operaEons  /  secondWednesday, March 7, 2012
  9. 9. But  the  scale  is  interesEng 14  billion  requests  /  month >100,000  DB  operaEons  /  second >50,000  DB  updates  /  secondWednesday, March 7, 2012
  10. 10. 2  Developers  to  do  it  all Typical  team  setup 4  product  managers 4  ar0sts 4  frontend  engineers 2  backend  engineers -­‐ design,  implementa6on,  opera6onWednesday, March 7, 2012
  11. 11. Wooga  has  dedicated  game  teamsWednesday, March 7, 2012
  12. 12. Wednesday, March 7, 2012
  13. 13. Wednesday, March 7, 2012
  14. 14. Oct  2009Wednesday, March 7, 2012
  15. 15. Oct  2009 Jan  2010Wednesday, March 7, 2012
  16. 16. Oct  2009 Jan  2010 Oct  2010Wednesday, March 7, 2012
  17. 17. Oct  2009 Jan  2010 Oct  2010 Aug  2012Wednesday, March 7, 2012
  18. 18. Architecture  EvoluEon  at  Wooga The  Start The  Next  Step Best  of  Two  Worlds Company  ValuesWednesday, March 7, 2012
  19. 19. Oct  2009:  1st  team  wanted  good  code  quality Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactorWednesday, March 7, 2012
  20. 20. Oct  2009:  1st  team  wanted  good  code  quality Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactorWednesday, March 7, 2012
  21. 21. EvoluEon  I:  Use  Ruby  (on  Rails) Oct  2009Wednesday, March 7, 2012
  22. 22. A  basic  setup  using  sharding  worked  fine lb app app app app app app app app app My My SQL SQL slave slaveWednesday, March 7, 2012
  23. 23. 250K  daily  users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life  was  good !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  24. 24. 250K  daily  users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life  was  good NO  MORE !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  25. 25. Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessaryWednesday, March 7, 2012
  26. 26. Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessary Numerous  small  fixes  regarding  DB  configWednesday, March 7, 2012
  27. 27. Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessary Numerous  small  fixes  regarding  DB  config More  shardsWednesday, March 7, 2012
  28. 28. Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessary Numerous  small  fixes  regarding  DB  config More  shards Even  more  shardsWednesday, March 7, 2012
  29. 29. Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessary Numerous  small  fixes  regarding  DB  config More  shards Even  more  shards SpliBng  the  model  to  get  more  shardsWednesday, March 7, 2012
  30. 30. Early  sharding  hell:  8  master  and  8  slaves lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slaveWednesday, March 7, 2012
  31. 31. At  500K  daily  users  we  were  at  a  dead  end &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  32. 32. OUCH!http://www.flickr.com/photos/billue_the_bear/Wednesday, March 7, 2012
  33. 33. Jan  2010:  Meanwhile  at  the  2nd  team Don’t  break  the  bank Make  it  faster Make  it  cheaper Make  it  simplerWednesday, March 7, 2012
  34. 34. Jan  2010:  Meanwhile  at  the  2nd  team Don’t  break  the  bank Make  it  faster Make  it  cheaper Make  it  simplerWednesday, March 7, 2012
  35. 35. EvoluEon  II:  Use  Redis  as  main  database Jan  2010 Oct  2009Wednesday, March 7, 2012
  36. 36. If  MySQL  is  a  truck Fast  enough Disk  based Robust Fast  enough                    disk  based                    robustWednesday, March 7, 2012
  37. 37. If  MySQL  is  a  truck,  Redis  is  a  race  car Super  fast RAM  based Fragile Super  fast                    RAM  based                    fragileWednesday, March 7, 2012
  38. 38. Bare  metal  for  low  latency! lb app app app app app app app Re-­‐ Re-­‐ dis dis disk (S3)Wednesday, March 7, 2012
  39. 39. How  could  we  apply  that  knowledge? &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  40. 40. On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  41. 41. On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  42. 42. On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  43. 43. On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  44. 44. On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  45. 45. On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  46. 46. On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  47. 47. Typical  migraEon  throughput  over  3  daysWednesday, March 7, 2012
  48. 48. Big  and  staEc  data  in  MySQL,  rest  goes  to  Redis 256  GB  data 60  GB  data 10%  writes 50%  writes hCp://www.flickr.com/photos/erix/245657047/Wednesday, March 7, 2012
  49. 49. One  team  saved  the  other  one &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  50. 50. One  team  saved  the  other  one &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  51. 51. We  now  have  more  than  2  million  users  /  day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  52. 52. We  now  have  more  than  2  million  users  /  day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" AWS  outage #!!$!!!" in  Ireland !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  53. 53. 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lb app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app My My My My My redis redis redis redis redis SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave slave slaveWednesday, March 7, 2012
  54. 54. http://www.flickr.com/photos/wolfsavard/OUCH!Wednesday, March 7, 2012
  55. 55. Architecture  EvoluEon  at  Wooga The  Start:  Ruby The  Next  Step Best  of  Two  Worlds Company  ValuesWednesday, March 7, 2012
  56. 56. Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  57. 57. Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  58. 58. Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  59. 59. Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  60. 60. Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  61. 61. Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  62. 62. Stateful  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  63. 63. Stateful  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  64. 64. Stateful  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  65. 65. Stateful  servers  and  DBs Server Database One  Game  SessionWednesday, March 7, 2012
  66. 66. Stateful  servers  and  DBs Server Database One  Game  SessionWednesday, March 7, 2012
  67. 67. Oct  2010:  3rd  team  used  a  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robustWednesday, March 7, 2012
  68. 68. Oct  2010:  3rd  team  used  a  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robustWednesday, March 7, 2012
  69. 69. EvoluEon  III:  Use  Erlang  for  a  stateful  server Oct  2010 Jan  2010 Oct  2009Wednesday, March 7, 2012
  70. 70. Stateful  servers  are  not  as  hard  as  you  think sessionWednesday, March 7, 2012
  71. 71. Stateful  servers  are  not  as  hard  as  you  think session session session sessionWednesday, March 7, 2012
  72. 72. Stateful  servers  are  not  as  hard  as  you  think Server session session session sessionWednesday, March 7, 2012
  73. 73. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  74. 74. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  75. 75. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  76. 76. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  77. 77. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  78. 78. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  79. 79. Stateful  servers  are  not  as  hard  as  you  think Server Server Server session session session session session session session session session session session session S3Wednesday, March 7, 2012
  80. 80. With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 7,500 0 database  operations  /  secWednesday, March 7, 2012
  81. 81. With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 700 7,500 0 database  operations  /  secWednesday, March 7, 2012
  82. 82. Deploying  with  a  stateful  server In  order  to  bring  up  a  new  versionWednesday, March 7, 2012
  83. 83. Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version Just  deploy  it Hot  code  replacement  is  great!Wednesday, March 7, 2012
  84. 84. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  boundWednesday, March 7, 2012
  85. 85. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recoverWednesday, March 7, 2012
  86. 86. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recover TransacEonal  logic -­‐ Invariants  instead  of  explicit  error  handlingWednesday, March 7, 2012
  87. 87. Example  “controller”  in  ErlangWednesday, March 7, 2012
  88. 88. Example  “controller”  in  ErlangWednesday, March 7, 2012
  89. 89. Example  “controller”  in  ErlangWednesday, March 7, 2012
  90. 90. Example  “controller”  in  ErlangWednesday, March 7, 2012
  91. 91. Example  “controller”  in  Erlang Central  handling  of  effectsWednesday, March 7, 2012
  92. 92. Example  “controller”  in  Erlang Central  handling  of  effects TransacEonal  behaviorWednesday, March 7, 2012
  93. 93. Example  model  in  ErlangWednesday, March 7, 2012
  94. 94. Example  model  in  ErlangWednesday, March 7, 2012
  95. 95. Example  model  in  ErlangWednesday, March 7, 2012
  96. 96. Example  model  in  ErlangWednesday, March 7, 2012
  97. 97. Example  model  in  ErlangWednesday, March 7, 2012
  98. 98. Example  model  in  Erlang Erlang  code  is  not  that  hard  to  read,  isn’t  it?Wednesday, March 7, 2012
  99. 99. http://www.flickr.com/photos/hotreactor/Wednesday, March 7, 2012
  100. 100. Architecture  EvoluEon  at  Wooga The  Start:  Ruby The  Next  Step:  Erlang Best  of  Two  Worlds Company  ValuesWednesday, March 7, 2012
  101. 101. Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera0onWednesday, March 7, 2012
  102. 102. Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera0on Ruby  is  great Concise,  expressive,  testable Great  for  developmentWednesday, March 7, 2012
  103. 103. Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera0on Ruby  is  great Concise,  expressive,  testable Great  for  developmentWednesday, March 7, 2012
  104. 104. Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera0on Ruby  is  great Concise,  expressive,  testable Great  for  developmentWednesday, March 7, 2012
  105. 105. EvoluEon  IV:  The  best  out  of  two  worlds Aug  2011 Oct  2010 Jan  2010 Oct  2009Wednesday, March 7, 2012
  106. 106. The  basic  setup  looks  exactly  like  before Server Server Server session session session session session session session session session session session session S3Wednesday, March 7, 2012
  107. 107. Example  controller  in  RubyWednesday, March 7, 2012
  108. 108. Example  controller  in  RubyWednesday, March 7, 2012
  109. 109. Example  controller  in  RubyWednesday, March 7, 2012
  110. 110. Example  controller  in  RubyWednesday, March 7, 2012
  111. 111. Example  controller  in  Ruby DSL-­‐like  definiEon  of  game  acEonWednesday, March 7, 2012
  112. 112. Example  controller  in  Ruby DSL-­‐like  definiEon  of  game  acEon Skinny  as  controllers  should  beWednesday, March 7, 2012
  113. 113. Example  model  in  RubyWednesday, March 7, 2012
  114. 114. Example  model  in  RubyWednesday, March 7, 2012
  115. 115. Example  model  in  RubyWednesday, March 7, 2012
  116. 116. Example  model  in  RubyWednesday, March 7, 2012
  117. 117. Example  model  in  Ruby Easily  unit  testableWednesday, March 7, 2012
  118. 118. Example  model  in  Ruby Easily  unit  testable Minimal  amount  of  codeWednesday, March 7, 2012
  119. 119. Bringing  2  worlds  together Server session session ... sessionWednesday, March 7, 2012
  120. 120. Bringing  2  worlds  together Server session sender session ... sessionWednesday, March 7, 2012
  121. 121. Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker session WorkerWednesday, March 7, 2012
  122. 122. Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session WorkerWednesday, March 7, 2012
  123. 123. Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session WorkerWednesday, March 7, 2012
  124. 124. Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session WorkerWednesday, March 7, 2012
  125. 125. Game  state Game  state  is  split  in  mulEple  parts user,  map,  fruit_trees  etc.Wednesday, March 7, 2012
  126. 126. Game  state Game  state  is  split  in  mulEple  parts user,  map,  fruit_trees  etc. Erlang  does  not  care  about  content Serialized  Ruby  objectsWednesday, March 7, 2012
  127. 127. Game  state Game  state  is  split  in  mulEple  parts user,  map,  fruit_trees  etc. Erlang  does  not  care  about  content Serialized  Ruby  objects Erlang  does  know  mapping  of  state  parts  to  URLs Mapping  provided  by  Ruby  on  startupWednesday, March 7, 2012
  128. 128. Looking  back  at  the  game  acEonWednesday, March 7, 2012
  129. 129. Looking  back  at  the  game  acEonWednesday, March 7, 2012
  130. 130. Looking  back  at  the  game  acEon Mapping  of  state  parts  to  game  acEonsWednesday, March 7, 2012
  131. 131. Looking  back  at  the  game  acEon Mapping  of  state  parts  to  game  acEons Worker  knows  mappingWednesday, March 7, 2012
  132. 132. Looking  back  at  the  game  acEon Mapping  of  state  parts  to  game  acEons Worker  knows  mapping Worker  pushes  mapping  to  Erlang  on  startupWednesday, March 7, 2012
  133. 133. Looking  back  at  the  game  acEon Mapping  of  state  parts  to  game  acEons Worker  knows  mapping Worker  pushes  mapping  to  Erlang  on  startup Erlang  can  query  mapping  if  neededWednesday, March 7, 2012
  134. 134. NICE! http://www.flickr.com/photos/aigle_dore/Wednesday, March 7, 2012
  135. 135. Architecture  EvoluEon  at  Wooga The  Start:  Ruby The  Next  Step:  Erlang Best  of  Two  Worlds Company  ValuesWednesday, March 7, 2012
  136. 136. Each  new  game  brought  us  innovaEon Aug  2011 Oct  2010 Jan  2010 Oct  2009Wednesday, March 7, 2012
  137. 137. We’ve  learned  to    value Small teams over big teamsWednesday, March 7, 2012
  138. 138. We’ve  learned  to    value Collaboration over competitionWednesday, March 7, 2012
  139. 139. We’ve  learned  to    value Generalists over specialistsWednesday, March 7, 2012
  140. 140. We’ve  learned  to    value Effort reduction over cost reductionWednesday, March 7, 2012
  141. 141. We’ve  learned  to    value Innovation over risk mitigationWednesday, March 7, 2012
  142. 142. A  good  value  system We’ve  learned  to  value Small  teams                              over   Big  teams   CollaboraEon                        over Compe00on Generalists                                over Specialists Effort  reducEon                over Cost  reduc0on InnovaEon                                  over Risk  mi0ga0onWednesday, March 7, 2012
  143. 143. It works!Wednesday, March 7, 2012
  144. 144. It works! Be fast, be bold!Wednesday, March 7, 2012
  145. 145. QuesEons? Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobsWednesday, March 7, 2012
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×