Games for the Masses (Jax)

3,715 views
2,667 views

Published on

Ein Backend für ein weltweit erfolgreiches Social Game zu entwickeln ist nicht einfach, aber die eigentliche Herausforderung ist der Betrieb der Systeme! Bei Wooga sind dieselben zwei bis drei Entwickler für beides verantwortlich. Über die letzten zwei Jahren hat sich ein halbes Dutzend Teams dieser Herausforderung gestellt. Dabei konnten sie auf die Erfahrungen der vorhergehenden Teams zurückgreifen und hatten die Freiheit, eigene Lösungsansätze zu verfolgen. Der Vortrag wird die entstandene Evolution der Backends nachvollziehen: Anfangs LAMP, dann Ruby statt PHP, dann NoSQL statt MySQL und am Ende ohne Datenbank auf Basis von Erlang OTP. Die Darstellung dieser Reise wird anschaulich zeigen, welche Vorteile es haben kann, Entwicklern auch einmal freie Hand zu lassen.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,715
On SlideShare
0
From Embeds
0
Number of Embeds
13
Actions
Shares
0
Downloads
51
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

Games for the Masses (Jax)

  1. GAMES  FOR  THE  MASSES Wie  DevOps  die  Entwicklung  von  Architektur  verändertJesper  Richter-­‐Reichhelm,  @jrirei
  2. DEV  -­‐  OPS
  3. DEV  -­‐  OPSSome  say:  Admins  learn  to  be  ‘Agile’ • Scrum  and  Kanban
  4. DEV  -­‐  OPSSome  say:  Admins  learn  to  be  ‘Agile’ • Scrum  and  KanbanOthers  say:  Admins  learn  to  program  their  setup • Chef,  Puppet
  5. DEV  -­‐  OPSSome  say:  Admins  learn  to  be  ‘Agile’ • Scrum  and  KanbanOthers  say:  Admins  learn  to  program  their  setup • Chef,  PuppetI  say:  This  is  all  nice,  but  not  enough • Developers  need  to  learn  a  lot  about  opera=on
  6. Classic  roles  in  IT Arch Dev Op
  7. Classic  roles  in  IT Arch Dev Op
  8. Classic  roles  in  IT Arch Dev Op
  9. Classic  roles  in  IT Arch Dev Op
  10. Classic  roles  in  IT Arch Dev Op
  11. Classic  roles  in  IT Arch Dev Op
  12. Classic  roles  in  IT Arch Dev Op
  13. Classic  roles  in  IT Arch Dev Op
  14. Classic  roles  in  IT Arch Dev Op
  15. Classic  roles  in  IT Arch Dev Op ???
  16. Our  games  all  look  the  same Flash  client Backend
  17. Our  games  all  look  the  same Flash  client Game  Session Asynch.  CommunicaMon
  18. Our  games  all  look  the  same Backend State  Changes ValidaMon Persistence
  19. But  the  scale  is  interesMng 14  billion  requests  /  month
  20. But  the  scale  is  interesMng 14  billion  requests  /  month
  21. But  the  scale  is  interesMng 14  billion  requests  /  month >100,000  DB  operaMons  /  second
  22. But  the  scale  is  interesMng 14  billion  requests  /  month >100,000  DB  operaMons  /  second >50,000  DB  updates  /  second
  23. Classic  roles  in  IT Arch Dev Op ???
  24. Roles  in  Wooga’s  IT
  25. Roles  in  Wooga’s  IT Arch
  26. Roles  in  Wooga’s  IT Arch Dev
  27. Roles  in  Wooga’s  IT Arch Dev Op
  28. Roles  in  Wooga’s  IT Arch Arch Dev Dev Op Op
  29. 2  Developers  to  do  it  all Typical  team  setup 4  product  managers 4  ar=sts 4  frontend  engineers 2  backend  engineers -­‐ design,  implementa7on,  opera7on
  30. Wooga  has  dedicated  game  teams
  31. Oct  2009
  32. Oct  2009 Jan  2010
  33. Oct  2009 Jan  2010Oct  2010
  34. Oct  2009 Jan  2010Oct  2010 Aug  2011
  35. Architecture  EvoluMon  at  Wooga The  Start The  Next  Step Best  of  Two  Worlds Company  Values
  36. Oct  2009:  1st  team  wanted  good  code  quality Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactor
  37. Oct  2009:  1st  team  wanted  good  code  quality Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactor
  38. EvoluMon  I:  Use  Ruby  (on  Rails) Oct  2009
  39. A  basic  setup  using  sharding  worked  fine lb app app app app app app app app app My My SQL SQL slave slave
  40. 250K  daily  users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" Life  was  good !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  41. 250K  daily  users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" Life  was  good NO  MORE !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  42. Welcome  to  6  weeks  of  pain! Heavy  opMmizaMons  were  necessary
  43. Welcome  to  6  weeks  of  pain! Heavy  opMmizaMons  were  necessary Numerous  small  fixes  regarding  DB  config
  44. Welcome  to  6  weeks  of  pain! Heavy  opMmizaMons  were  necessary Numerous  small  fixes  regarding  DB  config More  shards
  45. Welcome  to  6  weeks  of  pain! Heavy  opMmizaMons  were  necessary Numerous  small  fixes  regarding  DB  config More  shards Even  more  shards
  46. Welcome  to  6  weeks  of  pain! Heavy  opMmizaMons  were  necessary Numerous  small  fixes  regarding  DB  config More  shards Even  more  shards SpliNng  the  model  to  get  more  shards
  47. Early  sharding  hell:  8  master  and  8  slaves lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave
  48. At  500K  daily  users  we  were  at  a  dead  end&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  49. OUCH!http://www.flickr.com/photos/billue_the_bear/
  50. Jan  2010:  Meanwhile  at  the  2nd  team Don’t  break  the  bank Make  it  faster Make  it  cheaper Make  it  simpler
  51. Jan  2010:  Meanwhile  at  the  2nd  team Don’t  break  the  bank Make  it  faster Make  it  cheaper Make  it  simpler
  52. EvoluMon  II:  Use  Redis  as  main  database Jan  2010 Oct  2009
  53. If  MySQL  is  a  truck Fast  enough Disk  based Robust Fast  enough                    disk  based                    robust
  54. If  MySQL  is  a  truck,  Redis  is  a  race  car Super  fast RAM  based Fragile Super  fast                    RAM  based                    fragile
  55. Bare  metal  for  low  latency! lb app app app app app app app Re-­‐ Re-­‐ dis dis disk (S3)
  56. How  could  we  apply  that  knowledge?&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  57. Migrate  data  from  MySQL  to  Redis
  58. Migrate  data  from  MySQL  to  Redis No  downMmes  allowed
  59. Migrate  data  from  MySQL  to  Redis No  downMmes  allowed Migrate  data  on  demand
  60. Migrate  data  from  MySQL  to  Redis No  downMmes  allowed Migrate  data  on  demand On  every  access  check  if  data  needs  migra=on
  61. Migrate  data  from  MySQL  to  Redis No  downMmes  allowed Migrate  data  on  demand On  every  access  check  if  data  needs  migra=on Migrate  if  needed
  62. Migrate  data  from  MySQL  to  Redis No  downMmes  allowed Migrate  data  on  demand On  every  access  check  if  data  needs  migra=on Migrate  if  needed Con=nue  using  Redis
  63. Typical  migraMon  throughput  over  3  days
  64. Big  and  staMc  data  in  MySQL,  rest  goes  to  Redis 256  GB  data 60  GB  data 10%  writes 50%  writes hOp://www.flickr.com/photos/erix/245657047/
  65. One  team  saved  the  other  one&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  66. One  team  saved  the  other  one&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  67. We  now  have  more  than  2  million  users  /  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  68. We  now  have  more  than  2  million  users  /  day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" AWS  outage #!!$!!!" in  Ireland !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"
  69. 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app appMy My My My My redis redis redis redis redisSQL SQL SQL SQL SQLslave slave slave slave slave slave slave slave slave slave
  70. http://www.flickr.com/photos/wolfsavard/OUCH!
  71. Architecture  EvoluMon  at  Wooga The  Start:  Ruby The  Next  Step Best  of  Two  Worlds Company  Values
  72. Stateless  servers  and  DBs Server Database
  73. Stateless  servers  and  DBs Server Database
  74. Stateless  servers  and  DBs Server Database
  75. Stateless  servers  and  DBs Server Database
  76. Stateless  servers  and  DBs Server Database
  77. Stateless  servers  and  DBs Server Database
  78. Stateful  servers  and  DBs Server Database
  79. Stateful  servers  and  DBs Server Database
  80. Stateful  servers  and  DBs Server Database
  81. Stateful  servers  and  DBs Server Database One  Game  Session
  82. Stateful  servers  and  DBs Server Database One  Game  Session
  83. Oct  2010:  3rd  team  used  a  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robust
  84. Oct  2010:  3rd  team  used  a  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robust
  85. EvoluMon  III:  Use  Erlang  for  a  stateful  server Oct  2010 Jan  2010 Oct  2009
  86. Stateful  servers  are  not  as  hard  as  you  think session
  87. Stateful  servers  are  not  as  hard  as  you  think session session session session
  88. Stateful  servers  are  not  as  hard  as  you  think Server session session session session
  89. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  90. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  91. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  92. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  93. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  94. Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3
  95. Stateful  servers  are  not  as  hard  as  you  think Server Server Server session session session session session session session session session session session session S3
  96. With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 7,500 0 database  operations  /  sec
  97. With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 700 7,500 0 database  operations  /  sec
  98. Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version
  99. Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version Just  deploy  it Hot  code  replacement  is  great!
  100. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound
  101. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recover
  102. There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recover TransacMonal  logic -­‐ Invariants  instead  of  explicit  error  handling
  103. Example  “controller”  in  Erlang
  104. Example  “controller”  in  Erlang
  105. Example  “controller”  in  Erlang
  106. Example  “controller”  in  Erlang
  107. Example  “controller”  in  Erlang TransacMonal  behavior
  108. Example  “controller”  in  Erlang TransacMonal  behavior Erlang  code  is  not  that  hard  to  read,  isn’t  it?
  109. http://www.flickr.com/photos/hotreactor/
  110. Architecture  EvoluMon  at  Wooga The  Start:  Ruby The  Next  Step:  Erlang Best  of  Two  Worlds Company  Values
  111. Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera=on
  112. Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera=on Ruby  is  great Concise,  expressive,  testable Great  for  development
  113. Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera=on Ruby  is  great Concise,  expressive,  testable Great  for  development
  114. Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera=on Ruby  is  great Concise,  expressive,  testable Great  for  development
  115. EvoluMon  IV:  The  best  out  of  two  worlds Aug  2011 Oct  2010 Jan  2010 Oct  2009
  116. The  basic  setup  looks  exactly  like  before Server Server Server session session session session session session session session session session session session S3
  117. Bringing  2  worlds  together Server session session ... session
  118. Bringing  2  worlds  together Server session sender session ... session
  119. Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker session Worker
  120. Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session Worker
  121. Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session Worker
  122. Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session Worker
  123. Game  state Game  state  is  split  in  mulMple  parts user,  map,  fruit_trees  etc.
  124. Game  state Game  state  is  split  in  mulMple  parts user,  map,  fruit_trees  etc. Erlang  does  not  care  about  content Serialized  Ruby  objects
  125. Game  state Game  state  is  split  in  mulMple  parts user,  map,  fruit_trees  etc. Erlang  does  not  care  about  content Serialized  Ruby  objects Erlang  does  know  mapping  of  state  parts  to  URLs Mapping  provided  by  Ruby  on  startup
  126. Looking  back  at  the  game  acMon
  127. Looking  back  at  the  game  acMon
  128. Looking  back  at  the  game  acMon Mapping  of  state  parts  to  game  acMons
  129. Looking  back  at  the  game  acMon Mapping  of  state  parts  to  game  acMons Worker  knows  mapping
  130. Looking  back  at  the  game  acMon Mapping  of  state  parts  to  game  acMons Worker  knows  mapping Worker  pushes  mapping  to  Erlang  on  startup
  131. Looking  back  at  the  game  acMon Mapping  of  state  parts  to  game  acMons Worker  knows  mapping Worker  pushes  mapping  to  Erlang  on  startup Erlang  can  query  mapping  if  needed
  132. NICE! http://www.flickr.com/photos/aigle_dore/
  133. Architecture  EvoluMon  at  Wooga The  Start:  Ruby The  Next  Step:  Erlang Best  of  Two  Worlds Company  Values
  134. Each  new  game  brought  us  innovaMon Aug  2011 Oct  2010 Jan  2010 Oct  2009
  135. We’ve  learned  to    value Small teams over big teams
  136. We’ve  learned  to    value Collaboration over competition
  137. We’ve  learned  to    value Generalists over specialists
  138. We’ve  learned  to    value Effort reduction over cost reduction
  139. We’ve  learned  to    value Innovation over risk mitigation
  140. A  good  value  system We’ve  learned  to  value Small  teams                              over   Big  teams   CollaboraMon                        over Compe==on Generalists                                over Specialists Effort  reducMon                over Cost  reduc=on InnovaMon                                  over Risk  mi=ga=on
  141. It works!
  142. It works!Be fast, be bold!
  143. QuesMons?Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs

×