Games for the Masses (QCon London 2012)
Upcoming SlideShare
Loading in...5
×
 

Games for the Masses (QCon London 2012)

on

  • 4,111 views

Building a backend for a successful social game is always challenging: It needs to service over 1 million users per day that generate 10.000 http requests per second or more whereas the vast majority ...

Building a backend for a successful social game is always challenging: It needs to service over 1 million users per day that generate 10.000 http requests per second or more whereas the vast majority of those requests are changing persistent state. Using a conventional technology stack that leads to over 50,000 database writes per second. Throughout the last two years half a dozen teams at Wooga have set out to build a backends for social games, each trying to improve on previous solutions. Each team was able to leverage experiences made by other teams but was free to choose their own technology stack and hosting environment. They also operated the game themselves in a DevOps way. This talk will trace back that evolution of backends: Starting out with a simple LAMP stack, first replacing PHP by Ruby, then replacing relational by NoSQL databases and ending up in maintaining stateful application servers utilizing Erlang OTP - and more. We will discuss limitations and problems we faced in live operation and show how later teams improved on the overall design.

Statistics

Views

Total Views
4,111
Views on SlideShare
3,703
Embed Views
408

Actions

Likes
12
Downloads
134
Comments
0

12 Embeds 408

http://irr.posterous.com 322
http://blogs.ua.es 51
http://us-w1.rockmelt.com 9
http://unpocodemistics.blogspot.com 8
https://si0.twimg.com 6
http://www.onlydoo.com 4
http://unpocodemistics.blogspot.com.ar 2
http://www.unpocodemistics.blogspot.com 2
http://a0.twimg.com 1
http://fiddle.jshell.net 1
http://epignosis.talentlms.com 1
https://twimg0-a.akamaihd.net 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Games for the Masses (QCon London 2012) Games for the Masses (QCon London 2012) Presentation Transcript

  • GAMES  FOR  THE  MASSES How  DevOps  Affects  Architecture Jesper  Richter-­‐Reichhelm,  @jrireiWednesday, March 7, 2012
  • Wednesday, March 7, 2012
  • Our  games  all  look  the  same Flash  client BackendWednesday, March 7, 2012
  • Our  games  all  look  the  same Flash  client Game  Session Asynch.  CommunicaEonWednesday, March 7, 2012
  • Our  games  all  look  the  same Backend State  Changes ValidaEon PersistenceWednesday, March 7, 2012
  • But  the  scale  is  interesEng 14  billion  requests  /  monthWednesday, March 7, 2012
  • But  the  scale  is  interesEng 14  billion  requests  /  monthWednesday, March 7, 2012
  • But  the  scale  is  interesEng 14  billion  requests  /  month >100,000  DB  operaEons  /  secondWednesday, March 7, 2012
  • But  the  scale  is  interesEng 14  billion  requests  /  month >100,000  DB  operaEons  /  second >50,000  DB  updates  /  secondWednesday, March 7, 2012
  • 2  Developers  to  do  it  all Typical  team  setup 4  product  managers 4  ar0sts 4  frontend  engineers 2  backend  engineers -­‐ design,  implementa6on,  opera6onWednesday, March 7, 2012
  • Wooga  has  dedicated  game  teamsWednesday, March 7, 2012
  • Wednesday, March 7, 2012
  • Wednesday, March 7, 2012
  • Oct  2009Wednesday, March 7, 2012
  • Oct  2009 Jan  2010Wednesday, March 7, 2012
  • Oct  2009 Jan  2010 Oct  2010Wednesday, March 7, 2012
  • Oct  2009 Jan  2010 Oct  2010 Aug  2012Wednesday, March 7, 2012
  • Architecture  EvoluEon  at  Wooga The  Start The  Next  Step Best  of  Two  Worlds Company  ValuesWednesday, March 7, 2012
  • Oct  2009:  1st  team  wanted  good  code  quality Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactorWednesday, March 7, 2012
  • Oct  2009:  1st  team  wanted  good  code  quality Good  code  quality Easy  to  understand Easy  to  test Easy  to  refactorWednesday, March 7, 2012
  • EvoluEon  I:  Use  Ruby  (on  Rails) Oct  2009Wednesday, March 7, 2012
  • A  basic  setup  using  sharding  worked  fine lb app app app app app app app app app My My SQL SQL slave slaveWednesday, March 7, 2012
  • 250K  daily  users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life  was  good !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  • 250K  daily  users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life  was  good NO  MORE !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  • Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessaryWednesday, March 7, 2012
  • Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessary Numerous  small  fixes  regarding  DB  configWednesday, March 7, 2012
  • Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessary Numerous  small  fixes  regarding  DB  config More  shardsWednesday, March 7, 2012
  • Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessary Numerous  small  fixes  regarding  DB  config More  shards Even  more  shardsWednesday, March 7, 2012
  • Welcome  to  6  weeks  of  pain! Heavy  opEmizaEons  were  necessary Numerous  small  fixes  regarding  DB  config More  shards Even  more  shards SpliBng  the  model  to  get  more  shardsWednesday, March 7, 2012
  • Early  sharding  hell:  8  master  and  8  slaves lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slaveWednesday, March 7, 2012
  • At  500K  daily  users  we  were  at  a  dead  end &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  • OUCH!http://www.flickr.com/photos/billue_the_bear/Wednesday, March 7, 2012
  • Jan  2010:  Meanwhile  at  the  2nd  team Don’t  break  the  bank Make  it  faster Make  it  cheaper Make  it  simplerWednesday, March 7, 2012
  • Jan  2010:  Meanwhile  at  the  2nd  team Don’t  break  the  bank Make  it  faster Make  it  cheaper Make  it  simplerWednesday, March 7, 2012
  • EvoluEon  II:  Use  Redis  as  main  database Jan  2010 Oct  2009Wednesday, March 7, 2012
  • If  MySQL  is  a  truck Fast  enough Disk  based Robust Fast  enough                    disk  based                    robustWednesday, March 7, 2012
  • If  MySQL  is  a  truck,  Redis  is  a  race  car Super  fast RAM  based Fragile Super  fast                    RAM  based                    fragileWednesday, March 7, 2012
  • Bare  metal  for  low  latency! lb app app app app app app app Re-­‐ Re-­‐ dis dis disk (S3)Wednesday, March 7, 2012
  • How  could  we  apply  that  knowledge? &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  • On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  • On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  • On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  • On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  • On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  • On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  • On-­‐demand  migraEons  from  MySQL  to  RedisWednesday, March 7, 2012
  • Typical  migraEon  throughput  over  3  daysWednesday, March 7, 2012
  • Big  and  staEc  data  in  MySQL,  rest  goes  to  Redis 256  GB  data 60  GB  data 10%  writes 50%  writes hCp://www.flickr.com/photos/erix/245657047/Wednesday, March 7, 2012
  • One  team  saved  the  other  one &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  • One  team  saved  the  other  one &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  • We  now  have  more  than  2  million  users  /  day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  • We  now  have  more  than  2  million  users  /  day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" AWS  outage #!!$!!!" in  Ireland !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Wednesday, March 7, 2012
  • 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lb app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app My My My My My redis redis redis redis redis SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave slave slaveWednesday, March 7, 2012
  • http://www.flickr.com/photos/wolfsavard/OUCH!Wednesday, March 7, 2012
  • Architecture  EvoluEon  at  Wooga The  Start:  Ruby The  Next  Step Best  of  Two  Worlds Company  ValuesWednesday, March 7, 2012
  • Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  • Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  • Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  • Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  • Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  • Stateless  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  • Stateful  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  • Stateful  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  • Stateful  servers  and  DBs Server DatabaseWednesday, March 7, 2012
  • Stateful  servers  and  DBs Server Database One  Game  SessionWednesday, March 7, 2012
  • Stateful  servers  and  DBs Server Database One  Game  SessionWednesday, March 7, 2012
  • Oct  2010:  3rd  team  used  a  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robustWednesday, March 7, 2012
  • Oct  2010:  3rd  team  used  a  stateful  server If  DBs  are  the  problem Don’t  use  them Store  state  in  server Need  to  be  robustWednesday, March 7, 2012
  • EvoluEon  III:  Use  Erlang  for  a  stateful  server Oct  2010 Jan  2010 Oct  2009Wednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think sessionWednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think session session session sessionWednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think Server session session session sessionWednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think Server session session session session S3Wednesday, March 7, 2012
  • Stateful  servers  are  not  as  hard  as  you  think Server Server Server session session session session session session session session session session session session S3Wednesday, March 7, 2012
  • With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 7,500 0 database  operations  /  secWednesday, March 7, 2012
  • With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 700 7,500 0 database  operations  /  secWednesday, March 7, 2012
  • Deploying  with  a  stateful  server In  order  to  bring  up  a  new  versionWednesday, March 7, 2012
  • Deploying  with  a  stateful  server In  order  to  bring  up  a  new  version Just  deploy  it Hot  code  replacement  is  great!Wednesday, March 7, 2012
  • There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  boundWednesday, March 7, 2012
  • There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recoverWednesday, March 7, 2012
  • There  are  even  more  advantages Faster  than  Ruby  (5,000  rps  /  node) -­‐ CPU  bound Very  few  SPOFs -­‐ ...  and  those  are  easy  to  recover TransacEonal  logic -­‐ Invariants  instead  of  explicit  error  handlingWednesday, March 7, 2012
  • Example  “controller”  in  ErlangWednesday, March 7, 2012
  • Example  “controller”  in  ErlangWednesday, March 7, 2012
  • Example  “controller”  in  ErlangWednesday, March 7, 2012
  • Example  “controller”  in  ErlangWednesday, March 7, 2012
  • Example  “controller”  in  Erlang Central  handling  of  effectsWednesday, March 7, 2012
  • Example  “controller”  in  Erlang Central  handling  of  effects TransacEonal  behaviorWednesday, March 7, 2012
  • Example  model  in  ErlangWednesday, March 7, 2012
  • Example  model  in  ErlangWednesday, March 7, 2012
  • Example  model  in  ErlangWednesday, March 7, 2012
  • Example  model  in  ErlangWednesday, March 7, 2012
  • Example  model  in  ErlangWednesday, March 7, 2012
  • Example  model  in  Erlang Erlang  code  is  not  that  hard  to  read,  isn’t  it?Wednesday, March 7, 2012
  • http://www.flickr.com/photos/hotreactor/Wednesday, March 7, 2012
  • Architecture  EvoluEon  at  Wooga The  Start:  Ruby The  Next  Step:  Erlang Best  of  Two  Worlds Company  ValuesWednesday, March 7, 2012
  • Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera0onWednesday, March 7, 2012
  • Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera0on Ruby  is  great Concise,  expressive,  testable Great  for  developmentWednesday, March 7, 2012
  • Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera0on Ruby  is  great Concise,  expressive,  testable Great  for  developmentWednesday, March 7, 2012
  • Aug  2011:  4th  team  wanted  both Erlang  is  great Concurrency,  robustness Great  for  opera0on Ruby  is  great Concise,  expressive,  testable Great  for  developmentWednesday, March 7, 2012
  • EvoluEon  IV:  The  best  out  of  two  worlds Aug  2011 Oct  2010 Jan  2010 Oct  2009Wednesday, March 7, 2012
  • The  basic  setup  looks  exactly  like  before Server Server Server session session session session session session session session session session session session S3Wednesday, March 7, 2012
  • Example  controller  in  RubyWednesday, March 7, 2012
  • Example  controller  in  RubyWednesday, March 7, 2012
  • Example  controller  in  RubyWednesday, March 7, 2012
  • Example  controller  in  RubyWednesday, March 7, 2012
  • Example  controller  in  Ruby DSL-­‐like  definiEon  of  game  acEonWednesday, March 7, 2012
  • Example  controller  in  Ruby DSL-­‐like  definiEon  of  game  acEon Skinny  as  controllers  should  beWednesday, March 7, 2012
  • Example  model  in  RubyWednesday, March 7, 2012
  • Example  model  in  RubyWednesday, March 7, 2012
  • Example  model  in  RubyWednesday, March 7, 2012
  • Example  model  in  RubyWednesday, March 7, 2012
  • Example  model  in  Ruby Easily  unit  testableWednesday, March 7, 2012
  • Example  model  in  Ruby Easily  unit  testable Minimal  amount  of  codeWednesday, March 7, 2012
  • Bringing  2  worlds  together Server session session ... sessionWednesday, March 7, 2012
  • Bringing  2  worlds  together Server session sender session ... sessionWednesday, March 7, 2012
  • Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker session WorkerWednesday, March 7, 2012
  • Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session WorkerWednesday, March 7, 2012
  • Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session WorkerWednesday, March 7, 2012
  • Bringing  2  worlds  together Server Worker session sender Worker session Worker ... Worker receiver session WorkerWednesday, March 7, 2012
  • Game  state Game  state  is  split  in  mulEple  parts user,  map,  fruit_trees  etc.Wednesday, March 7, 2012
  • Game  state Game  state  is  split  in  mulEple  parts user,  map,  fruit_trees  etc. Erlang  does  not  care  about  content Serialized  Ruby  objectsWednesday, March 7, 2012
  • Game  state Game  state  is  split  in  mulEple  parts user,  map,  fruit_trees  etc. Erlang  does  not  care  about  content Serialized  Ruby  objects Erlang  does  know  mapping  of  state  parts  to  URLs Mapping  provided  by  Ruby  on  startupWednesday, March 7, 2012
  • Looking  back  at  the  game  acEonWednesday, March 7, 2012
  • Looking  back  at  the  game  acEonWednesday, March 7, 2012
  • Looking  back  at  the  game  acEon Mapping  of  state  parts  to  game  acEonsWednesday, March 7, 2012
  • Looking  back  at  the  game  acEon Mapping  of  state  parts  to  game  acEons Worker  knows  mappingWednesday, March 7, 2012
  • Looking  back  at  the  game  acEon Mapping  of  state  parts  to  game  acEons Worker  knows  mapping Worker  pushes  mapping  to  Erlang  on  startupWednesday, March 7, 2012
  • Looking  back  at  the  game  acEon Mapping  of  state  parts  to  game  acEons Worker  knows  mapping Worker  pushes  mapping  to  Erlang  on  startup Erlang  can  query  mapping  if  neededWednesday, March 7, 2012
  • NICE! http://www.flickr.com/photos/aigle_dore/Wednesday, March 7, 2012
  • Architecture  EvoluEon  at  Wooga The  Start:  Ruby The  Next  Step:  Erlang Best  of  Two  Worlds Company  ValuesWednesday, March 7, 2012
  • Each  new  game  brought  us  innovaEon Aug  2011 Oct  2010 Jan  2010 Oct  2009Wednesday, March 7, 2012
  • We’ve  learned  to    value Small teams over big teamsWednesday, March 7, 2012
  • We’ve  learned  to    value Collaboration over competitionWednesday, March 7, 2012
  • We’ve  learned  to    value Generalists over specialistsWednesday, March 7, 2012
  • We’ve  learned  to    value Effort reduction over cost reductionWednesday, March 7, 2012
  • We’ve  learned  to    value Innovation over risk mitigationWednesday, March 7, 2012
  • A  good  value  system We’ve  learned  to  value Small  teams                              over   Big  teams   CollaboraEon                        over Compe00on Generalists                                over Specialists Effort  reducEon                over Cost  reduc0on InnovaEon                                  over Risk  mi0ga0onWednesday, March 7, 2012
  • It works!Wednesday, March 7, 2012
  • It works! Be fast, be bold!Wednesday, March 7, 2012
  • QuesEons? Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobsWednesday, March 7, 2012