• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale)
 

Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale)

on

  • 1,712 views

(Slides in English) ...

(Slides in English)
Ein Backend für ein weltweit erfolgreiches Social Game zu entwickeln ist nicht einfach, aber die eigentliche Herausforderung ist der Betrieb der Systeme! Bei Wooga sind dieselben zwei bis drei Entwickler für beides verantwortlich. Über die letzten zwei Jahren hat sich ein halbes Dutzend Teams dieser Herausforderung gestellt. Dabei konnten sie auf die Erfahrungen der vorhergehenden Teams zurückgreifen und hatten die Freiheit, eigene Lösungsansätze zu verfolgen. Der Vortrag wird die entstandene Evolution der Backends nachvollziehen: Anfangs LAMP, dann Ruby statt PHP, dann NoSQL statt MySQL und am Ende ohne Datenbank auf Basis von Erlang OTP. Die Darstellung dieser Reise wird anschaulich zeigen, welche Vorteile es haben kann, Entwicklern auch einmal freie Hand zu lassen.

Statistics

Views

Total Views
1,712
Views on SlideShare
1,712
Embed Views
0

Actions

Likes
6
Downloads
33
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale) Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale) Presentation Transcript

    • GAMES  FOR  THE  MASSES Wie  DevOps  die  Entwicklung  von  Architektur  verändert Jesper  Richter-­‐Reichhelm,  @jrireiMonday, June 4, 2012
    • DEV  -­‐  OPSMonday, June 4, 2012
    • DEV  -­‐  OPS Some  say:  Admins  learn  to  be  ‘Agile’ Scrum  and  KanbanMonday, June 4, 2012
    • DEV  -­‐  OPS Some  say:  Admins  learn  to  be  ‘Agile’ Scrum  and  Kanban Others  say:  Admins  learn  to  program  their  setup Chef,  PuppetMonday, June 4, 2012
    • DEV  -­‐  OPS Some  say:  Admins  learn  to  be  ‘Agile’ Scrum  and  Kanban Others  say:  Admins  learn  to  program  their  setup Chef,  Puppet I  say:  This  is  all  nice,  but  not  enough Developers  need  to  learn  a  lot  about  opera?onMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev OpMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev OpMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev OpMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev OpMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev OpMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev OpMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev OpMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev OpMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev OpMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev Op ???Monday, June 4, 2012
    • Monday, June 4, 2012
    • Our  games  all  look  the  same Flash  client BackendMonday, June 4, 2012
    • Our  games  all  look  the  same Flash  client Game  Session Asynch.  CommunicaPonMonday, June 4, 2012
    • Our  games  all  look  the  same Backend State  Changes ValidaPon PersistenceMonday, June 4, 2012
    • But  the  scale  is  interesPng 14  billion  requests  /  monthMonday, June 4, 2012
    • But  the  scale  is  interesPng 14  billion  requests  /  monthMonday, June 4, 2012
    • But  the  scale  is  interesPng 14  billion  requests  /  month >100,000  DB  operaPons  /  secondMonday, June 4, 2012
    • But  the  scale  is  interesPng 14  billion  requests  /  month >100,000  DB  operaPons  /  second >50,000  DB  updates  /  secondMonday, June 4, 2012
    • Classic  roles  in  IT Arch Dev Op ???Monday, June 4, 2012
    • Roles  in  Wooga’s  ITMonday, June 4, 2012
    • Roles  in  Wooga’s  IT ArchMonday, June 4, 2012
    • Roles  in  Wooga’s  IT Arch DevMonday, June 4, 2012
    • Roles  in  Wooga’s  IT Arch Dev OpMonday, June 4, 2012
    • Roles  in  Wooga’s  IT Arch Arch Dev Dev Op OpMonday, June 4, 2012
    • Wooga  has  dedicated  game  teamsMonday, June 4, 2012
    • Architecture  EvoluPon  at  Wooga The  Start The  Next  Step Best  of  Two  Worlds Company  ValuesMonday, June 4, 2012
    • First  game:  Try  to  be  flexible ArchMonday, June 4, 2012
    • First  game:  Try  to  be  flexible •  Ruby  on  Rails ArchMonday, June 4, 2012
    • First  game:  Try  to  be  flexible •  Ruby  on  Rails •  Amazon  EC2 ArchMonday, June 4, 2012
    • First  game:  Try  to  be  flexible •  Ruby  on  Rails •  Amazon  EC2 •  MySQL ArchMonday, June 4, 2012
    • EvoluPon  I:  Use  Ruby  (on  Rails) Oct  2009Monday, June 4, 2012
    • Basic  setup  using  sharding lb app app app app app app app app app My My SQL SQL slave slaveMonday, June 4, 2012
    • 250K  daily  users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life  was  good !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012
    • 250K  daily  users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life  was  good NO  MORE !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012
    • Welcome  to  6  weeks  of  pain!Monday, June 4, 2012
    • Welcome  to  6  weeks  of  pain! Heavy  opPmizaPons  were  necessaryMonday, June 4, 2012
    • Welcome  to  6  weeks  of  pain! Heavy  opPmizaPons  were  necessary Numerous  small  fixes  regarding  DB  configMonday, June 4, 2012
    • Welcome  to  6  weeks  of  pain! Heavy  opPmizaPons  were  necessary Numerous  small  fixes  regarding  DB  config More  shardsMonday, June 4, 2012
    • Welcome  to  6  weeks  of  pain! Heavy  opPmizaPons  were  necessary Numerous  small  fixes  regarding  DB  config More  shards Even  more  shardsMonday, June 4, 2012
    • Welcome  to  6  weeks  of  pain! Heavy  opPmizaPons  were  necessary Numerous  small  fixes  regarding  DB  config More  shards Even  more  shards SpliOng  the  model  to  get  more  shardsMonday, June 4, 2012
    • Early  sharding  hell:  8+8  DBs lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slaveMonday, June 4, 2012
    • At  500K  daily  users  we  were  at  a  dead  end &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012
    • First  game:  Try  to  be  flexible •  Ruby  on  Rails •  Amazon  EC2 •  MySQL ArchMonday, June 4, 2012
    • First  game:  Try  to  be  flexible •  Ruby  on  Rails •  Amazon  EC2 •  MySQL Arch DevMonday, June 4, 2012
    • First  game:  Try  to  be  flexible •  Ruby  on  Rails •  Amazon  EC2 •  MySQL Arch Dev OpMonday, June 4, 2012
    • Pick  the  right  tool  for  the  jobMonday, June 4, 2012
    • Pick  the  right  tool  for  the  jobMonday, June 4, 2012
    • EvoluPon  II:  Use  Redis  as  main  DB Jan  2010 Oct  2009Monday, June 4, 2012
    • First  game:  Redis  to  the  mix •  Ruby  on  Rails •  Amazon  EC2 •  MySQL ArchMonday, June 4, 2012
    • First  game:  Redis  to  the  mix •  Ruby  on  Rails •  Amazon  EC2 •  MySQL •  Redis ArchMonday, June 4, 2012
    • Hybrid  approach 256  GB  data 60  GB  data 10%  writes 50%  writes hUp://www.flickr.com/photos/erix/245657047/Monday, June 4, 2012
    • One  team  saved  the  other  one &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012
    • One  team  saved  the  other  one &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012
    • Now  more  than  2M  users  /  day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012
    • Now  more  than  2M  users  /  day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" AWS  outage #!!$!!!" in  Ireland !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012
    • 10  single-­‐points-­‐of-­‐failure  -­‐  no  fun  at  all! lb lb app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app My My My My My redis redis redis redis redis SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave slave slaveMonday, June 4, 2012
    • First  game:  Add  Redis  to  the  mix •  Ruby  on  Rails •  Amazon  EC2 •  MySQL •  Redis ArchMonday, June 4, 2012
    • First  game:  Add  Redis  to  the  mix •  Ruby  on  Rails •  Amazon  EC2 •  MySQL •  Redis Arch DevMonday, June 4, 2012
    • First  game:  Add  Redis  to  the  mix •  Ruby  on  Rails •  Amazon  EC2 •  MySQL •  Redis Arch Dev OpMonday, June 4, 2012
    • Architecture  EvoluPon  at  Wooga The  Start:  Ruby The  Next  Step Best  of  Two  Worlds Company  ValuesMonday, June 4, 2012
    • Stateless  servers  and  DBs Server DatabaseMonday, June 4, 2012
    • Stateless  servers  and  DBs Server DatabaseMonday, June 4, 2012
    • Stateless  servers  and  DBs Server DatabaseMonday, June 4, 2012
    • Stateless  servers  and  DBs Server DatabaseMonday, June 4, 2012
    • Stateless  servers  and  DBs Server DatabaseMonday, June 4, 2012
    • Stateless  servers  and  DBs Server DatabaseMonday, June 4, 2012
    • Find  the  flawMonday, June 4, 2012
    • Find  the  flaw “Stateless  applicaPon  servers   guarantee  one  thing:Monday, June 4, 2012
    • Find  the  flaw “Stateless  applicaPon  servers   guarantee  one  thing: The  data  is  never where  you  need  it!” Paolo  Negri,  Developer  @  WoogaMonday, June 4, 2012
    • Strong  session  pamernMonday, June 4, 2012
    • Strong  session  pamern User  starts  playingMonday, June 4, 2012
    • Strong  session  pamern User  starts  playing Many  transformaPons  on   the  same  data  setMonday, June 4, 2012
    • Strong  session  pamern User  starts  playing Many  transformaPons  on   the  same  data  set User  stops  playingMonday, June 4, 2012
    • Strong  session  pamern User  starts  playing Many  transformaPons  on   the  same  data  set User  stops  playingMonday, June 4, 2012
    • Stateful  servers  and  DBs Server DatabaseMonday, June 4, 2012
    • Stateful  servers  and  DBs Server DatabaseMonday, June 4, 2012
    • Stateful  servers  and  DBs Server DatabaseMonday, June 4, 2012
    • Stateful  servers  and  DBs Server Database One  Game  SessionMonday, June 4, 2012
    • Stateful  servers  and  DBs Server Database One  Game  SessionMonday, June 4, 2012
    • Next  game:  Don’t  use  a  DB  at  all ArchMonday, June 4, 2012
    • Next  game:  Don’t  use  a  DB  at  all •  Erlang  OTP ArchMonday, June 4, 2012
    • Next  game:  Don’t  use  a  DB  at  all •  Erlang  OTP •  Amazon  EC2 ArchMonday, June 4, 2012
    • Next  game:  Don’t  use  a  DB  at  all •  Erlang  OTP •  Amazon  EC2 •  Amazon  S3 ArchMonday, June 4, 2012
    • EvoluPon  III:  Erlang  and  stateful  servers Oct  2010 Jan  2010 Oct  2009Monday, June 4, 2012
    • Stateful  servers  are  not  so  hard sessionMonday, June 4, 2012
    • Stateful  servers  are  not  so  hard session session session sessionMonday, June 4, 2012
    • Stateful  servers  are  not  so  hard Server session session session sessionMonday, June 4, 2012
    • Stateful  servers  are  not  so  hard Server session session session session S3Monday, June 4, 2012
    • Stateful  servers  are  not  so  hard Server session session session session S3Monday, June 4, 2012
    • Stateful  servers  are  not  so  hard Server session session session session S3Monday, June 4, 2012
    • Stateful  servers  are  not  so  hard Server session session session session S3Monday, June 4, 2012
    • Stateful  servers  are  not  so  hard Server session session session session S3Monday, June 4, 2012
    • Stateful  servers  are  not  so  hard Server session session session session S3Monday, June 4, 2012
    • Stateful  servers  are  not  so  hard Server Server Server session session session session session session session session session session session session S3Monday, June 4, 2012
    • With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 7,500 0 database  operations  /  secMonday, June 4, 2012
    • With  stateful  server  the  DB  is  less  used Ruby  Stateless Erlang  Stateful 30,000 22,500 15,000 700 7,500 0 database  operations  /  secMonday, June 4, 2012
    • Next  game:  Don’t  use  a  DB  at  all •  Erlang  OTP •  Amazon  EC2 •  Amazon  S3 ArchMonday, June 4, 2012
    • Next  game:  Don’t  use  a  DB  at  all •  Erlang  OTP •  Amazon  EC2 •  Amazon  S3 Arch DevMonday, June 4, 2012
    • Next  game:  Don’t  use  a  DB  at  all •  Erlang  OTP •  Amazon  EC2 •  Amazon  S3 Arch Dev OpMonday, June 4, 2012
    • Architecture  EvoluPon  at  Wooga The  Start:  Ruby The  Next  Step:  Erlang Best  of  Two  Worlds Company  ValuesMonday, June 4, 2012
    • Last  game:  The  ideal  architecture ArchMonday, June 4, 2012
    • Last  game:  The  ideal  architecture •  Erlang  OTP  (state) ArchMonday, June 4, 2012
    • Last  game:  The  ideal  architecture •  Erlang  OTP  (state) •  Ruby  (logic) ArchMonday, June 4, 2012
    • Last  game:  The  ideal  architecture •  Erlang  OTP  (state) •  Ruby  (logic) •  Amazon  EC2 ArchMonday, June 4, 2012
    • Last  game:  The  ideal  architecture •  Erlang  OTP  (state) •  Ruby  (logic) •  Amazon  EC2 •  Amazon  S3 ArchMonday, June 4, 2012
    • Last  game:  The  ideal  architecture •  Erlang  OTP  (state) •  Ruby  (logic) •  Amazon  EC2 •  Amazon  S3 Arch Dev OpMonday, June 4, 2012
    • EvoluPon  IV:  The  best  out  of  two  worlds Aug  2011 Oct  2010 Jan  2010 Oct  2009Monday, June 4, 2012
    • The  basic  setup  looks  exactly  like  before Server Server Server session session session session session session session session session session session session S3Monday, June 4, 2012
    • Bringing  2  worlds  together Server session URL  +  state session ... response  +  new  state sessionMonday, June 4, 2012
    • Bringing  2  worlds  together Server session sender URL  +  state session ... response  +  new  state sessionMonday, June 4, 2012
    • Bringing  2  worlds  together Server Worker session sender URL  +  state Worker session Worker ... Worker response  +  new  state session WorkerMonday, June 4, 2012
    • Bringing  2  worlds  together Server Worker session sender URL  +  state Worker session Worker ... Worker receiver response  +  new  state session WorkerMonday, June 4, 2012
    • Bringing  2  worlds  together Server Worker session sender URL  +  state Worker session Worker ... Worker receiver response  +  new  state session WorkerMonday, June 4, 2012
    • Bringing  2  worlds  together Server Worker session sender URL  +  state Worker session Worker ... Worker receiver response  +  new  state session WorkerMonday, June 4, 2012
    • Performance  impact  for  large  payloads 200  kB  payload 6000 5000 4000 requests  /  s 3000 2000 1000 0 1 #  of  workers 8Monday, June 4, 2012
    • Performance  impact  for  large  payloads 200  kB  payload 6000 5000 4000 requests  /  s Pure 3000 Erlang 2000 1000 Ruby ZMQ 0 1 #  of  workers 8Monday, June 4, 2012
    • Performance  impact  for  large  payloads 200  kB  payload 6000 5000 4000 requests  /  s Pure 3000 Erlang 2000 1000 Ruby 400  rps ZMQ 0 1 #  of  workers 8Monday, June 4, 2012
    • NOT  CPU  boundMonday, June 4, 2012
    • NOT  CPU  bound 25%Monday, June 4, 2012
    • NOT  CPU  bound 25% Fixed  bandwith  limit  @  300  MB/sMonday, June 4, 2012
    • Last  game:  The  ideal  architecture? •  Erlang  OTP  (state) •  Ruby  (logic) •  Amazon  EC2 •  Amazon  S3 ArchMonday, June 4, 2012
    • Last  game:  The  ideal  architecture? •  Erlang  OTP  (state) •  Ruby  (logic) •  Amazon  EC2 •  Amazon  S3 Arch DevMonday, June 4, 2012
    • Last  game:  The  ideal  architecture? •  Erlang  OTP  (state) •  Ruby  (logic) •  Amazon  EC2 •  Amazon  S3 Arch Dev OpMonday, June 4, 2012
    • Last  game:  The  ideal  architecture? •  Erlang  OTP  (state) •  Erlang  OTP  (logic)  !!! •  Amazon  EC2 •  Amazon  S3 Arch Dev OpMonday, June 4, 2012
    • Last  game:  The  ideal  architecture? •  Erlang  OTP  (state) •  Erlang  OTP  (logic)  !!! •  Amazon  EC2 •  Amazon  S3 Arch Dev OpMonday, June 4, 2012
    • Last  game:  The  ideal  architecture? •  Erlang  OTP  (state) •  Erlang  OTP  (logic)  !!! •  Amazon  EC2 •  Amazon  S3 Arch Dev OpMonday, June 4, 2012
    • Last  game:  The  ideal  architecture? •  Erlang  OTP  (state) •  Erlang  OTP  (logic)  !!! •  Amazon  EC2 •  Amazon  S3 Arch Dev OpMonday, June 4, 2012
    • So  nothing  new?Monday, June 4, 2012
    • So  nothing  new? We  could  not  combine  Ruby  and  Erlang,  too  bad. At  least  not  for  this  game.Monday, June 4, 2012
    • So  nothing  new? We  could  not  combine  Ruby  and  Erlang,  too  bad. At  least  not  for  this  game. Erlang  is  great  for  inter  process  messaging... User  sessions  are  represented  by  Erlang  processes.Monday, June 4, 2012
    • So  nothing  new? We  could  not  combine  Ruby  and  Erlang,  too  bad. At  least  not  for  this  game. Erlang  is  great  for  inter  process  messaging... User  sessions  are  represented  by  Erlang  processes. ...  and  Server  Side  Events  just  work. The  server  can  push  updates  to  the  client.Monday, June 4, 2012
    • So  nothing  new? We  could  not  combine  Ruby  and  Erlang,  too  bad. At  least  not  for  this  game. Erlang  is  great  for  inter  process  messaging... User  sessions  are  represented  by  Erlang  processes. ...  and  Server  Side  Events  just  work. The  server  can  push  updates  to  the  client. Now,  what  could  you  do  with  this?Monday, June 4, 2012
    • The  next  step  in  social  gamingMonday, June 4, 2012
    • The  next  step  in  social  gamingMonday, June 4, 2012
    • The  next  step  in  social  gamingMonday, June 4, 2012
    • The  next  step  in  social  gamingMonday, June 4, 2012
    • The  next  step  in  social  gamingMonday, June 4, 2012
    • The  next  step  in  social  gamingMonday, June 4, 2012
    • The  next  step  in  social  gamingMonday, June 4, 2012
    • Architecture  EvoluPon  at  Wooga The  Start:  Ruby The  Next  Step:  Erlang Best  of  Two  Worlds? Company  ValuesMonday, June 4, 2012
    • Each  new  game  brought  us  innovaPon Aug  2011 Oct  2010 Jan  2010 Oct  2009Monday, June 4, 2012
    • We’ve  learned  to  value Small teams over big teamsMonday, June 4, 2012
    • We’ve  learned  to  value Collaboration over competitionMonday, June 4, 2012
    • We’ve  learned  to  value Generalists over specialistsMonday, June 4, 2012
    • We’ve  learned  to  value Effort reduction over cost reductionMonday, June 4, 2012
    • We’ve  learned  to  value Innovation over risk mitigationMonday, June 4, 2012
    • A  good  value  system We’ve  learned  to  value Small  teams                              over   Big  teams   CollaboraPon                        over Compe??on Generalists                                over Specialists Effort  reducPon                over Cost  reduc?on InnovaPon                                  over Risk  mi?ga?onMonday, June 4, 2012
    • It works!Monday, June 4, 2012
    • It works! Be fast, be bold!Monday, June 4, 2012
    • QuesPons? Jesper  Richter-­‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobsMonday, June 4, 2012
    • Monday, June 4, 2012