Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale)

GAMES FOR THE MASSES Wie DevOps die Entwicklung von Architektur verändert Jesper Richter-‐Reichhelm, @jrireiMonday, June 4, 2012

DEV -‐ OPSMonday, June 4, 2012

DEV -‐ OPS Some say: Admins learn to be ‘Agile’ Scrum and KanbanMonday, June 4, 2012

DEV -‐ OPS Some say: Admins learn to be ‘Agile’ Scrum and Kanban Others say: Admins learn to program their setup Chef, PuppetMonday, June 4, 2012

DEV -‐ OPS Some say: Admins learn to be ‘Agile’ Scrum and Kanban Others say: Admins learn to program their setup Chef, Puppet I say: This is all nice, but not enough Developers need to learn a lot about opera?onMonday, June 4, 2012

Classic roles in IT Arch Dev OpMonday, June 4, 2012

Classic roles in IT Arch Dev Op ???Monday, June 4, 2012

Monday, June 4, 2012

Our games all look the same Flash client BackendMonday, June 4, 2012

Our games all look the same Flash client Game Session Asynch. CommunicaPonMonday, June 4, 2012

Our games all look the same Backend State Changes ValidaPon PersistenceMonday, June 4, 2012

But the scale is interesPng 14 billion requests / monthMonday, June 4, 2012

But the scale is interesPng 14 billion requests / month >100,000 DB operaPons / secondMonday, June 4, 2012

But the scale is interesPng 14 billion requests / month >100,000 DB operaPons / second >50,000 DB updates / secondMonday, June 4, 2012

Classic roles in IT Arch Dev Op ???Monday, June 4, 2012

Roles in Wooga’s ITMonday, June 4, 2012

Roles in Wooga’s IT ArchMonday, June 4, 2012

Roles in Wooga’s IT Arch DevMonday, June 4, 2012

Roles in Wooga’s IT Arch Dev OpMonday, June 4, 2012

Roles in Wooga’s IT Arch Arch Dev Dev Op OpMonday, June 4, 2012

Wooga has dedicated game teamsMonday, June 4, 2012

Architecture EvoluPon at Wooga The Start The Next Step Best of Two Worlds Company ValuesMonday, June 4, 2012

First game: Try to be ﬂexible ArchMonday, June 4, 2012

First game: Try to be ﬂexible • Ruby on Rails ArchMonday, June 4, 2012

First game: Try to be ﬂexible • Ruby on Rails • Amazon EC2 ArchMonday, June 4, 2012

First game: Try to be ﬂexible • Ruby on Rails • Amazon EC2 • MySQL ArchMonday, June 4, 2012

EvoluPon I: Use Ruby (on Rails) Oct 2009Monday, June 4, 2012

Basic setup using sharding lb app app app app app app app app app My My SQL SQL slave slaveMonday, June 4, 2012

250K daily users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life was good !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012

250K daily users &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" Life was good NO MORE !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012

Welcome to 6 weeks of pain!Monday, June 4, 2012

Welcome to 6 weeks of pain! Heavy opPmizaPons were necessaryMonday, June 4, 2012

Welcome to 6 weeks of pain! Heavy opPmizaPons were necessary Numerous small ﬁxes regarding DB conﬁgMonday, June 4, 2012

Welcome to 6 weeks of pain! Heavy opPmizaPons were necessary Numerous small ﬁxes regarding DB conﬁg More shardsMonday, June 4, 2012

Welcome to 6 weeks of pain! Heavy opPmizaPons were necessary Numerous small ﬁxes regarding DB conﬁg More shards Even more shardsMonday, June 4, 2012

Welcome to 6 weeks of pain! Heavy opPmizaPons were necessary Numerous small ﬁxes regarding DB conﬁg More shards Even more shards SpliOng the model to get more shardsMonday, June 4, 2012

Early sharding hell: 8+8 DBs lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slaveMonday, June 4, 2012

At 500K daily users we were at a dead end &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012

First game: Try to be ﬂexible • Ruby on Rails • Amazon EC2 • MySQL ArchMonday, June 4, 2012

First game: Try to be ﬂexible • Ruby on Rails • Amazon EC2 • MySQL Arch DevMonday, June 4, 2012

First game: Try to be ﬂexible • Ruby on Rails • Amazon EC2 • MySQL Arch Dev OpMonday, June 4, 2012

Pick the right tool for the jobMonday, June 4, 2012

EvoluPon II: Use Redis as main DB Jan 2010 Oct 2009Monday, June 4, 2012

First game: Redis to the mix • Ruby on Rails • Amazon EC2 • MySQL ArchMonday, June 4, 2012

First game: Redis to the mix • Ruby on Rails • Amazon EC2 • MySQL • Redis ArchMonday, June 4, 2012

Hybrid approach 256 GB data 60 GB data 10% writes 50% writes hUp://www.ﬂickr.com/photos/erix/245657047/Monday, June 4, 2012

One team saved the other one &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012

Now more than 2M users / day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012

Now more than 2M users / day &$!!!$!!!" %$#!!$!!!" %$!!!$!!!" AWS outage #!!$!!!" in Ireland !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"Monday, June 4, 2012

10 single-‐points-‐of-‐failure -‐ no fun at all! lb lb app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app app My My My My My redis redis redis redis redis SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave slave slaveMonday, June 4, 2012

First game: Add Redis to the mix • Ruby on Rails • Amazon EC2 • MySQL • Redis ArchMonday, June 4, 2012

First game: Add Redis to the mix • Ruby on Rails • Amazon EC2 • MySQL • Redis Arch DevMonday, June 4, 2012

First game: Add Redis to the mix • Ruby on Rails • Amazon EC2 • MySQL • Redis Arch Dev OpMonday, June 4, 2012

Architecture EvoluPon at Wooga The Start: Ruby The Next Step Best of Two Worlds Company ValuesMonday, June 4, 2012

Stateless servers and DBs Server DatabaseMonday, June 4, 2012

Find the ﬂawMonday, June 4, 2012

Find the ﬂaw “Stateless applicaPon servers guarantee one thing:Monday, June 4, 2012

Find the ﬂaw “Stateless applicaPon servers guarantee one thing: The data is never where you need it!” Paolo Negri, Developer @ WoogaMonday, June 4, 2012

Strong session pamernMonday, June 4, 2012

Strong session pamern User starts playingMonday, June 4, 2012

Strong session pamern User starts playing Many transformaPons on the same data setMonday, June 4, 2012

Strong session pamern User starts playing Many transformaPons on the same data set User stops playingMonday, June 4, 2012

Stateful servers and DBs Server DatabaseMonday, June 4, 2012

Stateful servers and DBs Server Database One Game SessionMonday, June 4, 2012

Next game: Don’t use a DB at all ArchMonday, June 4, 2012

Next game: Don’t use a DB at all • Erlang OTP ArchMonday, June 4, 2012

Next game: Don’t use a DB at all • Erlang OTP • Amazon EC2 ArchMonday, June 4, 2012

Next game: Don’t use a DB at all • Erlang OTP • Amazon EC2 • Amazon S3 ArchMonday, June 4, 2012

EvoluPon III: Erlang and stateful servers Oct 2010 Jan 2010 Oct 2009Monday, June 4, 2012

Stateful servers are not so hard sessionMonday, June 4, 2012

Stateful servers are not so hard session session session sessionMonday, June 4, 2012

Stateful servers are not so hard Server session session session sessionMonday, June 4, 2012

Stateful servers are not so hard Server session session session session S3Monday, June 4, 2012

Stateful servers are not so hard Server Server Server session session session session session session session session session session session session S3Monday, June 4, 2012

With stateful server the DB is less used Ruby Stateless Erlang Stateful 30,000 22,500 15,000 7,500 0 database operations / secMonday, June 4, 2012

With stateful server the DB is less used Ruby Stateless Erlang Stateful 30,000 22,500 15,000 700 7,500 0 database operations / secMonday, June 4, 2012

Next game: Don’t use a DB at all • Erlang OTP • Amazon EC2 • Amazon S3 ArchMonday, June 4, 2012

Next game: Don’t use a DB at all • Erlang OTP • Amazon EC2 • Amazon S3 Arch DevMonday, June 4, 2012

Next game: Don’t use a DB at all • Erlang OTP • Amazon EC2 • Amazon S3 Arch Dev OpMonday, June 4, 2012

Architecture EvoluPon at Wooga The Start: Ruby The Next Step: Erlang Best of Two Worlds Company ValuesMonday, June 4, 2012

Last game: The ideal architecture ArchMonday, June 4, 2012

Last game: The ideal architecture • Erlang OTP (state) ArchMonday, June 4, 2012

Last game: The ideal architecture • Erlang OTP (state) • Ruby (logic) ArchMonday, June 4, 2012

Last game: The ideal architecture • Erlang OTP (state) • Ruby (logic) • Amazon EC2 ArchMonday, June 4, 2012

Last game: The ideal architecture • Erlang OTP (state) • Ruby (logic) • Amazon EC2 • Amazon S3 ArchMonday, June 4, 2012

Last game: The ideal architecture • Erlang OTP (state) • Ruby (logic) • Amazon EC2 • Amazon S3 Arch Dev OpMonday, June 4, 2012

EvoluPon IV: The best out of two worlds Aug 2011 Oct 2010 Jan 2010 Oct 2009Monday, June 4, 2012

The basic setup looks exactly like before Server Server Server session session session session session session session session session session session session S3Monday, June 4, 2012

Bringing 2 worlds together Server session URL + state session ... response + new state sessionMonday, June 4, 2012

Bringing 2 worlds together Server session sender URL + state session ... response + new state sessionMonday, June 4, 2012

Bringing 2 worlds together Server Worker session sender URL + state Worker session Worker ... Worker response + new state session WorkerMonday, June 4, 2012

Bringing 2 worlds together Server Worker session sender URL + state Worker session Worker ... Worker receiver response + new state session WorkerMonday, June 4, 2012

Performance impact for large payloads 200 kB payload 6000 5000 4000 requests / s 3000 2000 1000 0 1 # of workers 8Monday, June 4, 2012

Performance impact for large payloads 200 kB payload 6000 5000 4000 requests / s Pure 3000 Erlang 2000 1000 Ruby ZMQ 0 1 # of workers 8Monday, June 4, 2012

Performance impact for large payloads 200 kB payload 6000 5000 4000 requests / s Pure 3000 Erlang 2000 1000 Ruby 400 rps ZMQ 0 1 # of workers 8Monday, June 4, 2012

NOT CPU boundMonday, June 4, 2012

NOT CPU bound 25%Monday, June 4, 2012

NOT CPU bound 25% Fixed bandwith limit @ 300 MB/sMonday, June 4, 2012

Last game: The ideal architecture? • Erlang OTP (state) • Ruby (logic) • Amazon EC2 • Amazon S3 ArchMonday, June 4, 2012

Last game: The ideal architecture? • Erlang OTP (state) • Ruby (logic) • Amazon EC2 • Amazon S3 Arch DevMonday, June 4, 2012

Last game: The ideal architecture? • Erlang OTP (state) • Ruby (logic) • Amazon EC2 • Amazon S3 Arch Dev OpMonday, June 4, 2012

Last game: The ideal architecture? • Erlang OTP (state) • Erlang OTP (logic) !!! • Amazon EC2 • Amazon S3 Arch Dev OpMonday, June 4, 2012

So nothing new?Monday, June 4, 2012

So nothing new? We could not combine Ruby and Erlang, too bad. At least not for this game.Monday, June 4, 2012

So nothing new? We could not combine Ruby and Erlang, too bad. At least not for this game. Erlang is great for inter process messaging... User sessions are represented by Erlang processes.Monday, June 4, 2012

So nothing new? We could not combine Ruby and Erlang, too bad. At least not for this game. Erlang is great for inter process messaging... User sessions are represented by Erlang processes. ... and Server Side Events just work. The server can push updates to the client.Monday, June 4, 2012

So nothing new? We could not combine Ruby and Erlang, too bad. At least not for this game. Erlang is great for inter process messaging... User sessions are represented by Erlang processes. ... and Server Side Events just work. The server can push updates to the client. Now, what could you do with this?Monday, June 4, 2012

The next step in social gamingMonday, June 4, 2012

Architecture EvoluPon at Wooga The Start: Ruby The Next Step: Erlang Best of Two Worlds? Company ValuesMonday, June 4, 2012

Each new game brought us innovaPon Aug 2011 Oct 2010 Jan 2010 Oct 2009Monday, June 4, 2012

We’ve learned to value Small teams over big teamsMonday, June 4, 2012

We’ve learned to value Collaboration over competitionMonday, June 4, 2012

We’ve learned to value Generalists over specialistsMonday, June 4, 2012

We’ve learned to value Effort reduction over cost reductionMonday, June 4, 2012

We’ve learned to value Innovation over risk mitigationMonday, June 4, 2012

A good value system We’ve learned to value Small teams over Big teams CollaboraPon over Compe??on Generalists over Specialists Eﬀort reducPon over Cost reduc?on InnovaPon over Risk mi?ga?onMonday, June 4, 2012

It works!Monday, June 4, 2012

It works! Be fast, be bold!Monday, June 4, 2012

QuesPons? Jesper Richter-‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobsMonday, June 4, 2012

Monday, June 4, 2012

Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale)

by Wooga on Jun 06, 2012

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Games for the Masses - Wie DevOps die Entwicklung von Architektur verändert (Webinale) Presentation Transcript