Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

ARCHITECTURE EVOLUTION AT WOOGA How does your company learn new things?Jesper Richter-‐Reichhelm, @jrirei

Our games all look the same Flash client Backend

But the scale is interesFng 14 billion requests / month

But the scale is interesFng 14 billion requests / month >100,000 DB operaFons / second

But the scale is interesFng 14 billion requests / month >100,000 DB operaFons / second >50,000 DB updates / second

Architecture EvoluFon at Wooga The Start The Next Step Best of Two Worlds

Oct 2009: Monster World based on Ruby Good code quality Easy to understand Easy to test Easy to refactor

A basic setup using sharding worked ﬁne lb app app app My My SQL SQL slave slave

A basic setup using sharding worked ﬁne lb app app app app app My My SQL SQL slave slave

A basic setup using sharding worked ﬁne lb app app app app app app app app app My My SQL SQL slave slave

250K daily users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" Life was good !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"

250K daily users&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" Life was good NO MORE !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"

Early sharding hell: 8 master and 8 slaves lb app app app app app app app app app app app app app app app app app app My My SQL SQL slave slave

Early sharding hell: 8 master and 8 slaves lb app app app app app app app app app app app app app app app app app app My My My My SQL SQL SQL SQL slave slave slave slave

Early sharding hell: 8 master and 8 slaves lb app app app app app app app app app app app app app app app app app app My My My My My My My My SQL SQL SQL SQL SQL SQL SQL SQL slave slave slave slave slave slave slave slave

At 500K daily users we were at a dead end&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"

OUCH!http://www.ﬂickr.com/photos/billue_the_bear/

Big and staFc data in MySQL, rest goes to Redis MySQL

Big and staFc data in MySQL, rest goes to Redis MySQL Redis

Big and staFc data in MySQL, rest goes to Redis MySQL Redis 256 GB data 10% writes

Big and staFc data in MySQL, rest goes to Redis MySQL Redis 256 GB data 60 GB data 10% writes 50% writes

Redis saved the day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"

We now have more than 2 million users / day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" #!!$!!!" !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"

We now have more than 2 million users / day&$!!!$!!!"%$#!!$!!!"%$!!!$!!!" AWS service #!!$!!!" disrupFon in Ireland !" ()*%!" +,-*%!" ./0*%!" +12*%%" ()*%%" +,-*%%" ./0*%%"

10 single-‐points-‐of-‐failure -‐ no fun at all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app app My My redis redis SQL SQL slave slave slave slave

10 single-‐points-‐of-‐failure -‐ no fun at all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app appMy My My My My redis redisSQL SQL SQL SQL SQLslave slave slave slave slave slave slave

10 single-‐points-‐of-‐failure -‐ no fun at all! lb lbapp app app app app app app app app app app app appapp app app app app app app app app app app app appapp app app app app app app app app app app app appMy My My My My redis redis redis redis redisSQL SQL SQL SQL SQLslave slave slave slave slave slave slave slave slave slave

http://www.ﬂickr.com/photos/wolfsavard/OUCH!

Architecture EvoluFon at Wooga The Start: Ruby + AutomaFon The Next Step Best of Two Worlds

Stateless servers and DBs Server Database

“Stateless applica,on servers guarantee one thing:

“Stateless applica,on servers guarantee one thing: The data is never where you need it!” Paolo Negri, Developer @ Wooga

Stateful servers and DBs Server Database

Stateful servers and DBs Server Database One Game Session

Oct 2010: Magic Land based on stateful server If DBs are the problem Don’t use them Store state in server Need to be robust

Stateful servers are not as hard as you think session

Stateful servers are not as hard as you think session session session session

Stateful servers are not as hard as you think Server session session session session

Stateful servers are not as hard as you think Server session session session session S3

Stateful servers are not as hard as you think Server Server Server session session session session session session session session session session session session S3

With stateful server the DB is less used Ruby Stateless Erlang Stateful 30,000 22,500 15,000 7,500 0 database operations / sec

With stateful server the DB is less used Ruby Stateless Erlang Stateful 30,000 22,500 15,000 700 7,500 0 database operations / sec

Deploying with a stateful server In order to bring up a new version

Deploying with a stateful server In order to bring up a new version Just deploy it Hot code replacement is great!

There are even more advantages Faster than Ruby (5,000 rps / node) -‐ CPU bound

There are even more advantages Faster than Ruby (5,000 rps / node) -‐ CPU bound Very few SPOFs -‐ ... and those are easy to recover

There are even more advantages Faster than Ruby (5,000 rps / node) -‐ CPU bound Very few SPOFs -‐ ... and those are easy to recover TransacFonal logic -‐ Invariants instead of explicit error handling

NICE! http://www.ﬂickr.com/photos/aigle_dore/

Architecture EvoluFon at Wooga The Start: Ruby + AutomaFon The Next Step: Erlang + S3 Company Values

A good value system We’ve learned to value

A good value system We’ve learned to value Small teams over Big teams

A good value system We’ve learned to value Small teams over Big teams CollaboraFon over Compe??on

A good value system We’ve learned to value Small teams over Big teams CollaboraFon over Compe??on Generalists over Specialists

A good value system We’ve learned to value Small teams over Big teams CollaboraFon over Compe??on Generalists over Specialists Eﬀort reducFon over Cost reduc?on

A good value system We’ve learned to value Small teams over Big teams CollaboraFon over Compe??on Generalists over Specialists Eﬀort reducFon over Cost reduc?on InnovaFon over Risk mi?ga?on

It works!

It works!Be fast, be bold!

QuesFons?Jesper Richter-‐Reichhelm @jrirei slideshare.net/wooga wooga.com/jobs

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,)

by Wooga

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 9

Statistics

Architecture Evolution at Wooga (AWS Cloud Computing for Developers,) Presentation Transcript

http://www.linkedin.com	5
https://si0.twimg.com	2
http://www.hanrss.com	2