• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Painful success - lessons learned while scaling up
 

Painful success - lessons learned while scaling up

on

  • 3,201 views

Full of hope we started developing our biggest games so far in 2009. In 2010 we released it and gradually scaled up to 1 million users a day. But we were forced to rewrite our whole persistence layer ...

Full of hope we started developing our biggest games so far in 2009. In 2010 we released it and gradually scaled up to 1 million users a day. But we were forced to rewrite our whole persistence layer and migrate most of our data from MySQL to Redis to make it work. In 2011 we even got 2 million daily users, but we also had to operate 200 servers which was painful - especially when the whole data center went down. So in 2012 we wanted to make everything better and started large refactoring projects - and made everything worse.
But in 2013 everything will run smoothly and painless - promised! At least we hope so.

Statistics

Views

Total Views
3,201
Views on SlideShare
1,680
Embed Views
1,521

Actions

Likes
10
Downloads
40
Comments
0

6 Embeds 1,521

http://www.wooga.com 1244
http://dataddict.wordpress.com 195
https://twitter.com 43
http://confluence.enkigames.com 36
http://translate.googleusercontent.com 2
http://instacurate.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Painful success - lessons learned while scaling up Painful success - lessons learned while scaling up Presentation Transcript

    • PAINFUL SUCCESSJesper Richter-Reichhelm (@jrirei) / session 4853
    • Pain
    • Flash client Backend
    • 7M players / month
    • 7M players / month10K API calls / second
    • 7M players / month10K API calls / second200K DB operations / second
    • 7M players / month10K API calls / second200K DB operations / second100K DB writes / second
    • 7M players / month10K API calls / second200K DB operations / second100K DB writes / second2 devops since 2009
    • Success
    • Painful Success
    • Painful Success
    • 2009Decisions
    • What is the rightsoftware stack?
    • Browser lbapp app app SQL SQL sql sql
    • 2010Scaling Up
    • 100%# 80%# 60%# 40%# 20%# 0%# 0# 5# 10# 15# 20# 25# 30# 35# 40#
    • 100%# 80%# 60%# 40%# 20%# 0%# 0# 5# 10# 15# 20# 25# 30# 35# 40#
    • 100%# 80%# 60%# 40%# 20%# 0%# 0# 5# 10# 15# 20# 25# 30# 35# 40#
    • Small things...
    • AMF responses
    • Checking connection ‘status’
    • ActiveRecord caching
    • Lesson 1:Always check back on reality!
    • Big Things...
    • Lot’s of tuning
    • Lot’s of tuningand more memory
    • appmasterslave
    • app 1. create new DBsmaster masterslave slave
    • app 1. create new DBsmaster master 2. setup replicationslave slave
    • app 1. create new DBsmaster master 2. setup replicationslave slave
    • app 1. create new DBsmaster master 2. setup replication 3. start using mastersslave slave
    • app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. cut replicationslave slave
    • app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. cut replicationslave slave 5. remove old DBs
    • app 1. create new DBs master 2. setup replication 3. start using masters 4. cut replication slave 5. remove old DBs
    • app 1. create new DBsmaster master 2. setup replication 3. start using mastersslave slave
    • app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. remove old DBsslave slave
    • app 1. create new DBs master ? 2. setup replication 3. start using masters 4. remove old DBs slave
    • app 1. create new DBs master 2. setup replication 3. start using masters 4. remove old DBs slave 5. cut replication
    • app 1. create new DBsmaster master 2. setup replication 3. start using mastersslave slave
    • app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
    • app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
    • app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
    • app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
    • app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. drop database;slave slave
    • app 1. create new DBs 2. setup replication 3. start using masters 4. drop database;
    • app 1. create new DBs 2. setup replication 3. start using masters 4. drop database; 5. Nooooooooo!
    • app 1. create new DBs 2. setup replication 3. start using masters 4. drop database; 5. Nooooooooo!
    • Lesson 2:You will make mistakes!
    • More things...
    • app 1. create new DBsmaster master 2. setup replication 3. start using masters 4. cut replicationslave slave 5. truncate data
    • app 1. create new DBsmaster master AB AB 2. setup replication 3. start using masters 4. cut replicationslave slave AB AB 5. truncate data
    • app A B 1. create new DBsmaster master AB AB 2. setup replication 3. start using mastersslave slave AB AB
    • app A B 1. create new DBsmaster master AB AB 2. setup replication 3. start using masters 4. cut replicationslave slave AB AB
    • app A B 1. create new DBsmaster master AB AB 2. setup replication 3. start using masters 4. cut replicationslave slave AB AB 5. truncate data
    • app A B 1. create new DBsmaster master AB AB 2. setup replication 3. start using masters 4. cut replicationslave slave AB AB 5. truncate data
    • appmaster master master master AC BDslave slave slave slave AC BD
    • appmaster master master master AC BD AC BDslave slave slave slave AC BD AC BD
    • appmaster master master master AC BD AC BDslave slave slave slave AC BD AC BD
    • appmaster master master master AC BD AC BDslave slave slave slave AC BD AC BD
    • The right thing...
    • Lesson 3:Software is easy, data is hard!
    • 44 GB in 8 days24 GB
    • Not enough memory
    • Not enough memory=> no backups
    • Not enough memory=> no backups=> no new slaves
    • Not enough memory=> no backups=> no new slaves=> big problem
    • 38 GB in 3 days24 GB
    • v2.2 in ixedF 38 GB in 3 days 24 GB
    • 2011Operation Hell
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app SQL app app app app app EBS EBS EBS EBSSQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
    • lb app app lb app appBrowser SQL EBS lb app app lb app app
    • lb app app lb app appBrowser SQL EBS lb app app lb app app
    • lb app app lb app appBrowser SQL EBS lb app app lb app app
    • lb app app lb app appBrowser SQL EBS lb app app lb app app
    • lb app app lb app appBrowser SQL EBS lb app app lb app app
    • lb app app lb app appBrowser SQL EBS lb app app lb app app
    • lb app app lb app appBrowser SQL EBS lb app app lb app app
    • lb app app lb app appBrowser SQL EBS lb app app lb app app
    • 2,000,000"1,500,000"1,000,000" 500,000" 0" Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"
    • 2,000,000"1,500,000"1,000,000" AWS  outage 500,000" in  Ireland 0" Apr*10" Jul*10" Oct*10" Jan*11" Apr*11" Jul*11" Oct*11"
    • http://en.wikipedia.org/wiki/File:St%C3%B6wer_Titanic.jpg
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
    • Installed in 3 years:
    • Installed in 3 years:98 MySQL machines
    • Installed in 3 years:98 MySQL machines195 Redis machines
    • 2012/2013Refactoring
    • Lessons learned
    • Server Database
    • Server Database
    • Server Database
    • Server Database
    • Server Database
    • Server Database
    • Server Database
    • Server Database
    • Server Database
    • Server Database One Game Session
    • Server Database One Game Session
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app Rd rd
    • lb lb lb lb app app app Rd rd
    • 1. Separate user and world DBs
    • 1. Separate user and world DBs2. Migrate to JRuby
    • 1. Separate user and world DBs2. Migrate to JRuby3. Introduce life cycle
    • 1. Separate user and world DBs2. Migrate to JRuby3. Introduce life cycle4. Remove (user) DBs
    • 1. Separate user and world DBs2. Migrate to JRuby3. Introduce life cycle4. Remove (user) DBs5. Enjoy!
    • Separate user DBs
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd rd rd
    • JRuby
    • http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/
    • JRuby v2
    • Throughput200150100 50 0 Rails 2.3.11 MRI 1.8.7
    • Throughput200150100 50 0 Rails 2.3.11 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7
    • Throughput200150100 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3
    • Throughput200150100 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
    • Throughput200 a rk150 c hm ben n o100 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
    • Throughput200 a rk150 c hm ben n o100 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
    • Throughput200 a rk150 c hm ben n o100 multi threaded 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
    • Throughput200 a rk150 c hm ben tuning n o100 multi threaded 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
    • Throughput200 cpu a rk saturation150 c hm ben tuning n o100 multi threaded 50 0 Rails 2.3.11 Rails 3.2.8 Rails 3.2.8 Rails 3.2.8 MRI 1.8.7 MRI 1.8.7 MRI 1.9.3 JRuby 1.7.2
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd
    • NEW OLD
    • NEW OLD
    • NEW OLD
    • NEW OLD
    • Life cycle
    • app
    • app
    • servlet app
    • servlet archiver app
    • servlet archiver User app Registry
    • servlet archiver User app Registry S3
    • servlet archiver User app RegistrySQL Rd S3
    • servlet archiver User app RegistrySQL Rd S3
    • servlet archiver User app Registry WorldSQL Rd Data S3
    • Load Balancer servlet archiver User app Registry WorldSQL Rd Data S3
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd rd rd rd
    • Removing DBs
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appSQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQL SQLsql sql sql sql sql sqlRd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app SQL SQL SQL SQL sql sql Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app SQL SQL SQL sql sql Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
    • Enjoy
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app SQL SQL SQL sql sql Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app SQL SQL sql Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
    • lb lb lb lbapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app appapp app app app app app app app app app app app app app app Rd Rd Rd Rd Rd Rd rd rd rd rd rd rd
    • http://www.flickr.com/photos/aigle_dore/
    • http://www.flickr.com/photos/aigle_dore/
    • Looking Back
    • What is the rightsoftware stack?
    • What is the rightsoftware stack?
    • What is the rightsoftware stack?
    • What kind of application are we building?
    • User
    • UserAvatar
    • User Cus-Avatar tomer
    • User Cus-Avatar Garden tomer
    • User Cus-Avatar Garden tomer Deco- Tiles rations
    • User Cus-Avatar Garden tomer Deco- Tiles rations
    • User User Cus- Cus-Avatar Garden Avatar Garden tomer tomer Deco- Deco- Tiles Tiles rations rations
    • User User Cus- Cus-Avatar Garden Avatar Garden tomer tomer Deco- Deco- Tiles Tiles rations rations
    • http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
    • http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
    • http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
    • http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
    • http://en.wikipedia.org/wiki/File:Columbus_Breaking_the_Egg%27_(Christopher_Columbus)_by_William_Hogarth.jpg
    • What kind of application are we building?
    • Plain File System
    • Handle statethe right way!
    • Lesson 3:Software is easy, data is hard!
    • Lesson 2:You will make mistakes!
    • Lesson 1:Always check back on reality!
    • Lesson 1: Always check backon your assumptions!
    • Thank you
    • Jesper Richter-Reichhelm @jrirei woo.ga/backend wooga.com/jobs
    • Jesper Richter-Reichhelm @jrirei wooga.com/jobs woo.ga/backend