Spil Games:outgrowing aninternet startupArt van ScheppingenHead of Database Engineering
Overview1.   Who is Spil Games?2.   How to professionalize?3.   Spil Storage Platform4.   Questions?                      ...
Who are we?Who is Spil Games?
Facts•   Company founded in 2001•   350+ employees world wide•   170M unique visitors per month•   100K unique visitors pe...
Geographic Reach       170 Million Monthly Active Users(*)      •        Over 40 localized portals in 19 languages      • ...
BrandsGirls, Teens and Family                          6
Games        7
Games        8
Games        9
Games        10
Games        11
Games• Inhouse game studios• Partnerships with Social Gaming studios• Over 1500 licensed games                         12
Spil Games is growing fast!                                                          DB Servers                           ...
DatabaseEngineeringHow to professionalize yourdepartment
Startup• Databases maintained by Systems Engineering• No focus on performance, structure or backups• Looking only one or t...
Lessons• Multiple migrations to new hardware• Ping-pong on Master-Master setups• Lack of insight into performance issues  ...
Professionalize• Plan ahead up to three months• Improve database platform  • Reduce number of repetitive tasks      • Writ...
LDAP isn’t suitable for the web• Scaling the LDAP platform• LDAP replaced by MySQL based solution (with help  from Percona...
Cloning• LVM snapshot method  • took 4 hours on average with manual intervention• Innobackupex + netcat + tar + script = q...
Improve monitoring• Different monitoring systems give different insights  • Different angles/metrics/purposes  • Early pro...
Growing pains• Uneven growth:  • Active master handling all write requests  • Vertical scaling• Write only  • More writes ...
Outgrowing our startup phase• Anticipate more than one year in advance• Acknowledge shortcomings/problems, look for  solut...
Spil StoragePlatformSharding is inevitable
What is this exciting project about?                    24
Functional sharding•   Natural growth•   Grown out of necessity for more functionality•   Adding functionality means more ...
Advantages• KISS• Problem isolationDisadvantages•   Uneven growth•   Difference in query patterns•   No data consistency• ...
Spil Storage Platform                   27
Bucket model• What is the bucket model?  • It is an abstraction layer between the database and    the datamodel  • Each re...
Advantages• Flexibility  • Database backend independent  • Seamless schema changes and upgrades  • Sharded on both functio...
How do GIDs work?• Globally sharded on GID• (local) GID Lookup                                         GID                ...
Pipeline flow                31
Bucket mapping and migration                                                     Current functional                       ...
Master-Master Sharding• Each cluster of two masters will contain two shards  • Data is written interleaved  • HA for both ...
How are we implementing this SSP?•   Erlang cluster with many workers•   Every GID has its own worker process•   (Inter)cl...
Advantages• Erlang node caching• Multiple backend connectors  • MySQL library  • Handlersockets  • Any other connectors if...
Do YOU want to be sexy?                 36
Questions?
Thank you!             38
Upcoming SlideShare
Loading in …5
×

Spil Games: outgrowing an internet startup

1,840 views

Published on

This presentation will explain how Spil Games has grown in a short time from an internet startup to a global online gaming company and how we currently are building a global cross datacenter storage solution with MySQL as its backend.

The first part will contain a short summary of where we started with our database engineering department (look ahead at most one week in time), to a more professionalized department (look ahead and plan three to four months in time) to currently growing out of the startup phase (look ahead and plan more than one year in time). This will be illustrated with some examples of the growing pains we encountered with scaling, replication and high availability and leading up to the conclusion that we need to acknowledge our problems and shortcomings to actually be able to overcome them.

The second part of the presentation will contain a comparison of our old architecture against the new architecture. In this new architecture we take into account that failure of a complete datacenter is certain to occur sometime and strive to give our users the best possible experience, even in worst case when data is inaccessible. We also introduce asynchronous calls which enable us to fire and forget most of our writes. The architecture is being built with MySQL, handler sockets, Erlang and Memcache as its building blocks.

  • Be the first to comment

Spil Games: outgrowing an internet startup

  1. 1. Spil Games:outgrowing aninternet startupArt van ScheppingenHead of Database Engineering
  2. 2. Overview1. Who is Spil Games?2. How to professionalize?3. Spil Storage Platform4. Questions? 2
  3. 3. Who are we?Who is Spil Games?
  4. 4. Facts• Company founded in 2001• 350+ employees world wide• 170M unique visitors per month• 100K unique visitors per month on spilgames.com 4
  5. 5. Geographic Reach 170 Million Monthly Active Users(*) • Over 40 localized portals in 19 languages • Focus on casual and social games • 170M MAU per month (30M YoY growth) • Over 40M registered usersSource: (*) Google Analytics, December 2011 5
  6. 6. BrandsGirls, Teens and Family 6
  7. 7. Games 7
  8. 8. Games 8
  9. 9. Games 9
  10. 10. Games 10
  11. 11. Games 11
  12. 12. Games• Inhouse game studios• Partnerships with Social Gaming studios• Over 1500 licensed games 12
  13. 13. Spil Games is growing fast! DB Servers MAU Employees Employees MAU 2006 2007 DB Servers 2008 2009 2010 2011 13
  14. 14. DatabaseEngineeringHow to professionalize yourdepartment
  15. 15. Startup• Databases maintained by Systems Engineering• No focus on performance, structure or backups• Looking only one or two weeks into the future 15
  16. 16. Lessons• Multiple migrations to new hardware• Ping-pong on Master-Master setups• Lack of insight into performance issues Read+write 16
  17. 17. Professionalize• Plan ahead up to three months• Improve database platform • Reduce number of repetitive tasks • Write them down step by step (wiki) • Automate where possible • Improve monitoring • A single monitoring system is not enough! • Forecast growth • Week / Month / Year • Look back and evaluate!• Extend department 17
  18. 18. LDAP isn’t suitable for the web• Scaling the LDAP platform• LDAP replaced by MySQL based solution (with help from Percona) DMM MMM 18
  19. 19. Cloning• LVM snapshot method • took 4 hours on average with manual intervention• Innobackupex + netcat + tar + script = quick cloning • Takes about 1 hour per 100GB • Foolproof • Can be run on active masters (if necessary) 19
  20. 20. Improve monitoring• Different monitoring systems give different insights • Different angles/metrics/purposes • Early problem detection • Signal abnormal use which could cause outage 20
  21. 21. Growing pains• Uneven growth: • Active master handling all write requests • Vertical scaling• Write only • More writes than reads• SOA problems • Connection spawning • Open file descriptors 21
  22. 22. Outgrowing our startup phase• Anticipate more than one year in advance• Acknowledge shortcomings/problems, look for solutions or alternatives • Dont commit to one single solution! • Be flexible!• Plan for capacity per instance, not for growth alone!• Start thinking globally! 22
  23. 23. Spil StoragePlatformSharding is inevitable
  24. 24. What is this exciting project about? 24
  25. 25. Functional sharding• Natural growth• Grown out of necessity for more functionality• Adding functionality means more interaction• Separation of database function • Profiles • Highscores • Comments • User Generated Content • etc 25
  26. 26. Advantages• KISS• Problem isolationDisadvantages• Uneven growth• Difference in query patterns• No data consistency• No clear ownership of data• Capacity planning on total number of reads/writes• Horizontal scaling is difficult 26
  27. 27. Spil Storage Platform 27
  28. 28. Bucket model• What is the bucket model? • It is an abstraction layer between the database and the datamodel • Each record has one unique owner attribute (GID) • The GID (Global IDentifier) identifies different data types • Different buckets per function • Attributes contain record data • Attributes do not have to correspond to schema 28
  29. 29. Advantages• Flexibility • Database backend independent • Seamless schema changes and upgrades • Sharded on both functional and GID level• Even distribution of queries possible • Capacity planning on number and type of entities• Asynchronous writes possible• Transparent data migrationDisadvantages• Harder to find data • At least two lookups needed! • Datawarehousing needs a different approach 29
  30. 30. How do GIDs work?• Globally sharded on GID• (local) GID Lookup GID lookup Persis stor Shard 1 Shard 2 30
  31. 31. Pipeline flow 31
  32. 32. Bucket mapping and migration Current functional shards LEGACY Legacy API adapter SPAPI Read only New GID based New SSP shards Application Read + write 32
  33. 33. Master-Master Sharding• Each cluster of two masters will contain two shards • Data is written interleaved • HA for both shards • No warmup needed• Both masters active and “warmed up”• Slave added for backups and Datawarehouse SSP Shard 1 Shard 2 33
  34. 34. How are we implementing this SSP?• Erlang cluster with many workers• Every GID has its own worker process• (Inter)cluster communication• (Near) linear scalability 34
  35. 35. Advantages• Erlang node caching• Multiple backend connectors • MySQL library • Handlersockets • Any other connectors if needed• Connection poolingDisadvantages• NOT SEXY? ( http://spil.com/notsexy ) 35
  36. 36. Do YOU want to be sexy? 36
  37. 37. Questions?
  38. 38. Thank you! 38

×