Spil Games:
outgrowing an
internet startup
Art van Scheppingen
Head of Database Engineering
2	
  
1.  Who	
  is	
  Spil	
  Games?	
  
2.  How	
  to	
  professionalize?	
  
3.  Spil	
  Storage	
  Pla<orm	
  
4.  Que...
Who are we?
Who	
  is	
  Spil	
  Games?	
  
	
  
4	
  
•  Company	
  founded	
  in	
  2001	
  
•  350+	
  employees	
  world	
  wide	
  
•  170M	
  unique	
  visitors	
  p...
5	
  
Geographic Reach
170	
  Million	
  Monthly	
  Ac@ve	
  Users(*)	
  
Source:	
  (*)	
  Google	
  Analy3cs,	
  Decembe...
6	
  
Girls,	
  Teens	
  and	
  Family	
  
Brands
7	
  
Games
8	
  
Games
9	
  
Games
10	
  
Games
11	
  
Games
12	
  
Games
•  Inhouse	
  game	
  studios	
  
•  Partnerships	
  with	
  Social	
  Gaming	
  studios	
  
•  Over	
  1500	...
13	
  
DB	
  Servers	
  
MAU	
  
Employees	
  
2006	
   2007	
   2008	
   2009	
   2010	
   2011	
  
DB	
  Servers	
  
MAU...
Database
Engineering
How	
  to	
  professionalize	
  your	
  
department	
  
	
  
15	
  
•  Databases	
  maintained	
  by	
  Systems	
  Engineering	
  
•  No	
  focus	
  on	
  performance,	
  structure	
 ...
16	
  
•  Mul@ple	
  migra@ons	
  to	
  new	
  hardware	
  
•  Ping-­‐pong	
  on	
  Master-­‐Master	
  setups	
  
•  Lack	...
17	
  
•  Plan	
  ahead	
  up	
  to	
  three	
  months	
  
•  Improve	
  database	
  pla<orm	
  
•  Reduce	
  number	
  of...
18	
  
•  Scaling	
  the	
  LDAP	
  pla<orm	
  
•  LDAP	
  replaced	
  by	
  MySQL	
  based	
  solu@on	
  (with	
  help	
 ...
19	
  
•  LVM	
  snapshot	
  method	
  
•  took	
  4	
  hours	
  on	
  average	
  with	
  manual	
  interven@on	
  
•  Inn...
20	
  
•  Different	
  monitoring	
  systems	
  give	
  different	
  insights	
  
•  Different	
  angles/metrics/purposes	
  ...
21	
  
•  Uneven	
  growth:	
  
•  Ac@ve	
  master	
  handling	
  all	
  write	
  requests	
  
•  Ver@cal	
  scaling	
  	
...
22	
  
•  An@cipate	
  more	
  than	
  one	
  year	
  in	
  advance	
  
•  Acknowledge	
  shortcomings/problems,	
  look	
...
Spil Storage
Platform
Sharding	
  is	
  inevitable	
  
	
  
24	
  
What is this exciting project about?
25	
  
•  Natural	
  growth	
  
•  Grown	
  out	
  of	
  necessity	
  for	
  more	
  func@onality	
  
•  Adding	
  func@on...
26	
  
•  KISS	
  
•  Problem	
  isola@on	
  
Advantages
Disadvantages
•  Uneven	
  growth	
  
•  Difference	
  in	
  query...
27	
  
Spil Storage Platform
28	
  
•  What	
  is	
  the	
  bucket	
  model?	
  
•  It	
  is	
  an	
  abstrac@on	
  layer	
  between	
  the	
  database...
29	
  
•  Flexibility	
  
•  Database	
  backend	
  independent	
  
•  Seamless	
  schema	
  changes	
  and	
  upgrades	
 ...
30	
  
•  Globally	
  sharded	
  on	
  GID	
  
•  (local)	
  GID	
  Lookup	
  
How do GIDs work?
GID
lookup
Shard 1 Shard ...
31	
  
Pipeline flow
Current functional
shards
SPAPI
LEGACY
adapter
New
Application
SSP
Legacy API
New GID based
shards
Read only
Read + write
...
33	
  
•  Each	
  cluster	
  of	
  two	
  masters	
  will	
  contain	
  two	
  shards	
  
•  Data	
  is	
  wrinen	
  inter...
34	
  
•  Erlang	
  cluster	
  with	
  many	
  workers	
  
•  Every	
  GID	
  has	
  its	
  own	
  worker	
  process	
  
•...
35	
  
•  Erlang	
  node	
  caching	
  
•  Mul@ple	
  backend	
  connectors	
  
•  MySQL	
  library	
  
•  Handlersockets	...
36	
  
Do YOU want to be sexy?
Questions?
38	
  
Thank you!
Upcoming SlideShare
Loading in...5
×

Outgrowing an internet startup: database administration in a fast growing company.

196

Published on

Database administration in a fast growing company: problems and solutions.

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
196
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Outgrowing an internet startup: database administration in a fast growing company.

  1. 1. Spil Games: outgrowing an internet startup Art van Scheppingen Head of Database Engineering
  2. 2. 2   1.  Who  is  Spil  Games?   2.  How  to  professionalize?   3.  Spil  Storage  Pla<orm   4.  Ques@ons?   Overview
  3. 3. Who are we? Who  is  Spil  Games?    
  4. 4. 4   •  Company  founded  in  2001   •  350+  employees  world  wide   •  170M  unique  visitors  per  month   •  100K  unique  visitors  per  month  on  spilgames.com   Facts
  5. 5. 5   Geographic Reach 170  Million  Monthly  Ac@ve  Users(*)   Source:  (*)  Google  Analy3cs,  December  2011   •  Over  40  localized  portals  in  19  languages   •  Focus  on  casual  and  social  games   •  170M  MAU  per  month  (30M  YoY  growth)   •  Over  40M  registered  users  
  6. 6. 6   Girls,  Teens  and  Family   Brands
  7. 7. 7   Games
  8. 8. 8   Games
  9. 9. 9   Games
  10. 10. 10   Games
  11. 11. 11   Games
  12. 12. 12   Games •  Inhouse  game  studios   •  Partnerships  with  Social  Gaming  studios   •  Over  1500  licensed  games  
  13. 13. 13   DB  Servers   MAU   Employees   2006   2007   2008   2009   2010   2011   DB  Servers   MAU   Employees   Spil Games is growing fast!
  14. 14. Database Engineering How  to  professionalize  your   department    
  15. 15. 15   •  Databases  maintained  by  Systems  Engineering   •  No  focus  on  performance,  structure  or  backups   •  Looking  only  one  or  two  weeks  into  the  future   Startup
  16. 16. 16   •  Mul@ple  migra@ons  to  new  hardware   •  Ping-­‐pong  on  Master-­‐Master  setups   •  Lack  of  insight  into  performance  issues   Lessons Read+write
  17. 17. 17   •  Plan  ahead  up  to  three  months   •  Improve  database  pla<orm   •  Reduce  number  of  repe@@ve  tasks   •  Write  them  down  step  by  step  (wiki)   •  Automate  where  possible   •  Improve  monitoring   •  A  single  monitoring  system  is  not  enough!   •  Forecast  growth   •  Week  /  Month  /  Year   •  Look  back  and  evaluate!   •  Extend  department   Professionalize
  18. 18. 18   •  Scaling  the  LDAP  pla<orm   •  LDAP  replaced  by  MySQL  based  solu@on  (with  help   from  Percona)   LDAP isn’t suitable for the web MMMDMM
  19. 19. 19   •  LVM  snapshot  method   •  took  4  hours  on  average  with  manual  interven@on   •  Innobackupex  +  netcat  +  tar  +  script  =  quick  cloning   •  Takes  about  1  hour  per  100GB   •  Foolproof   •  Can  be  run  on  ac@ve  masters  (if  necessary)   Cloning
  20. 20. 20   •  Different  monitoring  systems  give  different  insights   •  Different  angles/metrics/purposes   •  Early  problem  detec@on   •  Signal  abnormal  use  which  could  cause  outage   Improve monitoring
  21. 21. 21   •  Uneven  growth:   •  Ac@ve  master  handling  all  write  requests   •  Ver@cal  scaling     •  Write  only   •  More  writes  than  reads   •  SOA  problems   •  Connec@on  spawning   •  Open  file  descriptors   Growing pains
  22. 22. 22   •  An@cipate  more  than  one  year  in  advance   •  Acknowledge  shortcomings/problems,  look  for   solu@ons  or  alterna@ves   •  Don't  commit  to  one  single  solu@on!   •  Be  flexible!   •  Plan  for  capacity  per  instance,  not  for  growth  alone!   •  Start  thinking  globally!   Outgrowing our startup phase
  23. 23. Spil Storage Platform Sharding  is  inevitable    
  24. 24. 24   What is this exciting project about?
  25. 25. 25   •  Natural  growth   •  Grown  out  of  necessity  for  more  func@onality   •  Adding  func@onality  means  more  interac@on   •  Separa@on  of  database  func@on   •  Profiles   •  Highscores   •  Comments   •  User  Generated  Content   •  etc   Functional sharding
  26. 26. 26   •  KISS   •  Problem  isola@on   Advantages Disadvantages •  Uneven  growth   •  Difference  in  query  panerns   •  No  data  consistency   •  No  clear  ownership  of  data   •  Capacity  planning  on  total  number  of  reads/writes   •  Horizontal  scaling  is  difficult  
  27. 27. 27   Spil Storage Platform
  28. 28. 28   •  What  is  the  bucket  model?   •  It  is  an  abstrac@on  layer  between  the  database  and   the  datamodel   •  Each  record  has  one  unique  owner  anribute  (GID)   •  The  GID  (Global  IDen@fier)  iden@fies  different  data   types   •  Different  buckets  per  func@on   •  Anributes  contain  record  data   •  Anributes  do  not  have  to  correspond  to  schema   Bucket model
  29. 29. 29   •  Flexibility   •  Database  backend  independent   •  Seamless  schema  changes  and  upgrades   •  Sharded  on  both  func@onal  and  GID  level   •  Even  distribu@on  of  queries  possible   •  Capacity  planning  on  number  and  type  of  en@@es   •  Asynchronous  writes  possible   •  Transparent  data  migra@on   Advantages Disadvantages •  Harder  to  find  data   •  At  least  two  lookups  needed!   •  Datawarehousing  needs  a  different  approach  
  30. 30. 30   •  Globally  sharded  on  GID   •  (local)  GID  Lookup   How do GIDs work? GID lookup Shard 1 Shard 2 Persis stora
  31. 31. 31   Pipeline flow
  32. 32. Current functional shards SPAPI LEGACY adapter New Application SSP Legacy API New GID based shards Read only Read + write 32   Bucket mapping and migration
  33. 33. 33   •  Each  cluster  of  two  masters  will  contain  two  shards   •  Data  is  wrinen  interleaved   •  HA  for  both  shards   •  No  warmup  needed   •  Both  masters  ac@ve  and  “warmed  up”   •  Slave  added  for  backups  and  Datawarehouse   Master-Master Sharding SSP   Shard  1                                       Shard  2                                      
  34. 34. 34   •  Erlang  cluster  with  many  workers   •  Every  GID  has  its  own  worker  process   •  (Inter)cluster  communica@on   •  (Near)  linear  scalability   How are we implementing this SSP?
  35. 35. 35   •  Erlang  node  caching   •  Mul@ple  backend  connectors   •  MySQL  library   •  Handlersockets   •  Any  other  connectors  if  needed   •  Connec@on  pooling   Advantages Disadvantages •  NOT  SEXY?  (  hnp://spil.com/notsexy  )  
  36. 36. 36   Do YOU want to be sexy?
  37. 37. Questions?
  38. 38. 38   Thank you!
  1. Gostou de algum slide específico?

    Recortar slides é uma maneira fácil de colecionar informações para acessar mais tarde.

×