Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
© King Digital Entertainment plc 2014MySQL @ KingScaling Candy CrushOlle Nilsson, Senior Database Engineer @ KingTed Wennm...
© King Digital Entertainment plc 2014 
Table of contents 
2 
Introductions 
Scaling MySQL 
Scaling Candy Crush Saga 
Q...
© King Digital Entertainment plc 2014 
Introductions 
3 
Olle 
•Senior Database Engineer @ King 
•Previously in MySQL Con...
We make great games 
4 
-More than190 funtitles played in over 200 countries around the world 
-345 million averagemonthly...
© King Digital Entertainment plc 2014 5 
Some stats and factsMillions of players around the world 
•345mm MUUs as of Q2 20...
© King Digital Entertainment plc 2014 
Some of our games 
6
“Saga” concept 
7 
Play a leveland get a score and earn1-3 stars 
Progress tothe nextlevel 
Seethe progress ofyourfrien...
© King Digital Entertainment plc 2014 
Scaling MySQL 
8
© King Digital Entertainment plc 2014 
Scaling MySQL 
9 
Scale up / Vertical scaling 
•Buy more or faster disk 
•Buy more...
© King Digital Entertainment plc 2014 
Introduction to Sharding 
10
© King Digital Entertainment plc 2014 
What is sharding? 
11 
Divide your dataset into several pieces, think of horizonta...
© King Digital Entertainment plc 2014 
Why shard? 
12 
Performance 
•Smaller working set of data on each shard. Working s...
© King Digital Entertainment plc 2014 
Getting started 
13 
Choosing shard-key and type of shard distribution 
•RANGE, HA...
© King Digital Entertainment plc 2014 
Sharding challenges 
14 
JOINs 
•Tables are spread over multiple servers 
AUTO_IN...
© King Digital Entertainment plc 2014 
Sharding architecture 
15 
1.Lookup shard for User 100 
2.User 100 is on Shard 1 
3...
© King Digital Entertainment plc 2014 
Sharding Frameworks 
16 
MySQL Fabric 
•Based on plain old MySQL Replication 
•Req...
© King Digital Entertainment plc 2014 
Sharding Frameworks 
17 
MySQL Cluster 
•Automatic sharding 
•Shared nothing archi...
© King Digital Entertainment plc 2014 
Sharding Frameworks 
18 
Gizzard (Twitter) 
DbShards 
Hibernate Shards 
ScaleBa...
© King Digital Entertainment plc 2014 
Scaling Candy Crush Saga 
19
20 
Scaling Candy Crush Saga 
Candy Crush Numbers: 
849M game plays per day 
84M Candy Crush DAU 
Served by MySQL 
As o...
21 
ShardingCandy Crush Saga 
Sharding architecture in place before Candy Crush explosion 
Only 4 people in Operations 
...
© King Digital Entertainment plc 2014 
Architecture overview 
22
© King Digital Entertainment plc 2014 
Architecture overview 
23 
250 000requests / s 
50 000writes / s 
(60 MB/s) 
1 800 ...
© King Digital Entertainment plc 2014 
Regular operation 
24 
MySQL Sharding
© King Digital Entertainment plc 2014 
MySQL Numbers 
25 
User-Shards: 
•> 50TB 
•> 500K qps 
•10/1 Read/Write 
•Several ...
© King Digital Entertainment plc 2014 
MySQL Numbers 
26 
•Hardware 
#####################################################...
MySQL @ King 
27 
Main database technology at King 
Backs all our games 
Plenty of other applications are backed by MyS...
© King Digital Entertainment plc 2014 
Database tier 
28 
The butter and cream of the storage world: 
MySQL 
+ 
Memcached ...
© King Digital Entertainment plc 2014 
MySQL Requirements 
29 
Mission Critical 
Critical functions break without it 
M...
© King Digital Entertainment plc 2014 
Database Architecture and Setup 
30 
Game state data for all users are stored 
No...
© King Digital Entertainment plc 2014 
Application load patterns 
31 
Almost exclusively Key-Value 
Friend data read on ...
© King Digital Entertainment plc 2014 
Sharding Implementation 
32 
Central store used for 
•User/Shard lookup 
•Purchase...
© King Digital Entertainment plc 2014 
HA and Tooling 
33 
KISS 
Master/Slave pairs. No automatic failovers 
Shard cata...
© King Digital Entertainment plc 2014 
Monitoring of our databases! 
34
35 
Challenges @ King 
HDD performance 
Data Volume 
HDD => SSD 100G => SSD 200G => SSD 400G 
Delivery times for new HW...
36 
Challenges @ King 
Relations/Friendships/Joins: joins avoided 
Volume: one of our main issues 
Performace/Scalabili...
37 
Solutions 
Sharding. Just worked 
Throwing HWat problems 
MySQL, easy to use, well known, stable, plenty of 3rd par...
Conclusion 
38 
Pros: Scaled out with very few problems. All games on one DB tier. 
Cons: Very rawMySQL environment. No au...
© King Digital Entertainment plc 2014 
Current & Future Work 
39 
Managing the environment: Fire fighting, maintenance, n...
© King Digital Entertainment plc 2014 
Q & A 
Yes! We are hiring!! 
about.king.com/jobs 
40
Thank you 
© King Digital Entertainment plc 2014 41
Upcoming SlideShare
Loading in …5
×

MySQL@king

3,411 views

Published on

Our (Olle from King and myself) session at OOW2014 (MySQL Central).
You will learn about the setup at King and also have a brief introduction to scaling MySQL.

Published in: Engineering
  • Be the first to comment

MySQL@king

  1. 1. © King Digital Entertainment plc 2014MySQL @ KingScaling Candy CrushOlle Nilsson, Senior Database Engineer @ KingTed Wennmark, Senior Principal Sales Consultant @ Oracle
  2. 2. © King Digital Entertainment plc 2014 Table of contents 2 Introductions Scaling MySQL Scaling Candy Crush Saga Q & A
  3. 3. © King Digital Entertainment plc 2014 Introductions 3 Olle •Senior Database Engineer @ King •Previously in MySQL Consulting @ Oracle EMEA •Developer and Consultant @ Upsys (MySQL Pratner) •Often colleagues with Ted Ted •Senior Principal Sales Consultant @ Oracle EMEA •Developer and Consultant @ Upsys (MySQL Pratner) •Long time MySQL user & friend of Olle
  4. 4. We make great games 4 -More than190 funtitles played in over 200 countries around the world -345 million averagemonthlyuniqueusers(Q2 2014) -Studios in Stockholm, London, Barcelona, Bucharest, Malmo, Berlin and Singapore. Offices in San Francisco, Malta, Tokyo, Seoul and Shanghai About King © King Digital Entertainment plc 2014
  5. 5. © King Digital Entertainment plc 2014 5 Some stats and factsMillions of players around the world •345mm MUUs as of Q2 2014 •138mm DAUs as of Q2 2014 •More players playing at least one of our games each month than thepopulation of the US! Games popular across platforms, and can be played anywhere, anytime on any deviceGlobal leader in cross- platform casual games •Three games in all three major top ten grossing lists for Q2 2014 •Our Saga games allow players to switch platform without losing their progress •Four global franchises: Candy Crush Pet Rescue Farm Heroes Bubble Witch •Founded in 2003, studios in Stockholm, London, Barcelona, Malmo, Bucharest, Berlin and Singapore •900+ employees
  6. 6. © King Digital Entertainment plc 2014 Some of our games 6
  7. 7. “Saga” concept 7 Play a leveland get a score and earn1-3 stars Progress tothe nextlevel Seethe progress ofyourfriends! Progress is storedand accessiblecross platform/device. Historyand background
  8. 8. © King Digital Entertainment plc 2014 Scaling MySQL 8
  9. 9. © King Digital Entertainment plc 2014 Scaling MySQL 9 Scale up / Vertical scaling •Buy more or faster disk •Buy more memory •Buy more or faster CPU’s Scale out / Horizontal scaling •Shard by function •Shard by columns •Shard by rows
  10. 10. © King Digital Entertainment plc 2014 Introduction to Sharding 10
  11. 11. © King Digital Entertainment plc 2014 What is sharding? 11 Divide your dataset into several pieces, think of horizontal partitioning where each partition resides on dedicated instance/HW. Statements are spread amongst all partitions. Each shard can be located on separate servers.
  12. 12. © King Digital Entertainment plc 2014 Why shard? 12 Performance •Smaller working set of data on each shard. Working set fits (better) in memory •Fetch data in parallel from many shards. Size •Database too big and will not fit on disk/memory on one server GEO latency •You want to shard on GEO and have ’local’ shards close to your users
  13. 13. © King Digital Entertainment plc 2014 Getting started 13 Choosing shard-key and type of shard distribution •RANGE, HASH, etc •USER, TIME, etc Operations •Shard move •Shard split •Fail-over / Switch-over Monitoring •Identify quickly growing or ”Hot” shards Interfaces and metadata for sharding •DIY •MySQL Fabric or similar framework
  14. 14. © King Digital Entertainment plc 2014 Sharding challenges 14 JOINs •Tables are spread over multiple servers AUTO_INCREMENTs / Sequences •In MySQL these are per table Backups •Coordinating over multiple servers Complex operations •Schema changes, shard splitting, sharding SW Complex HA •Stand-by servers also need to be sharded Management (deploy / orchestration) •Automation a must
  15. 15. © King Digital Entertainment plc 2014 Sharding architecture 15 1.Lookup shard for User 100 2.User 100 is on Shard 1 3.Fetch data related to User 100 from shard Shard 1 4.User data for User 100 sent from Shard 1 sh4 sh3 sh2 sh1 META Sharding API Apps 1 2 3 4
  16. 16. © King Digital Entertainment plc 2014 Sharding Frameworks 16 MySQL Fabric •Based on plain old MySQL Replication •Requires GTID •Fabric aware connectors •Automatic failover
  17. 17. © King Digital Entertainment plc 2014 Sharding Frameworks 17 MySQL Cluster •Automatic sharding •Shared nothing architecture •No SPOF •IMDB •ndbmtd, ndb_mgm, and API nodes
  18. 18. © King Digital Entertainment plc 2014 Sharding Frameworks 18 Gizzard (Twitter) DbShards Hibernate Shards ScaleBase Many in-house custom made solutions
  19. 19. © King Digital Entertainment plc 2014 Scaling Candy Crush Saga 19
  20. 20. 20 Scaling Candy Crush Saga Candy Crush Numbers: 849M game plays per day 84M Candy Crush DAU Served by MySQL As of Q2 2014 © King Digital Entertainment plc 2014
  21. 21. 21 ShardingCandy Crush Saga Sharding architecture in place before Candy Crush explosion Only 4 people in Operations 10-20 new DB servers deployed every month at peak (+ ~50 App servers + Memcached + BI) No dedicated DB staff © King Digital Entertainment plc 2014
  22. 22. © King Digital Entertainment plc 2014 Architecture overview 22
  23. 23. © King Digital Entertainment plc 2014 Architecture overview 23 250 000requests / s 50 000writes / s (60 MB/s) 1 800 000reads / s 250 000writes / s As of Q2 2014
  24. 24. © King Digital Entertainment plc 2014 Regular operation 24 MySQL Sharding
  25. 25. © King Digital Entertainment plc 2014 MySQL Numbers 25 User-Shards: •> 50TB •> 500K qps •10/1 Read/Write •Several billion row tables •100+ instances •MySQL 5.1 and 5.6 (5.0 and 5.5 used for inhouse applications) •Memcached: Roughly 2M qps •App servers caches in front of Memcached As of Q2 2014
  26. 26. © King Digital Entertainment plc 2014 MySQL Numbers 26 •Hardware ################################################################################ #### hostXXX (debian/7.4 -Linux 3.2.0-4-amd64) #### #### XYXXYZXYZXYZXYZ member of: SOME IMPORTANT GROUP #### #### ProLiant DL380p Gen8 #### #### Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz 48x2.4 Ghz #### #### 379 GB #### ################################################################################ SSDs in RAID10 for data HDDs in RAID 10 for logs
  27. 27. MySQL @ King 27 Main database technology at King Backs all our games Plenty of other applications are backed by MySQL •Customer support •Custom built BI/Analysis applications •Buisness Systems (Collaboration, issue tracking, etc) •Other internal applications, e.g., developer support utilities © King Digital Entertainment plc 2014
  28. 28. © King Digital Entertainment plc 2014 Database tier 28 The butter and cream of the storage world: MySQL + Memcached Makes everything better!
  29. 29. © King Digital Entertainment plc 2014 MySQL Requirements 29 Mission Critical Critical functions break without it Mobile clients can playwithout connectivity. Client game state overrides on reconnect Sub millisecond response time expected. > 10ms generates alarm and throttling of database usage No bank
  30. 30. © King Digital Entertainment plc 2014 Database Architecture and Setup 30 Game state data for all users are stored No pruning or purging State stored as JSON blob Sharded MySQL databases UID sharding key One central MySQL shard catalogue data
  31. 31. © King Digital Entertainment plc 2014 Application load patterns 31 Almost exclusively Key-Value Friend data read on client start up. Populates Saga. No joins in shards Some JOIN and more SQL in shard catalogue Sharded, highly available, KV-Store, JSON (Recognize this architecture?)
  32. 32. © King Digital Entertainment plc 2014 Sharding Implementation 32 Central store used for •User/Shard lookup •Purchases •Registration •Application management console for Configuring Shard/Instance mappings Application user configuration Marketing etc Performance monitioring User Shards used for game state Data volume main issue Rich toolkit for application management Building tools for MySQL management
  33. 33. © King Digital Entertainment plc 2014 HA and Tooling 33 KISS Master/Slave pairs. No automatic failovers Shard catalogue has multiple slaves Home built tools for switching, installing, provisioning, moving shards, etc Cacti, VividCortex, MEM, Ganglia, Graphite Nagios Xtrabackup/mysqldump Percona toolkit
  34. 34. © King Digital Entertainment plc 2014 Monitoring of our databases! 34
  35. 35. 35 Challenges @ King HDD performance Data Volume HDD => SSD 100G => SSD 200G => SSD 400G Delivery times for new HW Racking Buy batches of pre-rackedHW Network capacity issues (Memcached/backups) Hetergenous environment © King Digital Entertainment plc 2014
  36. 36. 36 Challenges @ King Relations/Friendships/Joins: joins avoided Volume: one of our main issues Performace/Scalability: Working great Complex Operations: problematic, lots of new components needed + consolidations, standardization Backups: Taken lots of work. Needs more Monitoring: App good. DB not yet stable DB Connections: Typical app server issues. 10K+ connections to DB © King Digital Entertainment plc 2014
  37. 37. 37 Solutions Sharding. Just worked Throwing HWat problems MySQL, easy to use, well known, stable, plenty of 3rd party tools, multiple vendors, etc. Get more staff © King Digital Entertainment plc 2014
  38. 38. Conclusion 38 Pros: Scaled out with very few problems. All games on one DB tier. Cons: Very rawMySQL environment. No automationand tooling. Resource allocationskew. © King Digital Entertainment plc 2014
  39. 39. © King Digital Entertainment plc 2014 Current & Future Work 39 Managing the environment: Fire fighting, maintenance, new features, etc Data Volume management: InnoDB native compression, application compression, avoiding fragmentation (current workload fragments a lot) Automation, tooling, and Spring Cleaning of organically grown MySQL environment Document & Standardize Resource Utlization: CPU in abundance, storage scarce Migrate to 5.6 (so we can migrate to 5.7?) Improvementsto data-model, HW, new-games/new features Backupsolution improvements. DR improvements. Consolidationof minor in-house apps Handling number of connections
  40. 40. © King Digital Entertainment plc 2014 Q & A Yes! We are hiring!! about.king.com/jobs 40
  41. 41. Thank you © King Digital Entertainment plc 2014 41

×