MySQL@king

© King Digital Entertainment plc 2014
Table of contents
2
Introductions
Scaling MySQL
Scaling Candy Crush Saga
Q & A

Introductions
3
Olle
•Senior Database Engineer @ King
•Previously in MySQL Consulting @ Oracle EMEA
•Developer and Consultant @ Upsys (MySQL Pratner)
•Often colleagues with Ted
Ted
•Senior Principal Sales Consultant @ Oracle EMEA
•Developer and Consultant @ Upsys (MySQL Pratner)
•Long time MySQL user & friend of Olle

We make great games
4
-More than190 funtitles played in over 200 countries around the world
-345 million averagemonthlyuniqueusers(Q2 2014)
-Studios in Stockholm, London, Barcelona, Bucharest, Malmo, Berlin and Singapore. Offices in San Francisco, Malta, Tokyo, Seoul and Shanghai
About King

© King Digital Entertainment plc 2014 5
Some stats and factsMillions of players around the world
•345mm MUUs as of Q2 2014
•138mm DAUs as of Q2 2014
•More players playing at least one of our games each month than thepopulation of the US! Games popular across platforms, and can be played anywhere, anytime on any deviceGlobal leader in cross- platform casual games
•Three games in all three major top ten grossing lists for Q2 2014
•Our Saga games allow players to switch platform without losing their progress
•Four global franchises: Candy Crush Pet Rescue Farm Heroes Bubble Witch
•Founded in 2003, studios in Stockholm, London, Barcelona, Malmo, Bucharest, Berlin and Singapore
•900+ employees

Some of our games
6

“Saga” concept
7
Play a leveland get a score and earn1-3 stars
Progress tothe nextlevel
Seethe progress ofyourfriends!
Progress is storedand accessiblecross platform/device.
Historyand background

Scaling MySQL
8

Scaling MySQL
9
Scale up / Vertical scaling
•Buy more or faster disk
•Buy more memory
•Buy more or faster CPU’s
Scale out / Horizontal scaling
•Shard by function
•Shard by columns
•Shard by rows

Introduction to Sharding
10

What is sharding?
11
Divide your dataset into several pieces, think of horizontal partitioning where each partition resides on dedicated instance/HW.
Statements are spread amongst all partitions.
Each shard can be located on separate servers.

Why shard?
12
Performance
•Smaller working set of data on each shard. Working set fits (better) in memory
•Fetch data in parallel from many shards.
Size
•Database too big and will not fit on disk/memory on one server
GEO latency
•You want to shard on GEO and have ’local’ shards close to your users

Getting started
13
Choosing shard-key and type of shard distribution
•RANGE, HASH, etc
•USER, TIME, etc
Operations
•Shard move
•Shard split
•Fail-over / Switch-over
Monitoring
•Identify quickly growing or ”Hot” shards
Interfaces and metadata for sharding
•DIY
•MySQL Fabric or similar framework

Sharding challenges
14
JOINs
•Tables are spread over multiple servers
AUTO_INCREMENTs / Sequences
•In MySQL these are per table
Backups
•Coordinating over multiple servers
Complex operations
•Schema changes, shard splitting, sharding SW
Complex HA
•Stand-by servers also need to be sharded
Management (deploy / orchestration)
•Automation a must

Sharding architecture
15
1.Lookup shard for User 100
2.User 100 is on Shard 1
3.Fetch data related to User 100 from shard Shard 1
4.User data for User 100 sent from Shard 1
sh4
sh3
sh2
sh1
META
Sharding API
Apps
1
2
3
4

Sharding Frameworks
16
MySQL Fabric
•Based on plain old MySQL Replication
•Requires GTID
•Fabric aware connectors
•Automatic failover

Sharding Frameworks
17
MySQL Cluster
•Automatic sharding
•Shared nothing architecture
•No SPOF
•IMDB
•ndbmtd, ndb_mgm, and API nodes

Sharding Frameworks
18
Gizzard (Twitter)
DbShards
Hibernate Shards
ScaleBase
Many in-house custom made solutions

Scaling Candy Crush Saga
19

20
Scaling Candy Crush Saga
Candy Crush Numbers:
849M game plays per day
84M Candy Crush DAU
Served by MySQL
As of Q2 2014

21
ShardingCandy Crush Saga
Sharding architecture in place before Candy Crush explosion
Only 4 people in Operations
10-20 new DB servers deployed every month at peak (+ ~50 App servers + Memcached + BI)
No dedicated DB staff

Architecture overview
22

Architecture overview
23
250 000requests / s
50 000writes / s
(60 MB/s)
1 800 000reads / s
250 000writes / s
As of Q2 2014

Regular operation
24
MySQL Sharding

MySQL Numbers
25
User-Shards:
•> 50TB
•> 500K qps
•10/1 Read/Write
•Several billion row tables
•100+ instances
•MySQL 5.1 and 5.6 (5.0 and 5.5 used for inhouse applications)
•Memcached: Roughly 2M qps
•App servers caches in front of Memcached
As of Q2 2014

MySQL Numbers
26
•Hardware
################################################################################
#### hostXXX (debian/7.4 -Linux 3.2.0-4-amd64) ####
#### XYXXYZXYZXYZXYZ member of: SOME IMPORTANT GROUP ####
#### ProLiant DL380p Gen8 ####
#### Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz 48x2.4 Ghz ####
#### 379 GB ####
################################################################################
SSDs in RAID10 for data
HDDs in RAID 10 for logs

MySQL @ King
27
Main database technology at King
Backs all our games
Plenty of other applications are backed by MySQL
•Customer support
•Custom built BI/Analysis applications
•Buisness Systems (Collaboration, issue tracking, etc)
•Other internal applications, e.g., developer support utilities

Database tier
28
The butter and cream of the storage world:
MySQL
+
Memcached
Makes everything better!

MySQL Requirements
29
Mission Critical
Critical functions break without it
Mobile clients can playwithout connectivity. Client game state overrides on reconnect
Sub millisecond response time expected. > 10ms generates alarm and throttling of database usage
No bank

Database Architecture and Setup
30
Game state data for all users are stored
No pruning or purging
State stored as JSON blob
Sharded MySQL databases
UID sharding key
One central MySQL shard catalogue data

Application load patterns
31
Almost exclusively Key-Value
Friend data read on client start up. Populates Saga.
No joins in shards
Some JOIN and more SQL in shard catalogue
Sharded, highly available, KV-Store, JSON (Recognize this architecture?)

Sharding Implementation
32
Central store used for
•User/Shard lookup
•Purchases
•Registration
•Application management console for
Configuring Shard/Instance mappings
Application user configuration
Marketing etc
Performance monitioring
User Shards used for game state
Data volume main issue
Rich toolkit for application management
Building tools for MySQL management

HA and Tooling
33
KISS
Master/Slave pairs. No automatic failovers
Shard catalogue has multiple slaves
Home built tools for switching, installing, provisioning, moving shards, etc
Cacti, VividCortex, MEM, Ganglia, Graphite
Nagios
Xtrabackup/mysqldump
Percona toolkit

Monitoring of our databases!
34

35
Challenges @ King
HDD performance
Data Volume
HDD => SSD 100G => SSD 200G => SSD 400G
Delivery times for new HW
Racking
Buy batches of pre-rackedHW
Network capacity issues (Memcached/backups)
Hetergenous environment

36
Challenges @ King
Relations/Friendships/Joins: joins avoided
Volume: one of our main issues
Performace/Scalability: Working great
Complex Operations: problematic, lots of new components needed + consolidations, standardization
Backups: Taken lots of work. Needs more
Monitoring: App good. DB not yet stable
DB Connections: Typical app server issues. 10K+ connections to DB

37
Solutions
Sharding. Just worked
Throwing HWat problems
MySQL, easy to use, well known, stable, plenty of 3rd party tools, multiple vendors, etc.
Get more staff

Conclusion
38
Pros: Scaled out with very few problems. All games on one DB tier.
Cons: Very rawMySQL environment. No automationand tooling. Resource allocationskew.

Current & Future Work
39
Managing the environment: Fire fighting, maintenance, new features, etc
Data Volume management: InnoDB native compression, application compression, avoiding fragmentation (current workload fragments a lot)
Automation, tooling, and Spring Cleaning of organically grown MySQL environment
Document & Standardize
Resource Utlization: CPU in abundance, storage scarce
Migrate to 5.6 (so we can migrate to 5.7?)
Improvementsto data-model, HW, new-games/new features
Backupsolution improvements. DR improvements.
Consolidationof minor in-house apps
Handling number of connections

Q & A
Yes! We are hiring!!
about.king.com/jobs
40

MySQL@king

More Related Content

What's hot

Similar to MySQL@king

More from Ted Wennmark

Recently uploaded

MySQL@king