Eugene KovshilovskyTechnical Director, OperationsRick ChenTechnical DirectorCie Games, Inc.
Rick Chen Rick Chen is a Technical Director at CieGames, overseeing the company‟sengineering department. In 2012, Cie Gam...
Eugene Kovshilovsky Eugene Kovshilovsky is a Technical Director atCie Games, a leading developer and publisherof social a...
What is Sharding? (Horizontal) partitioning in a database where data is split byrows (collections) instead or columns
Why Shard in Games? Write Throughput Exceeds I/O Each click updates the user‟s currency amount.
Why Shard in Games? Memory isn‟t large enough to hold active dataset
Why Shard in Games? You want to run on commodity hardware and scalehorizontally vs. vertically
MongoDB Sharding Distribute objects within collections automatically Choose how data is partitioned Single master to sh...
Shard DesignShare the load amongst all shardsequallyHigh RandomnessHave the smallest working set possibleRoll with tim...
99.999% Most of our queries we have the ObjectId Client saves ObjectId for re-login at a later time I‟m this user and I...
Shard Behavior Attempt to insert data into pre-split chunks Increase cluster throughput by reducing overhead User Life ...
Pre Split Creates chunks and moves them to different shards ahead oftime Creates 0 byte chunks equally to all shards On...
3 Day Shard Key2^18 = 262144 seconds = 3.03 daysObjectId consists of the timestamp of when the_id is created“Consistent...
Mongo 2.4 and New HashSharding No need for Pre-splitting Easy setup of randomized shard key Better distribution of chun...
Development EnvironmentSharded Cluster
Deploy Start the Config Server (mongod) Instances Start the mongos Instances Add Shards to the Cluster Enable Sharding...
Start the Config Server DatabaseInstances Create data dir for each of the 3 config server instances Start the 3 config s...
Start the mongos Instances Ideally these lightweight instances run on the applicationserver(s) Start the mongos instance...
Add Shards to the Cluster Connect to mongos from mongo shell Add each shard to the cluster using sh.addShard() In produ...
Enable Sharding for a Database To shard collections you need to enable this first Add sharding to database usingsh.enabl...
Enable Sharding for a Collection Enable sharing usingsh.shardCollection("<database>.<collection>", shard-key-pattern)sh.s...
Demo Please check out our demo
Resources http://docs.mongodb.org/manual/sharding/ https://github.com/ekovshilovsky/mongo_java http://docs.mongodb.org/...
We are hiring! http://www.ciegames.com/careers
Upcoming SlideShare
Loading in …5
×

MongoDB Sharding

922
-1

Published on

This presentation describes the thought process for sharding in MongoDB for a social game that has or will have a large influx of users.

The presentation goes into how to setup a basic dev environment as well as provide some sample code to get you started.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
922
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

MongoDB Sharding

  1. 1. Eugene KovshilovskyTechnical Director, OperationsRick ChenTechnical DirectorCie Games, Inc.
  2. 2. Rick Chen Rick Chen is a Technical Director at CieGames, overseeing the company‟sengineering department. In 2012, Cie Games,the leading developer and publisher of socialand mobile games, was recognized as one ofthe Fastest Growing Company in NorthAmerica on Deloitte‟s Technology Fast 500 list.
  3. 3. Eugene Kovshilovsky Eugene Kovshilovsky is a Technical Director atCie Games, a leading developer and publisherof social and mobile games. Its popular game,Car Town, is the first Facebook game builtaround brands and has attracted more than 50million players worldwide. The company‟s firstiOS game, Car Town Streets, for iPhone, iPadand iPod touch, is one of the Top 10 Gameson the App Store in multiple countries.
  4. 4. What is Sharding? (Horizontal) partitioning in a database where data is split byrows (collections) instead or columns
  5. 5. Why Shard in Games? Write Throughput Exceeds I/O Each click updates the user‟s currency amount.
  6. 6. Why Shard in Games? Memory isn‟t large enough to hold active dataset
  7. 7. Why Shard in Games? You want to run on commodity hardware and scalehorizontally vs. vertically
  8. 8. MongoDB Sharding Distribute objects within collections automatically Choose how data is partitioned Single master to sharded cluster with = 0 down time Fully consistent Works well in the cloud Automated fail-over Java support with Spring integration for easy JSON to Objectmapping
  9. 9. Shard DesignShare the load amongst all shardsequallyHigh RandomnessHave the smallest working set possibleRoll with timeRecreate shard key from ObjectIdBe careful how you shard, no way back
  10. 10. 99.999% Most of our queries we have the ObjectId Client saves ObjectId for re-login at a later time I‟m this user and I want to do xyz with my car. No range based queries We don‟t do „find‟ this user in some range and query against ourmaster db, we can do this on our secondary data stores that arenot sharded or additional reporting systems.
  11. 11. Shard Behavior Attempt to insert data into pre-split chunks Increase cluster throughput by reducing overhead User Life Time of 3 Days Most new players expected to play for an average of 3 days Unbalanced chunks will be rebalanced during off-peak hours Rarely happens
  12. 12. Pre Split Creates chunks and moves them to different shards ahead oftime Creates 0 byte chunks equally to all shards One chunk may grow to 5-6 chunks depending on how manynew users come in within a 3 day window Used instead of waiting for the balancer to detect that thereare too many chunks in one shard
  13. 13. 3 Day Shard Key2^18 = 262144 seconds = 3.03 daysObjectId consists of the timestamp of when the_id is created“Consistent random” valueGenerate Md5 from String version of ObjectIdTake first 18 bit from Md5Append the 18 bit random value to thetimestamp
  14. 14. Mongo 2.4 and New HashSharding No need for Pre-splitting Easy setup of randomized shard key Better distribution of chunks on shards for isolated documentwrites and reads Uses first 64bit of md5 of the field Doesn‟t support compound indexes Don‟t use floating point numbers Range bases queries go to all shards Working set is distributed to all shards
  15. 15. Development EnvironmentSharded Cluster
  16. 16. Deploy Start the Config Server (mongod) Instances Start the mongos Instances Add Shards to the Cluster Enable Sharding for a Database Enable Sharding for a Collection
  17. 17. Start the Config Server DatabaseInstances Create data dir for each of the 3 config server instances Start the 3 config server instances If starting on the same machine for study purposes thenconfigure each one to run on a different portmkdir /data/configdbmongod –configsvr –dbpath /data/configdb –port 27019
  18. 18. Start the mongos Instances Ideally these lightweight instances run on the applicationserver(s) Start the mongos instances mongos –configdb <config server hostnames> If connecting to the demo configs on localhost then specify the config servers with the different ports. mongos run on port 27017 by defaultmongos –configdb cfg0.example.net:27019,cfg1.example.net:27019mongos –configdb localhost:27019,localhost:27020
  19. 19. Add Shards to the Cluster Connect to mongos from mongo shell Add each shard to the cluster using sh.addShard() In production always specify the replica setsh.addShard(“rs1/localhost:27017”)Mongo –host localhost –port 27017
  20. 20. Enable Sharding for a Database To shard collections you need to enable this first Add sharding to database usingsh.enableSharding((“<database>”)sh.enableSharding(“mydb”)
  21. 21. Enable Sharding for a Collection Enable sharing usingsh.shardCollection("<database>.<collection>", shard-key-pattern)sh.shardCollection(”mydb.shardcollection", {userId:1})
  22. 22. Demo Please check out our demo
  23. 23. Resources http://docs.mongodb.org/manual/sharding/ https://github.com/ekovshilovsky/mongo_java http://docs.mongodb.org/manual/administration/sharded-clusters/ http://docs.mongodb.org/manual/tutorial/deploy-shard-cluster/
  24. 24. We are hiring! http://www.ciegames.com/careers

×