Downloaded 37 times
![Check that everything is working!
Mongos
Config
Node 1
Secondary
Server
Shard
Mongod
[mongos] admin> db.runCommand({ listshards: 1 })
{
"shards": [
{
"_id": "shard0000",
"host": "cat1.mongodb.com:27018"
}
],
"ok": 1
}](https://image.slidesharecdn.com/randall-sharding-131207131548-phpapp01/85/Sharding-in-MongoDB-Days-2013-48-320.jpg)
Uploaded byRandall Hunt
Sharding in MongoDB Days 2013
This document provides an overview of MongoDB sharding. It begins with definitions of key terms like shards, chunks, config servers, and mongos. It explains how MongoDB partitions and distributes data across shards. The roles of config servers and mongos routers are outlined. Guidelines for choosing a shard key are presented, emphasizing characteristics like cardinality, write distribution, and query isolation. Best practices for setting up and using MongoDB sharding are also covered.
More Related Content
Similar to Sharding in MongoDB Days 2013
![Getting Started with Apache Spark: Big Data Made Simple [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/apachesparkgettingstarted-260203175547-8361bcc3-thumbnail.jpg?width=640&height=640&fit=bounds)
Sharding in MongoDB Days 2013
- 1. #MongoDBDays Chicago Introduction ToSharding J. Randall Hunt Hackathoner, MongoDB @jrhunt, randall@mongodb.com
- 2. In Today's Talk • What?Why? When? • How? • What's happening beind the scenes?
- 3.
- 4. This is apicture of my cat.
- 5. This is apicture of ~100 cats. http://a1.s6img.com/cdn/0011/p/3123272_8220815_lz.jpg
- 6. This is acat trying to find a home webserver mongod
- 7. 100 cats tryingto find a home. webserver (not to scale) mongod
- 8.
- 10. Data Store Scalability • CustomHardware • Custom Software In the past you've had two options for achieving data store scalability: 1) custom hardware (oracle?) 2) custom software (google, facebook) ! The reason these things were custom were that these problems were not yet common enough. The number of people on the internet 10 years ago is incredibly small compared to the number of people using web services 10 years from now.
- 11.
- 12.
- 13. The MongoDB ShardingSolution • Automatically partition your data • Worry about failover at the partition layer • Application independent • Free and open source
- 14. Why Do IShard?
- 15. Input/Output You input/output exceedsthe capacity of a single node or replica set. this is not easy to do!
- 16. Working Set ExceedsPhysical Memory RAM
- 17. Working Set ExceedsPhysical Memory Data RAM
- 18. Working Set ExceedsPhysical Memory Data RAM Indexes
- 19. Working Set ExceedsPhysical Memory Data RAM Sorts Indexes
- 20. Working Set ExceedsPhysical Memory Data RAM Sorts Indexes Aggregations
- 21. Working Set ExceedsPhysical Memory Data Indexes RAM Sorts Aggregations
- 22. Working Set ExceedsPhysical Memory
- 23. How Does ShardingWork?
- 24.
- 25. MongoDB's Sharding Infrastructure appserver mongod
- 26. MongoDB's Sharding Infrastructure appserver mongod mongod mongod
- 27. MongoDB's Sharding Infrastructure appserver shard
- 28. MongoDB's Sharding Infrastructure appserver shard
- 29. MongoDB's Sharding Infrastructure appserver mongos shard
- 30. MongoDB's Sharding Infrastructure appserver mongos mongod --configsvr shard
- 31. MongoDB's Sharding Infrastructure appserver mongos mongod --configsvr shard
- 32. Terminology • Shards • Chunks • Config Servers • mongos A shardis a server, or a collection of servers, that holds chunks of info which are split up according to a shard key, a shard holds a subset of a collection's data A chunk of info is a group of data falling in a particular range based on a shard key that can be moved logically from server to server config serves hold information about where chunks live mongos is the router and balancer -- it communicates with the config servers and figures out how to intelligently direct your query.
- 33. What exactly isa shard? • Shard is a node of the cluster • Can be a single mongod or an entire replica set Shard Mongod Shard or Primary Secondary Secondary Now what do shards hold? Chunks, which are partitions of your data that live in certain ranges.
- 34. Partitioning • User defines ashard key or uses hash based sharding • Shard key defines a range of data • The key space is like points on a line • A range is a segment of that line -∞ Remember interval notation? Key Space +∞
- 35. Data Distribution Initially asingle chunk Default Max Chunk Size: 64mb MongoDB willMongos Mongos split and migrate chunks as automatically Mongos they reach the max size Config Node 1 Secondary Server Shard 1 Mongod Shard 2
- 36. Shards and ShardKeys
- 37. Shards and ShardKeys Chunks!
- 38. Shards and ShardKeys Chunks! Shard Keys!
- 39. What is aconfig server? • A config server is for storing shard meta-data • It stores chunk ranges and locations • Run with 3 in production! Config Node 1 Secondary Server Config Node 1 Secondary Server or Config Node 1 Secondary Server Config Node 1 Secondary Server this is not a replica set, the three servers are purely for failover purposes. ! pro-tip use CNAMEs to identify these.
- 40. What is amongos? • Acts as a router / balancer for queries and ops • No local data (persists all info to the config servers) • Can run with just one or many App Server App Server App Server App Server or Mongos Mongos Mongos
- 41. MongoDB's Sharding Infrastructure AppServer Config Node 1 Secondary Server App Server App Server Mongos Mongos Mongos Shard Shard Shard Config Node 1 Secondary Server Config Node 1 Secondary Server
- 42. Get Started WithSharding? 1. Choose a shard key (we'll talk about this later) 2. Start config servers 3. Turn on sharding 4. Profit.
- 43. Mechanics of Sharding Ohhey there devops!
- 44. Start the ConfigurationServer Config Node 1 Secondary Server mongod --configsvr Starts a configuration server on the default port (27019)
- 45. Start the mongosrouter Mongos Config Node 1 Secondary Server mongos --configdb catconf.mongodb.com:27019
- 46. Start the mongod Mongos Config Node1 Secondary Server Shard Mongod mongod --shardsvr Starts a mongod with the default shard port (27018) Shard is not yet connected to the rest of the cluster Could have already been a part of the cluster
- 47. Add the Shard Mongos Config Node1 Secondary Server Shard Mongod On mongos: sh.addShard('cat1.mongodb.com:27018') For a replica set: sh.addShard('<rsname>/<seedlist>')
- 48. Check that everythingis working! Mongos Config Node 1 Secondary Server Shard Mongod [mongos] admin> db.runCommand({ listshards: 1 }) { "shards": [ { "_id": "shard0000", "host": "cat1.mongodb.com:27018" } ], "ok": 1 }
- 49. Now enable sharding • EnableSharding on a database sh.enableSharding("<dbname>") • Shard a collection (with a key): sh.shardCollection( "<dbname>.cat", {"name": 1}) • Use a compound shard key to prevent duplicates sh.shardCollection( "<dbname>.cats", {"name": 1, "uniqueid": 1})
- 50. Tag Aware Sharding • Totalcontrol over the distribution of your data! • Tag a range of shard keys: sh.addTagRange(<collection>,<min>,<max>,<tag>) • Tag a shard: sh.addShardTag("shard0000","NYC")
- 51. The Balancer • • Transparent todriver and application • try to minimize clock skew with ntpd Ensures even distribution of chunks across the cluster Very tuneable but defaults are often sensible
- 52. Routing Requests (Oh hithere application developers!)
- 53. Cluster Request Routing ScatterGather Targeted Choose your own adventure!
- 54.
- 55.
- 56. Request routed toappropriate shard 1 Mongos 2 Shard Shard Shard
- 57.
- 58. mongos returns resultsto client 1 4 Mongos 2 3 Shard Shard Shard
- 59.
- 60.
- 61. Farm request outto all shards 1 Mongos 2 Shard 2 Shard 2 Shard
- 62. shards return resultsto mongos 1 Mongos 2 3 Shard 2 2 3 Shard 3 Shard
- 63. mongos returns resultsto client 1 4 Mongos 2 3 Shard 2 2 3 Shard 3 Shard
- 64.
- 65. Things to remember! • • Shardkey values are immutable • Shard key must be indexed • It is limited to 512 bytes in size • Try to choose a field used in queries • should not be monotonically increasing! Shard Key is immutable Only the shard key can be guaranteed unique across shards
- 66. How to chooseyour key? • Cardinality • Write Distribution • Query Isolation • Reliability • Index Locality Cardinality – Can your data be broken down enough? Query Isolation - query targeting to a specific shard Reliability – shard outages ! A good shard key can: Optimize routing Minimize (unnecessary) traffic Allow best scaling ! consider pre splitting no unique indexes keys unless part of the shard key ! geokeys cannot be part of a shardkey $near won't work but the $geo commands work fine
- 67. Thanks! • What's Next? • Resources: https://education.mongodb.com/ https://www.mongodb.com/presentations • Me: @jrhunt, randall@mongodb.com Insummary -- and this is not a sales pitch... lots of other databases out there have sharding and replication... not many of them provide the granularity of control that you need for your applications while maintaining sensible defaults.