Mongosv 2011 - Sharding

2,060 views

Published on

Sharding talk from MongoSV 2011.

0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,060
On SlideShare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
45
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Mongosv 2011 - Sharding

  1. 1. ShardingJared Rosoff (@forjared)
  2. 2. Overview• Architecture• How it works• Use Cases
  3. 3. SHARDINGARCHITECTURE
  4. 4. Architecture Shards mongod mongod mongod ... mongod mongod mongod mongod mongod mongod Config Serversmongod mongos mongos ...mongodmongod client client client client
  5. 5. mongos • Shard Router Shards • Acts just like a MongoD mongod mongod mongod mongod mongod mongod ... • 1 or as many as you Config mongod mongod mongod want Serversmongod • Can run on Appmongod mongos mongos ... Servers • Caches meta-data frommongod client client client client config servers
  6. 6. Config Server • 3 of them Shards • Changes use 2 phase mongod mongod mongod mongod mongod mongod ... commit Config mongod mongod mongod • If any are down, meta Serversmongod data goes read onlymongod mongos mongos ... • System is online asmongod client client client client long as 1/3 is up
  7. 7. HOW IT WORKS
  8. 8. Keys> db.runCommand( { shardcollection: “test.users”, key: { email: 1 }} ) { name: “Jared”, email: “jsr@10gen.com”, } { name: “Scott”, email: “scott@10gen.com”, } { name: “Dan”, email: “dan@10gen.com”, }
  9. 9. Chunks-∞ + ∞
  10. 10. Chunks-∞ + ∞ dan@10gen.com scott@10gen.com jsr@10gen.com
  11. 11. Chunks Split!-∞ + ∞ dan@10gen.com scott@10gen.com jsr@10gen.com
  12. 12. Chunks This is a Split! This is a chunk chunk-∞ + ∞ dan@10gen.com scott@10gen.com jsr@10gen.com
  13. 13. Chunks-∞ + ∞ dan@10gen.com scott@10gen.com jsr@10gen.com
  14. 14. Chunks-∞ + ∞ dan@10gen.com scott@10gen.com jsr@10gen.com
  15. 15. Chunks Split!-∞ + ∞ dan@10gen.com scott@10gen.com jsr@10gen.com
  16. 16. ChunksMin Key Max Key Shard-∞ adam@10gen.com 1adam@10gen.com jared@10gen.com 1jared@10gen.com scott@10gen.com 1scott@10gen.com +∞ 1• Stored in the config servers• Cached in MongoS• Used to route requests and keep cluster balanced
  17. 17. Balancing mongos config balancer configChunks! config 1 2 3 4 13 14 15 16 25 26 27 28 37 38 39 40 5 6 7 8 17 18 19 20 29 30 31 32 41 42 43 44 9 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48 Shard 1 Shard 2 Shard 3 Shard 4
  18. 18. Balancing mongos config balancer config Imbalance Imbalance config1 2 3 45 6 7 89 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48 Shard 1 Shard 2 Shard 3 Shard 4
  19. 19. Balancing mongos config balancer config Move chunk 1 config to Shard 21 2 3 45 6 7 89 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48 Shard 1 Shard 2 Shard 3 Shard 4
  20. 20. Balancing mongos config balancer config config1 2 3 45 6 7 89 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48 Shard 1 Shard 2 Shard 3 Shard 4
  21. 21. Balancing mongos config balancer config Chunk 1 now lives on Shard 2 config 2 3 45 6 7 8 19 10 11 12 21 22 23 24 33 34 35 36 45 46 47 48 Shard 1 Shard 2 Shard 3 Shard 4
  22. 22. ROUTING
  23. 23. Routed Request 1 1. Query arrives at 4 mongos MongoS 2. MongoS routes query to a single shard 3. Shard returns results of query 2 4. Results returned to client 3Shard 1 Shard 2 Shard 3
  24. 24. Scatter Gather 1 4 1. Query arrives at mongos MongoS 2. MongoS broadcasts query to all shards 3. Each shard returns results for query 2 4. Results combined 2 2 and returned to client 3 3 3Shard 1 Shard 2 Shard 3
  25. 25. Distributed Merge Sort 1 6 1. Query arrives at mongos MongoS 2. MongoS broadcasts 5 query to all shards 3. Each shard locally sorts results 2 4. Results returned to 2 2 mongos 4 4 4 5. MongoS merge sorts individual results3 6. Combined sorted 3 3 result returned toShard 1 Shard 2 Shard 3 client
  26. 26. WritesInserts Requires shard db.users.insert({ key name: “Jared”, email: “jsr@10gen.com”})Removes Routed db.users.delete({ email: “jsr@10gen.com”}) Scattered db.users.delete({name: “Jared”})Updates Routed db.users.update( {email: “jsr@10gen.com”}, {$set: { state: “CA”}}) Scattered db.users.update( {state: “FZ”}, {$set:{ state: “CA”}} )
  27. 27. QueriesBy Shard Routed db.users.find( {email: “jsr@10gen.com”})KeySorted by Routed in order db.users.find().sort({email:-1})shard keyFind by non Scatter Gather db.users.find({state:”CA”})shard keySorted by Distributed db.users.find().sort({state:1}) merge sortnon shardkey
  28. 28. EXAMPLES
  29. 29. User Profiles{ • Shard by email name: “Jared”, email: “jsr@10gen.com”, • Lookup by email hits addresses: [ 1 node {state: “CA”} ] • Index on} {“addresses.state”:1}
  30. 30. Activity Stream{ • Shard by user_id user_id: “jsr@10gen.com”, event_id: “Logged in”, • Looking up a stream data: “…” hits 1 node} • Writing is evenly distributed • Index on {“event_id”:1} for deletes
  31. 31. Photos{ • What’s the right key? photo_id: ???, data: BinData(…) – Auto Increment?} – MD5( data ) – Now() + MD5(data) – Month() + MD5(data)

×