• Share
  • Email
  • Embed
  • Like
  • Private Content
2011 mongo sf-sharding
 

2011 mongo sf-sharding

on

  • 4,735 views

 

Statistics

Views

Total Views
4,735
Views on SlideShare
2,195
Embed Views
2,540

Actions

Likes
3
Downloads
57
Comments
0

9 Embeds 2,540

http://www.10gen.com 2355
http://www.mongodb.com 147
http://archive.10gen.com 23
http://fromwww.mongodb.org 4
http://drupal1.10gen.cc 3
https://www.mongodb.com 3
http://www.slideshare.net 2
url_unknown 2
http://translate.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n

2011 mongo sf-sharding 2011 mongo sf-sharding Presentation Transcript

  • The Secret Sauce of Sharding Eliot Horowitz @eliothorowitz MongoSF May 24 2011
  • Architecture Shards mongod mongod mongod ... mongod mongod mongod mongod mongod mongod Config Serversmongod mongos mongos ...mongodmongod client client client client
  • Common Setup• A common setup is 3 shards with 3 servers per shard: 3 masters, 6 slaves• Can add sharding later to an existing replica set with no down time• Can have sharded and non-sharded collections
  • Range Based MIN MAX LOCATION A F shard1 F M shard1 M R shard2 R Z shard3• collection is broken into chunks by range• chunks default to 64mb or 100,000 objects
  • Config Servers• 3 of them• changes are made with 2 phase commit• if any are down, meta data goes read only• system is online as long as 1/3 is up
  • mongos• Sharding Router• Acts just like a mongod to clients• Can have 1 or as many as you want• Can run on appserver so no extra network traffic• Cache meta data from config servers
  • Writes• Inserts : require shard key, routed• Removes: routed and/or scattered• Updates: routed or scattered
  • Queries• By shard key: routed• sorted by shard key: routed in order• by non shard key: scatter gather• sorted by non shard key: distributed merge sort
  • Splitting• Take a chunk and split it in 2• Splits on the median value• Splits only change meta data, no data change
  • SplittingT1 MIN MAX LOCATION A Z shard1T2 MIN MAX LOCATION A G shard1 G Z shard1T3 MIN MAX LOCATION A D shard1 D G shard1 G S shard1 S Z shard1
  • Balancing• Moves chunks from one shard to another• Done online while system is running• Balancing runs in the background
  • MigratingT3 MIN MAX LOCATION A D shard1 D G shard1 G S shard1 S Z shard1T4 MIN MAX LOCATION A D shard1 D G shard1 G S shard1 S Z shard2T5 MIN MAX LOCATION A D shard1 D G shard1 G S shard2 S Z shard2
  • Setting it Up• Start servers• add shards: db.runCommand( { addshard : "10.1.1.5" } )• turn on partitioning: db.runCommand( { enablesharding : "test" }• shard a collection: db.runCommand( { shardcollection : "test.data" , key : { num : 1 } } )
  • Live Migration• Copy initial data set• Copy anything modified since start• Commit start• Copy delta• Finish commit
  • Download MongoDB http://www.mongodb.org and
let
us
know
what
you
think @eliothorowitz



@mongodb 10gen is hiring!http://www.10gen.com/jobs