Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

2010 mongo berlin-shardinginternals (1)


Published on

Published in: Technology
  • Be the first to comment

2010 mongo berlin-shardinginternals (1)

  1. 1. Eliot Horowitz @eliothorowitz MongoBerlin October 4, 2010 Sharding Internals
  2. 2. MongoDB Sharding <ul><li>Scale horizontally for data size, index size, write and consistent read scaling </li></ul><ul><li>Distribute databases, collections or a objects in a collection </li></ul><ul><li>Auto-balancing, migrations, management happen with no down time </li></ul>
  3. 3. <ul><li>Choose how you partition data </li></ul><ul><li>Can convert from single master to sharded system with no downtime </li></ul><ul><li>Same features as non-sharding single master </li></ul><ul><li>Fully consistent </li></ul>
  4. 4. Range Based <ul><li>collection is broken into chunks by range </li></ul><ul><li>chunks default to 200mb or 100,000 objects </li></ul>
  5. 5. User profiles <ul><li>Partition by user_id </li></ul><ul><li>Secondary indexes on location, dates, etc... </li></ul><ul><li>Reads/writes know which shard to hit </li></ul>
  6. 6. User Activity Stream <ul><li>Shard by user_id </li></ul><ul><li>Loading a user’s stream hits a single shard </li></ul><ul><li>Writes are distributed across all shards </li></ul><ul><li>Can index on activity for deleting </li></ul>
  7. 7. Photos <ul><li>Can shard by photo_id for best read/write distribution </li></ul><ul><li>Secondary index on tags, date </li></ul>
  8. 8. Logging <ul><li>date </li></ul><ul><li>machine, date </li></ul><ul><li>logger name </li></ul>Possible Shard Keys
  9. 9. Architecture client mongos ... mongos mongod mongod mongodddd mongod mongod mongod ... Shards mongod mongod mongod Config Servers
  10. 10. Config Servers <ul><li>3 of them </li></ul><ul><li>changes are made with 2 phase commit </li></ul><ul><li>if any are down, meta data goes read only </li></ul><ul><li>system is online as long as 1/3 is up </li></ul>
  11. 11. Shards <ul><li>Can be master, master/slave or replica sets </li></ul><ul><li>Replica sets gives sharding + full auto-failover </li></ul><ul><li>Regular mongod processes </li></ul>
  12. 12. mongos <ul><li>Sharding Router </li></ul><ul><li>Acts just like a mongod to clients </li></ul><ul><li>Can have 1 or as many as you want </li></ul><ul><li>Can run on appserver so no extra network traffic </li></ul>
  13. 13. Writes <ul><li>Inserts : require shard key, routed </li></ul><ul><li>Removes: routed and/or scattered </li></ul><ul><li>Updates: routed or scattered </li></ul>
  14. 14. Queries <ul><li>By shard key: routed </li></ul><ul><li>sorted by shard key: routed in order </li></ul><ul><li>by non shard key: scatter gather </li></ul><ul><li>sorted by non shard key: distributed merge sort </li></ul>
  15. 15. Operations <ul><li>split: breaking a chunk into 2 </li></ul><ul><li>migrate: move a chunk from 1 shard to another </li></ul><ul><li>balancing: moving chunks automatically to keep system in balance </li></ul>
  16. 16. Setting it Up <ul><li>Start servers </li></ul><ul><li>add shards: db.runCommand( { addshard : &quot;; } ) </li></ul><ul><li>turn on partitioning: db.runCommand( { enablesharding : &quot;test&quot; } </li></ul><ul><li>shard a collection: db.runCommand( { shardcollection : &quot;; , key : { num : 1 } } ) </li></ul>
  17. 17. Download MongoDB and let us know what you think @eliothorowitz @mongodb 10gen is hiring!