2010 mongo berlin-shardinginternals (1)

2,152 views

Published on

Published in: Technology
  • Be the first to comment

2010 mongo berlin-shardinginternals (1)

  1. 1. Eliot Horowitz @eliothorowitz MongoBerlin October 4, 2010 Sharding Internals
  2. 2. MongoDB Sharding <ul><li>Scale horizontally for data size, index size, write and consistent read scaling </li></ul><ul><li>Distribute databases, collections or a objects in a collection </li></ul><ul><li>Auto-balancing, migrations, management happen with no down time </li></ul>
  3. 3. <ul><li>Choose how you partition data </li></ul><ul><li>Can convert from single master to sharded system with no downtime </li></ul><ul><li>Same features as non-sharding single master </li></ul><ul><li>Fully consistent </li></ul>
  4. 4. Range Based <ul><li>collection is broken into chunks by range </li></ul><ul><li>chunks default to 200mb or 100,000 objects </li></ul>
  5. 5. User profiles <ul><li>Partition by user_id </li></ul><ul><li>Secondary indexes on location, dates, etc... </li></ul><ul><li>Reads/writes know which shard to hit </li></ul>
  6. 6. User Activity Stream <ul><li>Shard by user_id </li></ul><ul><li>Loading a user’s stream hits a single shard </li></ul><ul><li>Writes are distributed across all shards </li></ul><ul><li>Can index on activity for deleting </li></ul>
  7. 7. Photos <ul><li>Can shard by photo_id for best read/write distribution </li></ul><ul><li>Secondary index on tags, date </li></ul>
  8. 8. Logging <ul><li>date </li></ul><ul><li>machine, date </li></ul><ul><li>logger name </li></ul>Possible Shard Keys
  9. 9. Architecture client mongos ... mongos mongod mongod mongodddd mongod mongod mongod ... Shards mongod mongod mongod Config Servers
  10. 10. Config Servers <ul><li>3 of them </li></ul><ul><li>changes are made with 2 phase commit </li></ul><ul><li>if any are down, meta data goes read only </li></ul><ul><li>system is online as long as 1/3 is up </li></ul>
  11. 11. Shards <ul><li>Can be master, master/slave or replica sets </li></ul><ul><li>Replica sets gives sharding + full auto-failover </li></ul><ul><li>Regular mongod processes </li></ul>
  12. 12. mongos <ul><li>Sharding Router </li></ul><ul><li>Acts just like a mongod to clients </li></ul><ul><li>Can have 1 or as many as you want </li></ul><ul><li>Can run on appserver so no extra network traffic </li></ul>
  13. 13. Writes <ul><li>Inserts : require shard key, routed </li></ul><ul><li>Removes: routed and/or scattered </li></ul><ul><li>Updates: routed or scattered </li></ul>
  14. 14. Queries <ul><li>By shard key: routed </li></ul><ul><li>sorted by shard key: routed in order </li></ul><ul><li>by non shard key: scatter gather </li></ul><ul><li>sorted by non shard key: distributed merge sort </li></ul>
  15. 15. Operations <ul><li>split: breaking a chunk into 2 </li></ul><ul><li>migrate: move a chunk from 1 shard to another </li></ul><ul><li>balancing: moving chunks automatically to keep system in balance </li></ul>
  16. 16. Setting it Up <ul><li>Start servers </li></ul><ul><li>add shards: db.runCommand( { addshard : &quot;10.1.1.5&quot; } ) </li></ul><ul><li>turn on partitioning: db.runCommand( { enablesharding : &quot;test&quot; } </li></ul><ul><li>shard a collection: db.runCommand( { shardcollection : &quot;test.data&quot; , key : { num : 1 } } ) </li></ul>
  17. 17. Download MongoDB http://www.mongodb.org and let us know what you think @eliothorowitz @mongodb 10gen is hiring! http://www.10gen.com/jobs

×