2010 mongo berlin-scaling


Published on

Published in: Technology
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • What is scaling? Well - hopefully for everyone here.
  • ec2 goes up to 64gb, maybe mention 256gb box here??? ($30-40k) maybe can but 256gb box, but i spin up 10 ec2 64gb boxes in 10 minutes
  • not schema less - dynamic schema schema is just as important, or more important than relational understand write vs read tradeoffs
  • compare to mysql here
  • most common performance problem why _id index can be ignored
  • data looked at per second/minute/hour/day are you indexes accessed randomly
  • 256gb ram $30-40k
  • Don’t pre-emptively shard - easy to add later
  • 2010 mongo berlin-scaling

    1. 1. Eliot Horowitz @eliothorowitz MongoBerlin October 4, 2010 Scaling with MongoDB
    2. 2. Scaling <ul><li>Storage needs only go up </li></ul><ul><li>Operations/sec only go up </li></ul><ul><li>Complexity only goes up </li></ul>
    3. 3. Scaling by Optimization <ul><li>Schema Design </li></ul><ul><li>Index Design </li></ul><ul><li>Hardware Configuration </li></ul>
    4. 4. Horizontal Scaling <ul><li>Vertical scaling is limited </li></ul><ul><li>Hard to scale vertically in the cloud </li></ul><ul><li>Can scale wider than higher </li></ul>
    5. 5. Schema <ul><li>Modeling the same data in different ways can change performance by orders of magnitude </li></ul><ul><li>Very often performance problems can be solved by changing Schema </li></ul>
    6. 6. Embedding <ul><li>Great for read performance </li></ul><ul><li>One seek to load entire object </li></ul><ul><li>One roundtrip to database </li></ul><ul><li>Writes can be slow if adding to objects all the time </li></ul><ul><li>Should you embed comments? </li></ul>
    7. 7. Indexes <ul><li>Index common queries </li></ul><ul><li>Make sure there aren’t duplicates: (A) and (A,B) aren’t needed </li></ul><ul><li>Right-balance indexes keep working set small </li></ul>
    8. 8. RAM Requirements <ul><li>Understand working set </li></ul><ul><li>What percentage of your data has to fit in RAM? </li></ul><ul><li>How do you figure this out? </li></ul>
    9. 9. Hardware <ul><li>Disk performance </li></ul><ul><li>How many drives </li></ul><ul><li>What about ec2? </li></ul><ul><li>Network performance </li></ul>
    10. 10. Read Scaling <ul><li>One master at any time </li></ul><ul><li>Programmer determines if read hits master or a slave </li></ul><ul><li>Pro: easy to setup, can scale reads very well </li></ul><ul><li>Con: reads are inconsistent on a slave </li></ul><ul><li>Writes don’t scale </li></ul><ul><li>Good for read heavy applications </li></ul>
    11. 11. One Master, Many Slaves <ul><li>Custom Master/Slave setup </li></ul><ul><li>Have as many slaves as you want </li></ul><ul><li>Can put them local to appservers </li></ul><ul><li>Good for 90+% read heavy applications (Wikipedia) </li></ul>
    12. 12. Replica Sets <ul><li>High Availability Cluster </li></ul><ul><li>One master at any time, up to 6 slaves </li></ul><ul><li>A slave automatically promoted to master if failure </li></ul><ul><li>Drivers [will] support auto routing of reads to slaves if programmer allows </li></ul><ul><li>Good for applications that need high write availability but mostly reads (Commenting System) </li></ul>
    13. 13. <ul><li>Many masters, even more slaves </li></ul><ul><li>Can scale reads and writes in two dimensions </li></ul><ul><li>Add slaves for inconsistent read scaling and redundancy </li></ul><ul><li>Add Shards for write and data size scaling </li></ul>Sharding
    14. 14. Common Setup <ul><li>Typical setup is 3 shards with 3 servers per shard: 3 masters, 6 slaves </li></ul><ul><li>One massive collection, dozen non-sharded </li></ul><ul><li>Can add sharding later to an existing replica set with no down time </li></ul><ul><li>Can have sharded and non-sharded collections </li></ul>
    15. 15. Use Cases <ul><li>Millions of user profiles </li></ul><ul><li>User activity stream </li></ul><ul><li>Photos </li></ul><ul><li>Logging </li></ul>
    16. 16. Download MongoDB http://www.mongodb.org and let us know what you think @eliothorowitz @mongodb 10gen is hiring! http://www.10gen.com/jobs