Downloaded 62 times
![Documents{ author : "roger",date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)", text : "Spirited Away",tags : [ "Tezuka", "Manga" ],comments : [ {author : "Fred",date : "Sat Jul 24 2010 20:51:03 GMT-0700 (PDT)",text : "Best Movie Ever” } ]}](https://image.slidesharecdn.com/scalingwithmongodb-110720123004-phpapp02/85/Scaling-with-mongo-db-SF-Mongo-User-Group-7-19-2011-11-320.jpg)
Uploaded byJared Rosoff
Scaling with mongo db - SF Mongo User Group 7-19-2011
- MongoDB allows scaling by using documents, optimizing indexes, and understanding your working data set size. - Replica sets can scale reads by adding secondary nodes for load balancing, while sharding scales writes and RAM usage by splitting data across multiple shards. - Proper disk configuration and replication are important to maximize performance when scaling with MongoDB.
More Related Content
Similar to Scaling with mongo db - SF Mongo User Group 7-19-2011
Scaling with mongo db - SF Mongo User Group 7-19-2011
- 1. Scaling with MongoDBJaredRosoff (jsr@10gen.com) - @forjared
- 3. How do wedo it today? We use a relational database but … We don’t use joinsWe don’t use transactionsWe add read-only slavesWe added a caching layerWe de-normalized our dataWe implemented custom shardingWe buy bigger servers
- 4. How’s that workingout for you?
- 5.
- 6.
- 7. By engineers, forengineers
- 8. The landscapeMemcachedKey /ValueScalability & PerformanceRDBMSDepth of functionality
- 9. Scaling your appUsedocuments Indexes make me happyKnowing your working setDisks are the bottleneckReplication makes reading funSharding for profit
- 10. Scaling your datamodel
- 11. Documents{ author: "roger",date : "Sat Jul 24 2010 19:47:11 GMT-0700 (PDT)", text : "Spirited Away",tags : [ "Tezuka", "Manga" ],comments : [ {author : "Fred",date : "Sat Jul 24 2010 20:51:03 GMT-0700 (PDT)",text : "Best Movie Ever” } ]}
- 12. Disk Seeks &Data LocalityRead = really really fastSeek = 5+ ms
- 13. Disk Seeks &Data LocalityPostCommentAuthor
- 14. Disk Seeks &Data LocalityPostAuthorCommentCommentCommentCommentComment
- 15.
- 16. Table scansFind wherex equals 71234567Looked at 7 objects
- 17. Tree LookupFind wherex equals 74627531Looked at 3 objects
- 18. Random IndexEntire indexmust fit in RAM
- 19. Right AlignedOnly smallportion in RAM
- 20.
- 21. Working SetActive Documents+ Used IndexesRAMDisk
- 22. Page FaultApp requestsdocumentDocument not in memoryEvict a page from memoryRead block from disk Return document from memoryApp152RAM34Disk
- 23. Figuring out workingSet> db.foo.stats() { "ns" : "test.foo", "count" : 1338330, "size" : 46915928, "avgObjSize" : 35.05557523181876, "storageSize" : 86092032, "numExtents" : 12, "nindexes" : 2, "lastExtentSize" : 20872960, "paddingFactor" : 1, "flags" : 0, "totalIndexSize" : 99860480, "indexSizes" : { "_id_" : 55877632, "x_1" : 43982848 }, "ok" : 1}Size of dataAverage document sizeSize on disk (and in memory!)Size of all indexesSize of each index
- 24.
- 25. Single Disk~200 seeks/ second
- 26. RAID0~200 seeks /second~200 seeks / second~200 seeks / second
- 27. RAID10~400 seeks /second~400 seeks / second~400 seeks / second
- 28.
- 29. Replica SetsRead /WriteSecondaryReadPrimaryReadSecondary
- 30. Replica SetsRead /WriteReadSecondarySecondaryReadPrimaryReadSecondarySecondaryRead
- 31.
- 32. SecondarySecondarySecondarySecondaryMongoSMongoSShard 10..10Shard 210..20Shard320..30Shard 430..40PrimaryPrimaryPrimaryPrimarySecondarySecondarySecondarySecondary
- 33.
- 34. 400GB Index?Shard 10..10Shard210..20Shard 320..30Shard 430..40100GBIndex!100GBIndex!100GBIndex!100GBIndex!
- 35.
- 36. SummaryUse documents toyour advantage!Optimize your indexes Understand your working setUse a sane disk configuratinoUse replicas to scale reads Use sharding to scale writes & working RAM
Editor's Notes
- #6 Let’s talk about infrastructure costs. You probably started building your application on top of an RDBMS. This is the way we have built enterprise and web applications for years. But the problem is that your RDBMS doesn’t have a smooth cost curve when you scale it up. When you start off, you may be running on a smaller server, totally adequate for your load. When you exceed the capacity of that small server, you need to buy a bigger server. You can’t add a second small server. This process repeats. You exceed the capacity of your new server, and upgrade your hardware. There are two long term problems with this: As you scale up, you end up paying more and more for each transaction that your system processes. A small server may cost you $1,000 per CPU, but when you need 128 processors, you might be paying as much as $100,000 per CPU. Each incremental step up in hardware gets more and more expensive, not cheaper and cheaper. You reach an end of this scaling approach. Once you have scaled up to the biggest hardware platform available on the market, there is nowhere to go; no bigger box to buy. At this point you need to change strategies, even if you can afford those ultra-high-end boxes.
- #7 And while we’ve been spending more and more money on Hardware, our developer productivity has gone down too. You will hear this storyover and over again from CIO’s and architects: “Well, we use <insert RDBMS> but we don’t use joins or transactions and we’ve de-normalized our schema.” As our hardware gets more and more expensive, we ask our developers to squeeze more and more performance out of the same box. To achieve this, they go through “herculean efforts” to strip their code of advanced features that once made them productive. De-normalizing data, eliminating joins and transactions, adding caching and sharding layers… These are risky projects that slow down feature velocity.