Webinar - Approaching 1 billion documents with MongoDB

Approaching 1 Billion Documents in MongoDB David Mytton 1/25 david@boxedice.com / www.mytton.net

Server Density Monitoring Processing Database UI 2/25 www.serverdensity.com

db.stats() Documents 981,289,332 Collections 47,962 Indexes 39,684 Data size 369GB Index size 241GB 3/25 As of 25th Apr 2010.

10 months 4/25 Why we moved: http://bit.ly/mysqltomongo

Initial Setup Replication Master Slave DC1 DC2 8GB RAM 8GB RAM 5/25

Vertical Scaling Replication Master Slave DC1 DC2 72GB RAM 8GB RAM 6/25

Tip #1 Keep your indexes in memory at all times. db.stats() 7/25

Manual Partitioning Replication Master A Slave A DC1 DC2 16GB RAM 16GB RAM Replication Master B Slave B DC1 DC2 8/25 16GB RAM 16GB RAM

Database vs collections • Many databases = many data ﬁles (small but quickly get large). • Many collections = watch namespace limit. 9/25

Namespaces = Number of collections + number of indexes 10/25

Tip #2 Monitor the 24,000 namespace limit. 11/25

Using Server Density 12/25

Console db.system.namespaces.count() 13/25

Replica Pairs = Failover Replica Pair Master A Slave A DC1 DC2 16GB RAM 16GB RAM Replica Pair Master B Slave B DC1 DC2 14/25 16GB RAM 16GB RAM

Tip #3 Pre-provision your oplog ﬁles. 15/25

A shell script to generate 75GB oplog ﬁles for i in {0..40} do echo $i head -c 2146435072 /dev/zero > local.$i done 16/25

Tip #4 Expect slower performance during initial replica sync. 17/25

Tip #5 You can rotate your log ﬁles from the console. 18/25

Rotating your log ﬁles db.runCommand("logRotate") 19/25

Tip #6 Index creation blocks by default. Use background indexing if necessary. 20/25 MongoDB Manual: http://bit.ly/mongobgindex

Tip #7 Increase your OS ﬁle descriptor limit + use persistent connections. 21/25

Too many open ﬁles! /etc/security/limits.conf mongo hard nofile 10000 mongo soft nofile 10000 user type limit /etc/ssh/sshd_conﬁg UsePAM yes 22/25

Space is not reused 23/25

Tip #8 10gen commercial support is worth paying for. 24/25

Summary 1. Keep indexes in memory. 2. Monitor the 24k namespace limit. 3. Pre-provision oplog ﬁles. 4. Expect slower performance on replica sync. 5. Rotate logs from the console. 6. Index creation blocks by default. 7. OS ﬁle descriptor limit + persistent connections. 25/25 8. Commercial support is worth it.

http://www.10gen.com	1110
http://blog.boxedice.com	668
http://www.nosqldatabases.com	659
http://www.ukd1.co.uk	122
http://www.slideshare.net	69
http://mongospanish.blogspot.com	28
http://blog.serverdensity.com	15
http://www.mongodb.org	10
http://translate.googleusercontent.com	5
https://www.10gen.com	3
http://trunk.ly	2
http://bitly.com	2
http://localhost:8080	2
http://dl.mongodb.org	1
http://dashboard.bloglines.com	1
http://blog.fasoulas.com	1

Webinar - Approaching 1 billion documents with MongoDB

by Boxed Ice on May 05, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

16 Embeds 2,698

Statistics

Webinar - Approaching 1 billion documents with MongoDB — Presentation Transcript