Indexing Matters: A MongoDB Optimization Primer

A MongoDB Optimization Primer Kyle Banker kyle@10gen.com @hwaet October 29, 2010 Indexing Matters: A MongoDB Optimization Primer

Agenda Indexing generalities Managing indexes in MongoDB How indexes match queries MongoDB’s query optimizer Proﬁling and explaning queries (Demos intermixed) Questions Indexing Matters: A MongoDB Optimization Primer

The trouble with indexing Widely misunderstood As is the case with relational systems, schema design and indexing go hand in hand... ... but you also need to know about your actual (not just predicted) query patterns. Indexing Matters: A MongoDB Optimization Primer

Need a new mental model? ”The Joy in Indexing” (http://kylebanker.com) Indexing Matters: A MongoDB Optimization Primer

Anti-patterns Lack of indexes. Too many indexes. Unused indexes. Redundant indexes. Failure to consider compound indexes. Poor understanding of how compound indexes work. Indexing Matters: A MongoDB Optimization Primer

Some indexing generalities MongoDB uses B-trees for indexes Many of the principles of indexing in an RDBMS can be applied to MongoDB e.g., Indexes entail additional work on inserts, updates, deletes. A collection may have at most 64 indexes. A query may only use 1 index (except for disjuncts of $or queries). Indexing Matters: A MongoDB Optimization Primer

Creating Indexes The id attribute is always indexed. Additional indexes can be created with ensureIndex(): // Create an index on the username attribute db.users.ensureIndex({ username : 1 }) // Create a compound index on // user_id and creation date db.posts.ensureIndex({ user-id : 1, created-at : -1 }) Indexing Matters: A MongoDB Optimization Primer

Creating Indexes // Create an index on the favorites // attribute, will index all values in list // Sample doc: { favorites: [’red’, ’blue’, ’green’] } db.users.ensureIndex({ favorites : 1 }) // Create a unique index on the username attribte db.users.ensureIndex({username: 1}, {unique:true}) // Create an index in the background. db.users.ensureIndex({user: 1}, {background:true}) Indexing Matters: A MongoDB Optimization Primer

Index maintenance // Drops an index on username db.users.dropIndex({username: 1}) // drops all indexes db.users.dropIndexes() // Rebuild indexes (need for this reduced in 1.6) db.users.reIndex() Indexing Matters: A MongoDB Optimization Primer

Indexes are smart about data types and structures Indexes on attributes whose values are of diﬀerent types in diﬀerent documents can speed up queries by skipping documents where the relevant attribute isn’t of the appropriate type. Indexes on attributes whose values are lists will index each element, speeding up queries that look into these attributes. (You really want to do this for querying on tags.) Indexing Matters: A MongoDB Optimization Primer

When can indexes be used? In short, if you can envision how the index might get used, it probably is. These will all use an index on age: db.users.find( { age: 25 } ) db.users.find( { age :{ $in : [1,2,3] } } ) db.users.find( { age : { $gt : 1 } } ) db.users.count( { age : 2 } ) db.users.distinct( { age : 2 } ) db.users.find().sort( { age: -1 } ) Indexing Matters: A MongoDB Optimization Primer

Trickier cases where indexes can be used db.posts.find({ user-id : 123 }).sort({ created-at : -1 }) will use an index on created-at for sorting, if there’s no index on user-id. (For this sort of case, use a compound index on both user-id and created-at in that order.) db.collection.update( { user-id : 123 } , {user-id : 456 } ) will use an index on user-id Indexing Matters: A MongoDB Optimization Primer

Inner attributes // Imagine you have a document like this { coord: { x: 86, y: -100 } addresses: [ {city: ’New York’, zip: 10010}, {city: ’Chicago’, zip: 60611} ] } db.items.ensureIndex({’coord.x’: 1, ’coord.y’: 1}) db.items.ensureIndex({coord: 1}) db.items.ensureIndex({’addresses.zip’: 1}) Indexing Matters: A MongoDB Optimization Primer

Some array examples The following queries will use an index on tags, and will match documents whose colors attribute is the array [’green’, ’blue’, ’red’] db.posts.find({ tags : ’green’ }) db.posts.find({ tags : { $gte : ’green’ } }) db.posts.find({ tags : { $all : [’green’, ’blue’] } }) db.posts.find({ tags : { $in : [’red’, ’black’] }}) Indexing Matters: A MongoDB Optimization Primer

When indexes cannot be used Many sorts of negations, e.g., $ne, $not. Tricky arithmetic, e.g., $mod. Most regular expressions (e.g., /a/). You can use on preﬁx ˆ queries (/b/) JavaScript expressions in $where clauses don’t take advantage of indexes. Of course $where clauses are mostly for complex queries that often can’t be indexed anyway, e.g., ‘‘where a > b’’. (If these cases matter to you, you can precompute the match and store that as an additional attribute, index it, and skip the $where clause entirely.) map/reduce can’t take advantage of indexes (mapping function is opaque to the query optimizer). However, you can pass a ﬁlter to map-reduce. Indexing Matters: A MongoDB Optimization Primer

Never forget about compound indexes Whenever you’re querying on multiple attributes, whether as part of the selector document or in a sort(), compound indexes can be used. Indexing Matters: A MongoDB Optimization Primer

When using compound indexes, careful with redundant indexes Imagine you have an index on { user-id : 1, votes : 1 } You don’t need the index on { user-id : 1 } But you would need an index on { votes : 1 } Indexing Matters: A MongoDB Optimization Primer

Compound order matters This: { user-id : 1, votes : 1 } Is not the same as this: { votes : 1, user-id : 1 } If this isn’t obvious, see ”The Joy in Indexing” (http://kylebanker.com) Indexing Matters: A MongoDB Optimization Primer

Matching compound indexes Imagine you have an index on { user-id: 1, votes: 1, : created-at: 1 } Permits: { user-id: 1 }, { user-id: 1, votes: 1 }, and { user-id: 1, votes: 1, created-at: 1 } Range has to be last: { user-id: 1, votes: {’$gt’: 1} } But not: { user-id: {$gt: 1}, votes: {’$gt’: 1} } Indexing Matters: A MongoDB Optimization Primer

Schema/index relationships Sometimes, question isn’t “given the shape of these documents, how do I index them?”, but “how might I shape the data so I can take advantage of indexing?” // Consider a schema that uses a list of // attribute/value pairs: db.elements.insert({ name : "Oxygen", attrs : [ { name : ’weight’, value: 8 }, { name : ’density’, value : 50 } ] }); db.c.ensureIndex({ ’attrs.name’ : 1, ’attrs.value’: 1 }); // All attribute queries can use one index. db.elements.find( { attrs : { $elemMatch : { name : ’weight’, value: {$gt: 10} } } } ) Indexing Matters: A MongoDB Optimization Primer

Index sizes Of course, indexes take up space. For many interesting databases, real query performance will depend on index sizes; so it’s useful to see these numbers. db.collection.stats() shows indexSizes, the size of each index in the collection. db.collection.totalIndexSize() displays the size of all indexes in the collection. Indexing Matters: A MongoDB Optimization Primer

explain() It’s useful to be able to ensure that your query is doing what you want it to do. For this, we have explain(). Query plans that use an index have cursor type BtreeCursor. db.collection.find({x:{$gt:5}}).explain() { "cursor" : "BtreeCursor x_1", ... "nscanned" : 12345, ... "n" : 100, "millis" : 4, ... } Indexing Matters: A MongoDB Optimization Primer

explain(), continued If the query plan doesn’t use the index, the cursor type will be BasicCursor. db.collection.find({x : { $gt : 5}}).explain() { "cursor" : "BasicCursor", ... "nscanned" : 12345, ... "n" : 42, "millis" : 4, ... } Indexing Matters: A MongoDB Optimization Primer

Query Optimizer MongoDB’s query optimizer is empirical, not cost-based. To test query plans, it tries several in parallel, and records the plan that ﬁnishes fastest. If a plan’s performance changes over time (e.g., as data changes), the database will reoptimize (i.e., retry all possible plans). Indexing Matters: A MongoDB Optimization Primer

Hinting the query plan Sometimes, you might want to force the query plan. For this, we have hint(). // Force the use of an index on attribute x: db.collection.find({x: 1, ...}).hint({x:1}) // Force indexes to be avoided! db.collection.find({x: 1, ...}).hint({$natural:1}) Indexing Matters: A MongoDB Optimization Primer

Going forward www.mongodb.org — downloads, docs, community mongodb-user@googlegroups.com — mailing list #mongodb on irc.freenode.net try.mongodb.org — web-based shell 10gen is hiring. Email jobs@10gen.com. 10gen oﬀers support, training, and advising services for mongodb Indexing Matters: A MongoDB Optimization Primer

http://www.10gen.com	169
http://www.nosqldatabases.com	127
http://samgee.pantip.com	1

Indexing Matters: A MongoDB Optimization Primer

by Kyle Banker on Nov 09, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

3 Embeds 297

Statistics

Indexing Matters: A MongoDB Optimization Primer — Presentation Transcript