2011 Mongo FR - Indexing in MongoDB

MongoDB Indexing and Query Optimizer Details Antoine Girbal Mongo FR March 23, 2011

What will we cover? Many details of how indexing and the query optimizer work

A full understanding of these details is not required to use mongo, but this knowledge can be helpful when making optimizations.

We’ll discuss functionality of Mongo 1.8 (for our purposes pretty similar to 1.6 and almost identical to 1.7 edge).

Much of the material will be presented through examples.

Diagrams are to aid understanding – some details will be left out.

Btree (conceptual diagram) 1 2 3 4 5 6 7 8 9 {_id:4,x:6}

Find One Document db.c.find( {x:6} ).limit( 1 )

Find One Document 1 2 3 4 5 6 7 8 9 6 ? {_id:4,x:6}

Find One Document > db.c.find( {x:6} ).limit( 1 ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 1, "nscannedObjects" : 1, "n" : 1, "millis" : 1, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : false, "indexOnly" : false, "indexBounds" : { "x" : [ [ 6, 6 ] ] } } Uses a btree cursor to find the object. Index ranges are around a single value.

Find One Document 1 2 3 4 5 6 6 6 9 6 ? {_id:4,x:6} Now we have duplicate x values

Find One Document 1 2 3 4 5 6 6 6 9 6 ? {_id:4,x:6}

Equality Match db.c.find( {x:6} )

Several documents to be returned

Equality Match 9 1 2 3 4 5 6 6 6 6 ? {_id:4,x:6} {_id:5,x:6} {_id:1,x:6}

Equality Match > db.c.find( {x:6} ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 3, "nscannedObjects" : 3, "n" : 3, "millis" : 1, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : false, "indexOnly" : false, "indexBounds" : { "x" : [ [ 6, 6 ] ] } }

Equality Match 1 2 3 4 5 6 6 6 9 6 ?

Full Document Matcher db.c.find( {x:6,y:1} )

Object content needs to be checked

Full Document Matcher 9 1 2 3 4 5 6 6 6 6 ? {y:4,x:6} {y:5,x:6} {y:1,x:6}

Full Document Matcher > db.c.find( {x:6,y:1} ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 3, "nscannedObjects" : 3, "n" : 1, "millis" : 1, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : false, "indexOnly" : false, "indexBounds" : { "x" : [ [ 6, 6 ] ] } } Documents for all matching index keys are scanned, but only one document matched on non index keys.

Range Match db.c.find( {x:{$gte:4,$lte:7}} )

Range Match 8 1 2 3 4 5 6 7 9 4 <= ? <= 7

Range Match > db.c.find( {x:{$gte:4,$lte:7}} ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 4, "nscannedObjects" : 4, "n" : 4, "millis" : 1, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : false, "indexOnly" : false, "indexBounds" : { "x" : [ [ 4, 7 ] ] } }

Exclusive Range Match db.c.find( {x:{$gt:4,$lt:7}} )

Range of index is same as inclusive range match

but boundaries are not scanned nor returned

Multikeys db.c.find( {x:{$gt:7}} )

documents contain lists with several values like [8,9].

Multikeys 1 2 3 4 5 6 7 9 ? > 7 {_id:4,x:[8,9]} 8

Multikeys > db.c.find( {x:{$gt:7}} ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 2, "nscannedObjects" : 2, "n" : 1, "millis" : 1, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : true, "indexOnly" : false, "indexBounds" : { "x" : [ [ 7, 1.7976931348623157e+308 ] ] } } All keys in valid range are scanned, but the matcher rejects duplicate documents making n == 1.

Range Types Explicit inequality db.c.find( {x:{$gt:4,$lt:7}} )

db.c.find( {x:{$ne:4}} ) Regular expression prefix db.c.find( {x:/^a/} ) Data type db.c.find( {x:/a/} )

Range Types db.c.find( {x:/â/} ) "indexBounds" : { "x" : [ [ "a", "b" ], [ /â/, /â/ ] ] } 2 ranges scanned of 2 different types: string and regex

Range Types db.c.find( {x:/a/} ) "indexBounds" : { "x" : [ [ "", { } ], [ /a/, /a/ ] ] } Here the index only helps to restrict type, not efficient in practice

Set Match db.c.find( {x:{$in:[3,6]}} )

Set Match 8 1 2 3 4 5 6 7 9 3 , 6

Set Match > db.c.find( {x:{$in:[3,6]}} ).explain() { "cursor" : "BtreeCursor x_1 multi", "nscanned" : 3, "nscannedObjects" : 2, "n" : 2, "millis" : 8, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : false, "indexOnly" : false, "indexBounds" : { "x" : [ [ 3, 3 ], [ 6, 6 ] ] }} Why is nscanned 3? This is an algorithmic detail, when there are disjoint ranges for a key nscanned may be higher than the number of matching keys.

All Match db.c.find( {x:{$all:[3,6]}} )

All Match 8 1 2 3 4 5 6 7 9 3 ? {_id:4,x:[3,6]}

All Match > db.c.find( {x:{$all:[3,6]}} ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 1, "nscannedObjects" : 1, "n" : 1, "millis" : 0, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : true, "indexOnly" : false, "indexBounds" : { "x" : [ [ 3, 3 ] ] } } The first entry in the $all match array is always used for index bounds. Note this may not be the least numerous indexed value in the $all array.

Limit db.c.find( {x:{$lt:6},y:3} ).limit( 3 )

Limit 8 1 2 3 4 5 6 7 9 6 ? < y:3 y:1 y:3 y:3 y:3

Limit > db.c.find( {x:{$lt:6},y:3} ).limit( 3 ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 4, "nscannedObjects" : 4, "n" : 3, "millis" : 1, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : true, "indexOnly" : false, "indexBounds" : { "x" : [ [ -1.7976931348623157e+308, 6 ] ] } } Scan until three matches are found, then stop.

Skip db.c.find( {x:{$lt:6},y:3} ).skip( 3 )

Skip 8 1 2 3 4 5 6 7 9 6 ? < y:3 y:1 y:3 y:3 y:3

Skip > db.c.find( {x:{$lt:6},y:3} ).skip( 3 ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 5, "nscannedObjects" : 5, "n" : 1, "millis" : 1, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : true, "indexOnly" : false, "indexBounds" : { "x" : [ [ -1.7976931348623157e+308, 6 ] ] } } All skipped documents are scanned.

Sort db.c.find( {x:{$lt:6}} ).sort( {x:1} )

Sorting along index key uses index btree ordering

Sort 8 1 2 3 4 5 6 7 9 6 ? < y:3 y:1 y:3 y:3 y:3

Sort > db.c.find( {x:{$lt:6},y:3} ).sort( {x:1} ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 5, "nscannedObjects" : 5, "n" : 4, "millis" : 1, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : true, "indexOnly" : false, "indexBounds" : { "x" : [ [ -1.7976931348623157e+308, 6 ] ] } } Find uses the btree cursor to easily sort data

Sort db.c.find( {x:{$lt:6}} ).sort( {y:1} )

Using non-indexed key to sort data will need to scan & order

Sort Results are sorted on the fly to match requested order. The scanAndOrder field is only printed when its value is true. > db.c.find( {x:{$lt:6},y:3} ).sort( {y:1} ).explain() { "cursor" : "BtreeCursor x_1", "nscanned" : 5, "nscannedObjects" : 5, "n" : 4, "scanAndOrder" : true, "millis" : 1, "nYields" : 0, "nChunkSkips" : 0, "isMultiKey" : true, "indexOnly" : false, "indexBounds" : { "x" : [ [ -1.7976931348623157e+308, 6 ] ] } }

Sort and scanAndOrder With “scanAndOrder” sort, all documents must be touched even if there is a limit spec.

With scanAndOrder, sorting is performed in memory and the memory footprint is constrained by the limit spec if present.

Count Count uses the same indexed but only scans in the index, not the object data in storage

With some operators the full document must be checked. Some of these cases: $all

2011 Mongo FR - Indexing in MongoDB

More Related Content

What's hot

Similar to 2011 Mongo FR - Indexing in MongoDB

2011 Mongo FR - Indexing in MongoDB