MongoDB Days UK: Indexing and Performance Tuning

Indexing and
Performance Tuning
Tom Schreiber
Senior Consulting Engineer, MongoDB

Indexes are the single biggest tunable
performance factor in MongoDB

Mail from a German customer after a MongoDB Performance Tuning Consulting:

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
db.test.find({a:5})

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
db.test.find({a:5})
Full collection scan

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
# documents scanned:
1
db.test.find({a:5})
!

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
!
db.test.find({a:5}) # documents scanned:
2

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
!
3

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
!
4

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
!
5

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
!
6

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
!
7

8
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
!

Time Complexity for searching in a list with n entries: O(n)
8
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
!

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 17
7
113
Index on 'a'
db.test.find({a:5})
Index scan
9

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 17
7
113
Index on 'a'
db.test.find({a:5})
Index scan
9
!
db.test.find({a:5})
# index entries scanned: 1
Index scan

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 17
7
113
Index on 'a'
db.test.find({a:5})
Index scan
9!
db.test.find({a:5})
Index scan

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 17
7
113
Index on 'a'
db.test.find({a:5})
Index scan
9
!
db.test.find({a:5})
Index scan

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 17
7
113
Index on 'a'
db.test.find({a:5})
Index scan
9
!

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 17
7
113
Index on 'a'
db.test.find({a:5})
Index scan
9
# documents scanned: 1
!

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
1 5 17
7
113
Index on 'a'
db.test.find({a:5})
Index scan
9
# documents scanned: 1
Time Complexity for
searching in a binary
search tree with n nodes:
O(log2 n)
log2 10.000 = 13
log2 100.000 = 16
log2 1.000.000 = 19
!

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
MongoDB Index Data Structure: B-Tree
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
db.test.createIndex({a:1})

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
db.test.find({a:{$gte:9, $lte:17}})

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
db.test.find().sort({a:1})

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
db.test.find().sort({a:-1})

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
db.test.createIndex({a:-1})
7 1
17 11 9 35 1

https://github.com/mongodb/mongo/tree/v3.0/src/mongo/db/storage/mmap_v1/btree

https://github.com/mongodb/mongo/tree/v3.0/src/third_party/wiredtiger/src/btree

WiredTiger Index- and Data Structures
{'a': 3,
'b': 20}
1
{'b': 30,
'a': 9 }
2
{'c': 40,
'a': 1 }
3
{'c': 9,
'b': 4,
'a': 7}
4
{'a': 1,
'c': 20}
5
{'a': 11}
6
{'a': 17}
7
{'a': 5,
'b': 80}
8
1 3 5 7
1 5
7
4
11
6
17
7
1 3 7 11
1 7
5
8
3
1
1
5
1
3
Collection B+-Tree
Index B+-Tree
Index on 'a'
https://github.com/mongodb/mongo/tree/v3.0/src/third_party/wiredtiger/src/btree
9
2

{...} {...} {...} {...}{...} {...} {...}{...}
MMAPv1
Flat Files for Collections
{...} {...}{...} {...}{...} {...}{...}
B-Trees for Indexes
B-Trees for Collections
WiredTiger
Index (and Collection-) Data Structures
are Storage Engine specific
B-Tree Indexes
LSM-Tree Indexes
https://github.com/mongodb/mongo/tree/
v3.0/src/mongo/db/storage/mmap_v1
v3.0/src/third_party/wiredtiger
Pluggable Storage Engine API
MongoDB Database Engine

{...} {...} {...} {...}{...} {...} {...} {...}
MMAPv1
Flat Files for Collections
{...} {...} {...} {...}{...} {...} {...}
B-Trees for Indexes
B-Trees for Collections
WiredTiger
. . .
Index (and Collection-) Data Structures
are Storage Engine specific
B-Tree Indexes
LSM-Tree Indexes
v3.0/src/mongo/db/storage/mmap_v1
v3.0/src/third_party/wiredtiger
Fractal-Tree
Indexes
TokuMXse
LSM-Tree
Indexes
RocksDB
Pluggable Storage Engine API
MongoDB Database Engine

Balancing Speed of Reads and WritesPerformance
Fast reads Fast writesBoth
Easy:
• Add indexes
Easy:
• No indexes
Hard:
• Smart schema design  
(hire a consultant) 
• LSM index structures
B-Tree Indexes LSM-Tree Indexes

WiredTiger
B-Tree Index
WiredTiger
LSM-Index
WiredTiger
LSM-Index
WiredTiger
You can choose…
Single Thread
Update-heavy
"
Multi Thread
Update-heavy
"
Write-heavy
"
Analytics
"
Special
"
3rd party
In-Memory
In-Memory
In-Memory
Encrypted
MMAPv1

…even in the same replica set
Write-heavy
"
Analytics
"
Special
"
Secondary
Primary
Secondary
In-Memory
Encrypted
WiredTiger
LSM-Index

tresult
The Query Optimizer
Chooses the most efficient query plan.
Information on query plans 
and their execution statistics:
db.col.query.explain()
full collection scan
index on x
index on y
t0
Choose and Remenber
Terminate
db.col.query(...)

stage
plan root
stage stage
stage
stage stage
db.col.query.explain()
{...} {...} {...} {...} {...} {...} {...} {...} {...}

db.test.find({a:5}).explain()
{
"queryPlanner" : {
[...],
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"a" : 5
},
"indexName" : "a_1",
"isMultiKey" : false,
"direction" : "forward",
"indexBounds" : {
"a" : [
"[5.0, 5.0]"
]
}
}
}
},
[...]
}

plan root
stage
FETCH
stage
IXSCAN
{
"queryPlanner" : {
[...],
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"a" : 5
},
"indexBounds" : {
"a" : [
"[5.0, 5.0]"
]
}
}
}
},
[...]
}

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
{
"queryPlanner" : {
[...],
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",
"keyPattern" : {
"a" : 5
},
"indexBounds" : {
"a" : [
"[5.0, 5.0]"
]
}
}
}
},
[...]
}
stage
FETCH
plan root
stage
IXSCAN

Explain Levels
queryPlanner
"Which plan will MongoDB choose to run my query?"
executionStats
"How is my query performing?"
allPlansExecution
"I want as much information as possible to diagnose a slow query."

{
[...],
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 1,
"executionTimeMillis" : 0,
"totalKeysExamined" : 1,
"totalDocsExamined" : 1,
"executionStages" : {
"stage" : "FETCH",
"nReturned" : 1,
"executionTime
MillisEstimate" : 0,
"works" : 2,
"advanced" : 1,
"needTime" : 0,
"needFetch" : 0,
"saveState" : 0,
"restoreState" : 0,
"isEOF" : 1,
"invalidates" : 0,
"docsExamined" : 1,
"alreadyHasObj" : 0,
"inputStage" : {
"stage" : "IXSCAN",
db.test.find({a:5}).explain("executionStats")db.test.find({a:5}).explain()
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
stage
IXSCAN
stage
FETCH
plan root

"invalidates" : 0, 
"docsExamined" : 1, 
"alreadyHasObj" : 0, 
 
"inputStage" : { 
 
"stage" : "IXSCAN", 
"nReturned" : 1, 
"executionTime
MillisEstimate" : 0, 
"works" : 1, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"restoreState" : 0, 
"isEOF" : 1, 
"keyPattern" : { 
"a" : 5 
}, 
"indexName" : "a_1", 
"isMultiKey" : false, 
"direction" : "forward", 
"indexBounds" : { 
"a" : [ 
"[5.0, 5.0]" 
] 
}, 
"keysExamined" : 1, 
"dupsTested" : 0, 
"dupsDropped" : 0, 
"seenInvalidated" : 0, 
"matchTested" : 0 
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
stage
IXSCAN
stage
FETCH
plan root
1
1
Loc1 Loc6Loc3 Loc4 Loc5Loc2 Loc7 Loc8Loc8
!!

 
"inputStage" : { 
 
"nReturned" : 1, 
"executionTime
"works" : 1, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
"keyPattern" : { 
"a" : 5 
}, 
"a" : [ 
"[5.0, 5.0]" 
] 
}, 
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
stage
IXSCAN
stage
FETCH
plan root
1
1
Loc8
Loc1 Loc6Loc3 Loc4 Loc5Loc2 Loc7 Loc8

"executionTimeMillis" : 0, 
"totalKeysExamined" : 1, 
"totalDocsExamined" : 1, 
 
"executionStages" : { 
 
"stage" : "FETCH", 
"nReturned" : 1, 
"executionTime
"works" : 2, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
 
"inputStage" : { 
 
"nReturned" : 1, 
"executionTime
"works" : 1, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
stage
IXSCAN
stage
FETCH
plan root
1
1
1
1
Loc8
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
!
2
Loc1 Loc6Loc3 Loc4 Loc5Loc2 Loc7 Loc8

 
"inputStage" : { 
 
"nReturned" : 1, 
"executionTime
"works" : 1, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
"keyPattern" : { 
"a" : 5 
}, 
"a" : [ 
"[5.0, 5.0]" 
] 
}, 
} 
Explain() method key metrics
# documents returned
How long did the query take
# index entries scanned
# documents scanned
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}

Index used? Which one?
 
"inputStage" : { 
 
"nReturned" : 1, 
"executionTime
"works" : 1, 
"advanced" : 1, 
"needTime" : 0, 
"needFetch" : 0, 
"saveState" : 0, 
"isEOF" : 1, 
"keyPattern" : { 
"a" : 5 
}, 
"a" : [ 
"[5.0, 5.0]" 
] 
}, 
} 
Explain() method key metrics
"IXSCAN"
"indexName" : "a_1"
# documents returned
How long did the query take
# index entries scanned
# documents scanned
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'a'
1 7
3 51 17119
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}

Performance Tuning with
Query Execution Monitoring

United States Postal Service ZIP-Codes
db.zips.find() 29353 documents
media.mongodb.org/zips.json
{ "zip" : "01001", "city" : "AGAWAM", "loc" : [ -72.622739, 42.070206 ], "pop" : 15338, "state" : "MA" }
{ "zip" : "01002", "city" : "CUSHMAN", "loc" : [ -72.51564999999999, 42.377017 ], "pop" : 36963, "state" : "MA" }
{ "zip" : "01005", "city" : "BARRE", "loc" : [ -72.10835400000001, 42.409698 ], "pop" : 4546, "state" : "MA" }
{ "zip" : "01007", "city" : "BELCHERTOWN", "loc" : [ -72.41095300000001, 42.275103 ], "pop" : 10579, "state" : "MA" }
{ "zip" : "01008", "city" : "BLANDFORD", "loc" : [ -72.936114, 42.182949 ], "pop" : 1240, "state" : "MA" }
{ "zip" : "01010", "city" : "BRIMFIELD", "loc" : [ -72.188455, 42.116543 ], "pop" : 3706, "state" : "MA" }
{ "zip" : "01011", "city" : "CHESTER", "loc" : [ -72.988761, 42.279421 ], "pop" : 1688, "state" : "MA" }
{ "zip" : "01012", "city" : "CHESTERFIELD", "loc" : [ -72.833309, 42.38167 ], "pop" : 177, "state" : "MA" }
{ "zip" : "01013", "city" : "CHICOPEE", "loc" : [ -72.607962, 42.162046 ], "pop" : 23396, "state" : "MA" }
{ "zip" : "01020", "city" : "CHICOPEE", "loc" : [ -72.576142, 42.176443 ], "pop" : 31495, "state" : "MA" }
{ "zip" : "01022", "city" : "WESTOVER AFB", "loc" : [ -72.558657, 42.196672 ], "pop" : 1764, "state" : "MA" }
{ "zip" : "01026", "city" : "CUMMINGTON", "loc" : [ -72.905767, 42.435296 ], "pop" : 1484, "state" : "MA" }
{ "zip" : "01027", "city" : "MOUNT TOM", "loc" : [ -72.67992099999999, 42.264319 ], "pop" : 16864, "state" : "MA" }
{ "zip" : "01028", "city" : "EAST LONGMEADOW", "loc" : [ -72.505565, 42.067203 ], "pop" : 13367, "state" : "MA" }
{ "zip" : "01030", "city" : "FEEDING HILLS", "loc" : [ -72.675077, 42.07182 ], "pop" : 11985, "state" : "MA" }
{ "zip" : "01031", "city" : "GILBERTVILLE", "loc" : [ -72.19858499999999, 42.332194 ], "pop" : 2385, "state" : "MA" }
{ "zip" : "01032", "city" : "GOSHEN", "loc" : [ -72.844092, 42.466234 ], "pop" : 122, "state" : "MA" }
{ "zip" : "01033", "city" : "GRANBY", "loc" : [ -72.52000099999999, 42.255704 ], "pop" : 5526, "state" : "MA" }
{ "zip" : "01034", "city" : "TOLLAND", "loc" : [ -72.908793, 42.070234 ], "pop" : 1652, "state" : "MA" }
{ "zip" : "01035", "city" : "HADLEY", "loc" : [ -72.571499, 42.36062 ], "pop" : 4231, "state" : "MA" }

Question
Zip codes in New York City with population more than
100,000 ordered by population in descending order
db.zips.find({state:'NY',city:'NEW YORK',pop:{'$gt':100000}})
.sort({pop:-1})
{"zip" : "10021", "city" : "NEW YORK", "pop" : 106564, "state" : "NY" }
{ "zip" : "99921", "city" : "CRAIG", "loc" : [ -133.117081, 55.47317 ], "pop" : 1398, "state" : "AK" }
{ "zip" : "99922", "city" : "HYDABURG", "loc" : [ -132.633175, 55.137406 ], "pop" : 891, "state" : "AK" }
{ "zip" : "99923", "city" : "HYDER", "loc" : [ -130.124915, 55.925867 ], "pop" : 116, "state" : "AK" }
{ "zip" : "99925", "city" : "KLAWOCK", "loc" : [ -133.055503, 55.552611 ], "pop" : 851, "state" : "AK" }
{ "zip" : "99926", "city" : "METLAKATLA", "loc" : [ -131.579001, 55.121491 ], "pop" : 1469, "state" : "AK" }
{ "zip" : "99927", "city" : "POINT BAKER", "loc" : [ -133.376372, 56.307858 ], "pop" : 426, "state" : "AK" }
{ "zip" : "99929", "city" : "WRANGELL", "loc" : [ -132.352918, 56.433524 ], "pop" : 2573, "state" : "AK" }
{ "zip" : "99950", "city" : "KETCHIKAN", "loc" : [ -133.18479, 55.942471 ], "pop" : 422, "state" : "AK" }

.sort({pop:-1})
No Index

.sort({pop:-1}).explain("executionStats")
No Index

{
"queryPlanner" : {
"winningPlan" : {
"stage" : "SORT",
"sortPattern" : {
"pop" : -1
},
"inputStage" : {
"stage" : "COLLSCAN",
"filter" : {
"$and" : [
{
"city" : {
"$eq" : "NEW YORK"
}
},
{
"state" : {
"$eq" : "NY"
}
No Index

plan root
{
"queryPlanner" : {
"winningPlan" : {
"stage" : "SORT",
"sortPattern" : {
"pop" : -1
},
"inputStage" : {
"filter" : {
"$and" : [
{
"city" : {
"$eq" : "NEW YORK"
}
},
{
"state" : {
"$eq" : "NY"
}
No Index

plan root
SORT
{
"queryPlanner" : {
"winningPlan" : {
"stage" : "SORT",
"sortPattern" : {
"pop" : -1
},
"inputStage" : {
"filter" : {
"$and" : [
{
"city" : {
"$eq" : "NEW YORK"
}
},
{
"state" : {
"$eq" : "NY"
}
No Index

{
"queryPlanner" : {
"winningPlan" : {
"stage" : "SORT",
"sortPattern" : {
"pop" : -1
},
"inputStage" : {
"filter" : {
"$and" : [
{
"city" : {
"$eq" : "NEW YORK"
}
},
{
"state" : {
"$eq" : "NY"
}
plan root
SORT
COLLSCAN {...} {...} {...} {...} {...} {...} {...} {...} {...}
Collection Scan!
No Index

}
},
{
"pop" : {
"$gt" : 100000
}
}
]
},
"direction" : "forward"
}
},
"rejectedPlans" : [ ]
},
"nReturned" : 2,
"stage" : "SORT",
"nReturned" : 2,
"executionTimeMillisEstimate" : 0,
"works" : 29359,
"advanced" : 2,
"needTime" : 29355,
"needFetch" : 0,
"saveState" : 229,
"restoreState" : 229,
"isEOF" : 1,
"invalidates" : 0,
"sortPattern" : {
"pop" : -1
.explain("executionStats")
{...} {...} {...} {...} {...} {...} {...} {...} {...}
plan root
SORT
COLLSCAN
Collection Scan!
No Index

}
},
{
"pop" : {
"$gt" : 100000
}
}
]
},
}
},
},
"nReturned" : 2,
"stage" : "SORT",
"nReturned" : 2,
"works" : 29359,
"advanced" : 2,
"needTime" : 29355,
"needFetch" : 0,
"saveState" : 229,
"isEOF" : 1,
"invalidates" : 0,
"sortPattern" : {
"pop" : -1
{...} {...} {...} {...} {...} {...} {...} {...} {...}
2 documents returnedplan root
SORT
COLLSCAN
Collection Scan!
No Index

}
},
{
"pop" : {
"$gt" : 100000
}
}
]
},
}
},
},
"nReturned" : 2,
"stage" : "SORT",
"nReturned" : 2,
"works" : 29359,
"advanced" : 2,
"needTime" : 29355,
"needFetch" : 0,
"saveState" : 229,
"isEOF" : 1,
"invalidates" : 0,
"sortPattern" : {
"pop" : -1
{...} {...} {...} {...} {...} {...} {...} {...} {...}
2 documents returned
29353 documents scanned
plan root
SORT
COLLSCAN
Collection Scan!
No Index

}
},
{
"pop" : {
"$gt" : 100000
}
}
]
},
}
},
},
"nReturned" : 2,
"stage" : "SORT",
"nReturned" : 2,
"works" : 29359,
"advanced" : 2,
"needTime" : 29355,
"needFetch" : 0,
"saveState" : 229,
"isEOF" : 1,
"invalidates" : 0,
"sortPattern" : {
"pop" : -1
{...} {...} {...} {...} {...} {...} {...} {...} {...}
plan root
SORT
COLLSCAN
In-Memory Sort!
Collection Scan!
No Index

Single Field Index
db.zips.createIndex({state:1})

Single Field Index
{
"queryPlanner": {
"winningPlan": {
"stage": "SORT",
"sortPattern": {
"pop": -1
},
"inputStage": {
"stage": "FETCH",
"filter": {
"$and": [
{
"city": {
"$eq": "NEW YORK"
}
},
{
"pop": {
"$gt": 100000
}
plan root

Single Field Index
{
"queryPlanner": {
"winningPlan": {
"stage": "SORT",
"sortPattern": {
"pop": -1
},
"inputStage": {
"stage": "FETCH",
"filter": {
"$and": [
{
"city": {
"$eq": "NEW YORK"
}
},
{
"pop": {
"$gt": 100000
}
plan root
SORT

{
"queryPlanner": {
"winningPlan": {
"stage": "SORT",
"sortPattern": {
"pop": -1
},
"inputStage": {
"stage": "FETCH",
"filter": {
"$and": [
{
"city": {
"$eq": "NEW YORK"
}
},
{
"pop": {
"$gt": 100000
}
Single Field Index
plan root
SORT
FETCH

"pop": -1
},
"inputStage": {
"stage": "FETCH",
"filter": {
"$and": [
{
"city": {
"$eq": "NEW YORK"
}
},
{
"pop": {
"$gt": 100000
}
}
]
},
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {
"state": 1
},
"indexName": "state_1",
"isMultiKey": false,
"direction": "forward",
"indexBounds": {
"state": [
"["NY", "NY"]"
]
}
}
}
},
Single Field Index
plan root
SORT
FETCH

"pop": -1
},
"inputStage": {
"stage": "FETCH",
"filter": {
"$and": [
{
"city": {
"$eq": "NEW YORK"
}
},
{
"pop": {
"$gt": 100000
}
}
]
},
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {
"state": 1
},
"indexBounds": {
"state": [
"["NY", "NY"]"
]
}
}
}
},
Single Field Index
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

},
"indexBounds": {
"state": [
"["NY", "NY"]"
]
}
}
}
},
"rejectedPlans": []
},
"executionStats": {
"executionSuccess": true,
"nReturned": 2,
"executionTimeMillis": 6,
"totalKeysExamined": 1595,
"totalDocsExamined": 1595,
"executionStages": {
"stage": "SORT",
"nReturned": 2,
"executionTimeMillisEstimate": 0,
"works": 1600,
"advanced": 2,
"needTime": 1596,
"needFetch": 0,
"saveState": 12,
"restoreState": 12,
"isEOF": 1,
"invalidates": 0,
"sortPattern": {
"pop": -1
Single Field Index
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

},
"indexBounds": {
"state": [
"["NY", "NY"]"
]
}
}
}
},
"rejectedPlans": []
},
"executionStats": {
"nReturned": 2,
"stage": "SORT",
"nReturned": 2,
"works": 1600,
"advanced": 2,
"needTime": 1596,
"needFetch": 0,
"saveState": 12,
"restoreState": 12,
"isEOF": 1,
"invalidates": 0,
"sortPattern": {
"pop": -1
Single Field Index
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

},
"indexBounds": {
"state": [
"["NY", "NY"]"
]
}
}
}
},
"rejectedPlans": []
},
"executionStats": {
"nReturned": 2,
"stage": "SORT",
"nReturned": 2,
"works": 1600,
"advanced": 2,
"needTime": 1596,
"needFetch": 0,
"saveState": 12,
"restoreState": 12,
"isEOF": 1,
"invalidates": 0,
"sortPattern": {
"pop": -1
Single Field Index
plan root
SORT
FETCH
IXSCAN
1595 index entries scanned
{...} {...} {...} {...} {...} {...} {...} {...} {...}

},
"indexBounds": {
"state": [
"["NY", "NY"]"
]
}
}
}
},
"rejectedPlans": []
},
"executionStats": {
"nReturned": 2,
"stage": "SORT",
"nReturned": 2,
"works": 1600,
"advanced": 2,
"needTime": 1596,
"needFetch": 0,
"saveState": 12,
"restoreState": 12,
"isEOF": 1,
"invalidates": 0,
"sortPattern": {
"pop": -1
Single Field Index
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

},
"indexBounds": {
"state": [
"["NY", "NY"]"
]
}
}
}
},
"rejectedPlans": []
},
"executionStats": {
"nReturned": 2,
"stage": "SORT",
"nReturned": 2,
"works": 1600,
"advanced": 2,
"needTime": 1596,
"needFetch": 0,
"saveState": 12,
"restoreState": 12,
"isEOF": 1,
"invalidates": 0,
"sortPattern": {
"pop": -1
Single Field Index
plan root
SORT
FETCH
IXSCAN
In-Memory Sort!
{...} {...} {...} {...} {...} {...} {...} {...} {...}

.sort({pop:-1})
NY NY NY NY NY NY NY NY NY NY NY NY NY NY NY OH OH OH OH OH OH OH OH OH O
NY TX
NY
NY OH OK OR
{ "state" : "NY", "city" : "NEW YORK", "pop" : 106564, "zip" : "10021", "loc" : [ -73.958805, 40.768476 ] }
{ "state" : "NY", "city" : "NORTH TARRYTOWN", "pop" : 20080, "zip" : "10591", "loc" : [ -73.859335, 41.078108 ] }
{ "state" : "NY", "city" : "YONKERS", "pop" : 19958, "zip" : "10703", "loc" : [ -73.885163, 40.951763 ] }
{ "state" : "NY", "city" : "WHITE PLAINS", "pop" : 17843, "zip" : "10605", "loc" : [ -73.755247, 41.014053 ] }
{ "state" : "NY", "city" : "EAST WHITE PLAIN", "pop" : 8899, "zip" : "10604", "loc" : [ -73.750727, 41.043295 ] }
{ "state" : "NY", "city" : "HASTINGS ON HUDS", "pop" : 8546, "zip" : "10706", "loc" : [ -73.875912, 40.99071 ] }
{ "state" : "NY", "city" : "PUTNAM VALLEY", "pop" : 7453, "zip" : "10579", "loc" : [ -73.85024, 41.372815 ] }

db.zips.createIndex({state:1, city:1})
Compound Index on two fields

{ 
"queryPlanner": { 
"winningPlan": { 
"stage": "SORT", 
"sortPattern": { 
"pop": -1 
}, 
"inputStage": { 
"stage": "FETCH", 
"filter": { 
"pop": { 
"$gt": 100000 
} 
}, 
"inputStage": { 
"stage": "IXSCAN", 
"keyPattern": { 
"state": 1, 
"city": 1 
}, 

{ 
"winningPlan": { 
"stage": "SORT", 
"sortPattern": { 
"pop": -1 
}, 
"inputStage": { 
"filter": { 
"pop": { 
"$gt": 100000 
} 
}, 
"inputStage": { 
"keyPattern": { 
"state": 1, 
"city": 1 
}, 
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

{ 
"winningPlan": { 
"stage": "SORT", 
"sortPattern": { 
"pop": -1 
}, 
"inputStage": { 
"filter": { 
"pop": { 
"$gt": 100000 
} 
}, 
"inputStage": { 
"keyPattern": { 
"state": 1, 
"city": 1 
}, 
"indexName": "state_1_city_1", 
"isMultiKey": false, 
"direction": "forward", 
"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
"["NEW YORK", "NEW YORK
"]" 
] 
} 
} 
} 
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
"]" 
] 
} 
} 
} 
} 
}, 
"executionStats": { 
"executionSuccess": true, 
"nReturned": 2, 
"executionTimeMillis": 4, 
"totalKeysExamined": 40, 
"totalDocsExamined": 40, 
"executionStages": { 
"stage": "SORT", 
"nReturned": 2, 
"executionTimeMillisEstimate": 4, 
"works": 45, 
"advanced": 2, 
"needTime": 41, 
"needFetch": 0, 
"saveState": 0, 
"restoreState": 0, 
"isEOF": 1, 
"invalidates": 0, 
"sortPattern": { 
"pop": -1 
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
"]" 
] 
} 
} 
} 
} 
}, 
"nReturned": 2, 
"stage": "SORT", 
"nReturned": 2, 
"works": 45, 
"advanced": 2, 
"needTime": 41, 
"needFetch": 0, 
"saveState": 0, 
"isEOF": 1, 
"sortPattern": { 
"pop": -1 
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
"]" 
] 
} 
} 
} 
} 
}, 
"nReturned": 2, 
"stage": "SORT", 
"nReturned": 2, 
"works": 45, 
"advanced": 2, 
"needTime": 41, 
"needFetch": 0, 
"saveState": 0, 
"isEOF": 1, 
"sortPattern": { 
"pop": -1 
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}
In-Memory Sort!

ABERDEENOH
NEW YORKNYNEW YORKNYW YORK NEW YORKNY NEWARKNY
NEW YORKNY NEWARKNY
.sort({pop:-1})

db.zips.createIndex({state:1,city:1,pop:1})
Compound Index on three fields

{ 
"winningPlan": { 
"inputStage": { 
"keyPattern": { 
"state": 1, 
"city": 1, 
"pop": 1 
}, 
"indexName": "state_1_city_1_pop_1", 
"direction": "backward", 
"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
"["NEW YORK", "NEW YORK"]" 

{ 
"winningPlan": { 
"inputStage": { 
"keyPattern": { 
"state": 1, 
"city": 1, 
"pop": 1 
}, 
"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
IXSCAN
FETCH
plan root
{...} {...} {...} {...} {...} {...} {...} {...} {...}

{ 
"winningPlan": { 
"inputStage": { 
"keyPattern": { 
"state": 1, 
"city": 1, 
"pop": 1 
}, 
"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
], 
"pop": [ 
"[inf.0, 100000.0)" 
] 
} 
IXSCAN
FETCH
plan root
{...} {...} {...} {...} {...} {...} {...} {...} {...}

{ 
"winningPlan": { 
"inputStage": { 
"keyPattern": { 
"state": 1, 
"city": 1, 
"pop": 1 
}, 
"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
], 
"pop": [ 
"[inf.0, 100000.0)" 
] 
} 
IXSCAN
FETCH
plan root
{...} {...} {...} {...} {...} {...} {...} {...} {...}?

"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
], 
"pop": [ 
"[inf.0, 100000.0)" 
] 
} 
} 
} 
}, 
"nReturned": 2, 
"nReturned": 2, 
"works": 3, 
"advanced": 2, 
"needTime": 0, 
"needFetch": 0, 
"saveState": 0, 
"isEOF": 1, 
"docsExamined": 2, 
"alreadyHasObj": 0, 
IXSCAN
FETCH
plan root
{...} {...} {...} {...} {...} {...} {...} {...} {...}

"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
], 
"pop": [ 
"[inf.0, 100000.0)" 
] 
} 
} 
} 
}, 
"nReturned": 2, 
"nReturned": 2, 
"works": 3, 
"advanced": 2, 
"needTime": 0, 
"needFetch": 0, 
"saveState": 0, 
"isEOF": 1, 
IXSCAN
FETCH
plan root
{...} {...} {...} {...} {...} {...} {...} {...} {...}

"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
], 
"pop": [ 
"[inf.0, 100000.0)" 
] 
} 
} 
} 
}, 
"nReturned": 2, 
"nReturned": 2, 
"works": 3, 
"advanced": 2, 
"needTime": 0, 
"needFetch": 0, 
"saveState": 0, 
"isEOF": 1, 
IXSCAN
FETCH
plan root
{...} {...} {...} {...} {...} {...} {...} {...} {...}
2 documents scanned

"indexBounds": { 
"state": [ 
"["NY", "NY"]" 
], 
"city": [ 
], 
"pop": [ 
"[inf.0, 100000.0)" 
] 
} 
} 
} 
}, 
"nReturned": 2, 
"nReturned": 2, 
"works": 3, 
"advanced": 2, 
"needTime": 0, 
"needFetch": 0, 
"saveState": 0, 
"isEOF": 1, 
IXSCAN
FETCH
plan root
{...} {...} {...} {...} {...} {...} {...} {...} {...}
2 documents scanned
Index used for Sorting!

NEW YORKNY 106564NEW YORKNY 100027NEW YORKNY 8414374643
NEW YORKNY 51224 NEWARKNY
ADENAOH
.sort({pop:-1})
"indexBounds": { 
“state": "["NY", "NY"]", 
"city":"["NEW YORK", "NEW YORK"]", 
"pop":"[inf.0, 100000.0)"

.sort({pop:-1})
Projection on Index-only Data

db.zips.find({state:'NY',city:'NEW YORK',pop:{'$gt':100000}},
{state:1, city:1, pop:1})
.sort({pop:-1})
{"state" : "NY", "city" : "NEW YORK", "pop" : 106564 }
{"state" : "NY", "city" : "NEW YORK", "pop" : 100027 }

{
"queryPlanner": {
"winningPlan": {
"stage": "PROJECTION",
"transformBy": {
"state": 1,
"city": 1,
"pop": 1,
"zip": 0
},
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {
"state": 1,
"city": 1,
"pop": 1
},
"indexName": "state_1_city_1_pop_1",

{
"queryPlanner": {
"winningPlan": {
"transformBy": {
"state": 1,
"city": 1,
"pop": 1,
"zip": 0
},
"inputStage": {
"stage": "IXSCAN",
"keyPattern": {
"state": 1,
"city": 1,
"pop": 1
},
"indexName": "state_1_city_1_pop_1",
IXSCAN
PROJECTION
plan root

"direction": "backward",
"indexBounds": {
"state": [
"["NY", "NY"]"
],
"city": [
"["NEW YORK", "NEW YORK"]"
],
"pop": [
"[inf.0, 100000.0)"
]
}
}
}
},
"executionStats": {
"nReturned": 2,
"nReturned": 2,
"works": 3,
"advanced": 2,
"needTime": 0,
"needFetch": 0,
"saveState": 0,
"restoreState": 0,
"isEOF": 1,
"invalidates": 0,
IXSCAN
PROJECTION
plan root

"indexBounds": {
"state": [
"["NY", "NY"]"
],
"city": [
],
"pop": [
"[inf.0, 100000.0)"
]
}
}
}
},
"executionStats": {
"nReturned": 2,
"nReturned": 2,
"works": 3,
"advanced": 2,
"needTime": 0,
"needFetch": 0,
"saveState": 0,
"restoreState": 0,
"isEOF": 1,
"invalidates": 0,
IXSCAN
PROJECTION
plan root 2 documents returned

"indexBounds": {
"state": [
"["NY", "NY"]"
],
"city": [
],
"pop": [
"[inf.0, 100000.0)"
]
}
}
}
},
"executionStats": {
"nReturned": 2,
"nReturned": 2,
"works": 3,
"advanced": 2,
"needTime": 0,
"needFetch": 0,
"saveState": 0,
"restoreState": 0,
"isEOF": 1,
"invalidates": 0,
IXSCAN
PROJECTION

"indexBounds": {
"state": [
"["NY", "NY"]"
],
"city": [
],
"pop": [
"[inf.0, 100000.0)"
]
}
}
}
},
"executionStats": {
"nReturned": 2,
"nReturned": 2,
"works": 3,
"advanced": 2,
"needTime": 0,
"needFetch": 0,
"saveState": 0,
"restoreState": 0,
"isEOF": 1,
"invalidates": 0,
IXSCAN
PROJECTION
0 documents scanned

"indexBounds": {
"state": [
"["NY", "NY"]"
],
"city": [
],
"pop": [
"[inf.0, 100000.0)"
]
}
}
}
},
"executionStats": {
"nReturned": 2,
"nReturned": 2,
"works": 3,
"advanced": 2,
"needTime": 0,
"needFetch": 0,
"saveState": 0,
"restoreState": 0,
"isEOF": 1,
"invalidates": 0,
IXSCAN
PROJECTION
0 documents scanned
"Covered Query "

NEW YORKNY 106564NEW YORKNY 100027NEW YORKNY 8414374643
NEW YORKNY 51224 NEWARKNY
ADENAOH
{ "state" : "NY", "city" : "NEW YORK", "pop" : 106564}
{ "state" : "NY", "city" : "NEW YORK", "pop" : 100027}
.sort({pop:-1})

Runtime (ms) Speedup
No Index 80.155 Baseline
Compound Index 126 400
Covered Query 40 2.000
Benchmark Results
Standalone MongoDB v3.2.0-rc1, WiredTiger (12 GB Cache Size and snappy block compressor)
Collection Storage-size: 3 GB
Collection Data-size: 6 GB
Compound-Index-Size: 1 GB
Total-Index-Size: 2 GB
2,2 GHz Intel Core i7, 16 GB 1600 MHz DDR3, SSD-Storage, OS X 10.11.1 (15B42)
100.000.000 ZIP-Code documents - via Datagenerator

Benchmark Results

Benchmark Results
But Memory Contention
can be a Showstopper…

Performance Tuning via
Memory Contention Reduction

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 5,
'b': 80}
{'a': 17}{'a': 11}
{'a': 1,
'c': 20}
Memory Contention
Main Memory (RAM)

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
Memory Contention
Main Memory (RAM)

Index on 'a'
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
3 51 17119
1 7
Memory Contention
Main Memory (RAM)

Index on 'a'
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
3 51
Page 1
17119
Page 2
1 7
Page 3
Memory Contention
Main Memory (RAM)

Index on 'a'
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
3 51
Page 1
17119
Page 2
1 7
Page 3
3 51
Page 1
17119
Page 2
1 7
Page 3
Memory Contention
Main Memory (RAM)

{'a': 8}
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
Index on 'a'
3 51
Page 1
17119
Page 2
1 7
Page 3
3 51
Page 1
17119
Page 2
1 7
Page 3
Memory Contention
Main Memory (RAM)
db.test.insert({a:8})

{'a': 8}
{'c': 9,
'b': 4,
'a': 7}
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
Index on 'a'
3 51
Page 1
17119
Page 2
1 7
Page 3
3 51
Page 1
17119
Page 2
1 7
Page 3
Memory Contention
Main Memory (RAM)
Swap Space

{'a': 8}
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
{'c': 9,
'b': 4,
'a': 7}
Index on 'a'
3 51
Page 1
17119
Page 2
1 7
Page 3
3 51
Page 1
17119
Page 2
1 7
Page 3
Memory Contention
Main Memory (RAM)
Swap Space

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
{'a': 8}
Index on 'a'
3 51
Page 1
17
11
9
Page 2
1 7
Page 3
3 51
Page 1
17119
Page 2
1 7
Page 3
8
Page 4
Memory Contention
Main Memory (RAM)
{'c': 9,
'b': 4,
'a': 7}
Swap Space

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
{'a': 8}
Index on 'a'
3 51
Page 1
17
11
9
Page 2
1 7
Page 3
3 51
Page 1 Page 2
1 7
Page 3
8
Page 4
1117
Memory Contention
Main Memory (RAM)
{'c': 9,
'b': 4,
'a': 7}
Swap Space

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
{'a': 8}
{'c': 9,
'b': 4,
'a': 7}
Index on 'a'
3 51
Page 1
17
11
Page 2
1 7
Page 3
3 51
Page 1
Page 2
1 7
Page 3
98
Page 4
1117
98
Page 4
Memory Contention
Swap Space
Main Memory (RAM)

{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
{'a': 8}
{'c': 9,
'b': 4,
'a': 7}
Index on 'a'
3 51
Page 1
17
11
Page 2
1 7
Page 3
3 51
Page 1
Page 2
1 7
Page 3
98
Page 4
111798
Page 4
Memory Contention
Swap Space
Main Memory (RAM)

{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'c': 40,
'a': 1 }
{'a': 17}
{'a': 8}
{'b': 30,
'a': 9 }
{'c': 9,
'b': 4,
'a': 7}
Index on 'a'
Main Memory (RAM)
3 51
Page 1
17
11
Page 2
1 7
Page 3
Page 2
1 7
Page 3
98
Page 4
111798
Page 4
3 51
Page 1
Swap Space
Reduce Memory Contention
Right-Balanced Index Access
db.test.insert([{a:18},{a:19}]) {'a': 18}
{'a': 19}

{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 11}
{'a': 5,
'b': 80}
{'a': 17}
{'a': 8}
{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
Index on 'a'
Main Memory (RAM)
3 51
Page 1
17
11
Page 2
1 7
Page 3
Page 2
1 7
Page 3
98
Page 4
111798
Page 4
3 51
Page 1
Swap Space
db.test.insert([{a:18},{a:19}]) {'a': 18}
{'a': 19}

{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 11}
{'a': 5,
'b': 80}
{'a': 17}
{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
Index on 'a'
Main Memory (RAM)
3 51
Page 1
17
11
Page 2
1 7
Page 3
Page 2
1 7
Page 3
98
Page 4
111798
Page 4
3 51
Page 1
Swap Space
db.test.insert([{a:18},{a:19}])

{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 11}
{'a': 5,
'b': 80}
{'a': 17}
{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
Index on 'a'
Main Memory (RAM)
3 51
Page 1
17
11
Page 2
1 7
Page 3
Page 2
1 7
Page 3
98
Page 4
111798
Page 4
3 51
Page 1
Swap Space
18
18

{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 11}
{'a': 5,
'b': 80}
{'a': 17}
{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
Index on 'a'
Main Memory (RAM)
3 51
Page 1
17
11
Page 2
1 7
Page 3
Page 2
1 7
Page 3
98
Page 4
111798
Page 4
3 51
Page 1
Swap Space
18
18
19
19

{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 11}
{'a': 5,
'b': 80}
{'a': 17}
{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
Index on 'a'
Main Memory (RAM)
3 51
Page 1
11
Page 2
1 7
Page 3
Page 2
1 7
Page 3
98
Page 4
111798
Page 4
3 51
Page 1
Swap Space
18 19
17 18 19
db.test.insert([{a:20},{a:21},{a:22}])
{'a': 22}
{'a': 21}
{'a': 20}

{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
{'a': 5,
'b': 80}{'a': 17}{'a': 11}
{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
Index on 'a'
Main Memory (RAM)
3 51
Page 1
11
Page 2
1 7
Page 3
Page 2
1 7
Page 3
98
Page 4
111798
Page 4
3 51
Page 1
18 19
17 18 19
{'a': 22}
{'a': 21}
{'a': 20}
Swap Space

{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
{'a': 5,
'b': 80}{'a': 17}{'a': 11}
{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 22}{'a': 21}{'a': 20}
Index on 'a'
Main Memory (RAM)
3 51
Page 1
11
Page 2
1 7
Page 3
Page 2
1 7
Page 3
98
Page 4
111798
Page 4
3 51
Page 1
18 19
17 18 19
Swap Space

{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
{'a': 5,
'b': 80}{'a': 17}{'a': 11}
{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 22}{'a': 21}{'a': 20}
Index on 'a'
Main Memory (RAM)
3 51
Page 1
111 7
Page 3
Page 2
1 7
Page 3
98
Page 4
1117
3 51
Page 1
Swap Space
18 19
Page 2
Page 5
17 18 19
98
Page 4

{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
{'a': 5,
'b': 80}{'a': 17}{'a': 11}
{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 22}{'a': 21}{'a': 20}
Index on 'a'
Main Memory (RAM)
3 51
Page 1
111 7
Page 3
Page 2
1 7
Page 3
98
Page 4
1117
98
Page 4
3 51
Page 1
Swap Space
18 19
Page 2
Page 5
17 18 19
Page 5

{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 22}{'a': 21}{'a': 20}
Index on 'a'
Main Memory (RAM)
3 51
Page 1
111 7
Page 3
Page 2
1 7
Page 3
98
Page 4
1117
18 19
Page 2
Page 5
17 18 19
Page 5
20
20
{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
{'a': 5,
'b': 80}{'a': 17}{'a': 11}98
Page 4
3 51
Page 1
Swap Space

{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 22}{'a': 21}{'a': 20}
Index on 'a'
Main Memory (RAM)
3 51
Page 1
111 7
Page 3
Page 2
1 7
Page 3
98
Page 4
1117
18 19
Page 2
Page 5
17 18 19
Page 5
20 21
20 21
{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
{'a': 5,
'b': 80}{'a': 17}{'a': 11}98
Page 4
3 51
Page 1
Swap Space

{'a': 3,
'b': 20}
{'a': 1,
'c': 20}
{'a': 8}{'a': 18} {'a': 19}
{'a': 22}{'a': 21}{'a': 20}
Index on 'a'
Main Memory (RAM)
3 51
Page 1
111 7
Page 3
Page 2
1 7
Page 3
98
Page 4
1117
18 19
Page 2
Page 5
17 18 19
Page 5
20 21 22
20 21 22
{'c': 9,
'b': 4,
'a': 7}
{'c': 40,
'a': 1 }
{'b': 30,
'a': 9 }
{'a': 5,
'b': 80}{'a': 17}{'a': 11}98
Page 4
3 51
Page 1
Swap Space

Random Index Access
The entire index is “hot”
The entire index must fit into RAM
Spatial Data Locality
Amount of Index Data in RAM
is significantly reduced

Main Memory (RAM)
Remove unneeded indexes
db.zips.find({state:'NY',city:'NEW YORK’,pop:{'$gt':100000}})
.sort({pop:-1})
db.zips.createIndex({state:1, city:1, pop:1})

Main Memory (RAM)
.sort({pop:-1})
db.zips.dropIndex({state:1, city:1})

Main Memory (RAM)
.sort({pop:-1})
db.zips.dropIndex({state:1, city:1})
db.zips.dropIndex({state:1})

Sparse Indexes
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'c'
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
db.test.createIndex({c:1})
null 20
409null
null null null

Sparse Indexes
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'c'
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
db.test.find({c: {$exists:false}})
.explain()
null 20
409null
null null null
"winningPlan" : { 
"stage" : "FETCH", 
"inputStage" : { 
"c" : [ 
"[null, null]" 
] 
} 
}

Sparse Indexes
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'c'
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
null 20
409null
null null null
db.test.createIndex({c:1}, {sparse: true})

Sparse Indexes
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'c'
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
20 409
db.test.createIndex({c:1}, {sparse: true})

Sparse Indexes
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
Index on 'c'
{'a': 3,
'b': 20}
{'b': 30,
'a': 9 }
{'c': 40,
'a': 1 }
{'c': 9,
'b': 4,
'a': 7}
{'a': 1,
'c': 20}
{'a': 11} {'a': 17}
{'a': 5,
'b': 80}
20 409
db.test.find({c: {$exists:false}})
.explain()
"stage" : "COLLSCAN", 
"filter" : { 
"$not" : { 
"a" : { 
"$exists" : true 
} 
} 
} 
}

MongoDB 3.2
https://docs.mongodb.org/manual/release-notes/3.2/#partial-indexes
Partial Indexes

Index compression
mongod --dbpath DBPATH --storageEngine wiredTiger
--wiredTigerIndexPrefixCompression
Prefix Compression for Indexes on disk and in RAM

There is (much) more
Multikey Indexes
Text Indexes
Geospatial Indexes
TTL Indexes
. . .
Hash Indexes
Sparse Indexes
Database Profiler
mongod Log Files
mongostat
mongotop
mongoperf
Cloud Manager
Schema Design
Query Patterns
. . .

Summary
Indexes are the single biggest tunable performance factor in MongoDB
Create indexes that support your queries
Create highly selective indexes
Analyse your query plans with explain()
Use covered queries for maximum read performance
"An Index for each Query and a Query for each Index"

"
ThankYou!
Go and build something humongous!

#MDBDays
mongodb.com
Get your technical questions
answered
Benjamin Britten lounge (3rd floor), 10:00 -
17:00
By appointment only – register in person

db.zipcodes.aggregate([
{ $match: { state : "CA" } },
{ $group: { _id: '$city', zipCodes: { $sum: 1 } } },
{ $sort: { zipCodes: -1 } },
{ $limit: 3 }
])
Aggregation Query: Three cities in CA with most zip codes

{ $limit: 3 }
])
{ "zip" : "LOS ANGELES",
"zipCodes" : 56 }
{ "zip" : "SAN DIEGO",
"zipCodes" : 34 }
{ "zip" : "SAN JOSE",
"zipCodes" : 29 }

{ $limit: 3 }
],{explain:true})
"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }

"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }
{
"waitedMS" : NumberLong(0),
"stages" : [
{
"$cursor" : {...}
},
{
"$group": {...}
},
{
"$sort" : {..."limit"...}
}
]
}
{ $limit: 3 }
],{explain:true})

"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }
Group
Limit
Sort
Cursor
{
"waitedMS" : NumberLong(0),
"stages" : [
{
"$cursor" : {...}
},
{
"$group": {...}
},
{
"$sort" : {..."limit"...}
}
]
}
{ $limit: 3 }
],{explain:true})

{ 
"waitedMS" : NumberLong(0), 
"stages" : [ 
{ 
"$cursor" : { 
"fields" : { 
"city" : 1, 
"zip" : 0 
}, 
"queryPlanner" : { 
"stage" : "PROJECTION", 
"transformBy" : { 
"city" : 1, 
"zip" : 0 
}, 
"inputStage" : { 
"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }
Group
Limit
Sort
Cursor
{ $limit: 3 }
],{explain:true})

aitedMS" : NumberLong(0), 
tages" : [ 
{ 
"$cursor" : { 
"fields" : { 
"city" : 1, 
"zip" : 0 
}, 
"city" : 1, 
"zip" : 0 
}, 
"inputStage" : { 
"indexName" : "state_1_city_1_pop_1", 
"state" : [ 
"["CA", "CA"]" 
], 
"city" : [ 
"[MinKey, MaxKey]" 
], 
"pop" : [ 
] 
"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }
Group
Limit
Sort
Cursor
{ $limit: 3 }
],{explain:true})

tages" : [ 
{ 
"$cursor" : { 
"fields" : { 
"city" : 1, 
"zip" : 0 
}, 
"city" : 1, 
"zip" : 0 
}, 
"inputStage" : { 
"state" : [ 
"["CA", "CA"]" 
], 
"city" : [ 
], 
"pop" : [ 
] 
"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }
Group
Limit
Sort
IXSCAN
PROJECTION
Cursor
{ $limit: 3 }
],{explain:true})

tages" : [ 
{ 
"$cursor" : { 
"fields" : { 
"city" : 1, 
"zip" : 0 
}, 
"city" : 1, 
"zip" : 0 
}, 
"inputStage" : { 
"state" : [ 
"["CA", "CA"]" 
], 
"city" : [ 
], 
"pop" : [ 
] 
"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }
Group
Limit
Sort
IXSCAN
PROJECTION
Cursor
{ $limit: 3 }
],{explain:true})
Projection Optimization

tages" : [ 
{ 
"$cursor" : { 
"fields" : { 
"city" : 1, 
"zip" : 0 
}, 
"city" : 1, 
"zip" : 0 
}, 
"inputStage" : { 
"state" : [ 
"["CA", "CA"]" 
], 
"city" : [ 
], 
"pop" : [ 
] 
"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }
Group
Limit
Sort
IXSCAN
PROJECTION
Cursor
{ $limit: 3 }
],{explain:true})
"Covered Query "

"["CA", "CA"]" 
], 
"city" : [ 
], 
"pop" : [ 
] 
} 
} 
} 
} 
} 
}, 
{ 
"$group" : { 
"zip" : "$city", 
"zipCodes" : { 
"$sum" : { 
"$const" : 1 
} 
} 
} 
}, 
{ 
"$sort" : { 
"sortKey" : { 
"zipCodes" : -1 
}, 
"limit" : NumberLong(3) 
} 
} 
] 
"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }
Group
Limit
Sort
IXSCAN
PROJECTION
Cursor
{ $limit: 3 }
],{explain:true})
"Covered Query "

"["CA", "CA"]" 
], 
"city" : [ 
], 
"pop" : [ 
] 
} 
} 
} 
} 
} 
}, 
{ 
"$group" : { 
"zip" : "$city", 
"zipCodes" : { 
"$sum" : { 
"$const" : 1 
} 
} 
} 
}, 
{ 
"$sort" : { 
"sortKey" : { 
"zipCodes" : -1 
}, 
"limit" : NumberLong(3) 
} 
} 
] 
"zipCodes" : 56 }
"zipCodes" : 34 }
"zipCodes" : 29 }
Group
Limit
Sort
IXSCAN
PROJECTION
Cursor
{ $limit: 3 }
],{explain:true})
"Covered Query "
$sort + $limit Coalescence

{ $group:{_id:{state:'$state', city:'$city'}, zipCodes: { $sum: 1 }}},
{ $sort : { zipCodes:-1} },
{ $limit : 3 }
])
Aggregation Pipeline and Sharded Collections

{ $limit : 3 }
])
Config 
ServerConfig 
ServerConfig 
Server
Router
(mongos)
{...} {...} {...} {...} {...} {...} {...} {...}
shard1 shard2

{ $limit : 3 }
])
Config 
ServerConfig 
ServerConfig 
Server
Router
(mongos)
{...} {...} {...} {...} {...} {...} {...} {...}
shard1 shard2
{"zip": {"state": "TX", "city": "HOUSTON"},
"zipCodes" : 93 }
{"zip": {"state": "CA", "city": "LOS ANGELES"},
"zipCodes" : 56 }
{"zip": {"state": "PA", "city": "PHILADELPHIA"},
"zipCodes" : 48 }

shard key: { "state" : 1 } 
 
chunks: shard1 2, shard2 2 
 
{ "state" : { "$minKey" : 1 } } -->>
{ "state" : "CT" } on : shard1
 
{ "state" : "CT" } -->>
{ "state" : "ME" } on : shard1 
{ "state" : "ME" } -->>
{ "state" : "NJ" } on : shard2 
{ "state" : "NJ" } -->>
{ "state" : { "$maxKey" : 1 } }
on : shard2
sh.status()
Config 
ServerConfig 
ServerConfig 
Server
Router
(mongos)
{...} {...} {...} {...} {...} {...} {...} {...}
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"
shard1 shard2
{ $limit : 3 }
])
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }

{"splitPipeline": {
"shardsPart": [
{
"$group": {...}
}
],
"mergerPart": [
{
"$group": {...}
},
{
"$sort": {..."limit"...}
}
]
},
"shards": {
"shard1": {...},
"shard2": {...}
}
{ $limit : 3 }
], {explain:true})
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2

{"splitPipeline": {
"shardsPart": [
{
"$group": {...}
}
],
"mergerPart": [
{
"$group": {...}
},
{
"$sort": {..."limit"...}
}
]
},
"shards": {
"shard1": {...},
"shard2": {...}
}
{ $limit : 3 }
], {explain:true})
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group

{"splitPipeline": {
"shardsPart": [
{
"$group": {...}
}
],
"mergerPart": [
{
"$group": {...}
},
{
"$sort": {..."limit"...}
}
]
},
"shards": {
"shard1": {...},
"shard2": {...}
}
{ $limit : 3 }
], {explain:true})
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group

{"splitPipeline": {
"shardsPart": [
{
"$group": {...}
}
],
"mergerPart": [
{
"$group": {...}
},
{
"$sort": {..."limit"...}
}
]
},
"shards": {
"shard1": {...},
"shard2": {...}
}
{ $limit : 3 }
], {explain:true})
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group
< 2.6: mongos
< 3.2: primary shard
3.2: any shard

{"splitPipeline": { 
"shardsPart": [ 
{ 
"$group": { 
"zip": { 
"state": "$state", 
"city": "$city" 
}, 
"zipCodes": { 
"$sum": { 
"$const": 1 
} 
} 
} 
} 
], 
"mergerPart": [ 
{ 
"$group": { 
"zip": "$$ROOT._id", 
{ $limit : 3 }
], {explain:true})
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group

"$group": { 
"zip": { 
"city": "$city" 
}, 
"zipCodes": { 
"$sum": { 
"$const": 1 
} 
} 
} 
} 
], 
"mergerPart": [ 
{ 
"$group": { 
"zipCodes": { 
"$sum": "$$ROOT.zipCodes" 
}, 
"$doingMerge": true 
} 
}, 
{ 
"$sort": { 
"sortKey": { 
"zipCodes": -1 
}, 
"limit": NumberLong(3) 
} 
} 
] 
}, 
{ $limit : 3 }
], {explain:true})
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group

"$group": { 
"zip": { 
"city": "$city" 
}, 
"zipCodes": { 
"$sum": { 
"$const": 1 
} 
} 
} 
} 
], 
"mergerPart": [ 
{ 
"$group": { 
"zipCodes": { 
"$sum": "$$ROOT.zipCodes" 
}, 
"$doingMerge": true 
} 
}, 
{ 
"$sort": { 
"sortKey": { 
"zipCodes": -1 
}, 
} 
} 
] 
}, 
{ $limit : 3 }
], {explain:true})
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group
$sort + $limit  
Coalescence

} 
}, 
{ 
"$sort": { 
"sortKey": { 
"zipCodes": -1 
}, 
} 
} 
] 
}, 
"shards": { 
"shard1": { 
"stages": [ 
{ 
"$cursor": { 
"fields": { 
"city": 1, 
"state": 1, 
"zip": 0 
}, 
"winningPlan": { 
"inputStage": { 
"stage": "COLLSCAN", 
} 
} 
} 
} 
}, 
{ 
{ $limit : 3 }
], {explain:true})
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group
$sort + $limit  
Coalescence

} 
}, 
{ 
"$sort": { 
"sortKey": { 
"zipCodes": -1 
}, 
} 
} 
] 
}, 
"shards": { 
"shard1": { 
"stages": [ 
{ 
"$cursor": { 
"fields": { 
"city": 1, 
"state": 1, 
"zip": 0 
}, 
"winningPlan": { 
"inputStage": { 
} 
} 
} 
} 
}, 
{ 
{ $limit : 3 }
], {explain:true})
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group
$sort + $limit  
Coalescence

} 
}, 
{ 
"$sort": { 
"sortKey": { 
"zipCodes": -1 
}, 
} 
} 
] 
}, 
"shards": { 
"shard1": { 
"stages": [ 
{ 
"$cursor": { 
"fields": { 
"city": 1, 
"state": 1, 
"zip": 0 
}, 
"winningPlan": { 
"inputStage": { 
} 
} 
} 
} 
}, 
{ 
{ $limit : 3 }
], {explain:true})
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group
Cursor
COLLSCAN
{...} {...} {...} {...} {...} {...} {...} {...}
Cursor
COLLSCAN
$sort + $limit  
Coalescence

} 
}, 
{ 
"$sort": { 
"sortKey": { 
"zipCodes": -1 
}, 
} 
} 
] 
}, 
"shards": { 
"shard1": { 
"stages": [ 
{ 
"$cursor": { 
"fields": { 
"city": 1, 
"state": 1, 
"zip": 0 
}, 
"winningPlan": { 
"inputStage": { 
} 
} 
} 
} 
}, 
{ 
{ $limit : 3 }
], {explain:true})
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group
Cursor
COLLSCAN
{...} {...} {...} {...} {...} {...} {...} {...}
Cursor
COLLSCAN
Projection 
Optimization
$sort + $limit  
Coalescence

"shard1": { 
"stages": [ 
{ 
"$cursor": { 
"fields": { 
"city": 1, 
"state": 1, 
"zip": 0 
}, 
"winningPlan": { 
"inputStage": { 
} 
} 
} 
} 
}, 
{ 
"$group": { 
"zip": { 
"city": "$city" 
}, 
"zipCodes": { 
"$sum": { 
"$const": 1 
} 
} 
} 
} 
] 
}, 
{ $limit : 3 }
], {explain:true})
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group
Cursor
COLLSCAN
{...} {...} {...} {...} {...} {...} {...} {...}
Cursor
COLLSCAN
$sort + $limit  
Coalescence
Projection 
Optimization

"city": "$city" 
}, 
"zipCodes": { 
"$sum": { 
"$const": 1 
} 
} 
} 
} 
] 
}, 
"shard2": { 
"stages": [ 
{ 
"$cursor": { 
"fields": { 
"city": 1, 
"state": 1, 
"zip": 0 
}, 
"winningPlan": { 
"inputStage": { 
} 
}, 
} 
} 
}, 
{ 
"$group": { 
{ $limit : 3 }
], {explain:true})
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group
Cursor
COLLSCAN
{...} {...} {...} {...} {...} {...} {...} {...}
Cursor
COLLSCAN
$sort + $limit  
Coalescence
Projection 
Optimization

"$cursor": { 
"fields": { 
"city": 1, 
"state": 1, 
"zip": 0 
}, 
"winningPlan": { 
"inputStage": { 
} 
}, 
} 
} 
}, 
{ 
"$group": { 
"zip": { 
"city": "$city" 
}, 
"zipCodes": { 
"$sum": { 
"$const": 1 
} 
} 
} 
} 
] 
} 
} 
}
{ $limit : 3 }
], {explain:true})
"zipCodes" : 93 }
"zipCodes" : 56 }
"zipCodes" : 48 }
shard1 shard2
Group Group
Limit
Sort
Group
Cursor
COLLSCAN
{...} {...} {...} {...} {...} {...} {...} {...}
Cursor
COLLSCAN
$sort + $limit  
Coalescence
Projection 
Optimization

{ $match : { state : "CA" } },
{ $group:{_id:'$city', zipCodes: { $sum: 1 }}},
{ $limit : 3 }
])
shard1 shard2
{ "zip" : "LOS ANGELES", "zipCodes" : 56 } 
{ "zip" : "SAN DIEGO", "zipCodes" : 34 } 
{ "zip" : "SAN JOSE", "zipCodes" : 29 }
Sharded Cluster - Targeted Query Routing

{ $limit : 3 }
], {explain:true})
shard1
shard2
shard1 shard2

{ $limit : 3 }
], {explain:true})
shard1 shard2
Group
Limit
Sort
Cursor

{ 
"splitPipeline": null, 
"shards": { 
"shard1": { 
"stages": [ 
{ 
"$cursor": { 
"query": { 
"state": "CA" 
}, 
"fields": { 
"city": 1, 
"zip": 0 
}, 
"winningPlan": { 
"stage": "PROJECTION", 
"transformBy": { 
"city": 1, 
{ $limit : 3 }
], {explain:true})
shard1 shard2
Group
Limit
Sort
Cursor

eline": null, 
{ 
": { 
es": [ 
$cursor": { 
"query": { 
"state": "CA" 
}, 
"fields": { 
"city": 1, 
"zip": 0 
}, 
"winningPlan": { 
"transformBy": { 
"city": 1, 
"zip": 0 
}, 
"inputStage": { 
"indexBounds": { 
"state": [ 
"["CA", "CA"]" 
], 
"city": [ 
], 
"pop": [ 
{ $limit : 3 }
], {explain:true})
shard1 shard2
Group
Limit
Sort
Cursor

eline": null, 
{ 
": { 
es": [ 
$cursor": { 
"query": { 
"state": "CA" 
}, 
"fields": { 
"city": 1, 
"zip": 0 
}, 
"winningPlan": { 
"transformBy": { 
"city": 1, 
"zip": 0 
}, 
"inputStage": { 
"indexBounds": { 
"state": [ 
"["CA", "CA"]" 
], 
"city": [ 
], 
"pop": [ 
{ $limit : 3 }
], {explain:true})
shard1 shard2
Group
Limit
Sort
Cursor
IXSCAN
PROJECTION
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"

"["CA", "CA"]" 
], 
"city": [ 
], 
"pop": [ 
] 
} 
} 
} 
} 
} 
}, 
{ 
"$group": { 
"zip": "$city", 
"zipCodes": { 
"$sum": { 
"$const": 1 
} 
} 
} 
}, 
{ 
"$sort": { 
"sortKey": { 
"zipCodes": -1 
}, 
} 
} 
] 
{ $limit : 3 }
], {explain:true})
shard1 shard2
Group
Limit
Sort
Cursor
IXSCAN
PROJECTION
"ME" -->>
"$maxKey"
"$minKey" -->>
"ME"

Negation
.sort({pop:-1})
db.zips.find({state:{$ne:'NY'},
city:{$ne:'NEW YORK'},
pop:{$not:{'$gt':100000}}})

{
"nReturned" : 27757,
"stage" : "SORT",
"works" : 55519,
"advanced" : 27757,
"needTime" : 27760,
"needFetch" : 0,
"saveState" : 433,
Negation
.sort({pop:-1})
pop:{$not:{'$gt':100000}}})
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

{
"stage" : "SORT",
"works" : 55519,
"advanced" : 27757,
"needTime" : 27760,
"needFetch" : 0,
"saveState" : 433,
Negation
.sort({pop:-1})
pop:{$not:{'$gt':100000}}})
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

{
"stage" : "SORT",
"works" : 55519,
"advanced" : 27757,
"needTime" : 27760,
"needFetch" : 0,
"saveState" : 433,
Negation
.sort({pop:-1})
pop:{$not:{'$gt':100000}}})
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}
In-Memory Sort!

"alreadyHasObj" : 0,
"inputStage" : {
"stage" : "IXSCAN",
"works" : 27759,
"advanced" : 27757,
"needTime" : 2,
"needFetch" : 0,
"saveState" : 433,
"isEOF" : 1,
"invalidates" : 0,
"keyPattern" : {
"state" : 1,
"city" : 1,
"pop" : 1
},
"indexName" : "state_1_city_1_pop_1",
"indexBounds" : {
"state" : [
"[MinKey, "NY")",
"("NY", MaxKey]"
],
"city" : [
"[MinKey, "NEW YORK")",
"("NEW YORK", MaxKey]"
],
"pop" : [
"[MinKey, 100000.0]",
"(inf.0, MaxKey]"
]
.sort({pop:-1})
pop:{$not:{'$gt':100000}}})
In-Memory Sort!
Negation
plan root
SORT
FETCH
IXSCAN
{...} {...} {...} {...} {...} {...} {...} {...} {...}

MongoDB
Performance MeasurementTools

Log files, Profiler, Query Optimizer
mongod
log file
profiler (collection)
query engine

mongod Log Files
Sun Jun 29 06:35:37.646 [conn2]
query test.docs query:
{ parent.company: "22794",
parent.employeeId: "83881" }
ntoreturn:1 ntoskip:0 nscanned:
806381 keyUpdates:0 numYields: 5
locks(micros) r:2145254
nreturned:0 reslen:20 1156ms
date and time thread
operation
namespace
n… 
counters
lock 
times
duration
number of
yields

mtools
• http://github.com/rueckstiess/mtools
• log file analysis for poorly performing queries
– Show me queries that took more than 1000 ms from 6
am to 6 pm:
$ mlogfilter mongodb.log --from 06:00 --to
18:00 --slow 1000 > mongodb-filtered.log

mtools graphs
% mplotqueries --type histogram --group namespace --bucketSize 3600

Database Profiler
• Collect actual samples from a running
MongoDB instance
• Tunable for level and slowness
• Can be controlled dynamically

• Enable to see slow queries
– (or all queries)
– Default 100ms
Using Database profiler
// Enable database profiler on the console, 0=off 1=slow 2=all
> db.setProfilingLevel(1, 50)
{ "was" : 0, "slowms" : 50, "ok" : 1 }
// View profile with
> show profile
// See the raw data
>db.system.profile.find().pretty()

Profiler
• 1MB capped collection named system.profile per database, per replica set
• One document per operation
• Examples:
> db.setProfilingLevel(1) // log all operations greater than 100ms
> db.setProfilingLevel(1, 20) // log all operations greater than 20ms
> db.setProfilingLevel(2) // log all operations regardless of duration
> db.setProfilingLevel(0) // turn off profiling
> db.getProfilingStatus() // display current profiling level
{
"slowms": 100,
"was": 2
}
• In a sharded cluster, you will need to connect to each shard's primary
mongod, not mongos

Command Line tools
• iostat
• dstat
• mongostat
• mongotop
• mongoperf

Cloud Manager
• Memory usage
• Opcounters
• Lock percentage
• Queues
• Background flush average
• Replication oplog window and lag

MongoDB Days UK: Indexing and Performance Tuning

More Related Content

What's hot

Similar to MongoDB Days UK: Indexing and Performance Tuning

More from MongoDB

Recently uploaded

MongoDB Days UK: Indexing and Performance Tuning