Avoid Query Pitfalls

Avoiding Query
Pitfalls
@nleite

2
Agenda Common Query
Pitfalls
How to think
about and how to
solve them
Tips and Tricks
for Query
Optimization

3
Howdy!
Disclaimer
Who's this guy?
Norberto Leite
Lead Engineer
@nleite
MongoDB

https://university.mongodb.com/

6
https://usatthebiglead.files.wordpress.com/2012/10/dez-drop-10-1-12.gif?w=1000
What's the
story about
query
pitfalls?

7
The Power of Query Optimization
Query tuning results in:
● Improved performance
● Reduced resource utilization
This may lead to:
● Improved stability and predictability
● Smaller hardware footprint
Not uncommon to observe efficiency improvements $gte 99%

Acme Games Introduces:
ShortFite
Tales of a release pre-launch testing

Getting
Ready for
Launch!
MongoDB Certified
Developers and DBA Stakeholders

Become a MongoDB
Certified Developer /
DBA

11
Stakeholder
Concerns
● Game nearly
complete
● Team starts to do
capacity testing

12
Stakeholder
Concerns
● Game nearly
complete
capacity testing
Indexes support the efficient
execution of queries in MongoDB

13
Stakeholder
Concerns
● Game nearly
complete
capacity testing
Ace Sue
… …
Bob
Indexes support the efficient
execution of queries in MongoDB

Stakeholder #1 Concern
App being stress tested
Concerns over current performance

Developers created index
db.games.createIndex({ gamerTag: 1 })
This query takes several seconds to execute:
db.games.find( { gamerTag: "Ace" } ).sort({score:-1})
Adding the index on score does not help!
db.games.createIndex({ score: -1 })

Developers created index
db.games.createIndex({ gamerTag: 1 })
This query takes several seconds to execute:
Adding the index on score does not help!
db.games.createIndex({ score: -1 })
“Clearly MongoDB
is not webscale!”

18
Blocking Operation
Formally:
“An operation which must process all input before it can begin to produce any output.”
Opposite of the often desirable “fully pipelined” plan which can stream results back as soon as they
are found.
Commonly observed when a sort is added to a query

Indexes that support sorting
db.games.createIndex({ gamerTag: 1, score:-1})

32
Blocking
Stages
$sort
In aggregation and find
$group
$bucket
$count
$facet
Are there any other blocking
operations?

34
Working with blocking stages
For sorting:
Add a supporting index
Worth the overhead in almost all circumstances
For other stages:
Do you need the blocking stage?
Offload to secondary member

Performance of
“Clearly MongoDB is not webscale!”

Performance of
db.games.createIndex({ gamerTag: 1, score:-1 })

Performance of
"That’ll work great!”

The $and version of a query
returns quickly:
db.games.find({
$and : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
But the $or version is slow:
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Concern #2: Multiple queries same index
The $and version of a query
returns quickly:
db.games.find({
$and : [
{ score: {$gt: 9000} }
]
})
But the $or version is slow:
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})
We just created an index with both
those fields… Can it be used?

$and example
Query on games:
db.games.find({
$and : [
{ score: {$gt: 9000} }
]
})
Matching games:
{ gamerTag: "Ace", score: 9500 }
Non-matching games:
{ gamerTag: "Ace", score: 500 },
{ gamerTag: "Bob", score: 9500 },
{ gamerTag: "Bob", score: 500 }

Groups of documents
score: {$gt: 9000}gamerTag: "Ace"
{ gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }
{ gamerTag: "Bob",
score: 500 }

gamerTag: "Ace"
$and Venn Diagram (logical)
score: {$gt: 9000}
{ gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }
{ gamerTag: "Bob",
score: 500 }
db.games.find(
{ $and : [
{ score: {$gt: 9000} }
]})

{ gamerTag: "Bob",
score: 500 }
gamerTag: "Ace"
{ gamerTag: "Ace",
score: 500 }
db.games.find(
{ $and : [
{ score: {$gt: 9000} }
]})

gamerTag: "Ace"
score: {$gt: 9000}
db.games.find(
{ $and : [
{ score: {$gt: 9000} }
]})

db.games.find(
{ $and : [
{ score: {$gt: 9000} }
]})

Ace Bob
{gamerTag:1
, score:-1}
500 9500
500 9500
$and Index Visualization
db.games.find(
{ $and : [
{ score: {$gt: 9000} }
]})

Ace Bob
{gamerTag:1
, score:-1}
500 9500
500 9500
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000.0)"
]
}
db.games.find(
{ $and : [
{ score: {$gt: 9000} }
]})

Ace Bob
500 9500
500 9500
{gamerTag:1
, score:-1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000.0)"
]
}
db.games.find(
{ $and : [
{ score: {$gt: 9000} }
]})

Bob
500 9500
500 9500
Ace
{gamerTag:1
, score:-1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000.0)"
]
}
db.games.find(
{ $and : [
{ score: {$gt: 9000} }
]})

Bob
500
500 9500
Ace
9500
{gamerTag:1
, score:-1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000.0)"
]
}
db.games.find(
{ $and : [
{ score: {$gt: 9000} }
]})

$or example
Query on games:
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})
Matching games:
Non-matching games:

gamerTag: "Ace"
$or Venn Diagram (logical)
score: {$gt: 9000}
{ gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }
{ gamerTag: "Bob",
score: 500 }
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

score: {$gt: 9000}
{ gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }
{ gamerTag: "Bob",
score: 500 }
gamerTag: "Ace"
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

score: {$gt: 9000}
{ gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Bob",
score: 500 }
{ gamerTag: "Ace",
score: 500 }
gamerTag: "Ace"
{ gamerTag: "Bob",
score: 9500 }
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

{ gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }
{ gamerTag: "Bob",
score: 500 }
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

score: {$gt: 9000}
gamerTag: "Ace"
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

$or (single) Index visualization
Ace Bob
{gamerTag:1
, score:-1}
500 9500
500 9500
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob
{gamerTag:1
, score:-1}
500 9500
500 9500
Expected Index Bounds:
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000]"
]}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob
500 9500
500 9500
{gamerTag:1
, score:-1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000]"
]}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Bob
500 9500
500 9500
{gamerTag:1
, score:-1}
Ace
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000]"
]}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Bob
500 9500
{gamerTag:1
, score:-1}
Ace
500 9500
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000]"
]}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

500
Bob
9500
{gamerTag:1
, score:-1}
Ace
500 9500
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000]"
]}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

500
Bob
9500
Actual (Hinted) Index Bounds:
"indexBounds" : {
"gamerTag" : [
"[MinKey, MaxKey]"
],
"score" : [
"[MaxKey, MinKey]"
]}
{gamerTag:1
, score:-1}
Ace
500 9500
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000]"
]}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob
500 9500
500 9500
{gamerTag:1
, score:-1}
So is there anything we can do to
improve the performance of this query?
Actual (Hinted) Index Bounds:
"indexBounds" : {
"gamerTag" : [
"[MinKey, MaxKey]"
],
"score" : [
"[MaxKey, MinKey]"
]}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"[inf.0, 9000]"
]}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Recommendations
Use multiple indexes!
db.data.createIndex({gamerTag: 1})
db.data.createIndex({score: 1})

$or (multiple) Index visualization
Ace Bob
{gamerTag:1
, score:-1}
500 9500
500 9500
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob
{gamerTag:1}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob
{gamerTag:1}
500
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob
{gamerTag:1}
500 9500
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob
{gamerTag:1}
500 9500
{score:1}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob
{gamerTag:1}
500 9500
{score:1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]”
]
}
"indexBounds" : {
"score" : [
"(9000.0, inf.0]”
]
}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Ace Bob 500 9500
{score:1}{gamerTag:1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]”
]
}
"indexBounds" : {
"score" : [
"(9000.0, inf.0]”
]
}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Bob 500 9500
{score:1}
Ace
{gamerTag:1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]”
]
}
"indexBounds" : {
"score" : [
"(9000.0, inf.0]”
]
}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Bob 500 9500
{score:1}{gamerTag:1}
Ace
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]”
]
}
"indexBounds" : {
"score" : [
"(9000.0, inf.0]”
]
}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Bob 500 9500
{gamerTag:1}
Ace
{score:1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]”
]
}
"indexBounds" : {
"score" : [
"(9000.0, inf.0]”
]
}
db.games.find({
$or : [
{ score: {$gt: 9000} }
]
})

Recommendations
We already have the {gamerTag:1, score:-1}
index, do we need both of these new ones?

Recommendations

db.games.find({
$or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]
})
Having the right index is critical
"Super!!”

93
“Wait wait wait, we can’t even FIND the gamers!”
A basic search on gamerTag takes several seconds already:
db.games.find({gamerTag: /^Ace$/i})
“This query is SLOWER with the index than it is without it!”

Matching games:
Non-matching games:
{ gamerTag: "ACE", score: 500 },
{ gamerTag: "aCe", score: 9500 },
{ gamerTag: "ace", score: 0 },
{ gamerTag: "acxyz", score: 9500 },
{ gamerTag: "Ace mdb", score: 9500 }
db.games.find({
gamerTag: /^Ace$/
})
//equivalent to
db.games.find({
gamerTag: “Ace”
})
Case Sensitive

Case Sensitive
ace aCe acxyz Ace
Ace
mdb
ACE Bob
"indexBounds" : {
"gamerTag" : [
"["Ace", "Acf")",
"[/^Ace$/, /^Ace$/]"
]}
Matching games:
Non-matching games:
{ gamerTag: "ace", score: 0 },

Matching games:
{ gamerTag: "ace", score: 0 }
Non-matching games:
Case Insensitive
db.games.find({
gamerTag: /^Ace$/i
})
//equivalent to
db.games.find({
gamerTag: {
$regex: “^Ace$”,
$options: “i”
}
})
//equivalent to
db.games.find({ gamerTag: “Ace”})
.collation({locale:’en’,strength:2})

Case Insensitive
db.games.find({
gamerTag: /^Ace$/i
})
//equivalent to
db.games.find({
gamerTag: {
$regex: “^Ace$”,
$options: “i”
}
})
//equivalent to
db.games.find({ gamerTag: “Ace”})
.collation({locale:’en’,
strength:2})
Would a $text search be the same as well?
Matching games:
Non-matching games:

Case Insensitive
ace aCe acxyz Ace
Ace
mdb
ACE Bob
"indexBounds" : {
"gamerTag" : [
“["", {})",
"[/^Ace$/i, /^Ace$/i]"
]
}
Matching games:
Non-matching games:

100
Recommendations
Case insensitive index!
Collations available since 3.4

101
Recommendations
Case insensitive index!
Collations available since 3.4
db.games.createIndex( { gamerTag: 1},
{ collation: { locale: 'en', strength: 2 } } )
Store a transformed (eg toLower()) copy of the string

102
https://docs.mongodb.com/manual/reference/collation/index.html
Collations

103
db.games.find({gamerTag: “Ace”})
.collation({locale:'en', strength:2})

104

105
“Wow, MongoDB can do anything!!!!1!”

107
Work Smarter Not Harder
Understand the business logic
Index appropriately
Is it the right index to support the query?
Be aware of:
Blocking Stages
Usage of $or
Case sensitivity
Leverage the Performance Advisor

108
Work Smarter Not Harder
Understand the business logic
Index appropriately
Is it the right index to support the query?
Be aware of:
Blocking Stages
Usage of $or
Case sensitivity
Leverage the Performance Advisor

109
Countdown to ShortFite
Powered by an optimized MongoDB
environment, ShortFite is sure to be a hit!

110
The Power of Query Optimization
Query tuning results in:
● Improved performance
● Reduced resource utilization
This may lead to:
● Improved stability and predictability
● A smaller hardware footprint
Not uncommon to observe efficiency improvements $gte 99%

111
Become the HERO!
- Plenty of resources out there
- http://university.mongodb.com
- http://docs.mongodb.com

Norberto Leite
Lead Engineer
norberto@mongodb.com
@nleite

Avoid Query Pitfalls

Recommended

Recommended

More Related Content

What's hot

What's hot (19)

Similar to Avoid Query Pitfalls

Similar to Avoid Query Pitfalls (20)

More from Norberto Leite

More from Norberto Leite (20)

Recently uploaded

Recently uploaded (20)

Avoid Query Pitfalls