Query performance can either be a constant headache or the unsung hero of an application. MongoDB provides extremely powerful querying capabilities when used properly. As a member of the support team I will share common mistakes observed as well as tips and tricks to avoiding them.

1. Tips & Tricks
for Avoiding Common Query Pitfalls
Amar Hamzeh, Technical Services, MongoDB

2. Amar Hamzeh
Team Lead – Technical Services, MongoDB

3. Roadmap
Motivation
Items to be aware of:
• Blocking Stages
• Using the $or operator
• Case insensitivity
Questions

4. The Power of Query Optimization
Query tuning results in:
• Improved performance
• Reduced resource utilization
This may lead to:
• Improved stability and predictability
• A smaller hardware footprint
• Not uncommon to observe efficiency
improvements greater than 99%

5. The Challenge @ Acme Games,
IncCharlie
• DBA at Acme Games, Inc
• MongoDB Champion
Stakeholders
• Developers
• Leadership

6. Acme Games Introduces
ShortFit
eBrand new battle
royale
Launching soon

7. Stakeholder Concerns
• Game nearly complete
• Developers learned a lot from Charlie

8. Stakeholder Concerns
• Game nearly complete
• Developers learned a lot from Charlie
Indexes support the efficient
execution of queries in MongoDB

9. Stakeholder Concerns
• Game nearly complete
• Developers learned a lot from Charlie
Indexes support the efficient
execution of queries in MongoDB

10. Stakeholder Concerns
• Game nearly complete
• Developers learned a lot from Charlie
Ace Bob Sue
… …
Indexes support the efficient
execution of queries in MongoDB

11. Stakeholder Concerns
• Game nearly complete
• Developers learned a lot from Charlie
• App is being stress-tested

12. Stakeholder Concerns
• Game nearly complete
• Developers learned a lot from Charlie
• App is being stress-tested
• Concerned over current performance

13. Stakeholder Concern #1
Developers Created an index
db.games.createIndex({ gamerTag: 1 })
This query takes several seconds to execute:
db.games.find({ gamerTag: "Ace" }).sort({ score: 1 })
Adding an index on score does not help!
db.games.createIndex({ score: 1 })

14. Blocking Operations

15. Blocking Operations
Formally:
“An operation which must process all input,
before it can begin to produce any output”
Opposite of the often more desirable “fully pipelined” operations which can stream
results back as soon as they are found
Commonly observed when a sort is added to a query

16. Sorting with Blocking (no index)

17. Sorting with Blocking (no index)

18. Sorting with Blocking (no index)

19. Sorting with Blocking (no index)

20. Sorting with Blocking (no index)

21. Sorting with Blocking (no index)

22. Sorting with Blocking (no index)

23. Sorting with Blocking (no index)

24. Sorting with Blocking (no index)

25. Sorting with Blocking (no index)

26. Sorting with Blocking (no index)

27. Sorting with Blocking (no index)

28. Sorting with Blocking (no index)

29. Sorting with Blocking (no index)

30. Sorting with Blocking (no index)

31. Sorting with Blocking (no index)

32. Sorting with Blocking (no index)

33. Sorting with Blocking (no index)

34. Sorting with Blocking (no index)

35. Sorting with Blocking (no index)

36. Sorting with Blocking (no index)

37. Sorting with Blocking (no index)

38. Sorting with Blocking (no index)

39. Sorting with Blocking (no index)

40. Sorting without Blocking

41. Sorting without Blocking

42. Sorting without Blocking

43. Sorting without Blocking

44. Blocking Stages
• $sort (in aggregation and find)
• $group
• $bucket
• $count
• $facet
Are there any other blocking
operations?

45. Working with Blocking Stages
For Sorting
• Add a supporting index
• Worth the overhead in almost all circumstances
For Other Stages
• Do you need the blocking stage?
• Can you offload it to a secondary member?
• Can it be a pre-computed field?

46. Stakeholder Concern #1
Performance of:
Db.games.find({ gamerTag: "Ace" }).sort({ score: 1 })

47. Stakeholder Concern #1
Performance of:
db.games.find({ gamerTag: "Ace" }).sort({ score: 1 })
db.games.createIndex({ gamerTag: 1, score: 1 })

48. Stakeholder Concern #1
Performance of:
db.games.find({ gamerTag: "Ace" }).sort({ score: 1 })
db.games.createIndex({ gamerTag: 1, score: 1 })
“That’ll work great!

49. Stakeholder Concern #2
$and
The $and version of a query returns
quickly:
db.games.find({
$and : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
$or
But the $or version is slow:
db.games.find({
$or : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})

50. Stakeholder Concern #2
$and
The $and version of a query returns
quickly:
db.games.find({
$and : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
$or
But the $or version is slow:
db.games.find({
$or : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
We just created an index with both
those fields… Can’t it be used?

51. $or

52. $and Example
Query on Games:
db.games.find({
$and : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
Matching Games:
{ gamerTag: "Ace", score: 9500 }
Non-matching Games:
{ gamerTag: "Ace", score: 500 },
{ gamerTag: ”Bob", score: 9500 },
{ gamerTag: ”Bob", score: 500 }

53. { gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }{ gamerTag: "Bob",
score: 500 }
score: {$gt: 9000}gamerTag: "Ace"
$and

54. { gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }{ gamerTag: "Bob",
score: 500 }
score: {$gt: 9000}gamerTag: "Ace"
$and

55. { gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }{ gamerTag: "Bob",
score: 500 }
score: {$gt: 9000}gamerTag: "Ace"
$and

56. score: {$gt: 9000}gamerTag: "Ace"
$and

57. score: {$gt: 9000}gamerTag: "Ace"
$and

58. score: {$gt: 9000}gamerTag: "Ace"
$and

59. score: {$gt: 9000}gamerTag: "Ace"
$and

60. score: {$gt: 9000}gamerTag: "Ace"
$and

61. $and
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$and : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})

62. $and
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$and : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
”(9000.0, inf.0]"
]
}

63. $and
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$and : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"(9000.0, inf.0]"
]
}

64. $and
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$and : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"(9000.0, inf.0]"
]
}

65. $and
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$and : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
"(9000.0, inf.0]"
]
}

66. $or Example
Query on Games:
db.games.find({
$or : [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
Matching Games:
{ gamerTag: "Ace", score: 9500 },
{ gamerTag: "Ace", score: 500 },
{ gamerTag: ”Bob", score: 9500 }
Non-matching Games:
{ gamerTag: ”Bob", score: 500 }

67. { gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }{ gamerTag: "Bob",
score: 500 }
score: {$gt: 9000}gamerTag: "Ace"
$or

68. { gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }{ gamerTag: "Bob",
score: 500 }
score: {$gt: 9000}gamerTag: "Ace"
$or

69. { gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }{ gamerTag: "Bob",
score: 500 }
score: {$gt: 9000}gamerTag: "Ace"
$or

70. { gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }{ gamerTag: "Bob",
score: 500 }
score: {$gt: 9000}gamerTag: "Ace"
$or

71. { gamerTag: "Ace",
score: 9500 }
{ gamerTag: "Ace",
score: 500 }
{ gamerTag: "Bob",
score: 9500 }{ gamerTag: "Bob",
score: 500 }
score: {$gt: 9000}gamerTag: "Ace"
$or

72. score: {$gt: 9000}gamerTag: "Ace"
$or

73. score: {$gt: 9000}gamerTag: "Ace"
$or

74. score: {$gt: 9000}gamerTag: "Ace"
$or

75. score: {$gt: 9000}gamerTag: "Ace"
$or

76. score: {$gt: 9000}gamerTag: "Ace"
$or

77. $or
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$or: [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})

78. $or
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$or: [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
Expected Index bounds:
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
”(9000.0, inf.0]"
]
}

79. $or
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$or: [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
Expected Index bounds:
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
”(9000.0, inf.0]"
]
}

80. $or
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$or: [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
Expected Index bounds:
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
”(9000.0, inf.0]"
]
}

81. $or
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$or: [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
Expected Index bounds:
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
”(9000.0, inf.0]"
]
}

82. $or
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$or: [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
Expected Index bounds:
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
”(9000.0, inf.0]"
]
}

83. $or
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$or: [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
Expected Index bounds:
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
”(9000.0, inf.0]"
]
}
Actual (hinted) Index bounds:
"indexBounds" : {
"gamerTag" : [
”[MinKey, MaxKey]"
],
"score" : [
”[MinKey, MaxKey]"
]
}

84. $or
Ace Bob
500
950
0 500
950
0
{gamerTag:1
, score:1}
db.games.find({
$or: [
{ gamerTag: "Ace" },
{ score: {$gt: 9000} }
]
})
Expected Index bounds:
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
],
"score" : [
”(9000.0, inf.0]"
]
}
Actual (hinted) Index bounds:
"indexBounds" : {
"gamerTag" : [
”[MinKey, MaxKey]"
],
"score" : [
”[MinKey, MaxKey]"
]
}
So is there anything we can do to
improve the performance of this
query?

85. Recommendations
Use Multiple Indexes!
db.games.createIndex({ gamerTag: 1})
db.games.createIndex({ score: 1 })

86. $or (Multiple Indexes)

87. $or (Multiple Indexes)
Ace

88. $or (Multiple Indexes)
Ace Bob

89. Ace Bob
{gamerTag:1}
$or (Multiple Indexes)

90. Ace Bob
{gamerTag:1}
500
$or (Multiple Indexes)

91. Ace Bob
{gamerTag:1}
500
950
0
$or (Multiple Indexes)

92. Ace Bob
{gamerTag:1}
500
950
0
{score:1}
$or (Multiple Indexes)

93. Ace Bob
{gamerTag:1}
500
950
0
{score:1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
}
$or (Multiple Indexes)

94. Ace Bob
{gamerTag:1}
500
950
0
{score:1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
}
$or (Multiple Indexes)

95. Ace Bob
{gamerTag:1}
500
950
0
{score:1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
}
"indexBounds" : {
”score" : [
”(9000.0, inf.0]"
}
$or (Multiple Indexes)

96. Ace Bob
{gamerTag:1}
500
950
0
{score:1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
}
"indexBounds" : {
”score" : [
”(9000.0, inf.0]"
}
$or (Multiple Indexes)

97. Ace Bob
{gamerTag:1}
500
950
0
{score:1}
"indexBounds" : {
"gamerTag" : [
"["Ace", "Ace"]"
}
"indexBounds" : {
”score" : [
”(9000.0, inf.0]"
}
$or (Multiple Indexes)

98. Recommendations
Use Multiple Indexes!
db.games.createIndex({ gamerTag: 1 })
db.games.createIndex({ score: 1 })

99. Recommendations
Use Multiple Indexes!
db.games.createIndex({ gamerTag: 1 })
db.games.createIndex({ score: 1 })
We already have the {gamerTag:1, score:1}
index, do we need both of these indexes?

100. Recommendations
Use Multiple Indexes!
db.games.createIndex({ gamerTag: 1 })
db.games.createIndex({ gamerTag: 1, score: 1 })
db.games.createIndex({ score: 1 })
We already have the {gamerTag:1, score:1}
index, do we need both of these indexes?

101. Recommendations
Use Multiple Indexes!
db.games.createIndex({ gamerTag: 1 })
db.games.createIndex({ gamerTag: 1, score: 1 })
db.games.createIndex({ score: 1 })
We already have the {gamerTag:1, score:1}
index, do we need both of these indexes?
Works with sorting
Generates a SORT_MERGE plan

102. Stakeholder Concern #2
db.games.find({
$or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
Having the right index is critical

103. Stakeholder Concern #2
db.games.find({
$or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
Having the right index is critical

104. Stakeholder Concern #2
db.games.find({
$or : [ { gamerTag: "Ace" }, { score: {$gt: 9000} } ]})
Having the right index is critical
“Great!”

105. Stakeholder Concern #3
“Wait! We can’t even FIND the gamers!”
A “basic” search on gamerTag takes several seconds already:
db.games.find({ gamerTag: /Ace/i)
“This query is even SLOWER with the index than without it!”

106. Case Insensitive

107. Case Sensitive
db.games.find({
gamerTag: /^Ace$/
})
Equivalent to:
db.games.find({
gamerTag: "Ace"
})
Matching Games:
{ gamerTag: "Ace", score: 9500 }
Non-matching Games:
{ gamerTag: ”ACE", score: 500 },
{ gamerTag: ”aCe", score: 500 },
{ gamerTag: ”ace", score: 500 },
{ gamerTag: ”Bob", score: 9500 },
{ gamerTag: ”Bob", score: 500 },
{ gamerTag: ”acxyz", score: 500 },
{ gamerTag: ”Ace mdb", score: 500 }

108. Case Sensitive
ace aCe
acxy
z
Ace
Ace
mdb
ACE Bob
Matching Games:
{ gamerTag: "Ace", score: 9500 }
Non-matching Games:
{ gamerTag: ”ACE", score: 500 },
{ gamerTag: ”aCe", score: 500 },
{ gamerTag: ”ace", score: 500 },
{ gamerTag: ”Bob", score: 9500 },
{ gamerTag: ”Bob", score: 500 },
{ gamerTag: ”acxyz", score: 500 },
{ gamerTag: ”Ace mdb", score: 500 }

109. Case Insensitive
Matching Games:
{ gamerTag: "Ace", score: 9500 },
{ gamerTag: "ACE", score: 500 },
{ gamerTag: "aCe", score: 500 },
{ gamerTag: "ace", score: 500 }
Non-matching Games:
{ gamerTag: "Bob", score: 9500 },
{ gamerTag: "Bob", score: 500 },
{ gamerTag: "acxyz", score: 500 },
{ gamerTag: "Ace mdb", score: 500 }
db.games.find({
gamerTag: /^Ace$/i})
Equivalent to:
db.games.find({
gamerTag: {
$regex: "^Ace$", $options: "i" }})
Equivalent to:
db.games.find({ gamerTag: "Ace"})
.collation({locale:'en’, strength: 2})

110. Case Insensitive
Matching Games:
{ gamerTag: "Ace", score: 9500 },
{ gamerTag: "ACE", score: 500 },
{ gamerTag: "aCe", score: 500 },
{ gamerTag: "ace", score: 500 }
Non-matching Games:
{ gamerTag: "Bob", score: 9500 },
{ gamerTag: "Bob", score: 500 },
{ gamerTag: "acxyz", score: 500 },
{ gamerTag: "Ace mdb", score: 500 }
db.games.find({
gamerTag: /^Ace$/i})
Equivalent to:
db.games.find({
gamerTag: {
$regex: "^Ace$", $options: "i" }})
Equivalent to:
db.games.find({ gamerTag: "Ace"})
.collation({locale:'en’, strength: 2})
Would a $text search be the same as well?

111. Case Insensitive
ace aCe
acxy
z
Ace
Ace
mdb
ACE Bob
"indexBounds": {
"gamerTag": [
"["", {})",
"[/^Ace$/i, /^Ace$/i]”
]
}

112. Recommendations
Case insensitive index
Collations available since 3.4

113. Recommendations
Case insensitive index
Collations available since 3.4
db.games.createIndex(
{ gamerTag: 1},
{ collation: { locale: 'en', strength: 2} } )

114. Recommendations
Case insensitive index
Collations available since 3.4
db.games.createIndex(
{ gamerTag: 1},
{ collation: { locale: 'en', strength: 2} } )
> db.games.find({ gamerTag: "Ace"}).collation({ collation: {locale: 'en’, strength: 2} })
{ "_id" : ObjectId("5b29dbee6c7d4f531bf73b5d"), "gamerTag" : "Ace", "score" : 9500 }
{ "_id" : ObjectId("5b29dbee6c7d4f531bf73b5e"), "gamerTag" : "ACE", "score" : 500 }
{ "_id" : ObjectId("5b29dbee6c7d4f531bf73b5f"), "gamerTag" : "aCe", "score" : 9500 }
{ "_id" : ObjectId("5b29dbee6c7d4f531bf73b60"), "gamerTag" : "ace", "score" : 0 }

115. Stakeholder Concern #3
Case insensitive index
db.games.createIndex(
{ gamerTag: 1},
{ collation: { locale: 'en', strength: 2} } )
Store a transformed (e.g. toLower()) copy of the string

116. Stakeholder Concern #3
db.games.find({ gamerTag: "Ace"})
.collation({ collation: {locale: 'en’, strength: 2} })

117. Stakeholder Concern #3
db.games.find({ gamerTag: "Ace"})
.collation(collation: {locale: 'en', strength: 2} })
“Wow, MongoDB can do anything!!!”

118. Summary

119. Work Smarter Not Harder
• Understand the business logic
• Index appropriately
• Is it the right index to support the query?
• Be aware of:
• Blocking stages
• Usage of $or
• Case sensitivity
• Leverage the Performance Advisor

120. Questions?

121. Thank You!