Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

MongoDB World 2019: Tips and Tricks++ for Querying and Indexing MongoDB

Query performance can either be a constant headache or the unsung hero of an application. MongoDB provides extremely powerful querying capabilities when used properly. As a senior member of the support team I will share more common mistakes observed and some tips and tricks to avoiding them.

  • Login to see the comments

MongoDB World 2019: Tips and Tricks++ for Querying and Indexing MongoDB

  1. 1. Chris Harris Tips and Tricks++ for Querying MongoDB
  2. 2. Chris Harris Lead Technical Expert for Queries and Indexing
  3. 3. Introductions Diego
  4. 4. Introductions I identified slow queries. What index do I make?
  5. 5. Roadmap § E-S-R § Overview § Predicate Type Check § Sequential Prefix § Exceptions? § Multikey Index § Definition § Semantics § Performance
  6. 6. Roadmap § E-S-R § Overview § Predicate Type Check § Sequential Prefix § Exceptions? § Multikey Index § Definition § Semantics § Performance
  7. 7. Equality Sort Range "Rule"
  8. 8. E – S - R The ordering of index keys in a compound index is critically important. E-S-R provides guidance that is useful in most cases: • Equality first • Sort next • Range last
  9. 9. E – S - R The ordering of index keys in a compound index is critically important. E-S-R provides guidance that is useful in most cases: • Equality first • Sort next • Range last What is the difference between Equality and Range?
  10. 10. Definitions Equality Fields An exact match on a single value. For example: • {x:123} • {x:{$eq:123}} • {x:"123"} • {"x.y":123}
  11. 11. Definitions Equality Fields An exact match on a single value. For example: • {x:123} • {x:{$eq:123}} • {x:"123"} • {"x.y":123} Sort The (entire) requested sort. .sort({x:1, y:-1})
  12. 12. Definitions Equality Fields An exact match on a single value. For example: • {x:123} • {x:{$eq:123}} • {x:"123"} • {"x.y":123} Sort The (entire) requested sort. .sort({x:1, y:-1}) Range Predicates Any predicates that are not exact matches. Some operators include: • {x:{$gt:0}} • {x:{$lte:1000}}
  13. 13. Equality Equality keys are placed first in any order If present in the query shape, equality fields should always form the prefix for the index. db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
  14. 14. Equality Equality keys are placed first in any order If present in the query shape, equality fields should always form the prefix for the index. db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
  15. 15. Equality db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {gamertag: "Ace", score: 100} {gamertag: "Ace", score: 99,999} {gamertag: "Bob", score: 15,000} {gamertag: "Bob", score: 50,000}
  16. 16. Equality db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {}{} 100 15,000 50,000 99,999 Ace Bob Bob Ace {gamertag: "Ace", score: 100} {gamertag: "Ace", score: 99,999} {gamertag: "Bob", score: 15,000} {gamertag: "Bob", score: 50,000}
  17. 17. Equality 100 15,000 50,000 99,999 Ace Bob Bob Ace db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  18. 18. Equality 100 15,000 50,000 99,999 Ace Bob Bob Ace db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  19. 19. Equality 100 15,000 50,000 99,999 Ace Bob Bob Ace db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  20. 20. Equality 100 15,000 50,000 99,999 Ace Bob Bob Ace db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  21. 21. Equality 100 15,000 50,000 99,999 Ace Bob Bob Ace db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  22. 22. Equality 100 15,000 50,000 99,999 Ace Bob Bob Ace db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  23. 23. Equality 100 15,000 50,000 99,999 Ace Bob Bob Ace db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  24. 24. Equality 100 15,000 50,000 99,999 Ace Bob Bob Ace db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  25. 25. Equality Ace Bob Bob Ace db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) 100 99,999 15,000 50,000 {} {} {} {}
  26. 26. Equality 100 Ace Bob 99,999 15,000 50,000 db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  27. 27. Equality 100 Ace Bob 99,999 15,000 50,000 db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  28. 28. Equality 100 Ace Bob 99,999 15,000 50,000 db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) {} {} {} {}
  29. 29. Equality before Range db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) R E R E
  30. 30. Sort Sort fields are placed next Placing sort predicates after sequential equality keys allow for the index to: • Provide a non-blocking sort. • Minimize the amount of scanning required. db.games.find( {gamertag: "Ace"} ).sort( {score: 1} )
  31. 31. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {}
  32. 32. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {}
  33. 33. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) {} 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {} {}
  34. 34. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {} 100
  35. 35. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {}
  36. 36. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {}
  37. 37. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {}
  38. 38. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {}
  39. 39. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {}
  40. 40. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {}
  41. 41. Sort db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) 100 15,000 50,000 99,999 Ace Bob Bob Ace {} {}{} {}
  42. 42. Sort Ace Bob Bob Ace 100 99,999 15,000 50,000 {} {} {} {} db.games.find( {gamertag: "Ace"} ).sort( {score: 1} )
  43. 43. Sort 100 Ace Bob {} 99,999 15,000 50,000 {} {} db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) {}
  44. 44. Sort 100 Ace Bob {} 99,999 15,000 50,000 {} {} db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) {}
  45. 45. Sort 100 Ace Bob {} 99,999 15,000 50,000 {} {} db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) {}
  46. 46. Sort 100 Ace Bob {} 99,999 15,000 50,000 {} {} db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) {}
  47. 47. Sort after Equality db.games.find( {gamertag: "Ace"} ).sort( {score: 1} ) S E S E
  48. 48. Range Range fields are usually last Generally range predicates should be placed last. This allows them to still participate in filtering the data, but does not force a blocking sort. db.games.find( {gamertag: "Ace", score: {$gt: 9000}} )
  49. 49. Range after Equality db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) R E R E
  50. 50. Range db.games.find( {gamertag: "Ace", score: {$gt: 9000}} ) Should range come after sort too? R E R E
  51. 51. Range db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1}) Should range come after sort too? R E R E
  52. 52. Range db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1}) 100 15,000 50,000 99,999 Ace Bob Cali Ace {} {}{} {}
  53. 53. Range db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1}) 100 15,000 50,000 99,999 Ace Bob Cali Ace {} {}{} {}{1}{3}{2}
  54. 54. {1}{3}{2} Range db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1}) 100 15,000 50,000 99,999 Ace Bob Cali Ace {}
  55. 55. {1}{3}{2} Range 100 15,000 50,000 99,999 Ace Bob Cali Ace {} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  56. 56. {1}{3}{2} Range 100 15,000 50,000 99,999 Ace Bob Cali Ace {} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  57. 57. {1}{3}{2} Range 100 15,000 50,000 99,999 Ace Bob Cali Ace {} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  58. 58. {1}{3}{2} Range 100 15,000 50,000 99,999 Ace Bob Cali Ace {} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  59. 59. {1}{3}{2} Range 100 15,000 50,000 99,999 Ace Bob Cali Ace {} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  60. 60. {1}{3}{2} Range 100 15,000 50,000 99,999 Ace Bob Cali Ace {} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  61. 61. {1}{3}{2} Range 100 15,000 50,000 99,999 Ace Bob Cali Ace {} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  62. 62. {1}{3}{2} Range {} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  63. 63. {1}{3}{2} Range db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1}) Blocking SortBlocking Sort BS
  64. 64. {1}{3}{2} Range db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1}) BS
  65. 65. Range 100 15,000 50,000 99,999 Ace Bob Cali Ace db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  66. 66. Range Ace Bob Cali Ace 100 15,000 50,00099,999 {1} {3}{2}{} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  67. 67. Range 100 15,000 50,00099,999 Ace Bob Cali {1} {3}{2}{} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  68. 68. Range 100 15,000 50,00099,999 Ace Bob Cali {1} {3}{2}{} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  69. 69. Range 100 15,000 50,00099,999 Ace Bob Cali {1} {3}{2}{} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  70. 70. Range 100 15,000 50,00099,999 Ace Bob Cali {1} {3}{2}{} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  71. 71. Range 100 15,000 50,00099,999 Ace Bob Cali {1} {3}{2}{} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  72. 72. Range 100 15,000 50,00099,999 Ace Bob Cali {1} {3}{2}{} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  73. 73. Range 100 15,000 50,00099,999 Ace Bob Cali {1} {3}{2}{} db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1})
  74. 74. Range after Sort db.games.find( {score: {$gt: 9000}} ).sort({gamertag: 1}) BS R S R S
  75. 75. Roadmap § E-S-R § Overview § Predicate Type Check § Sequential Prefix § Exceptions? § Multikey Index § Definition § Semantics § Performance
  76. 76. Predicate Type Check What are the types of the following operators?
  77. 77. Predicate Type Check - Inequality Inequality operators, such as the following, are E, S, or R? § $ne:123 § $nin:[1,2] Range!
  78. 78. Predicate Type Check - Inequality Inequality operators, such as the following, are E, S, or R? § $ne:123 § $nin:[1,2] Range! How do you know?
  79. 79. Predicate Type Check - Inequality Inequality operators, such as the following, are E, S, or R? § $ne:123 § $nin:[1,2] Range! "indexBounds" : { "x" : [ "[MinKey, 123.0)", "(123.0, MaxKey]" ] }
  80. 80. Predicate Type Check - Inequality Range! "indexBounds" : { "x" : [ "[MinKey, 123.0)", "(123.0, MaxKey]" ] } 50 123 999
  81. 81. Predicate Type Check - Inequality Range! "indexBounds" : { "x" : [ "[MinKey, 123.0)", "(123.0, MaxKey]" ] } 50 123 999
  82. 82. Predicate Type Check - Regex Regex operators, such as the following, are E, S, or R? § {str:/car/} § {str:/^car/i} Range!
  83. 83. Predicate Type Check - Regex Regex operators, such as the following, are E, S, or R? § {str:/car/} § {str:/^car/i} Range!
  84. 84. Predicate Type Check - Regex Regex operators, such as the following, are E, S, or R? § {str:/car/} § {str:/^car/i} Range! planecar racecar
  85. 85. Predicate Type Check - Regex Regex operators, such as the following, are E, S, or R? § {str:/car/} § {str:/^car/i} Range! planecar racecar
  86. 86. Predicate Type Check - Regex Regex operators, such as the following, are E, S, or R? § {str:/car/} § {str:/^car/i} Range! planecar racecar
  87. 87. Predicate Type Check - Regex Regex operators, such as the following, are E, S, or R? § {str:/car/} § {str:/^car/i} Range!
  88. 88. Predicate Type Check - Regex Regex operators, such as the following, are E, S, or R? § {str:/car/} § {str:/^car/i} Range! raincarry Carpool
  89. 89. Predicate Type Check - Regex Regex operators, such as the following, are E, S, or R? § {str:/car/} § {str:/^car/i} Range! raincarry Carpool
  90. 90. Predicate Type Check - Regex Regex operators, such as the following, are E, S, or R? § {str:/car/} § {str:/^car/i} Range! raincarry Carpool
  91. 91. Predicate Type Check - $in $in filters, as demonstrated below, are E, S, or R? § {field:{$in:[1,3]}} … it depends! Can be Equality or Range in terms of the key ordering
  92. 92. Predicate Type Check - $in {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]", "[3.0, 3.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } 1 2 3 C G B F A D
  93. 93. Predicate Type Check - $in {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]", "[3.0, 3.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } 1 2 3 C G B F A D
  94. 94. Predicate Type Check - $in {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]", "[3.0, 3.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } 1 2 3 C G B F A D
  95. 95. Predicate Type Check - $in {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]", "[3.0, 3.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } 1 2 3 C G B F A D
  96. 96. Predicate Type Check - $in {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]", "[3.0, 3.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } 1 2 3 C G B F A D
  97. 97. Predicate Type Check - $in {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]", "[3.0, 3.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } 1 2 3 C G B F A D BS
  98. 98. Predicate Type Check - $in {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]", "[3.0, 3.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] }
  99. 99. Predicate Type Check - $in {field:{$in:[1,3]}}
  100. 100. Predicate Type Check - $in {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } 1 2 3 C G B F A D {} {} {} {} {} {}
  101. 101. Predicate Type Check - $in 1 2 3 C G B F A D 1 2 3 C G B F A D {} {} {} {} {} {} {} {} {} {} {} {} {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } "indexBounds" : { "field" : [ "[3.0, 3.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] }
  102. 102. Predicate Type Check - $in {field:{$in:[1,3]}} "indexBounds" : { "field" : [ "[1.0, 1.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } "indexBounds" : { "field" : [ "[3.0, 3.0]" ], "sortField" : [ "[MinKey, MaxKey]" ] } 1 2 3 C G B F A D 1 2 3 C G B F A D Sort Merge {} {} {} {} {} {} {} {} {} {} {} {}
  103. 103. Predicate Type Check - $in BS Depends on the length of the $in list S M
  104. 104. Roadmap § E-S-R § Overview § Predicate Type Check § Sequential Prefix § Exceptions? § Multikey Index § Definition § Semantics § Performance
  105. 105. Sequential Equality Keys Does it matter if there is missing predicate in the equality key list? For example, given: § Query: {gamertag:"Ace", game: "Halo"} § Index: {gamertag:1, date:1, game:1} Is that index as efficient as the following one? {gamertag:1, game:1, date:1}
  106. 106. Sequential Equality Keys Answer: It depends! … but probably not "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  107. 107. Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  108. 108. Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  109. 109. Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  110. 110. Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  111. 111. Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  112. 112. Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  113. 113. Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  114. 114. Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  115. 115. Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  116. 116. "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "game" : [ "["Halo", "Halo"]" ] } Sequential Equality Keys Ace Mario Halo 2016 2017 Mario Halo 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] }
  117. 117. Halo Sequential Equality Keys Ace Mario Halo 2016 2017 Mario 2018 2019 "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "date" : [ "[MinKey, MaxKey]" ], "game" : [ "["Halo", "Halo"]" ] } "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "game" : [ "["Halo", "Halo"]" ] }
  118. 118. "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "game" : [ "["Halo", "Halo"]" ] } "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "game" : [ "["Halo", "Halo"]" ] } "indexBounds" : { "gamertag" : [ "["Ace", "Ace"]" ], "game" : [ "["Halo", "Halo"]" ], "date" : [ "[MinKey, MaxKey]" ] } Sequential Equality Keys Ace 2016 2017Mario2018 2019Halo
  119. 119. Sequential Equality Keys R E E E E E E
  120. 120. Roadmap § E-S-R § Overview § Predicate Type Check § Sequential Prefix § Exceptions? § Multikey Index § Definition § Semantics § Performance
  121. 121. Exceptions Is this "rule" always optimal? Not always. db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1})
  122. 122. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  123. 123. Exceptions Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019 db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1})
  124. 124. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  125. 125. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  126. 126. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  127. 127. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  128. 128. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  129. 129. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  130. 130. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  131. 131. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  132. 132. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  133. 133. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  134. 134. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019 "executionStats" : { "nReturned" : 2, "executionTimeMillis" : 23, "totalKeysExamined" : 9001, "totalDocsExamined" : 2,
  135. 135. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 1 5,000 9,0012 9,000 … … 2001 2000 2015 2019 2019
  136. 136. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace 2001 2000 2015 2019 2019 {score:2} {score:1} {score:5000} {score:9001} {score:9000} … …
  137. 137. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace … … 2000 2001 2015 2019 2019 {score:2} {score:1} {score:5000} {score:9001} {score:9000}
  138. 138. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace … … 2000 2001 2015 2019 2019 {score:2} {score:1} {score:5000} {score:9001} {score:9000}
  139. 139. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace … … 2000 2001 2015 2019 2019 {score:2} {score:1} {score:5000} {score:9001} {score:9000} BS "executionStats" : { "nReturned" : 2, "executionTimeMillis" : 0, "totalKeysExamined" : 2, "totalDocsExamined" : 2,
  140. 140. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) Ace … … 2000 2001 2015 2019 2019 {score:2} {score:1} {score:5000} "executionStats" : { "nReturned" : 2, "executionTimeMillis" : 0, "totalKeysExamined" : 2, "totalDocsExamined" : 2, "executionStages" : { "stage" : "SORT", "memUsage" : 154, "memLimit" : 33554432, {score:9001}{score:9000} BS
  141. 141. Exceptions db.games .find({gamertag:"Ace", date:{$gte:2019}}) .sort({score:1}) ……… … BS S R E R E
  142. 142. E-S-R Guidance A good starting place applicable to most use cases Place keys in the following order: • Equality first • Sort next • Range last Remember that equality is an exact match of a single value
  143. 143. E-S-R Guidance A good starting place applicable to most use cases Place keys in the following order: • Equality first • Sort next • Range last Remember that equality is an exact match of a single value I see isMultiKey:true in the explain output. What does that mean?
  144. 144. Roadmap § E-S-R § Overview § Predicate Type Check § Sequential Prefix § Exceptions? § Multikey Index § Definition § Semantics § Performance
  145. 145. Multikey Index A special type of index which supports efficient querying for array fields
  146. 146. Index Nomenclature Single Field Index The index entries reference a single field in the document Can be Multikey Compound Index The index entries reference more than one field in the document Can be Multikey Multikey Index There is separate entry in the index for every array in the document
  147. 147. Single Field Index Ace Doyen { gamertag: "Ace", games:["Mario", "Halo"] } { gamertag: "Doyen", games:["Fortnite", "GTA", "Minecraft"] }{} {}
  148. 148. Single Field Index - Multikey {} {} MarioHaloGTAFortnite Minecraft
  149. 149. DoyenAce Compound Index - Multikey {} {} MarioHaloGTAFortnite Minecraft
  150. 150. Compound Index - Multikey HaloMinecraftGTAFortnite Mario AceDoyen {}{} We tried to use arrays but get incorrect results and queries are slow. What gives?
  151. 151. Roadmap § E-S-R § Overview § Predicate Type Check § Sequential Prefix § Exceptions? § Multikey Index § Definition § Semantics § Performance
  152. 152. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] }
  153. 153. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] }
  154. 154. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] }
  155. 155. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  156. 156. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  157. 157. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  158. 158. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions
  159. 159. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions."
  160. 160. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  161. 161. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  162. 162. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  163. 163. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  164. 164. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  165. 165. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  166. 166. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  167. 167. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  168. 168. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  169. 169. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  170. 170. Semantics { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } { _id: 3, players: [ { name: "Doyen", rank : 1 }, { name: "Cali", rank : 2 } ] } "You can specify the query such that either a single array element meets these conditions or any combination of array elements meets the conditions." db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  171. 171. Semantics - $elemMatch { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } db.matches.find({ "players.name": "Ace", "players.rank": 1 })
  172. 172. Semantics - $elemMatch { _id: 1, players: [ { name: "Doyen", rank : 1 }, { name: "Ace", rank : 2 } ] } { _id: 2, players: [ { name: "Ace", rank : 1 }, { name: "Doyen", rank : 2 } ] } db.matches.find({ "players.name": "Ace", "players.rank": 1 }) db.matches.find({ "players": { $elemMatch: { name: "Ace", "rank": 1} } })
  173. 173. Semantics - $elemMatch Use $elemMatch to query multiple fields of a single array element Not necessary when querying on a single predicate
  174. 174. Semantics - $elemMatch Use $elemMatch to query multiple fields of a single array element Not necessary when querying on a single predicate Okay, that change got us correct results. But what about the performance?
  175. 175. Roadmap § E-S-R § Overview § Predicate Type Check § Sequential Prefix § Exceptions? § Multikey Index § Definition § Semantics § Performance
  176. 176. Multikey Path Tracking { "_id" : 1, "game" : "Halo", "players" : [ "Ace", "Doyen", "Cali", "Bob" ], "date" : ISODate("2019-02-15T00:00:00Z") }
  177. 177. Multikey Path Tracking { "_id" : 1, "game" : "Halo", "players" : [ "Ace", "Doyen", "Cali", "Bob" ], "date" : ISODate("2019-02-15T00:00:00Z") } db.matches.find({ players:"Ace", date:{ $gte:ISODate("2019-02-01"), $lte:ISODate("2019-02-28") } })
  178. 178. Multikey Path Tracking { "_id" : 1, "game" : "Halo", "players" : [ "Ace", "Doyen", "Cali", "Bob" ], "date" : ISODate("2019-02-15T00:00:00Z") } db.matches.find({ players:"Ace", date:{ $gte:ISODate("2019-02-01"), $lte:ISODate("2019-02-28") } }) .explain() … "indexBounds" : { "players" : [ "["Ace", "Ace"]" ], "date" : [ "(true, new Date(1551312000000)]" ] }
  179. 179. Multikey Path Tracking { "_id" : 1, "game" : "Halo", "players" : [ "Ace", "Doyen", "Cali", "Bob" ], "date" : ISODate("2019-02-15T00:00:00Z") } db.matches.find({ players:"Ace", date:{ $gte:ISODate("2019-02-01"), $lte:ISODate("2019-02-28") } }) .explain() … "indexBounds" : { "players" : [ "["Ace", "Ace"]" ], "date" : [ "(true, new Date(1551312000000)]" ] }
  180. 180. Multikey Path Tracking { "_id" : 1, "game" : "Halo", "players" : [ "Ace", "Doyen", "Cali", "Bob" ], "date" : ISODate("2019-02-15T00:00:00Z") } db.matches.find({ players:"Ace", date:{ $gte:ISODate("2019-02-01"), $lte:ISODate("2019-02-28") } }) .explain() … "indexBounds" : { "players" : [ "["Ace", "Ace"]" ], "date" : [ "(true, new Date(1551312000000)]" ] }
  181. 181. Multikey Path Tracking { "_id" : 1, "game" : "Halo", "players" : [ "Ace", "Doyen", "Cali", "Bob" ], "date" : ISODate("2019-02-15T00:00:00Z") } db.matches.find({ players:"Ace", date:{ $gte:ISODate("2019-02-01"), $lte:ISODate("2019-02-28") } }) .explain() … "indexBounds" : { "players" : [ "["Ace", "Ace"]" ], "date" : [ "(true, new Date(1551312000000)]" ] }
  182. 182. Multikey Path Tracking Before MongoDB 3.4: § Database only contained coarse isMultiKey flag for index § Did not know which field was the array § Conservative behavior to ensure correct results
  183. 183. Multikey Path Tracking Before MongoDB 3.4: "stage" : "IXSCAN", "indexName" : "players_1_date_1", "isMultiKey" : true, "indexBounds" : { "players" : [ "["Ace", "Ace"]" ], "date" : [ "(true, new Date(1551312000000)]" ] } 3.4+: ... "multiKeyPaths" : { "players" : [ "players" ], "date" : [ ] }, "indexBounds" : { "players" : [ "["Ace", "Ace"]" ], "date" : [ "[new Date(1548979200000), new Date(1551312000000)]" ] }
  184. 184. Multikey Path Tracking Take advantage of the improvement § Index needs to be rebuilt when running 3.4 binaries § Just running on 3.4 is not sufficient § Look for multiKeyPaths field in explain output § v:2 index would be a deterministic check § Multikey index cannot cover queries on the array field
  185. 185. Multikey Path Tracking Take advantage of the improvement § Index needs to be rebuilt when running 3.4 binaries § Just running on 3.4 is not sufficient § Look for multiKeyPaths field in explain output § v:2 index would be a deterministic check § Multikey index cannot cover queries on the array field Wow! Everything is running at Web Scale now!
  186. 186. Summary
  187. 187. § E-S-R § Overview § Predicate Type Check § Sequential Prefix § Exceptions? § Multikey Index § Definition § Semantics § Performance
  188. 188. Q&A about Q&I
  189. 189. Additional Resources
  190. 190. Key Ordering Resources Blog Post My colleague Jesse wrote an article which explores the performance characteristics of different arrangements. MongoDB University My colleague Kirby covers many indexing topics in greater detail, including key ordering, in M201: MongoDB Performance.
  191. 191. Please provide Session Feedback 1. Go to slido.com 2. Enter event code #MDBW19 3. Click on Room Name and Provide Session Feedback Feedback poll will remain open for 10 minutes after the talk ends Questions? Meet me now in the Leaf Lounge on the second floor of the Partner Pavilion YOU ARE IN MURRAY HILL

×