Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Reading the .explain() Output

234 views

Published on

Speaker: Charlie Swanson

Learn how MongoDB answers your queries from a query system engineer. If you've ever had a performance problem with a query but didn't know how to find the cause, or if you've ever needed to confirm that your shiny new index is being put to work, the explain command is an excellent place to start. MongoDB's explain system is a powerful tool for solving this type of problem, but can be intimidating and unwieldy to use. In this talk, we will discuss how the explain command works and break down its output into digestible pieces.

  • Be the first to comment

Reading the .explain() Output

  1. 1. O C T O B E R 1 2 , 2 0 1 7 | B E S P O K E | S AN F R AN C I S C O Deciphering Explain Output Charlie Swanson
  2. 2. # M D B l o c a l Understand how MongoDB answers queries Knowledge Figure out what's going on Debugging Learn some tricks to optimize your queries & aggregations Best Practices GOALS OF THIS TALK
  3. 3. # M D B l o c a l S E N I O R E N G I N E E R - Q U E R Y T E A M N Y C Charlie Swanson
  4. 4. # M D B l o c a l 01 02 03 04 05 Motivation "queryPlanner" Verbosity "executionStats" Verbosity "allPlansExecution" Versbosity Beyond Queries Why do you care? Describing considered plans More details about winning plan More details about plan selection Log messages What is explain? The profile Other commands
  5. 5. # M D B l o c a l 01 02 03 04 05 Motivation "queryPlanner" Verbosity "executionStats" Verbosity "allPlansExecution" Versbosity Beyond Queries Why do you care? Describing considered plans More details about winning plan More details about plan selection Log messages What is explain? The profile Other commands
  6. 6. # M D B l o c a l • Is your query using an index? Which one? Target Questions 🤔
  7. 7. # M D B l o c a l • Is your query using an index? Which one? • Is your query using an index to provide the sort? Target Questions 🤔
  8. 8. # M D B l o c a l • Is your query using an index? Which one? • Is your query using an index to provide the sort? • How many of the examined documents ended up matching? Target Questions 🤔
  9. 9. # M D B l o c a l • Is your query using an index? Which one? • Is your query using an index to provide the sort? • How many of the examined documents ended up matching? • Why did the server choose to answer the query the way it did? Target Questions 🤔
  10. 10. # M D B l o c a l • Is your query using an index? Which one? • Is your query using an index to provide the sort? • How many of the examined documents ended up matching? • Why was your winning plan chosen? • Can my queries go faster? Target Questions 🤔
  11. 11. # M D B l o c a l Example Query
  12. 12. # M D B l o c a l How Can The Server Answer This Query?
  13. 13. # M D B l o c a l OPTION 1: Collection scan TWITTER.TWEETS {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} ✅ ✅
  14. 14. # M D B l o c a l COLLECTION SCAN: TWITTER.TWEETS {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} ✅ ✅ 204587 190587 SORT 01 . 02 .
  15. 15. # M D B l o c a l {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} ✅ ✅ (190587, '@Charlie') (204587, '@TaylorSwift') 01. 02. SORT PROJECTION 02 01 {nFavorites: 204587, username: '@TaylorSwift'} {nFavorites: 109587, username: '@Charlie'} COLLECTION SCAN: TWITTER.TWEETS
  16. 16. # M D B l o c a l (190587, '@Charlie') (204587, '@TaylorSwift')01. 02. SORT PROJECTION 02 01 {nFavorites: 204587, username: '@TaylorSwift'} {nFavorites: 109587, username: '@Charlie'} COLLECTION SCAN {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} ✅ ✅
  17. 17. # M D B l o c a l OPTION 1: Collection scan SORT PROJECT COLLECTION SCAN
  18. 18. # M D B l o c a l OPTION 2: INDEX SCAN TWITTER.TWEETS {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} INDEX: {nFavorites: -1} 204587 190587 87983 83092 76032 29023 …
  19. 19. # M D B l o c a l {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} INDEX: {nFavorites: -1} 204587 190587 87983 83092 76032 29023 … Stop when entry is smaller than 100,000 TWITTER.TWEETS
  20. 20. # M D B l o c a l {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} INDEX: {nFavorites: -1} 204587 190587 87983 83092 76032 29023 … SORTED! TWITTER.TWEETS
  21. 21. # M D B l o c a l TWITTER.TWEET S{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… } INDEX 204587 190587 87983 83092 123 123 … INDEX SCAN SORT
  22. 22. # M D B l o c a l {… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… } INDEX 204587 190587 87983 83092 123 123 … INDEX SCAN FETCH 204587 190587 { _id: 400000, createdAt: ISODate(…), username: "@TaylorSwift", … } { _id: 400001, createdAt: ISODate(…), username: "@Charlie", … } TWITTER.TWEET S
  23. 23. # M D B l o c a l TWITTER.TWEET S { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } { … } INDEX 204587 190587 87983 83092 123 123 … INDEX SCAN FETCH { _id: 400000, createdAt: ISODate(…), username: "@TaylorSwift" } { _id: 400001, createdAt: ISODate(…), username: "@Charlie" } PROJECTION {nFavorites: 204587, username: '@TaylorSwift'} {nFavorites: 109587, username: '@Charlie'} { _id: 400000, createdAt: ISODate(…), username: "@TaylorSwift" } { _id: 400001, createdAt: ISODate(…), username: "@Charlie", }
  24. 24. # M D B l o c a l OPTION 2: INDEX scan FETCH PROJECT INDEX SCAN
  25. 25. # M D B l o c a l Many Ways to Answer a Query, Which Was It? SORT PROJECT COLLECTION SCAN FETCH PROJECT INDEX SCAN
  26. 26. # M D B l o c a l 1. Command to explain execution of various other commands 2. Helper on shell cursor object What is Explain?
  27. 27. O C T O B E R 1 2 , 2 0 1 7 | B E S P O K E | S AN F R AN C I S C O What is Explain? - Command
  28. 28. O C T O B E R 1 2 , 2 0 1 7 | B E S P O K E | S AN F R AN C I S C O What is Explain? - Shell Helper
  29. 29. O C T O B E R 1 2 , 2 0 1 7 | B E S P O K E | S AN F R AN C I S C O What is Explain? - Shell Helper ???
  30. 30. # M D B l o c a l Query Plans SORT PROJECT COLLECTION SCAN FETCH PROJECT INDEX SCAN …OR FETCH INDEX SCAN INDEX SCAN
  31. 31. # M D B l o c a l 01 02 03 04 05 Motivation "queryPlanner" Verbosity "executionStats" Verbosity "allPlansExecution" Versbosity Beyond Queries Why do you care? Describing considered plans More details about winning plan More details about plan selection Log messages What is explain? The profile Other commands
  32. 32. # M D B l o c a l > db.collection.find(…).explain("queryPlanner") { "queryPlanner" : { … "winningPlan" : {…}, "rejectedPlans" : […] }, … } Explain Output Optional, this is the default
  33. 33. # M D B l o c a l > db.collection.find(…).explain() {queryPlanner: { winningPlan: { stage: "SORT", inputStage: { stage: "FETCH", inputStage: { stage: "IXSCAN" } } } } }} Explain Output FETCH INDEX SCAN {nFavorites: -1} SORT
  34. 34. # M D B l o c a l > db.collection.find(…).explain() { … stage: "IXSCAN" keyPattern: {nFavorites: -1}, indexBounds: { a: [ "[inf.0, 100000]" ] }, … // Other index scan // specific stats. … }} Explain Output FETCH INDEX SCAN keyPattern: { nFavorites: -1 } indexBounds: […] … SORT
  35. 35. # M D B l o c a l > db.collection.find(…).explain() { "queryPlanner" : { … "winningPlan" : { // Encodes selected plan. }, "rejectedPlans" : […] }, … } Explain Output FETCH INDEX SCAN SORT
  36. 36. # M D B l o c a l > db.collection.find(…).explain() {"queryPlanner" : { "winningPlan" : { {"stage" : "SORT", "inputStage" : {…} }} }, "rejectedPlans" : [ {"stage" : "SORT", "inputStage" : {…} }} … ] }} Explain Output FETCH INDEX SCAN SORT COLL_SCAN SORT
  37. 37. # M D B l o c a l APPLYING THIS INFORMATION
  38. 38. # M D B l o c a l Is Your Query Using the Index You Expect?
  39. 39. # M D B l o c a l FETCH SORT ✓ COLLECTION SCAN SORT ✗ INDEX SCAN keyPattern: {nFollowers: -1} Is Your Query Using the Index You Expect?
  40. 40. # M D B l o c a l Is Your Query Using the Index You Expect?
  41. 41. # M D B l o c a l db.tweets.explain().find( {nFavorites: {$gte: 100000}}, {_id: 0, nFavorites: 1, username: 1}) .sort({nFavorites: -1}) Is Your Query Using the Index You Expect?
  42. 42. # M D B l o c a l db.tweets.explain().find( {nFavorites: {$gte: 100000}}, {_id: 0, nFavorites: 1, username: 1}).sort({nFavorites: -1}) { "queryPlanner": { "winningPlan": { "stage": "PROJECTION", "inputStage": { "stage": "FETCH", "inputStage": { "stage": "IXSCAN", "keyPattern": {"nFavorites": -1}, "indexBounds": { "nFavorites": ["[inf.0, 100000.0]"] } } } }, "rejectedPlans": [ ] } } Is Your Query Using the Index You Expect?
  43. 43. # M D B l o c a l Is Your Query Using an Index to Provide the Sort?
  44. 44. # M D B l o c a l FETCH INDEX SCAN ✓ ✗FETCH INDEX SCAN SORT Is Your Query Using an Index to Provide the Sort?
  45. 45. # M D B l o c a l Is Your Query Using an Index to Provide the Sort?
  46. 46. # M D B l o c a l db.tweets.explain().find( {nFavorites: {$gte: 100000}}, {_id: 0, nFavorites: 1, username: 1}).sort({nFavorites: -1}) { "queryPlanner": { "winningPlan": { "stage": "PROJECTION", "inputStage": { "stage": "FETCH", "inputStage": { "stage": "IXSCAN", "keyPattern": {"nFavorites": -1}, "indexBounds": { "nFavorites": ["[inf.0, 100000.0]"] } } } }, "rejectedPlans": [ ] } } NO SORT STAGE ✅ Is Your Query Using an Index to Provide the Sort?
  47. 47. # M D B l o c a l SORT_MERGE is OK ✓SORT_MERGE INDEX SCAN FETCH INDEX SCAN
  48. 48. # M D B l o c a l BONUS: Is Your Query Using an Index to Provide the Projection?
  49. 49. # M D B l o c a l Compound Index {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} INDEX: {username: 1, nFavorites: - 1} "@Charlie"29023 "@MongoDB"87983 "@MongoDB"60587 "@MongoDB"7983 "@MongoDBEn g" 83092 "@MongoDBEn g" 76032 … …
  50. 50. # M D B l o c a l db.tweets.find({ username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}}) Compound Index TWITTER.TWEETS {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} INDEX: {username: 1, nFavorites: -1} "@Charlie"29023 "@MongoDB"87983 "@MongoDB"60587 "@MongoDB"7983 "@MongoDBEng"83092 "@MongoDBEng"76032 … …
  51. 51. # M D B l o c a l db.tweets.find( {username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}}, {_id: 0, username: 1, nFavorites: 1}) Compound Index TWITTER.TWEETS {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} {…} INDEX: {username: 1, nFavorites: -1} "@Charlie"29023 "@MongoDB"87983 "@MongoDB"60587 "@MongoDB"7983 "@MongoDBEng"83092 "@MongoDBEng"76032 … …
  52. 52. # M D B l o c a l db.tweets.find( {username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}}, {_id: 0, username: 1, nFavorites: 1}) INDEX SCAN PROJECTION {… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… } INDEX: {username: 1, nFavorites: -1} "@Charlie " 29023 "@Mongo DB" 87983 "@Mongo DB" 60587 "@Mongo DB" 7983 "@Mongo DBEng" 83092 "@Mongo DBEng" 76032 … … Compound Index
  53. 53. # M D B l o c a l db.tweets.find( {username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}}, {_id: 0, username: 1, nFavorites: 1}) Compound Index INDEX SCAN PROJECTION {… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… } INDEX: {username: 1, nFavorites: -1} "@Charlie " 29023 "@Mongo DB" 87983 "@Mongo DB" 60587 "@Mongo DB" 7983 "@Mongo DBEng" 83092 "@Mongo DBEng" 76032 … …
  54. 54. # M D B l o c a l db.tweets.find( {username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}}, {_id: 0, username: 1, nFavorites: 1, other: 1}) Compound Index INDEX SCAN PROJECTION FETCH {… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… } INDEX: {username: 1, nFavorites: -1} "@Charlie " 29023 "@Mongo DB" 87983 "@Mongo DB" 60587 "@Mongo DB" 7983 "@Mongo DBEng" 83092 "@Mongo DBEng" 76032 … …
  55. 55. # M D B l o c a l INDEX SCAN PROJECTION db.tweets.find( {username: {$in: ["@MongoDBEng", "@MongoDB"]}, nFavorites: {$gt: 50000}}}, {_id: 0, username: 1, nFavorites: 1}) .sort({username: 1}) Compound Index {… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… }{… } {… } {… } {… } {… } {… } INDEX: {username: 1, nFavorites: -1} "@Charlie " 29023 "@Mongo DB" 87983 "@Mongo DB" 60587 "@Mongo DB" 7983 "@Mongo DBEng" 83092 "@Mongo DBEng" 76032 … …
  56. 56. # M D B l o c a l PROJECTION INDEX SCAN ✓ ✗ PROJECTION INDEX SCAN FETCH Is your query using an index to provide the projection?
  57. 57. # M D B l o c a l • Is your query using an index? Which one? • Is your query using an index to provide the sort? • Is your query using an index to provide the projection? The Power of "QueryPlanner"
  58. 58. # M D B l o c a l db.tweets.explain().find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) Next Up: "It's using an index, so what's taking so long?" FETCH INDEX SCAN keyPattern: {createdDate: 1}
  59. 59. # M D B l o c a l db.tweets.explain("executionStats").find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) It's using an index, so what's taking so long? FETCH INDEX SCAN keyPattern: {createdDate: 1} 12:02 12:03 12:04 … INDEX SCAN INDEX: {createdAt: 1}
  60. 60. # M D B l o c a l db.tweets.explain("executionStats").find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) {createdDate: 12:02, favorites: [ "@MongoDB", "@taylorswift" ]} FETCH FETCH filter: { favorites: "@eliothorowitz" } INDEX SCAN keyPattern: {createdDate: 1} It's using an index, so what's taking so long? 12:02 12:03 12:04 … INDEX: {createdAt: 1}
  61. 61. # M D B l o c a l db.tweets.explain("executionStats").find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) {createdDate: 12:02, favorites: [ "@MongoDB", "@taylorswift" ]} FETCH FETCH filter: { favorites: "@eliothorowitz" } INDEX SCAN keyPattern: {createdDate: 1} ❌ It's using an index, so what's taking so long? 12:02 12:03 12:04 … FETCH filter: { favorites: "@eliothorowitz" } INDEX SCAN keyPattern: {createdDate: 1} INDEX: {createdAt: 1}
  62. 62. # M D B l o c a l db.tweets.explain("executionStats").find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) {createdDate: 12:03, favorites: [ "@eliothorowitz", "@taylorswift" ]} FETCH It's using an index, so what's taking so long? 12:02 12:03 12:04 … FETCH filter: { favorites: "@eliothorowitz" } INDEX SCAN keyPattern: {createdDate: 1} INDEX: {createdAt: 1}
  63. 63. # M D B l o c a l • What percentage of the index keys in the scanned range ended up matching the predicate? • What's the selectivity? So how many of them were thrown out?
  64. 64. # M D B l o c a l 01 02 03 04 05 Motivation "queryPlanner" Verbosity "executionStats" Verbosity "allPlansExecution" Versbosity Beyond Queries Why do you care? Describing considered plans More details about winning plan More details about plan selection Log messages What is explain? The profile Other commands
  65. 65. # M D B l o c a l > db.collection.find(…).explain() {"queryPlanner" : { "winningPlan" : { {"stage" : "SORT", "inputStage" : {…} }} }, "rejectedPlans" : [ {"stage" : "SORT", "inputStage" : {…} }} … ] }} Explain Mode: "queryPlanner" FETCH INDEX SCAN (a, b) SORT FETCH INDEX SCAN (a) SORT
  66. 66. # M D B l o c a l • "executionStats" Explain Mode: "ExecutionStats" FETCH INDEX SCAN (a) SORT created by Mike Ashley from Noun Project created by Creative Stall from Noun Project
  67. 67. # M D B l o c a l > db.tweets.find(…).explain("executionStats") { "queryPlanner" : { … "winningPlan" : {…}, "rejectedPlans" : […] }, "executionStats": { // New! …, "executionStages": {…} } … } Explain Mode: "executionStats"
  68. 68. # M D B l o c a l > db.tweets.find(…).explain("executionStats") { "queryPlanner" : { /* Same as before. */ }, "executionStats": { // Top-level stats. "executionStages": { stage: "SORT", // Sort stats. inputStage: { // etc, etc. } } } … } Details: "ExecutionStats" FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  69. 69. # M D B l o c a l > db.tweets.find(…).explain("executionStats") { …, "executionStats" : { // Top-level stats. "nReturned" : 390000, "executionTimeMillis" : 4431, "totalKeysExamined" : 390000, "totalDocsExamined" : 390000, "executionStages" : {…} }, } Details: "ExecutionStats" FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  70. 70. # M D B l o c a l db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, … "sortPattern" : { "nFollowers" : 1 }, "memUsage" : 20280000, "memLimit" : 33554432, "inputStage" : {…} } } } Details: "ExecutionStats" FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  71. 71. # M D B l o c a l db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, … "sortPattern" : { "nFollowers" : -1 }, "memUsage" : 20280000, "memLimit" : 33554432, "inputStage" : {…} } } } Details: "ExecutionStats" FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  72. 72. # M D B l o c a l db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, "works" : 780003, "advanced" : 390000, "needTime" : 390002, "isEOF" : 1, "sortPattern" : { "b" : 1 }, … "inputStage" : {…} } } } ? Details: "ExecutionStats" FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  73. 73. # M D B l o c a l • These are all PlanStages • SortStage • FetchStage • IndexScanStage Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  74. 74. # M D B l o c a l • These are all PlanStages • Each PlanStage implements work() Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  75. 75. # M D B l o c a l • These are all PlanStages • Each PlanStage implements work(), returns one of: • ADVANCED • NEED_TIME • IS_EOF Execution Stats: works, advanced, etc. FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  76. 76. # M D B l o c a l EXECUTION STATS: WORKS, ADVANCED, ETC. FETCH SORT INDEX SCAN keyPattern: {hashtags: 1} work()
  77. 77. # M D B l o c a l EXECUTION STATS: WORKS, ADVANCED, ETC. work() FETCH SORT INDEX SCAN keyPattern: {hashtags: 1} work()
  78. 78. # M D B l o c a l EXECUTION STATS: WORKS, ADVANCED, ETC. work() work() FETCH SORT INDEX SCAN keyPattern: {hashtags: 1} work()
  79. 79. # M D B l o c a l EXECUTION STATS: WORKS, ADVANCED, ETC. work() ADVANCEDID FETCH SORT INDEX SCAN keyPattern: {hashtags: 1} work()
  80. 80. # M D B l o c a l EXECUTION STATS: WORKS, ADVANCED, ETC. ADVANCED{…} ADVANCED FETCH SORT INDEX SCAN keyPattern: {hashtags: 1} work()
  81. 81. # M D B l o c a l SORT EXECUTION STATS: WORKS, ADVANCED, ETC. NEED_TIME { … } ADVANCED ADVANCED FETCH INDEX SCAN keyPattern: {hashtags: 1} work()
  82. 82. # M D B l o c a l db.tweets.find(…).explain("executionStats") { "executionStats" : { // Top-level stats. "executionStages" : { "stage" : "SORT", "nReturned" : 390000, "executionTimeMillisEstimate" : 2030, "works" : 780003, "advanced" : 390000, "needTime" : 390002, "isEOF" : 1, "sortPattern" : { "b" : 1 }, … "inputStage" : {…} } } } Details: "executionStats" FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  83. 83. # M D B l o c a l > db.tweets.find(…).explain("executionStats") {"executionStats": { "executionStages": { stage: "SORT", // Sort stats, includes "works", "advanced", … inputStage: { stage: "FETCH", // Fetch stats, includes "works", "advanced", … inputStage: { // etc, etc. } } } } … } Details: "executionStats" FETCH SORT INDEX SCAN keyPattern: {hashtags: 1}
  84. 84. # M D B l o c a l > db.tweets.find(…).explain("executionStats") { "queryPlanner" : { … "winningPlan" : {…}, "rejectedPlans" : […] }, "executionStats": { // New! …, "executionStages": {…} } … } Explain Mode: "executionStats"
  85. 85. # M D B l o c a l Applying This Information
  86. 86. # M D B l o c a l How selective is your index?
  87. 87. # M D B l o c a l db.tweets.explain("executionStats").find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) How selective is your index? 12:02 12:03 12:04 12:06 … INDEX SCAN FETCH filter: { favorites: "@eliothorowitz" } INDEX SCAN keyPattern: {createdDate: 1}
  88. 88. # M D B l o c a l db.tweets.explain("executionStats").find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) { "executionStats" : { "nReturned" : 314, "totalKeysExamined" : 2704, // < 12% matched … } How selective is your index? FETCH filter: { favorites: "@eliothorowitz" } INDEX SCAN keyPattern: {createdDate: 1}
  89. 89. # M D B l o c a l What's the most expensive part of your plan?
  90. 90. # M D B l o c a l db.tweets.explain("executionStats").find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) What's the most expensive part of your plan? FETCH filter: { favorites: "@eliothorowitz" } INDEX SCAN keyPattern: {createdDate: 1}
  91. 91. # M D B l o c a l db.tweets.explain("executionStats").find(…) What's the most expensive part of your plan? FETCH executionTimeMillisEstimate: 431 INDEX SCAN executionTimeMillisEstimate: 67
  92. 92. # M D B l o c a l db.tweets.explain("executionStats").find(…) What's the most expensive part of your plan? FETCH works: 2705 advanced: 314 needTime: 2391 // 314/2705 ≈ 8% INDEX SCAN
  93. 93. # M D B l o c a l • "queryPlanner" • Is your query using the index you expect? • Is your query using an index to provide the sort? • Is your query using an index to provide the projection? • "executionStats" • How selective is your index? • Which part of your plan is the most expensive? Our Progress
  94. 94. # M D B l o c a l db.tweets.explain("executionStats").find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) • We had an index on {favorites: 1}, would that have been faster? next up: "Why was This plan chosen?" 🤔
  95. 95. # M D B l o c a l 01 02 03 04 05 Motivation "queryPlanner" Verbosity "executionStats" Verbosity "allPlansExecution" Versbosity Beyond Queries Why do you care? Describing considered plans More details about winning plan More details about plan selection Log messages What is explain? The profile Other commands
  96. 96. # M D B l o c a l > db.collection.find(…).explain() {"queryPlanner" : { "winningPlan" : { {"stage" : "SORT", "inputStage" : {…} }} }, "rejectedPlans" : [ {"stage" : "SORT", "inputStage" : {…} }} … ] }} Explain Mode: "QueryPlanner" ❓ FETCH INDEX SCAN (a, b) SORT FETCH INDEX SCAN (a) SORT
  97. 97. # M D B l o c a l Query Planning FETCH INDEX SCAN (a, b) SORT FETCH INDEX SCAN (a) SORT FETCH INDEX SCAN (a, b)
  98. 98. # M D B l o c a l • MultiPlanStage::pickBestPlan() Query Planning MATCH COLL SCAN … MATCH COLL SCAN COLL SCAN FETCH IX SCAN (ts) MultiPlanStage work() work() work()
  99. 99. # M D B l o c a l • MultiPlanStage::pickBestPlan() Query Planning ADVANCED NEED_TIME ADVANCED MATCH COLL SCAN … MATCH COLL SCAN COLL SCAN FETCH IX SCAN (ts) MultiPlanStage
  100. 100. # M D B l o c a l • MultiPlanStage::pickBestPlan() Query Planning Advances:78 22 50 MATCH COLL SCAN … MATCH COLL SCAN COLL SCAN FETCH IX SCAN (ts) MultiPlanStage
  101. 101. # M D B l o c a l • MultiPlanStage::pickBestPlan() Query Planning Advances:78 22 50 MATCH COLL SCAN … MATCH COLL SCAN COLL SCAN FETCH IX SCAN (ts) MultiPlanStage
  102. 102. # M D B l o c a l > db.tweets.find(…).explain("allPlansExecution") { "queryPlanner" : { … "winningPlan" : {…}, "rejectedPlans" : […] }, "executionStats": { …, "executionStages": {…}, "allPlansExecution": […] // New! } … } Explain Mode: "allPlansExecution"
  103. 103. # M D B l o c a l > db.collection.find(…).explain() {"queryPlanner" : { "winningPlan" : { {"stage" : "SORT", "inputStage" : {…} }} }, "rejectedPlans" : [ {"stage" : "SORT", "inputStage" : {…} }} … ] }} Verbosity: "queryPlanner" FETCH INDEX SCAN SORT COLL_SCAN SORT
  104. 104. # M D B l o c a l > db.collection.find(…).explain() {"queryPlanner" : { "winningPlan" : {…}, "rejectedPlans" : [ {…}, … ] }, "executionStats": { executionStages: {…} }} Verbosity: "executionStats" FETCH INDEX SCAN SORT SORT FETCH INDEX SCAN SORT
  105. 105. # M D B l o c a l > db.collection.find(…).explain() {"queryPlanner" : { "winningPlan" : {…}, "rejectedPlans" : [ {…}, … ] }, "executionStats": { executionStages: {…} }} Verbosity: "executionStats" FETCH INDEX SCAN SORT SORT FETCH INDEX SCAN SORT
  106. 106. # M D B l o c a l > db.collection.find(…).explain() {"queryPlanner" : { "winningPlan" : {…}, "rejectedPlans" : [ {…}, … ] }, "executionStats": { executionStages: {…}, allPlansExecution: [ {…}, {…}, … ] }} Verbosity: "executionStats" SORT FETCH INDEX SCAN SORT FETCH INDEX SCAN SORT FETCH INDEX SCAN SORT FETCH INDEX SCAN SORT
  107. 107. # M D B l o c a l db.tweets.explain("allPlansExecution").find({ createdDate: {$gte: <today>}, favorites: "@eliothorowitz" }) { "executionStats": { "allPlansExecution": [ {nReturned: 34, executionStages: { /* Index Scan on "favorites" */ } }, {nReturned: 101, executionStages: { /* Index Scan on "createdDate" */ } } ] } … } Explain Mode: "allPlansExecution"
  108. 108. # M D B l o c a l 01 02 03 04 05 Motivation "queryPlanner" Verbosity "executionStats" Verbosity "allPlansExecution" Versbosity Beyond Queries Why do you care? Describing considered plans More details about winning plan More details about plan selection Log messages What is explain? The profile Other commands
  109. 109. # M D B l o c a l • Queries with response time >100ms (server side) are logged: Slow Queries 2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find { find: "tweets", filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort: { nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites: 1.0, username: 1.0 } } planSummary: IXSCAN { nFavorites: -1 } keysExamined:359907 docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:2871 nreturned:20 reslen:1087 locks:{ Global: { acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872 } }, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms
  110. 110. # M D B l o c a l • Queries with response time >100ms (server side) are logged: Slow Queries 2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find { find: "tweets", filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort: { nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites: 1.0, username: 1.0 } } planSummary: IXSCAN { nFavorites: -1 } keysExamined:359907 docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:2871 nreturned:20 reslen:1087 locks:{ Global: { acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872 } }, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms
  111. 111. # M D B l o c a l • Queries with response time >100ms (server side) are logged: Slow Queries 2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find { find: "tweets", filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort: { nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites: 1.0, username: 1.0 } } planSummary: IXSCAN { nFavorites: -1 } keysExamined:359907 docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:2871 nreturned:20 reslen:1087 locks:{ Global: { acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872 } }, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms
  112. 112. # M D B l o c a l • Queries with response time >100ms (server side) are logged: Slow Queries 2017-05-25T10:01:58.917-0400 I COMMAND [conn7] command twitter.tweets appName: "MongoDB Shell" command: find { find: "tweets", filter: { nFavorites: { $gte: 10000.0 } }, limit: 20.0, singleBatch: false, sort: { nFavorites: -1.0, username: 1.0 }, projection: { _id: 0.0, nFavorites: 1.0, username: 1.0 } } planSummary: IXSCAN { nFavorites: -1 } keysExamined:359907 docsExamined:359907 hasSortStage:1 cursorExhausted:1 numYields:2871 nreturned:20 reslen:1087 locks:{ Global: { acquireCount: { r: 5744 } }, Database: { acquireCount: { r: 2872 } }, Collection: { acquireCount: { r: 2872 } } } protocol:op_command 1493ms
  113. 113. # M D B l o c a l • Queries with response time >100ms (server side) are logged. • Configurable via profiling parameter 'slowMs' Slow Queries
  114. 114. # M D B l o c a l • If turned on, queries show up in the system.profile collection The Profile { "op": "query", "ns": "twitter.tweets", "query": { "find": "tweets", "filter": { … }, "limit": 20, "sort": { … }, "projection": { … } }, "millis": 1355, "planSummary": "IXSCAN { nFavorites: -1 }", "execStats": { "stage": "PROJECTION", "inputStage": { "stage": "SORT", "inputStage": { "stage": "FETCH", "inputStage": { "stage": "IXSCAN", } } } } } }
  115. 115. # M D B l o c a l db.runCommand({ explain: {/* command */}, verbosity: <queryPlanner|executionStats|allPlansExecution> }) Other Commands
  116. 116. # M D B l o c a l db.runCommand({ explain: {findAndModify: {…}}, verbosity: <queryPlanner|executionStats|allPlansExecution> }) Other Commands
  117. 117. # M D B l o c a l db.runCommand({ explain: {update: {…}}, verbosity: <queryPlanner|executionStats|allPlansExecution> }) Other Commands
  118. 118. # M D B l o c a l db.runCommand({ explain: {aggregate: {…}}, verbosity: <queryPlanner|executionStats|allPlansExecution> }) Other Commands
  119. 119. # M D B l o c a l • Aggregation is special… Aggregation runCommand({ aggregate: "collection", pipeline: […], explain: <true|false> })
  120. 120. # M D B l o c a l • Aggregation is was special… • 3.4 and earlier: • 3.6 and beyond: Aggregation runCommand({ aggregate: "collection", pipeline: […], explain: <true|false> }) runCommand({explain: { aggregate: "collection", pipeline: […], }, verbosity: "…" })
  121. 121. # M D B l o c a l db.explain().aggregate([{$group: {…}}, {$project: {…}}]) { stages: [ {$cursor: {…}}, {$group: {…}}, {$project: {…}}, ] } Aggregation
  122. 122. # M D B l o c a l db.explain().aggregate([{$group: {…}}, {$project: {…}}]) { stages: [ {$cursor: { query: {…}, fields: {…}, queryPlanner: {/* same as query explain! */}, executionStats: {/* 3.6+ only, same as query explain! */} }}, {$group: {…}}, {$project: {…}} ] } Aggregation
  123. 123. # M D B l o c a l SUMMARY 01 02 03 04 05 Motivation "queryPlanner" Verbosity "executionStats" Verbosity "allPlansExecution" Versbosity Beyond Queries Why do you care? Describing considered plans More details about winning plan More details about plan selection Log messages What is explain? The profile Other commands
  124. 124. # M D B l o c a l Some Closing Thoughts
  125. 125. # M D B l o c a l 01: Output is VERY large
  126. 126. # M D B l o c a l Compass Shoutout
  127. 127. # M D B l o c a l Compass Shoutout
  128. 128. # M D B l o c a l Compass Shoutout
  129. 129. # M D B l o c a l 02: explain won't solve all your problems created by Mike Ashley from Noun Project Your Application MongoDB
  130. 130. # M D B l o c a l Can I see all the tweets with hashtag "#MDBlocal" from this hour? 02: explain won't solve all your problems Your Application MongoDB
  131. 131. # M D B l o c a l Hmm… Let me think about that… 02: explain won't solve all your problems Your Application MongoDB
  132. 132. # M D B l o c a l Ah! Here are your results! 02: explain won't solve all your problems Your Application MongoDB
  133. 133. # M D B l o c a l 02: explain won't solve all your problems What took you so long?! Your Application MongoDB
  134. 134. # M D B l o c a l - network latency 02: explain won't solve all your problems What took you so long?! Your Application MongoDB
  135. 135. # M D B l o c a l - network latency - large result set {… } 02: explain won't solve all your problems {… } {… } {… } {… } {… } {… } {… } {… } What took you so long?! Your Application MongoDB
  136. 136. # M D B l o c a l 02: explain won't solve all your problems Your Application MongoDB - network latency - large result set - server contention What took you so long?!
  137. 137. # M D B l o c a l - network latency - large result set - server contention - query planning problem 02: explain won't solve all your problems What took you so long?! Your Application MongoDB
  138. 138. # M D B l o c a l • Is your query using an index? Which one? • Is your query using an index to provide the sort? • How many of the examined documents ended up matching? • Why did the server choose to answer the query the way it did? 03: Target Questions 🤔

×