Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Query planner

1,885 views

Published on

A detailed explanation on how MongoDB's Query Planner works compared with relational databases.

Published in: Data & Analytics
  • Be the first to comment

Query planner

  1. 1. Miguel Angel Nieto miguel.nieto@mongodb.com Technical Services Engineer, MongoDB Query Planner
  2. 2. The Question ● I worked for 6 years as MySQL Technical Support Engineer. ● A large percentage of cases from customers were related to bad query plans/wrong index selection. ● Query Planning is a complex piece of code with many knobs that can be tuned. ● When I started working at MongoDB I found that the number of cases on that topic was very very (very) low. So I asked myself: Why?
  3. 3. Plan selection in other databases ● Traditional databases use a statistics approach to choose the best plan: ○ The information about data distribution is not accurate. ○ It is estimated by reading data with random dives in the index tree (MySQL). ○ When some prerequisites are met (like number of modified rows) statistics are automatically recalculated.
  4. 4. Plan selection in MongoDB ● MongoDB uses a empirical method: ○ If there is no cached plan, then all viable execution plans, based on the available indexes, are created. ○ MongoDB runs the query multiple times, one for each query plan and benchmarks them. It chooses the one that provides the best performance. ○ Once done, the plan is cached. ■ Future queries with the same shape will re-use this plan rather than re-running the candidate plans. ■ For each such query the performance of the cached plan is evaluated. If the plan's performance decreases beyond a given threshold, it is evicted from the cache and the candidate test phase runs again. This is known as re-planning (SERVER-15225)
  5. 5. Benchmarking the plans ● All possible plans are executed in round-robin fashion. ● It gathers execution metrics and then provide a score to each plan. ● Sort the plans by score and choose the best one.
  6. 6. Execution Metrics (I) ● These are the metrics: works advanced needTime isEOF
  7. 7. Execution Metrics (II) ● Number of works: ■ The planner asks each plan for the next document, via a call to work(). ■ If the plan can supply a document, it responds with 'advanced'. Otherwise, the plan responds with 'needsTime'. ● If all documents have been retrieved, then isEOF = 1.
  8. 8. Early stop of query execution ● The query could be expensive, so there are limits to early stop the execution. Execution stop if: ○ The maximum number of works has been reached. ○ The requested number of documents has been retrieved (advanced). ○ We get isEOF (the resultSet has no more documents). works work() isEOF advanced Break
  9. 9. Number of work() calls before stopping ● internalQueryPlanEvaluationWorks = 10000 For large collections we take a fraction of the number of documents: ● internalQueryPlanEvaluationCollFraction = 0.3 Then, get the maximum value. internalQueryPlanEvaluationWorks internalQueryPlanEvaluationCollFraction numRecords works
  10. 10. Number of documents to retrieve before stopping ● internalQueryPlanEvaluationMaxResults = 101 ● query.getQueryRequest().getNToReturn() ○ Used in the old OP_QUERY protocol. ○ Drivers set 'ntoreturn' to min('batchSize', 'limit') in order to fake the lack of 'limit' or 'batchSize' mechanism in the protocol. ● query.getQueryRequest().getLimit() ○ Used in OP_QUERY protocol from 3.2 onwards. getNToReturn advanced getNToReturn internalQueryPlanEvaluationMaxResults getLimit advanced getLimit internalQueryPlanEvaluationMaxResults advanced internalQueryPlanEvaluationMaxResults advanced
  11. 11. Pick the best plan, count the scores ● baseScore = 1 ● Productivity = queryResults / workUnits ● TieBreak (very small number) = min(1.0 / (10 * workUnits), 1e-4) ● noFetchBonus (covered index) = TieBreak or 0 ● noSortBonus (blocking sort) = TieBreak or 0 ● noIxisectBonus (avoiding index intersection) = TieBreak or 0 ● tieBreakers = noFetchBonus + noSortBonus + noIxisectBonus ● eofBonus (if during plan execution all possible documents are retrieved) = 0 | 1
  12. 12. Replanning: Automatic Plan Cache Eviction ● The stored data keep changing, it could possible that the cached plan is not the best one anymore. ● While the cached plan is being used, MongoDB re-runs the trial period for that plan and keeps a count of the work() function calls. ● If the new trial period takes more than 10 times as many works() as the original trial period, it evicts the plan from the cache and re-tests all candidate plans to pick a new winner. ● internalQueryCacheEvictionRatio = 10 maxWorksBeforeReplan internalQueryCacheEvictionRatio cachedWorks currentWorks maxWorksBeforeReplan replan()
  13. 13. Plans are not always cached ● In the following situations, the execution plan is not cached: ○ Collection scan without sort() ○ hint() ○ min() ○ max() ○ explain() ○ Tailable cursors (they don’t use indexes) ○ snapshot() ○ A single viable plan
  14. 14. Query Planner Troubleshoot Example (I) ● We check all query shapes: listQueryShapes
  15. 15. Query Planner Troubleshoot Example (II) ● Get the execution plan for that query: getPlansByQuery solution score works isEOF
  16. 16. Query Planner Troubleshoot Example (III) ● Remove the query plan for a particular query: clearPlansByQuery "plans": [ ]
  17. 17. Query Planner Troubleshoot Example (IV) ● Remove all query plans on a particular collection: clear
  18. 18. Query Planner Troubleshoot ● There are Plan Cache methods that can be used for troubleshooting: https://docs.mongodb.com/manual/reference/method/js-plan-cache/ ● Check all query shapes: ○ db.collection.getPlanCache().listQueryShapes() ● Get the plan for a particular query: ○ db.collection.getPlanCache().getPlansByQuery( <query>, <projection>, <sort> ) ● Clean the plans for a particular query: ○ db.collection.getPlanCache().clearPlansByQuery() ● Clean all plans: ○ db.collection.getPlanCache().clear()
  19. 19. Thanks!

×