Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Understanding and optimizing your N1QL queries – Couchbase Connect New York 2017

736 views

Published on

Every flight has a flight plan. Every query has a query plan. In Couchbase Server, the query optimizer is responsible for creating the best query plan that it can for every query. But query plan optimization is a complex subject and improving query performance requires expert knowledge – just ask the RDBMS folks.
This session will teach you how the query optimizer selects the appropriate indices, index scan methods, and joins. It will teach you how to evaluate optimizer behavior using EXPLAIN PLAN and how to change the choices the optimizer makes, when needed. We’ll explore in detail how the query optimizer works, how to examine the query plan, and how you can tune your query performance with the USE INDEX clause. We will also discuss the new adaptive indexing feature being introduced in Couchbase Server 5.0 and how it simplifies index creation.

Published in: Software
  • Be the first to comment

Understanding and optimizing your N1QL queries – Couchbase Connect New York 2017

  1. 1. ©2017 Couchbase Inc. Understanding and Optimizing N1QL Queries Asif Kazi. Director,Worldwide Professional Service | asif@couchbase.com Keshav Murthy | Senior Director, Couchbase R&D | keshav@couchbase.com 1
  2. 2. ©2017 Couchbase Inc. Agenda • Brief Introduction to N1QL • Understand Query Plan • How Query Plan Generated • Query + Indexing features • Q&A 2
  3. 3. ©2017 Couchbase Inc.©2017 Couchbase Inc. 3 Introduction to N1QL
  4. 4. ©2017 Couchbase Inc. SQL 4 ResultSet Input: Relations Output: Relation
  5. 5. ©2017 Couchbase Inc. NoSQL 5 { "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ] } LoyaltyInfo ResultDocuments Orders CUSTOMER Built Manually; Expensive Input: JSON Documents Output: JSON Documents
  6. 6. ©2017 Couchbase Inc. N1QL Changes the Possibilities for NoSQL
  7. 7. ©2017 Couchbase Inc. NoSQL { "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ] } LoyaltyInfo ResultDocuments Orders CUSTOMER Input: JSON Documents Output: JSON Documents
  8. 8. ©2017 Couchbase Inc. Goal of N1QL: SQL for JSON Give developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data.
  9. 9. ©2017 Couchbase Inc. 9 SELECT Customers.ID, Customers.Name, SUM(OrderLine.Amount) FROM Orders UNNEST Orders.LineItems AS OrderLine INNER JOIN Customers ON KEYS Orders.CustID GROUP BY Customers.ID, Customers.Name HAVING SUM(OrderLine.Amount) > 10000 ORDER BY SUM(OrderLine.Amount) DESC LIMIT 100 ;
  10. 10. ©2017 Couchbase Inc. 10 SELECT Customers.ID, Customers.Name, SUM(OrderLine.Amount) FROM Orders UNNEST Orders.LineItems AS OrderLine INNER JOIN Customers ON KEYS Orders.CustID GROUP BY Customers.ID, Customers.Name HAVING SUM(OrderLine.Amount) > 10000 ORDER BY SUM(OrderLine.Amount) DESC LIMIT 100 ;
  11. 11. ©2017 Couchbase Inc.©2017 Couchbase Inc. 11 Index
  12. 12. ©2017 Couchbase Inc. Index Options 12 Index Type Description 1 Primary Index Index on the document key on the whole bucket 2 Named Primary Index Give name for the primary index. Allows multiple primary indexes in the cluster 3 Secondary Index Index on the key-value or document-key 4 Secondary Composite Index Index on more than one key-value 5 Functional Index Index on function or expression on key-values 6 Array Index Index individual elements of the arrays 7 Partial Index Index subset of items in the bucket 8 Covering Index Query able to answer using the the data from the index and skips retrieving the item. 9 Duplicate Index This is not type of index. Feature of indexing that allows load balancing. Thus providing scale-out, multi-dimensional scaling, performance, and high availability.
  13. 13. ©2017 Couchbase Inc.©2017 Couchbase Inc. 13 Understanding Query Plan
  14. 14. ©2017 Couchbase Inc. N1QL: Query Execution Flow Clients 1. Submit the query over REST API 8. Query result 2. Parse, Analyze, create Plan 7. Evaluate: Filter, Join, Aggregate, Sort, Paginate 3. Scan Request; index filters 6. Fetch the documents Index Service Query Service Data Service 4. Get qualified doc keys 5. Fetch Request, doc keys SELECT c_id, c_first, c_last, c_max FROM CUSTOMER WHERE c_id = 49165; { "c_first": "Joe", "c_id": 49165, "c_last": "Montana", "c_max" : 50000 }
  15. 15. ©2017 Couchbase Inc. Inside a Query Service Client FetchParse Plan Join Filter Pre-Aggregate Offset Limit ProjectSortAggregateScan Query Service Index Service Data Service
  16. 16. ©2017 Couchbase Inc. Inside a Query Service FetchParse Plan Join Filter Pre-Aggregate Offset Limit ProjectSortAggregateScan Query Service Index Service Data Service Prepare Phase Projection Phase Run Phase Scan Phase Fetch Phase
  17. 17. ©2017 Couchbase Inc. Query Execution: Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggregate Scan • Each query can be executed in several ways • Create the query execution plan • Access path for each keyspace referenced • Decide on the filters to push down • Determine Join order and join method • Create the execution tree • For each keyspace reference: • Look at the available indices • Match the filters in the query with index condition and keys • Choose one or more indices for each keyspace • Decide if a covering index can be used • Covering index is preferred • Create index filters and post scan, post join filters
  18. 18. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • Covering Index Scan • Predicate pushdown to Indexer • Pushdown limit to IndexScan (4.5) • Use Index Order (4.5) • OFFSET pushdown (5.0) • Index projection (5.0) • ASC, DESC index (5.0) • Intersect scan performance (5.0) CREATE INDEX ts_c_id ON `travel-sample` (country, id) WHERE type = "airline"; EXPLAIN SELECT country, id FROM `travel-sample` t WHERE type = "airline" AND country = "United States" AND id BETWEEN 10 AND 1000 ORDER BY country, id OFFSET 5 LIMIT 10; { "#operator": "IndexScan", "covers": [ "cover ((`t`.`country`))", "cover ((`t`.`id`))", "cover ((meta(`t`).`id`))" ], "filter_covers": { "cover ((`t`.`type`))": "airline" }, "index": "ts_c_id", "index_id": "ccd0f0c297114733", "keyspace": "travel-sample", "limit": "(5 + 10)", "namespace": "default", "spans": [ { "Range": { "High": [ ""United States"", "1000" ], "Inclusion": 3, "Low": [ ""United States"", "10" ] } } ], "using": "gsi" }
  19. 19. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • NonCovering Index Scan • Predicate pushdown to Indexer • Pushdown limit to IndexScan (4.5) • Use Index order (4.5) CREATE INDEX ts_c_id ON `travel- sample`(country, id) WHERE type = "airline"; EXPLAIN SELECT country, id, name FROM `travel-sample` t WHERE type = "airline" AND country = "United States" AND id BETWEEN 10 AND 1000 ORDER BY country, id OFFSET 5 LIMIT 10; { "#operator": "IndexScan", "index": "ts_c_id", "index_id": "ccd0f0c297114733", "keyspace": "travel-sample", "limit": "(5 + 10)", "namespace": "default", "spans": [ { "Range": { "High": [ ""United States"", "1000" ], "Inclusion": 3, "Low": [ ""United States"", "10" ] } } ], "using": "gsi" }
  20. 20. ©2017 Couchbase Inc. N1QL: Covering Index and Covered Query Clients 1. Submit the query over REST API 6. Query result 2. Parse, Analyze, create Plan 5. Evaluate: Filter, Join, Aggregate, Sort, Paginate 3. Scan Request; index filters Index Service Query Service 4. Get qualified doc keys create index i1 on CUSTOMER(c_id, c_max, c_first, c_last) SELECT c_id, c_first, c_last, c_max FROM CUSTOMER WHERE c_id = 49165; { "c_first": "Joe", "c_id": 49165, "c_last": "Montana", "c_max" : 50000 }
  21. 21. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • NonCovering Query • Fetch (Only for Non Covering Scan) • Filter CREATE INDEX ts_c_id ON `travel- sample`(country, id) WHERE type = "airline"; EXPLAIN SELECT country, id, name FROM `travel-sample` t WHERE type = "airline" AND country = "United States" AND id BETWEEN 10 AND 1000 ORDER BY country, id OFFSET 5 LIMIT 10; { "#operator": "Fetch", "keyspace": "travel-sample", "namespace": "default" } { "#operator": "Filter", "condition": "((((`t`.`type`) = "airline") and ((`t`.`country`) = "United States")) and ((`t`.`id`) between 10 and 1000))" }
  22. 22. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • IntersectScan • For non covered queries only CREATE INDEX ts_c ON `travel- sample`(country) WHERE type = "airline" ; CREATE INDEX ts_n ON `travel- sample`(name) WHERE type = "airline" ; EXPLAIN SELECT country, name FROM `travel-sample` t WHERE type = "airline" AND country = "United States" AND name = "US Airways"; {"#operator": "IntersectScan", "scans": [ { "#operator": "IndexScan", "index": "ts_c", "keyspace": "travel-sample", "spans": [ { "Range": { "High": [ ""United States”"], "Inclusion": 3, "Low": [ ""United States””] } } ] }, { "#operator": "IndexScan", "index": "ts_n", "keyspace": "travel-sample", "spans": [ { "Range": { "High": [""US Airways”” ], "Inclusion": 3, "Low": [""US Airways”” ] } } ] } ] }
  23. 23. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • Array Index Key Scan (4.5) CREATE INDEX ts_aix1 ON `travel- sample` (DISTINCT ARRAY v.day FOR v IN schedule END) WHERE type = "route"; SELECT META(t).id FROM `travel-sample` t WHERE type = "route" AND ANY v IN schedule SATISFIES v.day IN [1,2] END; { "#operator": "DistinctScan", "scan": { "#operator": "IndexScan", "index": "ts_aix1", "keyspace": "travel-sample", "spans": [ { "Range": { "High": [ "1" ], "Inclusion": 3, "Low": [ "1" ] } }, { "Range": { "High": [ "2" ], "Inclusion": 3, "Low": [ "2" ] } } ] } }
  24. 24. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • IndexCountScan (4.5) CREATE INDEX ts_c_id ON `travel- sample`(country, id) WHERE type = "airline"; EXPLAIN SELECT COUNT(1) AS count FROM `travel-sample` t WHERE type = "airline" AND country = "United States" AND id BETWEEN 10 AND 1000; { "#operator": "IndexCountScan", "covers": [ "cover ((`t`.`country`))", "cover ((`t`.`id`))", "cover ((meta(`t`).`id`))" ], "index": "ts_c_id", "keyspace": "travel-sample", "spans": [ { "Range": { "High": [ ""United States"", "1000" ], "Inclusion": 3, "Low": [ ""United States"", "10" ] } } ] } { "#operator": "IndexCountProject", "result_terms": [ { "as": "count", "expr": "count(1)" } ] }
  25. 25. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • JOIN, LEFT JOIN. • NEST, LEFT NEST will have operator NEST. • LEFT side keyspace of JOIN can be covered. • Non-covered query. CREATE INDEX ts_rix1 ON `travel- sample` (DISTINCT ARRAY v.day FOR v IN schedule END, airlineid, schedule) WHERE type = "route"; EXPLAIN SELECT r, a FROM `travel-sample` r LEFT JOIN `travel-sample` a ON KEYS r.airlineid WHERE r.type = "route" AND ANY v IN r.schedule SATISFIES v.day = 1 END LIMIT 1; "scan": { "#operator": "IndexScan", "index": "ts_rix1", "keyspace": "travel-sample", "spans": [ { "Range": { "High": [ "1" ], "Inclusion": 3, "Low": [ "1" ] } } ] } { "#operator": "Join", "as": "a", "keyspace": "travel-sample", "namespace": "default", "on_keys": "(`r`.`airlineid`)", "outer": true }
  26. 26. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • JOIN LEFT side covered query. CREATE INDEX ts_rix1 ON `travel- sample` (DISTINCT ARRAY v.day FOR v IN schedule END, airlineid, schedule) WHERE type = "route"; EXPLAIN SELECT r.schedule, META(r).id FROM `travel-sample` r LEFT JOIN `travel-sample` a ON KEYS r.airlineid WHERE r.type = "route" AND ANY v IN r.schedule SATISFIES v.day = 1 END LIMIT 1; "scan": { "#operator": "IndexScan", "covers": [ "cover ((distinct (array (`v`.`day`) for `v` in (`r`.`schedule`) end)))", "cover ((`r`.`airlineid`))", "cover ((`r`.`schedule`))", "cover ((meta(`r`).`id`))" ], "filter_covers": { "cover ((`r`.`type`))": "route" } "index": "ts_rix1", "keyspace": "travel-sample", "spans": [ { "Range": { "High": [ "1" ], "Inclusion": 3, "Low": [ "1" ] } } ] } { "#operator": "Join", "as": "a", "keyspace": "travel-sample", "namespace": "default", "on_keys": ”cover ((`r`.`airlineid`))", "outer": true }
  27. 27. ©2017 Couchbase Inc. How does this work? CREATE INDEX brewery_state ON `beer-sample`(state); CREATE INDEX beer_brewery_id ON `beer-sample`(brewery_id); SELECT brewery.name as breweryname, ARRAY_AGG(beer.name) beerlist FROM `beer-sample` brewery LEFT OUTER JOIN `beer-sample` beer ON KEY beer.brewery_id FOR brewery WHERE brewery.state = "California" GROUP BY brewery.name; brewerybrewery_state [“California”, “21st_sfo_ale”] beer_brewery_id (brewery_id = “21st_sfo_ale”) beer [“21st_sfo_ale” , ”pale_ale”] (state = ‘California’)
  28. 28. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • Index JOIN, LEFT Index JOIN. • Index NEST, LEFT Index NEST will have operator NEST. • LEFT side keyspace of JOIN can be covered. • In 4.6 RIGHT side keyspace of Index JOIN can be covered. NOT for Index NEST. CREATE INDEX ts_c ON `travel- sample`(country) WHERE type = "airline" ; CREATE INDEX ts_aid ON `travel- sample`(airlineid, airline) WHERE type = "route" ; EXPLAIN SELECT r, a FROM `travel-sample` a JOIN `travel-sample` r ON KEY r.airlineid FOR a WHERE r.type = "route" AND a.type = "airline" AND a.country = "United States" LIMIT 1; { "#operator": "IndexScan", "index": "ts_c", "keyspace": "travel-sample", "spans": [ { "Range": { "High": [ ""United States"" ], "Inclusion": 3, "Low": [ ""United States"" ] } } ] } { "#operator": "IndexJoin", "as": "r", "for": "a", "keyspace": "travel-sample", "on_key": "(`r`.`airlineid`)", "scan": { "index": "ts_aid" } }
  29. 29. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • Covered Index Join (4.5) CREATE INDEX ts_c ON `travel- sample`(country) WHERE type = "airline" ; CREATE INDEX ts_aid ON `travel- sample`(airlineid, airline) WHERE type = "route" ; EXPLAIN SELECT a.country, r.airline FROM `travel-sample` a JOIN `travel-sample` r ON KEY r.airlineid FOR a WHERE r.type = "route" AND a.type = "airline" AND a.country = "United States"; { "#operator": "IndexScan", "covers": [ "cover ((`a`.`country`))", "cover ((meta(`a`).`id`))" ], "filter_covers": { "cover ((`a`.`type`))": "airline" }, "index": "ts_c", "keyspace": "travel-sample", "spans": [ { "Range": { "High": [ ""United States"" ], "Inclusion": 3, "Low": [ ""United States"" ] } } ] } { "#operator": "IndexJoin", "as": "r", "for": "a", "keyspace": "travel-sample", "on_key": "(`r`.`airlineid`)", "scan": { "index": "ts_aid" } }
  30. 30. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • LEFT & RIGHT Covered Index Join (4.6) CREATE INDEX ts_c ON `travel- sample`(country) WHERE type = "airline" ; CREATE INDEX ts_aid ON `travel- sample`(airlineid, airline) WHERE type = "route" ; EXPLAIN SELECT a.country, r.airline FROM `travel-sample` a JOIN `travel-sample` r ON KEY r.airlineid FOR a WHERE r.type = "route" AND a.type = "airline" AND a.country = "United States"; {"#operator": "IndexScan", "covers": [ "cover ((`a`.`country`))", "cover ((meta(`a`).`id`))" ], "filter_covers": { "cover ((`a`.`type`))": "airline" }, "index": "ts_c” ……… } { "#operator": "IndexJoin", "as": "r", "for": "a", "keyspace": "travel-sample", "on_key": "cover ((`r`.`airlineid`))", "scan": { "covers": [ "cover ((`r`.`airlineid`))", "cover ((`r`.`airline`))", "cover ((meta(`r`).`id`))" ], "filter_covers": { "cover ((`r`.`type`))": "route" }, "index": "ts_aid" } }
  31. 31. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggregate Scan • Aggregate CREATE INDEX ts_c_id ON `travel- sample`(country, id) WHERE type = "airline"; EXPLAIN SELECT country, count(1) count FROM `travel-sample` t WHERE type = "airline" GROUP BY country; { "#operator": "InitialGroup", "aggregates": [ "count(1)" ], "group_keys": [ "(`t`.`country`)" ] } { "#operator": "FinalGroup", "aggregates": [ "count(1)" ], "group_keys": [ "(`t`.`country`)" ] } { "#operator": "IntermediateGroup", "aggregates": [ "count(1)" ], "group_keys": [ "(`t`.`country`)" ] }
  32. 32. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • Sort • Offset • Limit CREATE INDEX ts_c_id ON `travel- sample`(country, id) WHERE type = "airline"; EXPLAIN SELECT country, id, name FROM `travel-sample` t WHERE type = "airline" AND country = "United States" AND id BETWEEN 10 AND 1000 ORDER BY country, id DESC OFFSET 5 LIMIT 10; { "#operator": "Order", "limit": "10", "offset": "5", "sort_terms": [ { "expr": "(`t`.`country`)" }, { "desc": true, "expr": "(`t`.`id`)" } ] } { "#operator": "Offset", "expr": "5" } { "#operator": "Limit", "expr": "10" }
  33. 33. ©2017 Couchbase Inc. Understanding Query Plan Fetch Parse Plan Join Filter Offset Limit Project Sort Aggreg ate Scan • Project CREATE INDEX ts_c_id ON `travel- sample`(country, id) WHERE type = "airline"; EXPLAIN SELECT country, id, name FROM `travel-sample` t WHERE type = "airline" AND country = "United States" AND id BETWEEN 10 AND 1000 ORDER BY country, id DESC OFFSET 5 LIMIT 10; { "#operator": "InitialProject", "result_terms": [ { "expr": "(`t`.`country`)" }, { "expr": "(`t`.`id`)" }, { "expr": "(`t`.`name`)" } ] } { "#operator": "FinalProject" }
  34. 34. ©2017 Couchbase Inc.©2017 Couchbase Inc. 34 How Query Plan Generated
  35. 35. ©2017 Couchbase Inc. Query Plan Generation • Rule based optimization • The index chosen by N1QL will satisfy the query • Each query block has is optimized separately & stitched. 35
  36. 36. ©2017 Couchbase Inc. Query Plan Generation • Index Selection Criteria • If the query contains a USE KEYS clause, no index scan or full / primary scan is performed. The input document keys are taken directly from the USE KEYS clause. • If there is an explicit or implicit query predicate: • Covering secondary scan; • Regular secondary scan -- longest satisfying keys, intersect scan; • UNNEST scan; • Regular primary scan • If the query does not contain a predicate • covering primary scan; • regular primary scan. • If a primary scan is selected, and there is no primary index available, the query errors out. 36
  37. 37. ©2017 Couchbase Inc. Query Plan Generation • Covering Primary Scan • A covering primary scan is a primary scan that does not perform a subsequent document fetch. It is used for queries that need a full / primary scan and only reference META().id. • Regular Primary Scan • A regular primary scan also performs a subsequent document fetch. It is used for queries that need a full / primary scan and reference some document data other than META().id. 37 SELECT META(t).id FROM `travel-sample` t; SELECT META(t).cas FROM `travel-sample` t; SELECT * FROM `travel-sample` t; SELECT t.type FROM `travel-sample` t;
  38. 38. ©2017 Couchbase Inc. Query Plan Generation Preliminaries : • Online indexes • Only online indexes are considered • Preferred indexes • USE INDEX hint is provided the indexes in that list are only considered • Satisfying Index condition • Partial / filtered indexes that index condition is super set of query predicate are considered • Satisfying Index keys • Indexes whose leading keys satisfy query predicate are considered • Longest satisfying index keys • Redundancy is eliminated båy keeping longest satisfying index keys in same order. • Index with satisfying keys (a,b,c) is retained over index with satisfying (a,b) 38
  39. 39. ©2017 Couchbase Inc. Query Plan Generation Covering Secondary Scan • Each satisfied index with most number of index keys is examined for query coverage • Shortest covering index will be used. Regular Secondary Scan • Indexes in with most number of matching index keys are used • When more than one index are qualified, IntersectScan is used. • To avoid IntersectScan provide hint with USE INDEX. 39 CREATE INDEX ts_name ON `travel-sample`(country, name) WHERE type = "airline"; SELECT country, name, type, META().id FROM `travel-sample` WHERE type = "airline" AND country = "United States"; SELECT country, name, type, id FROM `travel-sample` WHERE type = "airline" AND country = "United States";
  40. 40. ©2017 Couchbase Inc. Query Plan Generation UNNEST Scan • Only array indexes are considered. And only queries with UNNEST clauses are considered IndexCountScan • Queries with single projection of COUNT aggregate, NO JOIN’s, GROUP BY is considered • Chosen Index needs to be covered with single range, exact range will be able to push to indexer and argument to COUNT needs to be constant or leading key 40 CREATE INDEX ts_c ON `travel-sample`(country) WHERE type = "airline"; SELECT COUNT(1) AS count FROM `travel-sample` WHERE type = "airline" AND country = "United States";
  41. 41. ©2017 Couchbase Inc. Query Plan Generation • In order to satisfy anAND predicate, an index only needs to satisfy any term in the AND predicate. • In order to satisfy an OR predicate, an index must satisfy every term in the OR predicate. 41
  42. 42. ©2017 Couchbase Inc. 42 Query Plan Generation Index Satisfies these predicates Index doesn’t satisfy these predicates CREATE INDEX idx_dept ON employee( dept_id, hire_date, base_comp ); hire_date >= "2010" AND dept_id = "xyz" hire_date >= "2010" AND dept_id = "xyz" AND last_name = "Smith" dept_id = "abc" AND base_comp > 50000 dept_id = "abc" AND bonus > 2000 dept_id = "abc" OR dept_id = "xyz" (hire_date >= "2010" AND dept_id >= "xyz") OR (hire_date >= "2010" AND dept_id = "abc") dept_id = "abc" OR (hire_date >= "2010" AND dept_id >= "xyz") (hire_date >= "2010" AND dept_id >= "xyz") OR (dept_id = "abc" AND bonus > 2000) hire_date >= "2010" AND base_comp > 50000 hire_date >= "2010" AND bonus > 2000 bonus > 2000 AND base_comp > 50000 bonus > 2000 AND last_name = "Smith" dept_id = "abc" OR hire_date >= "2010" dept_id = "abc" OR dept_id = "xyz" OR base_comp > 50000 dept_id = "abc" OR dept_id = "xyz" OR bonus > 2000 hire_date > "2010" OR base_comp > 50000
  43. 43. ©2017 Couchbase Inc.©2017 Couchbase Inc. 43 Query OptimizationTo Exploit Index Features
  44. 44. ©2017 Couchbase Inc. Query OptimizationTo Exploit Index Features • N1QL Query engine can exploit the following index features. • By creating right index and query that takes advantage performs better. • Pushing predicates hint to Index • Index Filters • Using Index Order • Pushing LIMIT to Index • Pushing Offset as LIMIT (limit = limit + offset) to Index • Top n Heap for Order • Index Count (GSI Index only) • Pushing MIN to Index • Duplicate Indexes and Load balancing • USE INDEX 44
  45. 45. ©2017 Couchbase Inc. Query OptimizationTo Exploit Index Features USE INDEX • Queries can specify which index to use with USE INDEX clause • Multiple indexes qualify for query and one performs better than others • To avoid Intersect Scan • If there are many indexes on data source and queries are ad-hoc USE INDEX can reduce prepare time Pushing predicates • Predicates are pushed to indexer when possible • These can be examined as spans in Index section of the EXPLAIN plan • The spans can be exact or superset and never be subset of the predicates • If the spans are exact the applying query predicates again will not eliminate the document Index Filters • If Index filter has equal on expression, that expression not required to include in the index keys.This allows keep the index LEAN . Queries can exploit index filter condition for covered queries. 45 CREATE INDEX ts_c ON `travel-sample`(country) WHERE type = "airline" ; SELECT type, country FROM `travel-sample` WHERE type = "airline" AND country = "United States";
  46. 46. ©2017 Couchbase Inc. Query OptimizationTo Exploit Index Features Using Index Order • Query ORDER BY list matches with index keys list order left to right with single range span • Query doesn’t have any JOINs, GROUP BY or other clauses that can change the order produced by indexer • The query can exploit index order and avoid expensive sort and fetching lot of data unnecessarily in some cases 46 CREATE INDEX ts_c_id ON `travel-sample`(country, id) WHERE type = "airline" ; SELECT country, id FROM `travel-sample` WHERE type = "airline" AND country = "United States” ORDER BY id; --4.6 Uses Index order without any change SELECT country, id FROM `travel-sample` WHERE type = "airline" AND country = "United States” ORDER BY country, id; --4.5 Uses Index order CREATE INDEX ts_c_nid ON `travel-sample`(country, -id) WHERE type = "airline" ; SELECT country, -(-id) FROM `travel-sample` WHERE type = "airline" AND country = "United States” ORDER BY country, -id; -- id prodcued in DESC order
  47. 47. ©2017 Couchbase Inc. Query OptimizationTo Exploit Index Features Push LIMIT to Index • Pushing the LIMIT is hint to indexer when to stop • Pushing LIMIT can be done only when the predicates are pushed to indexer, spans are exact and matches index order • In case multiple spans LIMIT can be pushed to each span 47 CREATE INDEX ts_c_id ON `travel-sample`(country, id) WHERE type = "airline" ; SELECT country, id FROM `travel-sample` WHERE type = "airline" AND country = "United States" AND id BETWEEN 0 AND 1000 LIMIT 10; --LIMIT can be pushed SELECT country, id FROM `travel-sample` WHERE type = "airline" AND country IN ["United States", "United Kingdom"] AND id BETWEEN 0 AND 1000 LIMIT 10; --LIMIT can be pushed CREATE INDEX ts_c_id ON `travel-sample`(id, country) WHERE type = "airline" ; SELECT country, id FROM `travel-sample` WHERE type = "airline" AND country = "United States" AND id BETWEEN 0 AND 1000 LIMIT 10; --LIMIT can’t be pushed. Indexer produces {"id":11, "country":"France"}
  48. 48. ©2017 Couchbase Inc. Query OptimizationTo Exploit Index Features Push Offset • If LIMIT is present offset is added to limit and pushed limit to Indexer Top-n Heap for Order • Query requires order and has LIMIT less than 64K it usesTop-n Heap to eliminate unwanted documents before order. Index Count • Count is done by indexer • Only GSI index supports Index count • Queries with single projection of COUNT aggregate, NO JOIN’s, GROUP BY is considered • Chosen Index needs to be covered with single range, exact range will be able to push to indexer and argument to COUNT needs to be constant or leading key 48 CREATE INDEX ts_c ON `travel-sample`(country) WHERE type = "airline"; SELECT COUNT(1) AS count FROM `travel-sample` WHERE type = "airline" AND country = "United States";
  49. 49. ©2017 Couchbase Inc. Query OptimizationTo Exploit Index Features MIN • Queries with single projection of MIN aggregate, NO JOIN’s, GROUP BY is considered. • Chosen Index needs to be covered with single range, exact range will be able to push to indexer and argument to MIN needs to be leading key. • If the required conditions are satisfied limit 1 pushed to indexer. 49 CREATE INDEX ts_c ON `travel-sample`(country) WHERE type = "airline"; SELECT MIN(country) FROM `travel-sample` t WHERE type = "airline" AND country > "United States"; { "#operator": "IndexScan", "covers": [ "cover ((`t`.`country`))", "cover ((meta(`t`).`id`))" ], "filter_covers": { "cover ((`t`.`type`))": "airline" }, "index": "ts_c", "keyspace": "travel-sample", "limit": "1", "spans": [ { "Range": { "Inclusion": 0, "Low": [ ""United States"" ] } } ] }
  50. 50. ©2017 Couchbase Inc. Query OptimizationTo Exploit Index Features Duplicate Indexes and Load balancing • Allows duplicate indexes (i.e. different name but identical schema) on same or different Indexer services • Query Service will choose one of the index during prepare time • During execution Indexer client does load balancing and uses one of the indexer to service query 50
  51. 51. ©2017 Couchbase Inc. QueryTuning Checklist • Query should have predicates to avoid primary index scan. • Explore all index options (composite secondary index, partial composite index, covered partial composite index ,…) • Include as many predicate keys as possible in leading index keys • Query processing is bit more efficient when there is equality predicates on leading index keys • Explore avoiding Intersect scan. If required provide hint with USE INDEX • For ANY predicate clause, use ARRAY index keys. • Explore using index key order and pushing limit, offset to indexer • Rewrite query if required • Use array fetch when possible • Execute query and look the monitoring stats for each phase of query (ex: system:completed_requests) and tune it. • Check the final query plan through the explain • Set pretty=false in Query Service or query parameter • Increase parallel processing via max_parallelism • Fetching large set of data for non covered index increase pipeline-cap, pipeline-batch in Query-Service • Duplicate indexes and memory optimized indexes • Add more Query nodes or Indexer nodes . 51
  52. 52. ©2017 Couchbase Inc.©2017 Couchbase Inc. 52 Q&A
  53. 53. ©2017 Couchbase Inc. 53 Asif Kazi Director asif@couchbase.com Keshav Murthy Senior Director keshav@couchbase.com
  54. 54. ©2017 Couchbase Inc.©2017 Couchbase Inc. ThankYou! 54
  55. 55. ©2017 Couchbase Inc. 55 Share your opinion on Couchbase 1. Go here: http://gtnr.it/2eRxYWn 2. Create a profile 3. Provide feedback (~15 minutes)
  56. 56. ©2017 Couchbase Inc.©2017 Couchbase Inc. Follow us on social media Twitter @couchbase Facebook /couchbase Instagram @couchbase Linkedin /company/couchbase

×