Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

N1QL+GSI: Language and Performance Improvements - Connect Silicon Valley 2018

59 views

Published on

Speaker: Keshav Murthy, Senior Director, Couchbase R&D, Couchbase

N1QL gives developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data. We’ll begin this session with a brief overview of N1QL and then explore some key enhancements we’ve made in the latest versions of Couchbase Server. Couchbase Server 5.0 has language and performance improvements for pagination, index exploitation, integration, index availability, and more. Couchbase Server 5.5 will offer even more language and performance features for N1QL and global secondary indexes (GSI), including ANSI joins, aggregate performance, index partitioning, auditing, and more. We’ll give you an overview of the new features as well as practical use case examples.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

N1QL+GSI: Language and Performance Improvements - Connect Silicon Valley 2018

  1. 1. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. N1QL+GSI: LANGUAGE AND PERFORMANCE IMPROVEMENTS IN COUCHBASE 5.0 AND 5.5 September, 19th, 2018 Silicon Valley, California Keshav Murthy | Senior Director, Couchbase R&D
  2. 2. AGENDA 02 03 N1QL & GSI Features in Couchbase 5.0 N1QL & GSI Features in Couchbase 5.5 01 N1QL for newbies : 5 minutes
  3. 3. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 1 N1QL FOR NEWBIES
  4. 4. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 4 ResultSet Relations/Tuples
  5. 5. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 5 { "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ] } LoyaltyInfo Results Orders CUSTOMER • NoSQL systems provide specialized APIs • Key-Value get and set • Each task requires custom built program • Should test & maintain it
  6. 6. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 6 Find High-Value Customers with Orders > $10000 Query customer objects from database • Complex codes and logic • Inefficient processing on client side For each customer object Find all the order objects for the customer Calculate the total amount for each order Sum up the grand total amount for all orders If grand total amount > $10000, Extract customer data Add customer to the high-value customer list Sort the high-value customer list LOOPING OVER MILLIONS OF CUSTOMERS IN APPLICATION!!!
  7. 7. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 7 { "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ] } LoyaltyInfo ResultDocuments Orders CUSTOMER
  8. 8. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 8 N1QL = SQL + JSON Give developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data.
  9. 9. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 9 Why SQL for NoSQL?
  10. 10. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 10 N1QL : Data Types from JSON Data Type Example Numbers { "id": 5, "balance":2942.59 } Strings { "name": "Joe", "city": "Morrisville" } Boolean { "premium": true, "balance pending": false} Null { "last_address": Null } Array { "hobbies": ["tennis", "skiing", "lego"]} Object { "address": {"street": "1, Main street", "city": Morrisville, "state":"CA", "zip":"94824"}} MISSING Arrays of objects of arrays [ { "type": "visa", "cardnum": "5827-2842-2847-3909", "expiry": "2019-03" }, { "type": "master", "cardnum": "6274-2542-5847-3949", "expiry": "2018-12" } ]
  11. 11. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 11 N1QL: Data Manipulation Statements •SELECT Statement- •UPDATE … SET … WHERE … •DELETE FROM … WHERE … •INSERT INTO … ( KEY, VALUE ) VALUES … •INSERT INTO … ( KEY …, VALUE … ) SELECT … •MERGE INTO … USING … ON … WHEN [ NOT ] MATCHED THEN … Note: Couchbase provides per-document atomicity.
  12. 12. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 12 N1QL: SELECT Statement SELECT * FROM customers c WHERE c.address.state = 'NY' AND c.status = 'premium' ORDER BY c.address.zip Project Everything From the bucket customers Sort order Predicate (Filters)
  13. 13. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 13 N1QL: SELECT Statement SELECT customers.id, customers.NAME.lastname, customers.NAME.firstname Sum(orderline.amount) FROM orders UNNEST orders.lineitems AS orderline INNER JOIN customers ON KEYS orders.custid WHERE customers.state = 'NY' GROUP BY customers.id, customers.NAME.lastname, customers.NAME.firstname HAVING sum(orderline.amount) > 10000 ORDER BY sum(orderline.amount) DESC • Dotted sub-document reference • Names are CASE-SENSITIVE UNNEST to flatten the arrays` JOINS with Document KEY of customers
  14. 14. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 14 N1QL: SELECT Statement Highlights • Querying across relationships • INNER JOIN, LEFT OUTER JOIN, RIGHT OUTER JOIN (5.5) • Subqueries • Aggregation (HUGE PERFORMANCE IMPROVEMENT IN 5.5) • MIN, MAX • SUM, COUNT, AVG, ARRAY_AGG [ DISTINCT ] • Combining result sets using set operators • UNION, UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL
  15. 15. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 15 N1QL : Query Operators [ 1 of 2 ] •USE KEYS … • Direct primary key lookup bypassing index scans • Ideal for hash-distributed datastore • Available in SELECT, UPDATE, DELETE •JOIN … ON KEYS … • Nested loop JOIN using key relationships • 5.5: HASH JOIN for significantly better performance with larger amount of data. • Ideal for hash-distributed datastore • Current implementation supports INNER and LEFT OUTER joins
  16. 16. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 16 N1QL : Query Operators [ 2 of 2 ] • UNNEST • Flattening JOIN that surfaces nested objects as top-level documents • Ideal for decomposing JSON hierarchies • Example: Flatten customer document to customer-orders •NEST • Does the opposite of UNNEST • Special JOIN that embeds external child documents under their parent • Ideal for JSON encapsulation •JOIN, NEST, and UNNEST can be chained in any combination
  17. 17. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 17 UNNEST { "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ] } "c": { "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ] }, "type" : "master", "cardnum" : "6274-2842-2847-3909” } SELECT c, b.type, b.cardnum FROM customer c UNNEST c.Billing AS b "c": { "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ] }, "type" : "visa", "cardnum" : "5827-2842-2847-3909” }
  18. 18. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 18 N1QL : Expressions for JSON Ranging over collections • WHERE ANY c IN children SATISFIES c.age > 10 END • WHERE EVERY r IN ratings SATISFIES r > 3 END Mapping with filtering • ARRAY c.name FOR c IN children WHEN c.age > 10 END Deep traversal, SET, and UNSET • WHERE ANY node WITHIN request SATISFIES node.type = “xyz” END • UPDATE doc UNSET c.field1 FOR c WITHIN doc END Dynamic Construction • SELECT { “a”: expr1, “b”: expr2 } AS obj1, name FROM … // Dynamic object • SELECT [ a, b ] FROM … // Dynamic array Nested traversal • SELECT x.y.z, a[0] FROM a.b.c … IS [ NOT ] MISSING • WHERE name IS MISSING
  19. 19. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 19 DELETE THIS Ranging over collections • WHERE ANY c IN children SATISFIES c.age > 10 END • WHERE EVERY r IN ratings SATISFIES r > 3 END Mapping with filtering • ARRAY c.name FOR c IN children WHEN c.age > 10 END Deep traversal, SET, and UNSET • WHERE ANY node WITHIN request SATISFIES node.type = “xyz” END • UPDATE doc UNSET c.field1 FOR c WITHIN doc END Dynamic Construction • SELECT { “a”: expr1, “b”: expr2 } AS obj1, name FROM … // Dynamic object • SELECT [ a, b ] FROM … // Dynamic array Nested traversal • SELECT x.y.z, a[0] FROM a.b.c … IS [ NOT ] MISSING • WHERE name IS MISSING
  20. 20. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 20 Architecture App App Couchbase App App Couchbase App App Couchbase App App Couchbase Couchbase Cluster App App Couchbase Couchbase Single Node Deployment Couchbase Cluster Deployment
  21. 21. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 21 Couchbase Server Cluster Service Deployment STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service STORAGE Couchbase Server 2 Managed Cache Cluster ManagerCluster Manager Data Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Index Service Managed Cache Storage Managed Cache Storage Storage STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Analytics Service Storage Managed Cache Managed Cache SDK SDK
  22. 22. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 22 Couchbase Server Cluster Service Deployment STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service STORAGE Couchbase Server 2 Managed Cache Cluster ManagerCluster Manager Data Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Query Service N1QL for Query STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Index Service Managed Cache Storage Managed Cache Storage Storage STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Analytics Service N1QL for Analytics Storage Managed Cache Managed Cache SDK SDK
  23. 23. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 23 N1QL: Query Execution Flow Clients 1. Submit the query over REST API 8. Query result 2. Parse, Analyze, create Plan 7. Evaluate: Documents to results 3. Scan Request; index filters 6. Fetch the documents Index Service Query Service Data Service 4. Get qualified doc keys 5. Fetch Request, doc keys SELECT c_id, c_first, c_last, c_max FROM CUSTOMER WHERE c_id = 49165; { "c_first": "Joe", "c_id": 49165, "c_last": "Montana", "c_max" : 50000 }
  24. 24. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 24 N1QL: Inside the Query Service Client FetchParse Plan Join Filter Pre-Aggregate Offset Limit ProjectSortAggregateScan Query Service Index Service Data Service
  25. 25. READ THIS!
  26. 26. DO THIS: http://query-tutorial.couchbase.com
  27. 27. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. * N1QL: PATH OF PROGRESS
  28. 28. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 28 N1QL features in Couchbase releases 28 Couchbase 4.0: N1QL GA Query language for JSON, Integrated Query Service, Global Secondary Index, REST API, Simba ODBC, JDBC Drivers Couchbase 4.1: INSERT, UPDATE, DELETE, MERGE Covering Index Optimization Couchbase 4.1.1: Index JOINs Couchbase 4.5: Array Indexes, Workbench, CBQ Shell++, INFER Memory Optimized Index, IndexScanCount YCSB Performance Optimizations++, Language++ Couchbase 4.5.1: Pretty=false; Fetch; SUFFIXES; Index Selection; UPDATE improvement Oct 2015 Dec 2015 March 2016 June 2016 Sep 2016
  29. 29. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 29 N1QL features in Couchbase releases. 29 Couchbase 4.6.1: TOKENS (Simple Search/Faster LIKE), Optimizer improvements Couchbase 5.0: Subqueries over nested data ; Pagination; RBAC; Curl, Super Charged Indexing; Monitoring & Profiling; New workbench, UI, monitoring, profiler, visual EXPLAIN Performance++, Bitwise functions Q1 2017 4Q 2017 Couchbase 4.6.2: Optimizer improvements, intersect scan performanceQ2 2017 Couchbase 5.5: ANSI joins, Hash join, grouping & aggregate performance, index partitioning, general performance improvement, x.509, backfill, alter index, visual explain.2Q 2018 Couchbase 6.0: Now in BETA N1QL for Analytics3Q 2018
  30. 30. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 30 SQL is English for Relational Database SQL Invented by Don Chamberlin & Raymond Boyce at IBM N1QL, based on SQL, is English for JSON N1QL was invented by Gerald Sangudi at Couchbase SQL Instance Database Table Row Column Index Datatypes N1QL Cluster Bucket Bucket, Keyspace Document Attribute Index JSON Datatypes SQL Input and Output: Set(s) of Tuples N1QL STMT CREATE BUCKET CREATE INDEX None SELECT INSERT UPDATE DELETE MERGE Subqueries JOIN: INNER, LEFT, RIGHT OUTER only GROUP BY ORDER BY OFFSET, LIMIT EXPLAIN PREPARE EXECUTE GRANT ROLE REVOKE ROLE INFER PREPARE EXECUTE FLUSH Tuples SQL Model Set of JSON N1QL for Query Model Set of Tuples Set of JSON N1QL Tooling Web Console Monitoring Profiling Dev workbench SDK Simba, Cdata BI Slamdata SQL Tooling ODBC, JDBC, .NET Hibernate BI Tools erwin TOAD N1QLResources query.couchbase.com SQL Indexes Primary Key Secondary Key Composite Range Partitioned Expression-Function Spatial Search N1QL Indexes Primary Secondary Composite Hash Partition Partial Expression-Function Array Index Replica(HA) Adaptive Spatial SQL Logic 3 valued logic TRUE, FALSE, NULL/UNKNOWN N1QL Logic 4 valued logic TRUE, FALSE, NULL/UNKNOWN, MISSING SQL Transactions ACID Multi-Statement Savepoints Commit/Rollback Redo, Undo N1QL Transactions Single Document atomicity SQL Datatypes Numeric Decimal Character Date Time Timezone BLOB Spatial JSON N1QL Datatype Numeric Boolean Character Array Object Null JSON Conversion Functions SQL Optimizer Rule Based Cost Based Index Selection Query Rewrites NL, Hash, Merge join N1QL Optimizer Rule based Index Selection NL, Hash join SQL ACID ATOMIC Consistent Isolated Durable N1QL BASE Single doc Atomic Consistent Data* Optimistic Concurrency N1QL Index Scan Consistency* Unbounded AT_PLUS REQUEST_PLUS SQL Engine (SMP Scale UP) N1QL Engine (MPP Cluster Scale OUT) Additional SQL Features Triggers Stored Procedures XML Constraints RAC SQL STMT CREATE TABLE CREATE INDEX ALTER TABLE SELECT INSERT UPDATE DELETE MERGE Subqueries JOINS GROUP BY ORDER BY OFFSET, LIMIT EXPLAIN PLAN PREPARE EXECUTE GRANT REVOKE DESCRIBE PREPARE EXECUTE TRUNCATE N1QL Input and Output: Set(s) of JSON Services Search Eventing (~Post action trigger)
  31. 31. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 3 N1QL FEATURES IN COUCHBASE 5.0
  32. 32. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 32 Couchbase N1QL and GSI features Query-Indexing Features • Large Indexing Keysize • Index key collation: ASC, DESC on each key • Index replicas, just like data replication • New storage engine: Plasma Query Language & Infrastructure • Subquery Expressions • Additional Date & time functions • Bitwise functions • CURL() within N1QL Query Optimizer • Complex Filters Pushdown • Pagination optimization • Optimization for ASC, DESC keys • Query-Index API optimization (projection, etc.) • Index projections, Intersect scans • Adaptive Indexes Security, Administration & Functionality • Security: RBAC: Statement level security • Query Monitoring, Profiling with UI • Query work bench and UI: Fully upgraded • Query UI: Visual Explain • Query on Ephemeral buckets • Application Continuity, Seamless Upgrade Performance • Core daily workload • YCSB • YCSB-JSON for Engagement Database http://query.couchbase.com
  33. 33. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 33 Query-Indexing Enhancements Index key collation: ASC, DESC on each key • Prior to 5.0, each index key was sorted and kept in ASCENDING order only • To sort the key in descending order, you did • CREATE INDEX i1 ON t(c1 ASC, -c2, -c3) • SELECT * FROM t WHERE c1 = 10 and -c2 > -20 ORDER BY c1, -c2 • Query formulations becomes confusing • Cannot use this trick on all data types and expressions In Couchbase 5.0: • CREATE INDEX i1 ON t(c1 ASC, c2 DESC, c3 DESC) • SELECT * FROM t WHERE c1 = 10 and c2 < 20 ORDER BY c1,c2 DESC • You need to create an index to match the ORDER BY order • Reverse scans are still unsupported
  34. 34. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 34 Query-Indexing Enhancements Large Indexing Keysize • Prior to 5.0, the sum of index key size could be up to 4096 bytes • This was controlled by the setting • For ARRAY keys, sum of all array key sizes could be up to 10240. • This is controlled by the setting max_array_seckey_size In Couchbase 5.0: • The total keysize could be pretty high – high up to 20 MB • This is true for single key, composite key, expressions and array indexes as well. • Simply do nothing, except create the index and issue the query. • The index entries that exceed 20MB will still generate error in the index log
  35. 35. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 35 Query-Indexing Enhancements Index replicas, just like data replication • Prior to 5.0, you could create multiple indexes with same keys & condition • This is needed for load balancing and index high availability CREATE INDEX i1 ON t(c1, c2, c3) CREATE INDEX i2 ON t(c1, c2, c3) CREATE INDEX i3 ON t(c1, c2, c3) • Indexer automatically recognizes these to be equivalent and does load balancing on all o these. In Couchbase 5.0: • Simply create one index and set the num_replica at CREATE or ALTER time • CREATE INDEX i1 ON t(c1, c2, c3) WITH {"num_replica":2} • Number of replicas can be up to number of nodes in the cluster • You can ALTER the number of replica dynamically
  36. 36. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 36 Query-Indexing Enhancements New storage engine: Plasma • Index size can be arbitrarily large • Uses lock-free skip list • All the performance benefits of MOI – Memory Optimized Index • Automatically does IO as needed • From usage point of view: • Choose the standard secondary Index during installation • simply create any kind of index and use it.
  37. 37. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 37 Query Language & Infrastructure Subquery Expressions • Provides rich functionality and Powerful subquery-expressions • Can be used in FROM-clause, projection, LET/WHERE-clauses etc., SELECT word, cnt FROM ARRAY split(i) FOR i IN (SELECT raw name FROM `travel-sample` WHERE type = "hotel") END AS words UNNEST words w GROUP BY w LETTING cnt = COUNT(w) ORDER BY cnt DESC;
  38. 38. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 38 Query Language & Infrastructure Additional Date, time, timestamp functions • JSON does not directly support date and time related data types • Store the date and time in extended ISO 8901 format • "2017-10-16T18:44:43.308-07:00” • Need extract, conversion and arithmetic functions • Detailed article with all the functions and Oracle to Couchbase mapping https://dzone.com/articles/comparing-oracle-and-n1ql-support-for-the-date-tim • If you can’t do something, let us know!
  39. 39. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 39 Query Language & Infrastructure CURL() within N1QL • CURL (URL, [options]) • The first argument is the URL, which represents any URL that points to a JSON endpoint. • Only URLs with the http:// or the https:// protocol are supported. • Redirection is disabled. • The second argument is a list of options. • This is a JSON object that contains a list of curl options and their corresponding values. • For a full list of options that we support, please refer to the Dzone article on CURL in N1QL by Isha Kandaswamy •
  40. 40. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 40 CURL() from N1QL • Search for Santa Cruz in Spain using my Google dev api key SELECT CURL("GET","https://maps.googleapis.com/maps/api/geocode/json", {"data":"address=santa+cruz&components=country:ES&key=AIzaSyCT6niGCMsgegJkQ SYasfoLZ4_rSO59XQQ"}) ; • Live translate your text to another language. SELECT ginfo FROM ( SELECT r.content as english, curl("https://translation.googleapis.com/language/translate/v2?key=PUT YOUR KEYS HERE", {"request": "POST", "header":"Content-Type: application/json", "data": mydata }) AS french FROM `travel-sample` h USE KEYS "hotel_10142" UNNEST h.reviews r LET mydata = '{ "q":"' || r.content || '", "target": "fr"}') AS ginfo
  41. 41. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 41 Query Language & Infrastructure CURL() within N1QL
  42. 42. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 42 Query Language & Infrastructure BITWISE Functions • All bitwise functions can only take a number. All numbers are 64 bit signed numbers (integers). • If the Number is not an integer and for other data types, we throw an error. • When looking at the value in binary form, bit 1 is the Least Significant Bit (LSB) and bit 64 is the Most Significant Bit. • (MSB) Bit 64→ 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 ← Bit 1 (LSB) BitAND BitOR BitNOT BitXOR BitSHIFT BitSET BitCLEAR BitTEST/ IsBitSET
  43. 43. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 43 Query Optimizer & Execution: Stable Scans • IndexScan use to do single range scan (i.e single Span) • If the query has multiple ranges (i.e. OR, IN, NOT clauses) Query service used to do separate IndexScan for each range. • This causes Indexer can use different snapshot for each scan (make it unstable scan) • Number of IndexScans can grow and result increase in index connections • In 5.0.0 multiple ranges are passed into indexer and indexer uses same snapshot for all the ranges. • This makes stable Scan for given IndexScan (i.e. IndexScan2 in the EXPLAIN). • This will not make stable scan for query due to Subqueries, Joins etc • Example: CREATE INDEX ix1 ON default(k0); EXPLAIN SELECT META().id FROM default WHERE k0 IN [10,12,13];
  44. 44. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 44 Query Optimizer & Execution: Pushdown Composite Filters • For composite Index the spans that pushed to indexer contains single range for all composite keys together. • Indexer will not applying range for each part of the key separately. This result in lot of false positives. • In 5.0.0 with IndexScan2 we push the each index key range separately and indexer will apply keys separately. • This results in no/less false positives and aides push more information to indexer. CREATE INDEX ix1 ON default(k0,k1); EXPLAIN SELECT meta().id FROM default WHERE k0 BETWEEN 0 AND 100 AND k1 = 200;
  45. 45. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 45 Query Optimizer: ORDER, OFFSET, LIMIT pushdown • Pagination queries can contain any combination of ORDER, LIMIT, OFFSET clauses. • Performance of these queries are critical to applications. • When Predicates are completely and exactly pushed to indexer, by pushing offset, limit to indexer can improve query performance significantly. If that happened IndexScan2 section of EXPLAIN will have limit,offset. • If query ORDER BY matches index key order query can avoid sort and performance can be improved significantly. If that happened order operator is not present in the EXPLAIN. • Example: CREATE INDEX ix1 ON default(k0,k1); EXPLAIN SELECT meta().id FROM default WHERE k0 > 10 AND k1 > 20 ORDER BY k0 LIMIT 10 OFFSET 100;
  46. 46. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 46 Query Optimizer: MAX pushdown • If the MAX arguments matched with Index leading key exploit index order for MAX. • MAX can only DESC on index key. • MIN can only use ASC on index key. • Example : CREATE INDEX ix5 ON default(k0 DESC); SELECT MAX(k0) FROM default WHERE k0 > 10; • Above query able to exploit index order. In that case IndexScan2 section of EXPLAIN will have “limit” 1.
  47. 47. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 47 Query Optimizer: Index Projection • The index can have many keys but query might be interested only subset of keys. • By only requesting required information can save lot of network transportation, memory, cpu, backfill etc. All this can help in performance and scaling the cluster. • The requested information can be found in “IndexScan2” Section of EXPLAIN as “index_projection” "index_projection": { "entry_keys": [1, 5 ], "primary_key": true } CREATE INDEX ix1 ON default(k0,k1,k2,k3,k4, k5); EXPLAIN SELECT meta().id, k1, k5 FROM default WHERE k0 > 10 AND k1 > 20;
  48. 48. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 48 Query Optimizer: Index Projection CREATE INDEX ix1 ON default(k0,k1); Covered query SELECT k0 FROM default WHERE k0 = 10 AND k1 = 100; "index_projection": {"entry_keys": [0,1]} SELECT k0 FROM default WHERE k0 = 10; "index_projection": {"entry_keys": [0]} SELECT k0 ,META().idFROM default WHERE k0 = 10; "index_projection": {"entry_keys": [0],“primary_key”: true} Non-covered query SELECT k0 ,k5 FROM default WHERE k0 = 10 AND k1 = 100; "Index_projetion": { “primary_key”: true }
  49. 49. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 49 Query Execution: CAS & Expiration • In 5.0.0 META().cas, META().expiration can be indexed and used in queries. • Example: • CREATE INDEX ix1 ON default( meta().id, meta().cas, meta().expiration); • SELECT meta().id , meta().cas, meta().expiration FROM default where meta().id > "" • Note: META().expiration will work in covered queries. For non covered queries it gives 0
  50. 50. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 50 Query Execution: COUNT (DISTINCT expr) • If the expr matched with Index leading key COUNT DISTINCT can be pushed to indexer • Complete predicate needs to pushed to indexer exactly • No false positives are possible • No group or JOIN • Only single projection • Example : CREATE INDEX ix5 ON default(k0); SELECT COUNT(DISTINCT k0) FROM default WHERE k0 > 10; • Above query uses IndexCountDistinctScan2
  51. 51. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 51 Customer Scenario • Customer document has 100 fields • They have multiple business entities sharing the same data • Each entity want to FILTER, GROUP, ORDER on distinct criteria • For Index selection, order of the keys in the composite index is important. Fields: c1 through c100 Filter fields: c1 through c50 Group, order and projection: Any from c1 through c100 SELECT c1, c2, c3, COUNT(c10), SUM(c5) FROM CUSTOMER WHERE c4 = "CXT-MULTI" AND c8 = "iPhone6" AND c9 BETWEEN 10 IN 20 GROUP BY c1, c2, c3; SELECT c12, COUNT(c19), SUM(c15) FROM CUSTOMER WHERE c44 = "CXT-MULTI" AND c18 = "Gpixel 2" AND c29 BETWEEN 10 IN 20 GROUP BY c12;
  52. 52. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 52 Customer Scenario • What indexes to create for this? SELECT c1, c2, c3, COUNT(c10), SUM(c5) FROM CUSTOMER WHERE c4 = "CXT-MULTI" AND c8 = "iPhone6" AND c9 BETWEEN 10 IN 20 GROUP BY c1, c2, c3; CREATE INDEX i1 ON CUSTOMER(c8, c4, c9) CREATE INDEX i1 ON CUSTOMER(c8, c4, c9, c1, c2, c3, c10, c5); For Covering the query What about this? SELECT c12, COUNT(c19), SUM(c15) FROM CUSTOMER WHERE c44 = "CXT-MULTI" AND c18 = "Gpixel 2" AND c29 BETWEEN 10 IN 20 GROUP BY c12;
  53. 53. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 53 Large, wide, composite indexes Filter fields: c1 through c50 To support all combinations of 50 predicates via composite indexes, you’ll need LOT of indexes. 50! =30414093201713378043612608166064768844377641568 960512000000000000
  54. 54. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 54 Customer Scenario Solution: Intersection • Option 1 • Create indexes on individual fields • Scan individual indexes • Apply the full set of predicates (boolean expression from WHERE clause) • Then do the post processing. CREATE INDEX i1 on CUSTOMER(c1); CREATE INDEX i2 on CUSTOMER(c2); CREATE INDEX i3 on CUSTOMER(c3); • Option 2 • Too many indexes to maintain and manage. • Don’t even talk about equivalent indexes for each of these. CREATE INDEX i1to50 on CUSTOMER(DISTINCT PAIRS({c1, c2, c3, c4, c5,c6, c7, c8, c9, c10, c11, c23, c13, c14, …});
  55. 55. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 55 Solution: Intersection • Option 3 • Too many keys to manage/specify • The document is flexible. I want the index to be flexible. CREATE INDEX ixpairon CUSTOMER(DISTINCT PAIRS(self)); SELECT * FROM CUSTOMER WHERE a = 10 and b < 20 and c between 30 and 40; "#operator": "IntersectScan", "scans": [ { "#operator": "DistinctScan", "scan": { "#operator": "IndexScan2", "index": "ixpair", "index_id": "466c0c5c4c3b21c1", "index_projection": { "primary_key": true }, "keyspace": "test", "namespace": "default", "spans": [ { "exact": true, "range": [ { "high": "["a", 10]", "inclusion": 3, "low": "["a", 10]" } "range": [ { "high": "["b", 20]", "inclusion": 1, "low": "["b", false]" } "range": [ { "high": "[successor("c")]", "inclusion": 1, "low": "["c", 30]" } ]
  56. 56. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 56 Flexible Indexing • This is not a silver bullet, yet. • TRY THIS OUT • SIZING is a concern because we {“Key“:“value“} • Give us feedback
  57. 57. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 57 SECURITY : GRANT and REVOKE to roles • Query_select, query_insert, query_update, query_delete roles • Parameterized: query_select[customers] or query_insert[*] • Query_manage_index[foo] • Create, delete, build indexes on bucket foo • Query_system_catalog • Full access to the system tables (which are controlled now) • Query_external_access • Allows access to CURL() function (disabled by default) GRANT cluster_admin TO spock GRANT query_select ON default TO kirk REVOKE query_insert, query_delete ON bridge, engineering FROM mccoy, scotty
  58. 58. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 58 Monitoring in UI 58Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  59. 59. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 59 Profiling in UI 59Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  60. 60. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 60 Profiling • We can collect execution timings and document processed on a per operator basis • If the functionality is turned on, timings are reported • with the metrics at the end of execution • in system:active_requests • in system:completed_requests • Profiling is turned on • at the request level via the “profile” REST API parameter, EG from cbq: • set –profile timings; • at the node level via the “profile” command line parameter or admin settings REST API parameter • takes 3 values, “off”, “phases”, “timings” • “phases” supplies total times for each operator class • “timings” supplies detailed information for each operator
  61. 61. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 61 Profiling cbq> select * from `travel-sample` where source-airport is not missing; … "executionTimings": { "~children": [ { "#operator": "IndexScan2", "#stats": { "#itemsOut": 24024, "#phaseSwitches": 96099, "execTime": "55.370283ms", "kernTime": "5.397199311s" }, "index": "def_sourceairport", "index_id": "29702e564c9d2ca4", "index_projection": { "primary_key": true }, "keyspace": "travel-sample", "namespace": "default", "spans": [ { "exact": true, "range": [ { "inclusion": 1,
  62. 62. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 62 Developer Tooling
  63. 63. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 63 N1QL Performance: 5.0 vs. 4.5 • Run internally • YCSB is the public YCSB • other queries are written on Couchbase dataset • 50% higher throughput in YCSB workload E • 10-40x faster pagination queries • 10-30x better performance of queries with composite filters • 10-40x faster queries with COUNT function • 6-9x better performance of basic queries (Q1 & Q2) • 55x faster queries with UNNEST clause
  64. 64. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 64 N1QL Performance: 5.0 vs. 4.5 • Up to 10x faster array indexing • Fast text search with TOKENS() • 10x better performance of lookup and index joins • Query performance on Windows is on par with Linux • Up to 100K index scans per second in DGM scenarios
  65. 65. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 4 N1QL FEATURES IN COUCHBASE 5.5
  66. 66. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 66 Language Features • ANSI Joins support • INNER JOIN • LEFT OUTER • RIGHT OUTER • NEST and UNNEST • JOIN on arrays Security & Infra Features • PREPARE Infrastructure • N1QL Auditing • X.509 Support • IPV6 Support • Backfill Performance Features • GROUP BY performance • Aggregation performance • Index Partitioning • parallelization with Partitioned index • Query pipeline performance • Hash join • YCSB-JSON Query Workbench Features • Visual Explain improvements • Tabular document editor • Parameters for Query • Easy copy results to Excel N1QL & Indexing features in Couchbase 5.5
  67. 67. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 67 Vulcan Features: ANSI JOIN What? • ANSI standard for SQL join specification • Supported in all major relational databases Why? • Lowering barrier for migration to Couchbase • Especially from relational databases • Address limitation of N1QL joins • Lookup join and index join requires joining on document key • Parent-child or child-parent join only • Only equi-join • Proprietary syntax How? • ON-clause to specify join condition, which can be any expression
  68. 68. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 68 ANSI JOIN Examples SELECT c.lastName, c.firstName, c.customerId, o.ordersId FROM customer c INNER JOIN orders o ON c.customerId = o.customerId; SELECT c.lastName, c.firstName, c.customerId, o.ordersId FROM customer c LEFT OUTER JOIN orders o ON c.customerId = o.customerId SELECT c.lastName, c.firstName, c.customerId, o.ordersId FROM customer c RIGHT OUTER JOIN orders o ON c.customerId = o.customerId SELECT meta(brewery).id brewery_id, brewery.name brewery_name FROM `beer-sample` brewery INNER JOIN `beer-sample` beer ON beer.brewery_id = LOWER(REPLACE(brewery.name, " ", "_")) AND beer.type = "beer" WHERE brewery.type = "brewery" AND brewery.state = ”Kansas"
  69. 69. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 69 ANSI JOIN Syntax SELECT … FROM keyspace1 <join_type> JOIN keyspace2 ON <join_expression> WHERE <filter_expression> • Supported JOIN Types • INNER, LEFT OUTER, RIGHT OUTER • ON-clause specifies join condition • <join_expression> is evaluated at time of join • Can have multiple JOIN clauses in one query block • WHERE-clause specifies filter condition • <filter_expression> is evaluated after the join is done, or “post-join” • One per query block
  70. 70. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 70 ANSI JOIN : Designing Indexes • ANSI joins use indexes on both sides of the join. • JOINs are evaluated LEFT to RIGHT • For the first keyspace, optimizer chooses the index based on predicates in the WHERE clause and the ON clause. • For the second keyspace, only the ON clause is considered. • There should be an index on at least one key. • Composite lookup is used if there are matching composite keys • For hash join, need to have index on the build side. The probe side can make use of primary index, but not advisable.
  71. 71. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 71 N1QL : Arrays Array { "hobbies": ["tennis", "skiing", "lego"]} { "orders": [582, 9721, 3814]} Object { "address": {"street": "1, Main street", "city": Morrisville, "state":"CA", "zip":"94824"} } Arrays of objects of arrays [ { "type": "visa", "cardnum": "5827-2842-2847-3909", "expiry": "2019-03" }, { "type": "master", "cardnum": "6274-2542-5847-3949", "expiry": "2018-12" } ]
  72. 72. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 72 ANSI JOIN Support for Arrays • Array is an important construct in NoSQL world • Although the SQL standard does not include array support, we added support for arrays in our implementation of ANSI JOIN • Support arrays on the left-hand-side of ANSI JOIN, on the right-hand- side of ANSI JOIN, or on both sides of ANSI JOIN • Right-hand-side: use array index • Left-hand-side: use IN clause or UNNEST operation • Both sides: combination of above
  73. 73. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 73 Play with ANSI JOIN Support for Arrays - Setup CREATE PRIMARY INDEX ON product; "product01", {"productId": "product01", "category": "Toys", "name": "Truck", "unitPrice": 9.25} "product02", {"productId": "product02", "category": "Kitchen", "name": "Bowl", "unitPrice": 5.50} "product03", {"productId": "product03", "category": "utensil", "name": "Spoons", "unitPrice": 2.40} CREATE PRIMARY INDEX ON purchase; "purchase01", {"purchaseId": "purchase01", "customerId": "customer01", "lineItems": [ {"productId": "product01", "count": 3}, {"productId": "product02", "count": 1} ], "purchasedAt": "2017-11-24T15:03:22”} "purchase02", {"purchaseId": "purchase02", "customerId": "customer02", "lineItems": [ {"productId": "product03", "count": 2} ], "purchasedAt": "2017-11-27T09:08:37”}
  74. 74. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 74 ANSI JOIN Support for Arrays – Right-hand-side • Utilize array index defined on the right-hand-side keyspace CREATE INDEX purchase_ix1 ON purchase(DISTINCT ARRAY l.productId FOR l IN lineItems END) USING GSI SELECT p.name, pu.purchasedAt FROM product p JOIN purchase pu ON ANY l IN pu.lineItems SATISFIES l.productId = p.productId END
  75. 75. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 75 ANSI JOIN Support for Arrays – Left-hand-side with UNNEST • Utilize UNNEST to flatten the left-hand-side array first CREATE INDEX product_ix1 ON product(productId) USING GSI SELECT p.name, pu.purchasedAt FROM purchase pu UNNEST pu.lineItems AS pl JOIN product p ON pl.productId = p.productId
  76. 76. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 76 ANSI JOIN Support for Arrays – Left-hand-side with IN • Utilize IN-clause with array SELECT p.name, pu.purchasedAt FROM purchase pu JOIN product p ON p.productId IN ARRAY l.productId FOR l IN pu.lineItems END
  77. 77. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 77 Difference Between UNNEST and IN-clause • UNNEST first make copies of the left-hand-side document, one for each element of the array. There is no copying if using IN-clause • If there are duplicates in the array: • UNNEST makes copies for all duplicates • IN-clause does not care about duplicates • If performing LEFT OUTER JOIN • UNNEST makes copies and preserves all copies • IN-clause only preserves the original document
  78. 78. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 78 ANSI JOIN Support for Arrays – Both-side with UNNEST SELECT pu1.purchaseId pid1, pu2.purchaseId pid2 FROM purchase pu1 UNNEST pu1.lineItems AS pl JOIN purchase pu2 ON ANY l IN pu2.lineItems SATISFIES l.productId = pl.productId END
  79. 79. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 79 ANSI JOIN Support for Arrays – Both-side with IN-clause SELECT pu1.purchaseId pid1, pu2.purchaseId pid2 FROM purchase pu1 JOIN purchase pu2 ON ANY l2 IN pu2.lineItems SATISFIES l2.productId IN ARRAY l1.productId FOR l1 IN pu1.lineItems END END
  80. 80. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 80 Block nested loop join SELECT COUNT(1) FROM `beer-sample` brewery JOIN `beer-sample` beer ON (beer.brewery_id = LOWER(REPLACE(brewery.name, " ", "_")) AND beer.updated = brewery.updated) AND beer.type = "beer” WHERE brewery.type = "brewery" AND brewery.state = "California"
  81. 81. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 81 HASH join SELECT COUNT(1) FROM `beer-sample` brewery JOIN `beer-sample` beer USE HASH(probe) ON (beer.brewery_id = LOWER(REPLACE(brewery.name, " ", "_")) AND beer.updated = brewery.updated) AND beer.type = "beer” WHERE brewery.type = "brewery" AND brewery.state = "California"
  82. 82. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 82 HASH join SELECT COUNT(1) FROM `beer-sample` brewery JOIN `beer-sample` beer USE HASH(build) ON (beer.brewery_id = LOWER(REPLACE(brewery.name, " ", "_")) AND beer.updated = brewery.updated) AND beer.type = "beer” WHERE brewery.type = "brewery" AND brewery.state = "California"
  83. 83. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 83 Hash JOIN • beer is the build side. • Scan beer to create the hash table • Brewery automatically becomes the probe. • Each keyspace is scanned once. SELECT COUNT(1) FROM `beer-sample` brewery JOIN `beer-sample` beer USE HASH(build) ON (beer.brewery_id = LOWER(REPLACE(brewery.name, " ", "_")) AND beer.updated = brewery.updated) AND beer.type = "beer” WHERE brewery.type = "brewery" AND brewery.state = "California"
  84. 84. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 84 5.5 Features: HASH JOIN • Enterprise Edition only – Performance feature • ANSI JOIN query only • Only considered when (new) USE HASH hint is specified • USE HASH(build) or USE HASH(probe) • Specify USE HASH hint on right-hand side keyspace • Can combine USE HASH with USE INDEX or USE KEYS • Requires equality join predicate(s) • Hash join is preferred when USE HASH is specified, however, if hash join cannot be generated for some reason, nested-loop join is considered • Beneficial for “large” joins SELECT DISTINCT route.destinationairport FROM `travel-sample` airport INNER JOIN `travel-sample` route USE HASH(probe) INDEX(route_airports) ON airport.faa = route.sourceairport AND route.type = "route" WHERE airport.type = "airport" AND airport.city = "San Jose";
  85. 85. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 85 HASH join • At least one equality expression between two keyspaces • When this is absent, silently defaults to nested loop join. • Ditto on community edition • Requires memory to build the hash table on the BUILD side. • Unsuitable when only few documents qualify and can be done via index lookup. • The build side has be the keyspace with LESS qualified documents. • This selection is manual until cost based decisions can be made automatically • Typically good for reporting kind of queries. • Once the query latency goes to seconds, consider hash joins!
  86. 86. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 86 5.5: Hash JOIN
  87. 87. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 87 Grouping and Aggregation Performance (EE) CUSTOMER PROBLEM OR SCENARIO • Grouping and Aggregation are expensive operations • Latencies are high and cluster is not scaling BENEFITS • High scalability • Low query latencies • Low TCO • Automatic: No changes to query or index SOLUTION • If query is covered by index, Let Indexer perform Grouping and Aggregation. • Eliminate network transportation and disk I/0 due to backfill. These are slowest operations. EXAMPLE: • CREATE INDEX idx ON `travel- sample`(type); • SELECT type, COUNT(1) AS typecount FROM `travel-sample` WHERE type IS NOT NULL GROUP BY type;
  88. 88. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 88 Aggregation Queries SELECT type FROM `travel-sample` WHERE type is NOT NULL GROUP BY type; SELECT COUNT(1) AS typecount FROM `travel-sample` WHERE type IS NOT NULL; SELECT type, COUNT(1) AS cnt FROM `travel-sample` WHERE type IS NOT NULL GROUP BY type;
  89. 89. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 89 Aggregation Example
  90. 90. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 90 What is Group and Aggregation? • Grouping and Aggregations are part of SQL syntax. Used in Dashboards, Analytical, Reporting queries. • Query contains GROUP BY clause • Example : SELECT type FROM `travel-sample` WHERE type IS NOT NULL GROUP BY type; • Query contains Aggregate functions. • Aggregate functions are MIN(), MAX(), COUNT(), SUM(), AVG(), ARRAY_AGG() • Aggregate functions can only be in Projection, Having, Letting clause • Aggregate functions can be present with or without GROUP BY. • When GROUP BY is absent it considered as single group. • SELECT type, COUNT(1) AS cnt FROM `travel-sample` WHERE type IS NOT NULL GROUP BY type; • Returns how many documents present in each group.
  91. 91. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 91 N1QL: Inside the Query Service Client FetchParse Plan Join Filter Pre-Aggregate Offset Limit ProjectSortAggregateScan Query Service Index Service Data Service
  92. 92. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 92 CREATE INDEX idx ON `travel-sample`(type); Client FetchParse Plan Join Filter Pre-Aggregate Offset Limit ProjectSortAggregateScan Query Service Index Service Data Service SELECT type, COUNT(type) AS typecount FROM `travel-sample` WHERE type is not missing GROUP BY TYPE ORDER BY typecount Covered Query Path Note: Grayed out phases (Fetch, Join) are skipped in the query because the query is completely covered by the index scan.
  93. 93. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 93 CREATE INDEX idx ON `travel-sample`(type); Client FetchParse Plan Join Filter Pre-Aggregate Offset Limit ProjectSortAggregateScan Query Service Index Service Data Service SELECT type, COUNT(type) AS typecount FROM `travel-sample` WHERE type is not missing GROUP BY TYPE ORDER BY typecount Query with GROUP & Aggregate Pushdown Note: Grayed out phases (Fetch, Join, filter, pre-aggregate, aggregate, sort) are skipped in the query because indexer can in Couchbase 5.5, when the query qualifies.
  94. 94. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 94 5.5: Index grouping and aggregation Performance SELECT store, AVG(quantity) AS avg_quantity FROM store_sales WHERE DATE_PART_STR(sold_date, 'year') = $1 AND customer.state = $2 GROUP BY store;
  95. 95. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 95 5.5: Index grouping and aggregation Performance SELECT DATE_PART_STR(sold_date, 'year') AS year, item.brand, SUM(sales_price) AS sum_agg FROM store_sales WHERE item.manufacturer_id = $1 AND DATE_PART_STR(sold_date, 'month') = $1 GROUP BY DATE_PART_STR(sold_date, 'year'), item.brand ORDER BY DATE_PART_STR(sold_date, 'year'), sum_agg DESC, item.brand;
  96. 96. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 96 5.5: Index Partitioning Manageability: Scale out GSI Index create index route on bucket(airline, flight, source_airport, destination_airport) partition by hash(airline) • Scale out partitions as cluster size grows • Partition key must be immutable -- meta().id, immutable secondary key (e.g. airline) Performance: Partition Elimination Select flight from bucket where airline is not missing and source_airport = “SFO” • Scatter-gather across all partitions Select flight from bucket where airline in [“UA”, “AA”] and source_airport = “SFO” • Partition Elimination when predicate contains partition key : Only scan the partitions specified in predicate for faster range query response Performance: Parallelize Aggregate Scan Select count(flight) from bucket where airline is not missing group by source_airport, destination_airport • Ability to parallelize scan on aggregate query across partitions
  97. 97. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 97 5.5 Features: Index Partitioning Scan availability: Index scan can span partitions across replica create index route on bucket(airline, flight, source_airport, destination_airport) partition by hash(airline) with {“num_replica”:1} • A single index scan can pick any available partition across all replica Manageability: Repair Lost Partition • If partitions are lost due to node failover, those partitions can be repaired (rebuilt) on remaining nodes during rebalancing Performance: Scan Load Balancing • Scan traffic is load balanced on partitioned index across replica • Partition of a replica can be skipped if it is falling behind from other replica
  98. 98. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 98 5.5: BACKFILL SETTING CUSTOMER PROBLEM OR SCENARIO • Results from index scan come back faster than query can consume. • These results are saved in a temporary file, known as backfill • In 5.0, this location was /tmp by default. • Customers can (and have) run out of space. • Make the backfill location configurable. • Settable in Web console or REST API • By default the path is /tmp – this is now settable • By default the Quota is • If set to -1 it means unlimited • If set to 0 then backfill is disabled • There is no upper limit, it depends on user system SOLUTION
  99. 99. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 99 Vulcan: ALTER INDEX CUSTOMER PROBLEM OR SCENARIO • An imbalance occurs due to a particular index growing faster than expected and is needed on a different node. • An imbalance occurs due to a cluster of indexes being dropped on a single node. • A machine is scheduled for removal, so its indexes need to move off its current node. • The automated process of rebalancing does not give the expected results. SOLUTION Alter Index support to change the placement of an existing index or replica among different GSI nodes. For example, if a node fails and you need to move it from node 172.23.130.24 to node 172.23.130.25 ALTER INDEX `travel-sample`.idx1 WITH {"action":"move","nodes": ["172.23.120.25:8091"]}
  100. 100. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 100 5.5 Features: Query Workbench Improvements Auto Explain / Visual Explain • EXPLAIN is automatically run before every query • User can check plans to see why query ran slow or fast • Improved Query Plan Visualization • Layout in any direction, better panning/zooming • Easier to read • Improved tooltips High Performance Tabular View • Tabular results view used to get slow with 750kb data • Now scales to > 100MB • Column headers always visible Export/Copy as Tab-separated Text • Send results to Excel by exporting or copying as tab-separated values. Copy button
  101. 101. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 101 Vulcan Features: Query Workbench Improvements Positional and Named Parameters • Run prepared queries • Parameters specified in Preferences dialog Tabular Document Editor, with N1QL! • Previous document editor text only • Only supported searching by document key • New version: • Shows document in editable table • Can filter documents with N1QL WHERE clause • One button to copy, delete, edit, save document
  102. 102. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 102 5.5: N1QL Auditing Problem: no idea who is doing what in the system. Solution: N1QL auditing • Auditing available for all statement types • SELECT, INSERT, UPDATE, … • Auditing also available for all API endpoints of query engine. • /admin/stats, /admin/config, /admin/prepareds, … • Configurable from UI • Security/Audit tab • Selectable • Choose what query types to audit • Whitelist of trusted users who will not be audited • Cost varies depending on query type and how much to audit • Worst case: many small queries, everything audited (approx 20% throughput loss)
  103. 103. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 103 5.5 Features : Prepared Statement • Prepared statements now automatically distributed across N1QL nodes • Both in push and pull configuration • N1QL service monitors resource usage prior of execution • Statements silently prepared again if indexes or keyspaces change • …and automatically distributed to other nodes if reprepared
  104. 104. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 104 5.5 Features: X.509 support • Couchbase Server uses X.509 certificates to encrypt its client-server communication • Query service automatically refreshes certificates when server certificates are updated • Query doesn’t mandate the presence of authorization headers • It supports client certificate authorization • To run a query successfully with client certificates – curl --cacert ./root/ca.pem --cert-type PEM --cert ./client/client/chain.pem --key-type PEM –key ./client/client/client.key https://localhost:18093/query/ser vice -d "statement=select * from system:keyspaces"
  105. 105. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 105 5.5: IPv6 support • Make query IPv6 compliant. • Server passes in --ipv6 to query which takes the value true or false. This determines the mode in which the query service needs to operate. The default value is false (IPv4). • IPv6 equivalents of 127.0.0.1 and 0.0.0.0 is ‘::1’ and ‘::’ respectively. • Construction of URLs • If we are using hostnames or fully qualified domain names there will be no difference. • For constructing URLs with raw IPv6 addresses – • the IPv6 address must be enclosed within ‘[‘ and ‘]’ brackets. When we construct urls using localhost for example. For example: http://[::1]:8091/pools/default • CBQ shell also supports connection to IPV6 addresses. curl --cacert ./root/ca.pem --cert-type PEM --cert ./client/client/chain.pem --key-type PEM –key ./client/client/client.key https://localhost:18093/query/service -d "statement=select * from system:keyspaces"
  106. 106. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 106 Vulcan Features : Curl Whitelist CUSTOMER PROBLEM OR SCENARIO • The curl_whitelist.json file needed to be created on every query node in … /var/lib/couchbase/n1qlcerts/ by the user • Could have different values for each query node • Needed to be part of cbcollect info SOLUTION • UI now supports setting curl whitelist. • This is propagated to all query nodes
  107. 107. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. 107 5.5: Query performance Query workload - TCO Improvement Queries/Sec • Average N1QL throughput improved by 50%+ • Latency improved by 20% in performance test bed • Memory consumption reduced substantially • Faster document loading from KV • Memory and CPU usage improvements in projector as well Index grouping and aggregation • Latency 5 times to 10 times lower • Throughput 10 times to 20 times higher Workload 5.0 5.5 Improvement USE KEYS 33K 64K 95% Equality predicate 22K 33K 55%
  108. 108. READ THIS!
  109. 109. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2018. All rights reserved. THANK YOU! * query.couchbase.com @N1QL

×