Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

N1QL: what's new in Couchbase 5.0 – Connect Silicon Valley 2017

194 views

Published on

Speaker: Keshav Murthy

N1QL = SQL + JSON. N1QL gives developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data. We begin with a brief overview. Couchbase 5.0 has language and performance improvements for pagination, index exploitation, integration, and more. We’ll walk through scenarios, features, and best practices.

Published in: Technology
  • Be the first to comment

N1QL: what's new in Couchbase 5.0 – Connect Silicon Valley 2017

  1. 1. N1QL : WHAT’S NEW IN COUCHBASE 5.0 Keshav Murthy Senior Director, Couchbase R&D
  2. 2. AGENDA 01 02 N1QL: Overview -- 5 minutes, hopefully! N1QL Features in 5.0 -- Rest of the time
  3. 3. 1 N1QL: OVERVIEW
  4. 4. 4 ResultSet Relations/Tuples
  5. 5. 5 { "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ] } LoyaltyInfo Results Orders CUSTOMER • NoSQL systems provide specialized APIs • Key-Value get and set • Each task requires custom built program • Should test & maintain it
  6. 6. 6 Find High-Value Customers with Orders > $10000 Query customer objects from database • Complex codes and logic • Inefficient processing on client side For each customer object Find all the order objects for the customer Calculate the total amount for each order Sum up the grand total amount for all orders If grand total amount > $10000, Extract customer data Add customer to the high-value customer list Sort the high-value customer list LOOPING OVER MILLIONS OF CUSTOMERS IN APPLICATION!!!
  7. 7. 7 { "Name" : "Jane Smith", "DOB" : "1990-01-30", "Billing" : [ { "type" : "visa", "cardnum" : "5827-2842-2847-3909", "expiry" : "2019-03" }, { "type" : "master", "cardnum" : "6274-2842-2847-3909", "expiry" : "2019-03" } ], "Connections" : [ { "CustId" : "XYZ987", "Name" : "Joe Smith" }, { "CustId" : "PQR823", "Name" : "Dylan Smith" } { "CustId" : "PQR823", "Name" : "Dylan Smith" } ], "Purchases" : [ { "id":12, item: "mac", "amt": 2823.52 } { "id":19, item: "ipad2", "amt": 623.52 } ] } LoyaltyInfo ResultDocuments Orders CUSTOMER
  8. 8. 8 N1QL = SQL + JSON Give developers and enterprises an expressive, powerful, and complete language for querying, transforming, and manipulating JSON data.
  9. 9. 9 Why SQL for NoSQL?
  10. 10. 10 N1QL : Data Types from JSON Data Type Example Numbers { "id": 5, "balance":2942.59 } Strings { "name": "Joe", "city": "Morrisville" } Boolean { "premium": true, "balance pending": false} Null { "last_address": Null } Array { "hobbies": ["tennis", "skiing", "lego"]} Object { "address": {"street": "1, Main street", "city": Morrisville, "state":"CA", "zip":"94824"}} MISSING Arrays of objects of arrays [ { "type": "visa", "cardnum": "5827-2842-2847-3909", "expiry": "2019-03" }, { "type": "master", "cardnum": "6274-2542-5847-3949", "expiry": "2018-12" } ]
  11. 11. 11 N1QL: Data Manipulation Statements • SELECT Statement- • UPDATE … SET … WHERE … • DELETE FROM … WHERE … • INSERT INTO … ( KEY, VALUE ) VALUES … • INSERT INTO … ( KEY …, VALUE … ) SELECT … • MERGE INTO … USING … ON … WHEN [ NOT ] MATCHED THEN … Note: Couchbase provides per-document atomicity.
  12. 12. 12 N1QL: SELECT Statement SELECT * FROM customers c WHERE c.address.state = 'NY' AND c.status = 'premium' ORDER BY c.address.zip Project Everything From the bucket customers Sort order Predicate
  13. 13. 13 N1QL: SELECT Statement SELECT customers.id, customers.NAME.lastname, customers.NAME.firstname Sum(orderline.amount) FROM orders UNNEST orders.lineitems AS orderline INNER JOIN customers ON KEYS orders.custid WHERE customers.state = 'NY' GROUP BY customers.id, customers.NAME.lastname, customers.NAME.firstname HAVING sum(orderline.amount) > 10000 ORDER BY sum(orderline.amount) DESC • Dotted sub-document reference • Names are CASE- SENSITIVE UNNEST to flatten the arrays JOINS with Document KEY of customers
  14. 14. 14 N1QL: SELECT Statement Highlights • Querying across relationships • JOINs • Subqueries • Aggregation • MIN, MAX • SUM, COUNT, AVG, ARRAY_AGG [ DISTINCT ] • Combining result sets using set operators • UNION, UNION ALL, INTERSECT, INTERSECT ALL, EXCEPT, EXCEPT ALL
  15. 15. 15 N1QL : Query Operators [ 1 of 2 ] • USE KEYS … • Direct primary key lookup bypassing index scans • Ideal for hash-distributed datastore • Available in SELECT, UPDATE, DELETE • JOIN … ON KEYS … • Nested loop JOIN using key relationships • Ideal for hash-distributed datastore • Current implementation supports INNER and LEFT OUTER joins • ANSI JOINS • We’re working on it. Be part of BETA for the next release.
  16. 16. 16 N1QL : Query Operators [ 2 of 2 ] • NEST • Special JOIN that embeds external child documents under their parent • Ideal for JSON encapsulation • UNNEST • Flattening JOIN that surfaces nested objects as top-level documents • Ideal for decomposing JSON hierarchies • JOIN, NEST, and UNNEST can be chained in any combination
  17. 17. 17 N1QL : Expressions for JSON Ranging over collections • WHERE ANY c IN children SATISFIES c.age > 10 END • WHERE EVERY r IN ratings SATISFIES r > 3 END Mapping with filtering • ARRAY c.name FOR c IN children WHEN c.age > 10 END Deep traversal, SET, and UNSET • WHERE ANY node WITHIN request SATISFIES node.type = “xyz” END • UPDATE doc UNSET c.field1 FOR c WITHIN doc END Dynamic Construction • SELECT { “a”: expr1, “b”: expr2 } AS obj1, name FROM … // Dynamic object • SELECT [ a, b ] FROM … // Dynamic array Nested traversal • SELECT x.y.z, a[0] FROM a.b.c … IS [ NOT ] MISSING • WHERE name IS MISSING
  18. 18. 18 Couchbase Server Cluster Service Deployment STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service STORAGE Couchbase Server 2 Managed Cache Cluster ManagerCluster Manager Data Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Index Service Managed Cache Storage Managed Cache Storage Storage STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Index Service Storage Managed Cache Managed Cache SDK SDK
  19. 19. 19 N1QL: Query Execution Flow Clients 1. Submit the query over REST API 8. Query result 2. Parse, Analyze, create Plan 7. Evaluate: Documents to results 3. Scan Request; index filters 6. Fetch the documents Index Service Query Service Data Service 4. Get qualified doc keys 5. Fetch Request, doc keys SELECT c_id, c_first, c_last, c_max FROM CUSTOMER WHERE c_id = 49165; { "c_first": "Joe", "c_id": 49165, "c_last": "Montana", "c_max" : 50000 }
  20. 20. 20 N1QL: Inside the Query Service Client FetchParse Plan Join Filter Pre-Aggregate Offset Limit ProjectSortAggregateScan Query Service Index Service Data Service
  21. 21. * N1QL: PATH OF PROGRESS
  22. 22. 22 N1QL features in Couchbase releases 22 Couchbase 4.0: N1QL GA Query language for JSON, Integrated Query Service, Global Secondary Index, REST API, Simba ODBC, JDBC Drivers Couchbase 4.1: INSERT, UPDATE, DELETE, MERGE Covering Index Optimization Couchbase 4.1.1: Index JOINs Couchbase 4.5: Array Indexes, Workbench, CBQ Shell++, INFER Memory Optimized Index, IndexScanCount YCSB Performance Optimizations++, Language++ Couchbase 4.5.1: Pretty=false; Fetch; SUFFIXES; Index Selection; UPDATE improvement Oct 2015 Dec 2015 March 2016 June 2016 Sep 2016
  23. 23. 23 N1QL features in Couchbase releases. 23 Couchbase 4.6.1: TOKENS (Simple Search/Faster LIKE), Optimizer improvements Couchbase 5.0: Subqueries over nested data ; Pagination; RBAC; Curl, Super Charged Indexing; Monitoring & Profiling; New workbench, UI, monitoring, profiler, visual EXPLAIN Performance++, Bitwise functions Q1 2017 4Q 2017 Couchbase 4.6.2: Optimizer improvements, intersect scan performance Q2 2017
  24. 24. N1QL FEATURES IN 5.0 01 02 04 List of Features Query-Indexing Enhancements Optimizer 03 Language & Infrastructure 05 Flexible Indexing 06 Security 07 Monitoring & Profiling 08 Developer Tools 09 Performance
  25. 25. 1 LIST OF FEATURES
  26. 26. 26 Couchbase N1QL and GSI features Query-Indexing Features • Large Indexing Keysize • Index key collation: ASC, DESC on each key • Index replicas, just like data replication • New storage engine: Plasma Query Language & Infrastructure • Subquery Expressions • Additional Date & time functions • Bitwise functions • CURL() within N1QL Query Optimizer • Complex Filters Pushdown • Pagination optimization • Optimization for ASC, DESC keys • Query-Index API optimization (projection, etc.) • Index projections, Intersect scans • Adaptive Indexes Security, Administration & Functionality • Security: RBAC: Statement level security • Query Monitoring, Profiling with UI • Query work bench and UI: Fully upgraded • Query UI: Visual Explain • Query on Ephemeral buckets • Application Continuity, Seamless Upgrade Performance • Core daily workload • YCSB • YCSB-JSON for Engagement Database http://query.couchbase.com
  27. 27. 2 QUERY-INDEX FEATURES
  28. 28. 28 Query-Indexing Enhancements Index key collation: ASC, DESC on each key • Prior to 5.0, each index key was sorted and kept in ASCENDING order only • To sort the key in descending order, you did • CREATE INDEX i1 ON t(c1 ASC, -c2, -c3) • SELECT * FROM t WHERE c1 = 10 and -c2 < -20 ORDER BY c1, -c2 • Query formulations becomes confusing • Cannot use this trick on all data types and expressions In Couchbase 5.0: • CREATE INDEX i1 ON t(c1 ASC, c2 DESC, c3 DESC) • SELECT * FROM t WHERE c1 = 10 and c2 < 20 ORDER BY c1,c2 DESC • You need to create an index to match the ORDER BY order • Reverse scans are still unsupported
  29. 29. 29 Query-Indexing Enhancements Large Indexing Keysize • Prior to 5.0, the sum of index key size could be up to 4096 bytes • This was controlled by the setting • For ARRAY keys, sum of all array key sizes could be up to 10240. • This is controlled by the setting max_array_seckey_size In Couchbase 5.0: • The total keysize could be pretty high – high up to 20 MB • This is true for single key, composite key, expressions and array indexes as well. • Simply do nothing, except create the index and issue the query. • The index entries that exceed 20MB will still generate error in the index log
  30. 30. 30 Query-Indexing Enhancements Index replicas, just like data replication • Prior to 5.0, you could create multiple indexes with same keys & condition • This is needed for load balancing and index high availabilitt CREATE INDEX i1 ON t(c1, c2, c3) CREATE INDEX i2 ON t(c1, c2, c3) CREATE INDEX i3 ON t(c1, c2, c3) • Indexer automatically recognizes these to be equivalent and does load balancing on all o these. In Couchbase 5.0: • Simply create one index and set the num_replica at CREATE or ALTER time • CREATE INDEX i1 ON t(c1, c2, c3) WITH {"num_replica":2} • Number of replicas can be up to number of nodes in the cluster • You can ALTER the number of replica dynamically
  31. 31. 31 Query-Indexing Enhancements New storage engine: Plasma • Uses lock-free skip list • All the performance benefits of MOI – Memory Optimized Index • Automatically does IO as needed • From usage point of view: • Choose the standard secondary Index during installation • simply create any kind of index and use it. • More on Plasma in the Indexing Talk this afternoon.
  32. 32. 3 QUERY LANGUAGE & INFRASTRUCTURE
  33. 33. 33 Query Language & Infrastructure Subquery Expressions • Sub-queries are N1QL expressions that can embed a full N1QL SELECT query • Can be used in projection, FROM and WHERE clause • Before 5.0, the table expression in FROM-clause is limited to: • Keyspace name or Sub-query • constants, expressions, functions, nested paths are disallowed • Correlated Sub-queries on nested objects is expensive • Inner sub-query is evaluated for every document from outer query • Same document is accessed/read for every level of the query nesting
  34. 34. 34 Query Language & Infrastructure Subquery Expressions • Provides rich functionality and Powerful subquery-expressions • Can be used in FROM-clause, projection, LET/WHERE-clauses etc., SELECT word, cnt FROM ARRAY split(i) FOR i IN (SELECT raw name FROM `travel-sample` WHERE type = "hotel") END AS words UNNEST words w GROUP BY w LETTING cnt = COUNT(w) ORDER BY cnt DESC;
  35. 35. 35 Query Language & Infrastructure Additional Date, time, timestamp function • JSON does not directly support date and time related functions • Store the date and time in extended ISO 8901 format • "2017-10-16T18:44:43.308-07:00” • Need extract, conversion and arithmetic functions • Detailed article with all the functions and Oracle to Couchbase mapping https://dzone.com/articles/comparing-oracle-and-n1ql-support-for-the-date-tim • If you can’t do something, let us know!
  36. 36. 36 Query Language & Infrastructure CURL() within N1QL • CURL (URL, [options]) • The first argument is the URL, which represents any URL that points to a JSON endpoint. • Only URLs with the http:// or the https:// protocol are supported. • Redirection is disabled. • The second argument is a list of options. • This is a JSON object that contains a list of curl options and their corresponding values. • For a full list of options that we support, please refer to the Dzone article on CURL in N1QL by Isha Kandaswamy •
  37. 37. 37 Query Language & Infrastructure CURL() within N1QL • Search for Santa Cruz in Spain using my Google dev api key • SELECT CURL("GET","https://maps.googleapis.com/maps/api/geocode/json", {"data":"address=santa+cruz&components=country:ES&key=AIzaSyCT6n iGCMsgegJkQSYasfoLZ4_rSO59XQQ"}) ; • Search for Half Moon Bay • SELECT CURL("GET","https://maps.googleapis.com/maps/api/geocode/json",{ "data":"address=Half+Moon+Bay"} ).results rs; •
  38. 38. 38 Query Language & Infrastructure BITWISE Functions • All bitwise functions can only take a number. All numbers are 64 bit signed numbers (integers). If the Number is not an integer and for other data types, we throw an error. • When looking at the value in binary form, bit 1 is the Least Significant Bit (LSB) and bit 32 is the Most Significant Bit. (MSB) Bit 32 → 0000 0000 0000 0000 0000 0000 0000 0000 ← Bit 1 (LSB) • Supported functions : • BitAND • BitOR • BitNOT • BitXOR • BitSHIFT • BitSET • BitCLEAR • BitTEST/ IsBitSET
  39. 39. 4 QUERY OPTIMIZER & EXECUTION
  40. 40. 40 Query Optimizer & Execution: Stable Scans • IndexScan use to do single range scan (i.e single Span) • If the query has multiple ranges (i.e. OR, IN, NOT clauses) Query service use to do separate IndexScan for each range. • This causes Indexer can use different snapshot for each scan (make it unstable scan) • Number of IndexScans can grow and result increase in index conneconnections • In 5.0.0 multiple ranges are passed into indexer and indexer uses same snapshot for all the ranges. • This makes stable Scan for given IndexScan (i.e. IndexScan2 in the EXPLAIN). • This will not make stable scan for query due to Subqueries, Joins etc • Example: • create index ix1 on default(k0); • EXPLAIN SELECT META().id FROM default WHERE k0 IN [10,12,13];
  41. 41. 41 Query Optimizer & Execution: Pushdown Composite Filters • For composite Index the spans that pushed to indexer contains single range for all composite keys together. • Indexer will not applying range for each part of the key separately. This result in lot of false positives. • In 5.0.0 with IndexScan2 we push the each index key range separately and indexer will apply keys separately. • This results in no/less false positives and aides push more information to indexer. • Example: • CREATE INDEX ix1 ON default(k0,k1); • EXPLAIN SELECT meta().id FROM default WHERE k0 BETWEEN 0 AND 100 AND k1 = 200;
  42. 42. 42 Query Optimizer: ORDER, OFFSET, LIMIT pushdown • Pagination queries can contain any combination of ORDER, LIMIT, OFFSET clauses. • Performance of these queries are critical to applications. • When Predicates are completely and exactly pushed to indexer, by pushing offset, limit to indexer can improve query performance significantly. If that happened IndexScan2 section of EXPLAIN will have limit,offset. • If query ORDER BY matches index key order query can avoid index sort and performance can be improved significantly. If that happened order operator is not present in the EXPLAIN. • Example: • CREATE INDEX ix1 ON default(k0,k1); • EXPLAIN SELECT meta().id FROM default WHERE k0 > 10 AND k1 > 20 ORDER BY k0 LIMIT 10 OFFSET 100;
  43. 43. 43 Query Optimizer: MAX pushdown • If the MAX arguments matched with Index leading key exploit index order for MAX. • MAX can only DESC on index key. • MIN can only use ASC on index key. • Example : • CREATE INDEX ix5 ON default(k0 DESC); • SELECT MAX(k0) FROM default WHERE k0 > 10; • Above query able to exploit index order. In that case IndexScan2 section of EXPLAIN will have “limit” 1.
  44. 44. 44 Query Optimizer: Index Projection • The index can have many keys but query might be interested only subset of keys. • By only requesting required information can save lot of network transportation, memory, cpu, backfill etc. All this can help in performance and scaling the cluster. • The requested information can be found in “IndexScan2” Section of EXPLAIN as “index_projection” "index_projection": { "entry_keys": [ xxx,....... ], “primary_key”: true } •
  45. 45. 45 Query Optimizer: Index Projection CREATE INDEX ix1 ON default(k0,k1); Covered query SELECT k0 FROM default WHERE k0 = 10 AND k1 = 100; "index_projection": { "entry_keys": [0,1] } SELECT k0 FROM default WHERE k0 = 10; "index_projection": { "entry_keys": [0] } SELECT k0 ,META().idFROM default WHERE k0 = 10; "index_projection": { "entry_keys": [0], “primary_key”: true } Non-covered query SELECT k0 ,k5 FROM default WHERE k0 = 10 AND k1 = 100; "Index_projetion": { “primary_key”: true }
  46. 46. 46 Query Execution: CAS & Expiration • In 5.0.0 META().cas, META().expiration can be indexed and used in queries. • Example: • CREATE INDEX ix1 ON default( meta().id, meta().cas, meta().expiration); • SELECT meta().id , meta().cas, meta().expiration FROM default where meta().id > "" • Note: META().expiration will work in covered queries. For non covered queries it gives 0
  47. 47. 47 Query Execution: COUNT (DISTINCT expr) • If the expr matched with Index leading key COUNT DISTINCT can be pushed to indexer • Complete predicate needs to pushed to indexer exactly • No false positives are possible • No group or JOIN • Only single projection • Example : • CREATE INDEX ix5 ON default(k0); • SELECT COUNT(DISTINCT k0) FROM default WHERE k0 > 10; • Above query uses IndexCountDistinctScan2
  48. 48. 5 N1QL: FLEXIBLE INDEXING
  49. 49. 49 Customer Scenario • Customer document has 100 fields • They have multiple business entities sharing the same data • Each entity want to FILTER, GROUP, ORDER on distinct criteria • For Index selection, order of the keys in the composite index is important. Fields: c1 through c100 Filter fields: c1 through c50 Group, order and projection: Any from c1 through c100 SELECT c1, c2, c3, COUNT(c10), SUM(c5) FROM CUSTOMER WHERE c4 = "CXT-MULTI" AND c8 = "iPhone6" AND c9 BETWEEN 10 IN 20 GROUP BY c1, c2, c3; SELECT c12, COUNT(c19), SUM(c15) FROM CUSTOMER WHERE c44 = "CXT-MULTI" AND c18 = "Gpixel 2" AND c29 BETWEEN 10 IN 20 GROUP BY c12;
  50. 50. 50 Customer Scenario • What indexes to create for this? SELECT c1, c2, c3, COUNT(c10), SUM(c5) FROM CUSTOMER WHERE c4 = "CXT-MULTI" AND c8 = "iPhone6" AND c9 BETWEEN 10 IN 20 GROUP BY c1, c2, c3; CREATE INDEX i1 ON CUSTOMER(c8, c4, c9) CREATE INDEX i1 ON CUSTOMER(c8, c4, c9, c1, c2, c3, c10, c5; For Covering the query What about this? SELECT c12, COUNT(c19), SUM(c15) FROM CUSTOMER WHERE c44 = "CXT-MULTI" AND c18 = "Gpixel 2" AND c29 BETWEEN 10 IN 20 GROUP BY c12;
  51. 51. 51 Large, wide, composite indexes Filter fields: c1 through c50 To support all combinations of 50 predicates via composite indexes, you’ll need LOT of indexes. 50! =304140932017133780436126081660647688443776415689605 12000000000000
  52. 52. 52 Customer Scenario Solution: Intersection • Option 1 • Create indexes on individual fields • Scan individual indexes • Apply the full set of predicates (boolean expression from WHERE clause) • Then do the post processing. CREATE INDEX i1 on CUSTOMER(c1); CREATE INDEX i2 on CUSTOMER(c2); CREATE INDEX i3 on CUSTOMER(c3); • Option 2 • Too many indexes to maintain and manage. • Don’t even talk about equivalent indexes for each of these. CREATE INDEX i1to50 on CUSTOMER(DISTINCT PAIRS({c1, c2, c3, c4, c5,c6, c7, c8, c9, c10, c11, c23, c13, c14, …});
  53. 53. 53 Solution: Intersection • Option 3 • Too many keys to manage/specify • The document is flexible. I want the index to be flexible. CREATE INDEX ixpairon CUSTOMER(DISTINCT PAIRS(self)); SELECT * FROM CUSTOMER WHERE a = 10 and b < 20 and c between 30 and 40; "#operator": "IntersectScan", "scans": [ { "#operator": "DistinctScan", "scan": { "#operator": "IndexScan2", "index": "ixpair", "index_id": "466c0c5c4c3b21c1", "index_projection": { "primary_key": true }, "keyspace": "test", "namespace": "default", "spans": [ { "exact": true, "range": [ { "high": "["a", 10]", "inclusion": 3, "low": "["a", 10]" } "range": [ { "high": "["b", 20]", "inclusion": 1, "low": "["b", false]" } "range": [ { "high": "[successor("c")]", "inclusion": 1, "low": "["c", 30]" } ]
  54. 54. 54 Flexible Indexing • This is not a silver bullet, yet. • TRY THIS OUT • SIZING is a concern because we {“Key“:“value“} • Give us feedback
  55. 55. 6 SECURITY
  56. 56. 56 SECURITY • Query_select, query_insert, query_update, query_delete roles • Parameterized: query_select[customers] or query_insert[*] • Query_manage_index[foo] • Create, delete, build indexes on bucket foo • Query_system_catalog • Full access to the system tables (which are controlled now) • Query_external_access • Allows access to CURL() function (disabled by default) • GRANT cluster_admin TO spock • GRANT query_select ON default TO kirk • REVOKE query_insert, query_delete ON bridge, engineering FROM mccoy, scotty
  57. 57. 7 MONITORING & PROFILING
  58. 58. 58 Monitoring in UI 58Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  59. 59. 59 Profiling in UI 59Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
  60. 60. 60 Profiling • We can collect execution timings and document processed on a per operator basis • If the functionality is turned on, timings are reported • with the metrics at the end of execution • in system:active_requests • in system:completed_requests • Profiling is turned on • at the request level via the “profile” REST API parameter, EG from cbq: • set –profile timings; • at the node level via the “profile” command line parameter or admin settings REST API parameter • takes 3 values, “off”, “phases”, “timings” • “phases” supplies total times for each operator class • “timings” supplies detailed information for each operator
  61. 61. 61 Profiling cbq> select * from `travel-sample` where source-airport is not missing; … "executionTimings": { "~children": [ { "#operator": "IndexScan2", "#stats": { "#itemsOut": 24024, "#phaseSwitches": 96099, "execTime": "55.370283ms", "kernTime": "5.397199311s" }, "index": "def_sourceairport", "index_id": "29702e564c9d2ca4", "index_projection": { "primary_key": true }, "keyspace": "travel-sample", "namespace": "default", "spans": [ { "exact": true, "range": [ { "inclusion": 1, "low": "null"
  62. 62. 8 DEVELOPER TOOLING
  63. 63. 63 Developer Tooling
  64. 64. 9 PERFORMANCE
  65. 65. 65 N1QL Performance: 5.0 vs. 4.5 • Run internally • YCSB is the public YCSB • other queries are written on Couchbase dataset • 50% higher throughput in YCSB workload E • 10-40x faster pagination queries • 10-30x better performance of queries with composite filters • 10-40x faster queries with COUNT function • 6-9x better performance of basic queries (Q1 & Q2) • 55x faster queries with UNNEST clause
  66. 66. 66 N1QL Performance: 5.0 vs. 4.5 • Up to 10x faster array indexing • Fast text search with TOKENS() • 10x better performance of lookup and index joins • Query performance on Windows is on par with Linux • Up to 100K index scans per second in DGM scenarios
  67. 67. * N1QL: FUTURE
  68. 68. 68 68 N1QL Roadmap for 5.1 • Performance Improvements: Phase 1 • View Replacement : Phase 1 • Aggregate Performance : Phase 1 • ANSI Joins: Phase 1 • Statement Level Auditing • ALTER INDEX, Configuration • Support for XATTRS
  69. 69. 69 SUMMARY of N1QL features in Couchbase 5.0 Query-Indexing Features • Large Indexing Keysize • Index key collation: ASC, DESC on each key • Index replicas, just like data replication • New storage engine: Plasma Query Language & Infrastructure • Subquery Expressions • Additional Date & time functions • Bitwise functions • CURL() within N1QL Query Optimizer • Complex Filters Pushdown • Pagination optimization • Optimization for ASC, DESC keys • Query-Index API optimization (projection, etc.) • Index projections, Intersect scans • Adaptive Indexes Security, Administration & Functionality • Security: RBAC: Statement level security • Query Monitoring, Profiling with UI • Query work bench and UI: Fully upgraded • Query UI: Visual Explain • Query on Ephemeral buckets • Application Continuity, Seamless Upgrade Performance • Core daily workload • YCSB • YCSB-JSON for Engagement Database http://query.couchbase.com

×