Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015

3,624 views

Published on

The new indexing technology in Couchbase Server 4.0 advances query performance to new heights. In this talk, Director of Product Management, Cihan Biyikoglu will explore the capabilities of the new indexing technology from head-to-toe. First, we will cover primary and secondary index types. We will then explore indexers: M/R views vs global secondary indexes, index administration, monitoring, best practices and much more. If you are a developer, architect or an administrator looking to use N1QL in your next application, you won't want to miss this talk.

Published in: Technology

Introducing Global Secondary Indexes: New High Performance Indexing with Couchbase Server 4.0: Couchbase Connect 2015

  1. 1. GLOBAL SECONDARY INDEXES NEW HIGH PERFORMANCE INDEXER Cihan Biyikoglu Dir. Product Management
  2. 2. ©2015 Couchbase Inc. 2 Goals  Get to know Global Secondary Indexes (GSI) – the new high performance indexer for N1QL  Look at Indexing lifecycle & management with GSI  Cover top best practices and tips with GSI
  3. 3. ©2015 Couchbase Inc. 3 Agenda  Overview  Indexing in Couchbase Server 4.0  Couchbase Server 4.0 Architecture  Indexers in Couchbase Server 4.0  Indexing today and Indexing with GSI  Working with Global Secondary Indexes  GSI Architecture  GSI Lifecycle - Creation & Maintenance  IndexAvailability & Rebalance  Index Placement and Load Balancing  Monitoring GSI  Best Practices for GSI  Q&A
  4. 4. Overview
  5. 5. ©2015 Couchbase Inc. 5 Couchbase Server Cluster Architecture STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service Managed Cache Storage Managed Cache Storage Managed Cache Storage Managed Cache Storage Managed Cache Storage
  6. 6. ©2014 Couchbase Inc. Couchbase Server Cluster Architecture STORAGE Couchbase Server 1 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Managed Cache Storage Data Service Index Service Query Service STORAGE Couchbase Server 2 Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 3 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 4 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 5 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service STORAGE Couchbase Server 6 SHARD 7 SHARD 9 SHARD 5 SHARDSHARDSHARD Managed Cache Cluster ManagerCluster Manager Data Service Index Service Query Service Managed Cache Storage Managed Cache Storage Managed Cache Storage Managed Cache Storage Managed Cache Storage
  7. 7. ©2015 Couchbase Inc. 7 Indexing in Couchbase Server 4.0  Multiple Indexers  GSI – Index Service New indexing for N1QL for low latency queries without compromising on mutation performance (insert/update/delete) Independently partitioned and independently scalable indexes in Indexing Service  Map/ReduceViews – Data Service Powerful programmable indexer for complex reporting and indexing logic. Full partition alignment and paired scalability with Data Service.  SpatialView – Data Service Incremental R-tree indexing for powerful bounding-box queries Full partition alignment and paired scalability with Data Service New
  8. 8. ©2015 Couchbase Inc. 8 Query and IndexToday Once upon a time in a User Profile System….  Q1: Find the top 10 most “active” customer by #logins in Jan 2015 {… “customer_name” : ”Cihan”, “total_logins”: {… “aug_2015”:100, …} “type” : “customer_profile” …} … Q1 Active @ Jan 2015
  9. 9. ©2015 Couchbase Inc. 9 Query and IndexToday INDEX ON Customer_bucket(customer_name, total_logins.jan_2015) WHERE type=“customer_profile”; SELECT customer_name, total_logins.jan_2015 FROM customer_bucket WHERE type=“customer_profile” ORDER BY total_logins.jan_2015 DESC LIMIT 10; … Q1 Active @ Jan 2015 Q1: Execution Plan on N nodes • Scatter: Execute Q1 on N nodes • Gather: gather N results • Finalize: Execute Q1 on governor node 1 2 2 2 2 2 3 1 2 3
  10. 10. ©2015 Couchbase Inc. 10 Query and Index with GSI INDEX ON Customer_bucket(customer_name, total_logins.jan_2015) WHERE type=“customer_profile”; SELECT customer_name, total_logins.jan_2015 FROM customer_bucket WHERE type=“customer_profile” ORDER BY total_logins.jan_2015 DESC LIMIT 10; … Q1 Active @ Jan 2015 Q1: Execution Plan on N nodes • Execute Q1 on N1QL Service node • Scan index on Index Service node 12 1 3
  11. 11. ©2015 Couchbase Inc. 11 Introducing Global Secondary Indexes What are Global Secondary Indexes? High performance indexes for low latency queries with powerful caching, storage and independent placement.  Power of GSI  Fully integrated into N1QL Query Optimization and Execution  Independent Index Distribution for Limiting scatter-gather  Independent Scalability with Index Service – more on this later  Powerful caching and storage with ForestDB
  12. 12. ©2015 Couchbase Inc. 12 Which to choose – GSI vsViews Workloads New GSI in v4.0 Map/ReduceViews Complex Reporting Just InTime Aggregation Pre-aggregated Workload Optimization Optimized for Scan Latency & Throughput Optimized for Insertion Flexible Index Logic N1QL Functions Javascript Secondary Lookups Single Node Lookup Scatter-Gather Tunable Consistency Staleness false or ok or everything in between Staleness false or ok
  13. 13. ©2015 Couchbase Inc. 13 Which to choose – GSI vsViews Capabilities New GSI in v4.0 Map/ReduceViews Partitioning Model Independent – Indexing Service Aligned to Data – Data Service Scale Model IndependentlyScale Index Service Scale with Data Service Fetch with Index Key Single Node Scatter-Gather Range Scan Single Node Scatter-Gather Grouping,Aggregates With N1QL Built-in withViews API Caching Managed Not Managed Storage ForestDB Couchstore Availability Multiple Identical Indexes load balanced Replica Based
  14. 14. Deep Dive
  15. 15. GSI Architecture
  16. 16. ©2015 Couchbase Inc. 16 Data Service Projector & Router Indexing Service Query ServiceIndex Service Supervisor Index maintenance & Scan coordinator Index#2Index#1 Query Processor cbq-engine Bucket#1 Bucket#2 DCP Stream Index#4Index#3 ... B u c k e t # 2 B u c k e t # 1 Projector and Router: 1 Projector and Router per node 1 stream of changes per buckets per supervisor ForestDB Storage Engine Supervisor 1 Supervisor per node Many indexes per Supervisor
  17. 17. ©2015 Couchbase Inc. 17 Deeper Dive into Architecture @3.45 - ArchitectureTrack Deep Dive into Global Secondary Indexing Architecture in Couchbase Server 4.0 John Liang, Architect, Couchbase
  18. 18. GSI Lifecycle
  19. 19. ©2015 Couchbase Inc. 19 Indexing Lifecycle  Primary vs Secondary  Primary Index is a full list of document keys within a given bucket CREATE PRIMARY INDEX index_name ON bucket_name USING GSI|VIEW WITH `{"nodes”: [“node_name”], “defer_build”:true}`; //GSI-ONLY  Secondary Index is an index on a field/expression on a subset of documents for lookups CREATE INDEX index_name ON bucket_name (field/expression, …) USING GSI|VIEW WHERE filter_expressions WITH `{"nodes”: [“node_name”], “defer_build”:true}`; //GSI-ONLY
  20. 20. ©2015 Couchbase Inc. 20 Deferred Index Building  Index building can be deferred to build multiple indexes all at once with greater scan efficiency. CREATE INDEX … WITH {…“defer_build”:true}; CREATE INDEX … WITH {…“defer_build”:true}; … BUILD INDEX ON bucket_name(index_name1, …) USING GSI;
  21. 21. DEMO Quick tour of GSI
  22. 22. GSI Partitioning and Placement
  23. 23. ©2015 Couchbase Inc. 23 GSI Placement and Partitioning Place GSI Indexes using NODES clause  Each GSI reside on 1 node  You can specify the node using nodes clause  You can scale out the index by creating identical indexes (load balanced) CREATE INDEX i1 … WITH {“nodes”:”node1”}; CREATE INDEX i1 … WITH {“nodes”:”node1”}; CREATE INDEX i2 … WITH {“nodes”:”node2”}; …
  24. 24. ©2015 Couchbase Inc. 24 GSI Placement and Partitioning Partition Indexes Manually with WHERE clause  You can partition with theWHERE clause and place on various nodes for scaling out CREATE INDEX i1 … WHERE zipcode between “94000” and “94999” … CREATE INDEX i2 … WHERE zipcode between “95000” and “96000” …
  25. 25. GSI Availability and Rebalance
  26. 26. ©2015 Couchbase Inc. 26 GSI Availability and Rebalance Use multiple identical indexes for availability  GSIAvailability  Create multiple identical indexes on separate nodes for availability  GSI will auto divert traffic if any copy goes down  GSI & Rebalance  Removing/Failing a node with index service, remove the GSI indexes on that node  Adding a node with index service, won’t automatically move some indexes to the node.
  27. 27. Monitoring GSI
  28. 28. ©2015 Couchbase Inc. 28 Monitoring GSI Indexes Index Size and Maintenance Stats Index Scan Stats
  29. 29. Best Practices with GSI
  30. 30. ©2015 Couchbase Inc. 30 New Consistency Settings!  View Stale-ness  Ok: unbounded – query what’s available in the index/view now  False: query after all changes up to the request timestamp (and maybe more) has been indexed for a given index or view.  New Indexes with Couchbase Server 4.0  Improves granularity of the consistency logical-timestamp.  New: ScanConsistency can be set to any logical timestamp Indicate stale=false to stale=ok and everything in between
  31. 31. ©2015 Couchbase Inc. 31 Flexible Consistency Settings  Time t1 insert (k1, v1) … t2 do other business logic computation … t3 issue query/read on (k1,v1) with t3 vs t1 Catch up all the indexes to t3 and then issue query Identical to “stale=false” Catch up all the indexes to t1 and then issue query Improved efficiency over “stale=false”
  32. 32. ©2015 Couchbase Inc. 32 ComplexTypes and GSI  Indexing ComplexTypes  Sub-documents attributes CREATE INDEX ifriend_id ON default(friends.id) USING GSI; SELECT * FROM default WHERE friends.id= "002819”; { "id": "00000000000001", "desc": ”---", "type": "friends", "tags": [0,1,2,3,4,5,6,7,8,9], "friends": { "id": "002819", "class": "005" } }
  33. 33. ©2015 Couchbase Inc. 33 ComplexTypes and GSI  Indexing ComplexTypes  Compound Keys CREATE INDEX ifriends_id_class ON default(friends.id, friends.class) USING GSI; SELECT * FROM default WHERE friends.id="002819" and friends.class="005”; { "id": "00000000000001", "desc": ”---", "type": "friends", "tags": [0,1,2,3,4,5,6,7,8,9], "friends": { "id": "002819", "class": "005" } }
  34. 34. ©2015 Couchbase Inc. 34 ComplexTypes and GSI  Indexing ComplexTypes  Sub-documents CREATE INDEX ifriend ON default(friends) USING GSI; SELECT * FROM default WHERE friends= {"class": "005","id":"002819"}; { "id": "00000000000001", "desc": ”---", "type": "friends", "tags": [0,1,2,3,4,5,6,7,8,9], "friends": { "id": "002819", "class": "005" } }
  35. 35. ©2015 Couchbase Inc. 35 ComplexTypes and GSI  Indexing ComplexTypes  Arrays CREATE INDEX itags_sorted ON default(ARRAY_SORT(tags)) USING GSI; SELECT * FROM default WHERE tags= ARRAY_SORT([0,1,2,3,4,5,6,7,8,9]); { "id": "00000000000001", "desc": ”---", "type": "friends", "tags": [0,1,2,3,4,5,6,7,8,9], "friends": { "id": "002819", "class": "005" } }
  36. 36. Q&A Cihan Biyikoglu cihan@couchbase.com @cihangirb
  37. 37. Couchbase.com/beta
  38. 38. Thank you.

×