Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Index management in Couchbase 5.0 for easier and higher availability – Couchbase Connect New York 2017

193 views

Published on

How many indexes do you need in order to cover your application queries, and how do you ensure that they are always reliable and balanced? This session will focus on the new index manageability features being added in Couchbase Server 5.0. We’ll be exploring several key indexing features, including automatic index replication and index rebalancing. If you’re an operations expert looking to improve index reliability and distribution, this is the session for you!

Published in: Software
  • Be the first to comment

Index management in Couchbase 5.0 for easier and higher availability – Couchbase Connect New York 2017

  1. 1. ©2017 Couchbase Inc. Index Management in 5.0 Perry Krug, Principal Architect 1
  2. 2. ©2017 Couchbase Inc. Index Service • “Global Secondary Indexes” • Index Service can be deployed on a dedicated node • Index Service can be co-deployed with other services on the same node 2
  3. 3. ©2017 Couchbase Inc. Pre-5.0 • GSI introduced in 4.0 • First release didn’t have automatic replicas or rebalancing • It was _possible_ to setup and manage manually • But it wasn’t elegant… • Enter Couchbase Server 5.0, administrators rejoice! 3
  4. 4. ©2017 Couchbase Inc. Index Replicas: • Provides HA for indexing service • Auto-placement: Couchbase picks index nodes for initial placement • Rule 1: Do not place replica onto the same node • Rule 2: Spread out replica onto as many server groups as possible • Rule 3: Pick the nodes that have fewest number of indexes (room for improvement in the future) • Overridden by specifying list of nodes • Auto-load balanced: • Any indexes or replicas with matching definitions are load balanced by query engine • Query engine uses ever-smarter algorithm for determining which index to use 4
  5. 5. ©2017 Couchbase Inc. Index Replicas: • Auto-placed: CREATE INDEX index_name on bucket_name(“type”) WITH {“num_replica”:1}; • Application specified: CREATE INDEX index_name on bucket_name(“type”) WITH {“nodes”:[192.168.1.1:8091, 192.168.1.2:8091]}; • All replicas dropped together: DROP INDEX bucket_name.index_name; 5
  6. 6. ©2017 Couchbase Inc. Replica Status • Index Replica 1 – 127.0.0.1:9001 • Index Replica 2 – 127.0.0.1:9004 • Index Replica 3 – 127.0.0.1:9005 6
  7. 7. ©2017 Couchbase Inc. Replica Statistics 7
  8. 8. ©2017 Couchbase Inc. GSI Rebalancing: Planned and unplanned • Planned: • Scaling out (doesn’t re-place existing indexes we don’t have enough data to do this reliably…yet) • Scaling in • Upgrades of hardware or software • Unplanned: • Node failures: recreate lost indexes • It does the “right” thing: • Indexes are moved gracefully when possible • Replicas are recreated if missing (only upon rebalance: to prevent overloads and cascading failures) • Individual indexes or their replicas are only activated once fully built 8
  9. 9. ©2017 Couchbase Inc. Add capacity for new index • Adding new indexer node for additional indexes to be created • Need to add 2 more indexer nodes for additional indexes. • Solution : Add 2 new nodes and do not eject any node • Will not place existing index onto new node if there is no ejected node 9 Index Node 1 Index#2Index#1 Index#4Index#3 Index Node 2 Replica#2Replica#1 Replica#4Replica#3 Index Node 3 Index#6Index#5 Index Node 4 Replica#6Replica#5
  10. 10. ©2017 Couchbase Inc. Reclaim excess capacity • Reduce capacity (scale-in) • There are 3 indexer nodes. Want to reduce to 2 nodes when peak season is over. • Solution : Eject 1 node • Will move indexes from ejected nodes to available nodes while balancing resource utilization 10 Index Node 1 Index#2Index#1 Index Node 2 Replica#2Replica#1 Index Node 3 Index#4Replica#4 Replica#3Index#3
  11. 11. ©2017 Couchbase Inc. Node Maintenance • Node Maintenance • Need to swap out 1 node for maintenance • Solution : Add 1 new node and eject 1 node • Will move indexes from ejected node to new node while preserving index layout 11 Index Node 1 Index#2Index#1 Index Node 2 Replica#2Replica#1 Index Node 3 Index#4Replica#4 Replica#3Index#3
  12. 12. ©2017 Couchbase Inc. Add capacity for congested nodes or capacity planning (new capacity) • Adding new capacity for resource planning • Need to increase capacity for 2 nodes for upcoming peak season. Want to add 2 nodes. • Solution : Add 4 new nodes and eject 2 nodes • Will move indexes from ejected nodes to available nodes while balancing resource utilization 12 Index Node 1 Index Node 2 Index Node 1 Replica#2Index#1 Index Node 2 Replica#3Index#2 Index Node 4 Replica#1Index#4 Index#2Index#1 Replica#3 Index#4Index#3 Replica#1Replica#4 Replica#2 Ejected Nodes New Nodes Index Node 3 Replica#4Index#3
  13. 13. ©2017 Couchbase Inc. Add capacity for congested nodes or capacity planning (scale-up) • Adding new capacity to to relieve congested node (scale-up) • There are 2 indexer nodes are under heavy load. Want to replace with 2 larger nodes. • Solution : Add 2 new nodes and eject 2 congested nodes • Will move indexes from 2 ejected nodes to new nodes while preserving index layout • Also for software upgrades (Couchbase or OS) 13 Index Node 1 Index Node 2 Index Node 1 Index#2Index#1 Index Node 2 Index#4Index#3 Replica#3 Replica#1 Index#2Index#1 Replica#3 Index#4Index#3 Replica#1Replica#4 Replica#2 Replica#4 Replica#2 Ejected Smaller Nodes Larger Nodes with Additional Capacity
  14. 14. ©2017 Couchbase Inc.©2017 Couchbase Inc. 14 Demo
  15. 15. ©2017 Couchbase Inc. What else you need to know • Index rebalancing happens after data service rebalancing • Index scan will not be interrupted during rebalancing • Index scan will automatically redirect/retry on new node after index is moved • As index is moved to new node, it becomes “ready” only after it catches up with latest traffic • Rebalancing will try to minimize variations of memory and CPU usages across nodes • Memory and CPU usages are measured at the time when rebalancing triggered • Since indexes come in different sizes, cannot guarantee perfect resource balance • Will attempt to balance resource in best-effort manner • If rebalancing fails or stops, it will not revert back to the original layout • Make sure that you have enough capacity when sizing down • You cannot perform Create/Drop/Build index during rebalancing • Rebalancing does not take into consideration on any explicit index placement (create index with {“nodes”:”193.2.33.104:8091”}) 15
  16. 16. ©2017 Couchbase Inc. Move Index - cbindex cbindex -type move -auth "Administrator:asdasd" -server "127.0.0.1:9000" -index country -bucket default -with '{"nodes":["127.0.0.1:9002","127.0.0.1:9003","127.0.0.1:9004"]}’ • Move index replica manually from source node to target node • Index currently resided in • Replica 0: 127.0.0.1:9001 • Replica 1:127.0.0.1:9003 • Replica 2: 127.0.0.1:9004 • Move index figures out new layout requires moving index from 127.0.0.1:9001 to 127.0.0.1:9002 16
  17. 17. ©2017 Couchbase Inc. In the future: ALTER_INDEX • Once setup, there are various needs for modifying an index configuration: • Changing number of replicas • Editing index fields • Today that needs to be done manually: 1. Create new index with desired configuration 2. Drop old index • In development is ALTER_INDEX to take care of the change automatically 17
  18. 18. ©2017 Couchbase Inc. 18 Perry Krug PrincipalArchitect perry@couchbase.com IMAGE GOES HERE
  19. 19. ©2017 Couchbase Inc.©2017 Couchbase Inc. ThankYou! 19

×