Couchbase 105 - Cross Datacenter Replication and Elasticsearch

2,458 views

Published on

Learn the architectural principles behind Cross Datacenter Replication (XDCR), configuration parameters, atomicity implications, and some use cases. Learn how to integrate Couchbase and Elasticsearch for flexible full text searching to compliment Couchbase Views with additional query flexibility.

What will be covered during this training:

CROSS DATACENTER REPLICATION

Cross-Data-Center-Replication (XDCR) Architecture
Setup XDCR between two clusters
XDCR Configuration Parameters



ELASTICSEARCH

Elasticsearch (ES) Integration setup
Setup Couchbase Transport plugin in ES
Create replication into ES
Run Queries against ES and retrieve results against Couchbase

Published in: Technology, Sports
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,458
On SlideShare
0
From Embeds
0
Number of Embeds
364
Actions
Shares
0
Downloads
87
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Couchbase 105 - Cross Datacenter Replication and Elasticsearch

  1. 1. Couchbase  105:     XDCR  and  elasEcsearch Jasdeep  Jaitla Technical  Evangelist email:  jasdeep@couchbase.com twi0er:  @scalabl3
  2. 2. (XDCR)
  3. 3. CROSS  DATA  CENTER  REPLICATION (XDCR)
  4. 4. Replication Use Cases • Unidirectional - Create Backup Cluster - Create Geographic Cluster, Chained Propagation - Create Development/Test Cluster with Active Data - Integrate with Elastic Search - Integrate with Custom Consumer • Bi-Directional - Populate Multiple Clusters with Data Modified in each Cluster
  5. 5. Active-Active Replication • XDCR configured Per Bucket • Within Bucket, Each Partition (vbucket) Replicated Directly - Data is replicated from Active Partition -> Active Partition - Each Partition (1024) has a separate append only file ! - XDCR process: ๏ Queues documents for replication ๏ Transmits Documents via HTTP in batches ๏ Keeps track of Checkpoint Intervals if Replication Streams are Disconnected ๏ Upon disconnection between clusters, automatically handles reconnection attempts
  6. 6. XDCR Ports-Connections 8091 8091 MAP MAP 8092 8092
  7. 7. XDCR Ports-Connections Retrieve  Target  Cluster  Map  &   Cluster  Map 8091 HTTP 8091 MAP MAP 8092 8092
  8. 8. XDCR Ports-Connections Retrieve  Target  Cluster  Map  &   Cluster  Map 8091 HTTP 8091 MAP MAP 8092 HTTP Data  Replica4on 8092
  9. 9. Active-to-Active MAP MAP Partitions Partitions 0 1 2 3 0 1 2 3 1020 1020 1021 1021 1022 1022 1023 1023
  10. 10. Replication Follows Cluster Map MAP 512 3 IO Workers Partitions 512 512 MAP Partitions 8 GB RAM MAP 6 IO Workers 3 IO Workers Partitions 16 GB RAM 512 6 IO Workers Partitions 8 GB RAM 16 GB RAM MAP
  11. 11. Replication Follows Cluster Map MAP 512 3 IO Workers Partitions 512 512 MAP Partitions 8 GB RAM MAP 6 IO Workers 3 IO Workers Partitions 16 GB RAM 512 6 IO Workers Partitions 8 GB RAM 16 GB RAM MAP
  12. 12. Replication Follows Cluster Map MAP 512 3 IO Workers Partitions 512 512 MAP Partitions 8 GB RAM MAP 6 IO Workers 3 IO Workers Partitions 16 GB RAM 512 6 IO Workers Partitions 8 GB RAM 16 GB RAM MAP
  13. 13. Replication Follows Cluster Map MAP 512 3 IO Workers Partitions 512 512 MAP Partitions 8 GB RAM MAP 6 IO Workers 3 IO Workers Partitions 16 GB RAM 512 6 IO Workers Partitions 8 GB RAM 16 GB RAM Different  Configura4on  OK MAP
  14. 14. Cluster Sizes Don’t Have to Match 256 3 IO Workers Partitions 8 GB RAM 256 3 IO Workers 512 Partitions Partitions 8 GB RAM 256 3 IO Workers 8 GB RAM 512 3 IO Workers Partitions Partitions 8 GB RAM 256 3 IO Workers 3 IO Workers Partitions 8 GB RAM 8 GB RAM
  15. 15. Cluster Sizes Don’t Have to Match 256 3 IO Workers Partitions 8 GB RAM 256 3 IO Workers 512 Partitions Partitions 8 GB RAM 256 3 IO Workers 3 IO Workers 8 GB RAM 512 3 IO Workers Partitions Partitions 8 GB RAM 8 GB RAM Smaller  Cluster 256 3 IO Workers Partitions 8 GB RAM
  16. 16. Cluster Sizes Don’t Have to Match 3 IO Workers 256 Partitions 8 GB RAM 512 3 IO Workers 3 IO Workers Partitions 256 Partitions 8 GB RAM 8 GB RAM 512 3 IO Workers 3 IO Workers Partitions 256 Partitions 8 GB RAM 8 GB RAM 3 IO Workers 256 Partitions 8 GB RAM
  17. 17. Cluster Sizes Don’t Have to Match 3 IO Workers 256 Partitions 8 GB RAM 512 3 IO Workers 3 IO Workers Partitions 256 Partitions 8 GB RAM 8 GB RAM 512 3 IO Workers 3 IO Workers Partitions 256 Partitions 8 GB RAM 8 GB RAM 3 IO Workers 256 Partitions 8 GB RAM Larger  Cluster
  18. 18. Chaining XDCR Partitions Partitions Partitions 0 1 2 3 0 1 2 3 0 1 2 3 1020 1020 1020 1021 1021 1021 1022 1022 1022 1023 1023 1023
  19. 19. Chaining XDCR All  Writes Partitions Partitions Partitions 0 1 2 3 0 1 2 3 0 1 2 3 1020 1020 1020 1021 1021 1021 1022 1022 1022 1023 1023 1023
  20. 20. Chaining XDCR All  Writes Partitions Partitions Partitions 0 1 2 3 0 1 2 3 0 1 2 3 1020 1020 1020 1021 1021 1021 1022 1022 1022 1023 1023 1023 Unidirec4onal  XDCR
  21. 21. Chaining XDCR All  Writes Partitions Partitions Partitions 0 1 2 3 0 1 2 3 0 1 2 3 1020 1020 1020 1021 1021 1021 1022 1022 1022 1023 1023 1023 Unidirec4onal  XDCR 1.5  
  22. 22. Chaining XDCR
  23. 23. Storage to XDCR Replication Couchbase Server RAM Cache EP Engine Disk Write Queue View Engine Indexers Application Server Replication Queue Intra-Cluster Node XDCR Replication Queue Remote     Cluster
  24. 24. Storage to XDCR Replication Couchbase Server RAM Cache storage ops EP Engine Disk Write Queue View Engine Indexers Application Server Replication Queue Intra-Cluster Node XDCR Replication Queue Remote     Cluster
  25. 25. Storage to XDCR Replication Couchbase Server RAM Cache storage ops EP Engine Disk Write Queue View Engine Indexers Application Server Replication Queue Intra-Cluster Node XDCR Replication Queue Remote     Cluster
  26. 26. Storage to XDCR Replication Couchbase Server RAM Cache storage ops EP Engine Disk Write Queue View Engine Indexers Application Server Replication Queue Intra-Cluster Node XDCR Replication Queue Remote     Cluster
  27. 27. XDCR: Eventually Consistent Couchbase Server RAM Cache EP Engine Disk Write Queue View Engine Indexers Application Server Replication Queue Intra-Cluster Node XDCR Replication Queue Remote     Cluster
  28. 28. XDCR: Eventually Consistent Time  1 RAM Cache storage ops Couchbase Server EP Engine Disk Write Queue View Engine Indexers Application Server Replication Queue Intra-Cluster Node XDCR Replication Queue Remote     Cluster
  29. 29. XDCR: Eventually Consistent Time  1 RAM Cache storage ops Couchbase Server EP Engine View Engine Disk Write Queue Application Server Indexers Time  2 Replication Queue Intra-Cluster Node XDCR Replication Queue Remote     Cluster
  30. 30. XDCR: Eventually Consistent Time  1 RAM Cache storage ops Couchbase Server EP Engine View Engine Disk Write Queue Application Server Indexers Time  2 Replication Queue Intra-Cluster Node XDCR Replication Queue Remote     Cluster Time  3
  31. 31. Bi-Directional Conflicts What happens when you write to the same key in two clusters? Cluster 1 App Servers Cluster 2 App Servers
  32. 32. Bi-Directional Conflicts What happens when you write to the same key in two clusters? Cluster 1 App Servers replace(“key”, data) Cluster 2 App Servers replace(“key”, data)
  33. 33. Bi-Directional Conflicts What happens when you write to the same key in two clusters? Cluster 1 Cluster 2 App Servers replace(“key”, data) App Servers (conflict) replace(“key”, data)
  34. 34. CONFLICT  RESOLUTION
  35. 35. Conflict Resolution Rules • Completely Automatic, Currently Not Customizable ! - Rule 1: Revision #/Count ๏ Whichever cluster has higher revision number will replicate to other cluster ๏ If document is mutated in cluster with lower revision number, the document will not be put in replication queue ! - Rule 2: Metadata ๏ If Revision Count equal, use CAS, Flags, and TTL ๏ Basically Random ! • It is not Time/NTP based
  36. 36. Bi-Directional Caution! Avoid (as in don’t) update the same document in multiple clusters with bi-directional XDCR, as it may not have expected behavior due to the conflict resolution rules. ! Recommended Separate Key spaces (use DC prefix) so that you avoid conflicts on individual documents. DC1::user:a9838-­‐s92-­‐s00   DC2::user:293ba-­‐293-­‐922
  37. 37. The Document Life • Storage operation (set/add/replace/incr/decr) • Stored in RAM Cache ๏ Can be Read by get operation • Added to Disk Write Queue/Replication Queue • Pushed to Replica Node RAM Cache (and DWQ) ๏ Exists after Failover • Persisted to Active Partition Disk • Added to XDCR Replication Queue • Indexed by View Indexers (Design Document) ๏ Can be Queried in View • Pushed to XDCR Remote Cluster RAM Cache ๏ Can be Read on Remote Cluster get operation • Indexed by Elastic Search ๏ Can be Queried in Elastic Search
  38. 38. INTERFACE  DEMO
  39. 39. Q  &  A
  40. 40. ELASTIC  SEARCH
  41. 41. Integrated via XDCR
  42. 42. Integrated via XDCR
  43. 43. Architecture Couchbase Server RAM Cache EP Engine Disk Write Queue Application Server Replication Queue Replica Couchbase Intra-Cluster Machine XDCR Replication Queue View Engine Indexers
  44. 44. Architecture Couchbase Server RAM Cache EP Engine Disk Write Queue storage ops Application Server Replication Queue Replica Couchbase Intra-Cluster Machine XDCR Replication Queue View Engine Indexers
  45. 45. Architecture Couchbase Server RAM Cache EP Engine Disk Write Queue storage ops Application Server Replication Queue Replica Couchbase Intra-Cluster Machine XDCR Replication Queue View Engine Indexers
  46. 46. Architecture Couchbase Server RAM Cache EP Engine Disk Write Queue storage ops Application Server Replication Queue Replica Couchbase Intra-Cluster Machine XDCR Replication Queue View Engine Indexers
  47. 47. Architecture Couchbase Server RAM Cache EP Engine Disk Write Queue storage ops Application Server Replication Queue Replica Couchbase Intra-Cluster Machine XDCR Replication Queue View Engine Indexers
  48. 48. Architecture Couchbase Server RAM Cache EP Engine Disk Write Queue storage ops Application Server Replication Queue Replica Couchbase Intra-Cluster Machine XDCR Replication Queue View Engine Indexers
  49. 49. Architecture Couchbase Server RAM Cache EP Engine Disk Write Queue storage ops Application Server Replication Queue Replica Couchbase Intra-Cluster Machine XDCR Replication Queue View Engine Indexers
  50. 50. Using Elastic Search Application Workflow Applica>on
  51. 51. Using Elastic Search Application Workflow result  set Applica>on
  52. 52. Using Elastic Search Application Workflow Applica>on docs
  53. 53. INTERFACE  DEMO
  54. 54. Q  &  A

×