Solu%ons	
  Architect	
  
email:	
  don.stacy@couchbase.com	
  
Don	
  Stacy	
  
Couchbase	
  105:	
  	
  
XDCR	
  and	
  ...
Cross	
  Data	
  Center	
  Replica0on	
  
(XDCR)	
  
•  Deliver	
  highly	
  performant,	
  asynchronous	
  data	
  replica0on	
  
•  Provide	
  for	
  disaster	
  recovery	
 ...
Intracluster	
  Replica0on	
  
Intercluster	
  Replica0on	
  
Key	
  Features	
  
• Replicates existing and modified data FROM source cluster
• Compares document revisions before transf...
Storage	
  to	
  XDCR	
  Replica0on	
  
Remote	
  	
  
Cluster	
  
XDCR:	
  Eventually	
  Consistent	
  
Remote	
  	
  
Cluster	
  
Time	
  2	
  
Time	
  3	
  
Time	
  1	
  
Bucket	
  A	
  
Bucket	
  B	
  
Bucket	
  C	
  
Cluster	
  1	
  
Bucket	
  A	
  
Bucket	
  B	
  
Bucket	
  C	
  
Cluster	
...
Topology	
  and	
  Use	
  Cases	
  
Uni-­‐Direc0onal	
  
•  Hot	
  spare	
  /	
  Disaster	
  recovery	
  
•  Development/Tes0ng	
  copies	
  
•  Heavy	
  repo...
Chain	
  
Data	
  aggrega0on	
  
Propaga0on	
  
Bi-­‐Direc0onal	
  (aka	
  Ac0ve/Ac0ve)	
  
•  Mul0ple	
  Ac0ve	
  Masters	
  
•  Disaster	
  Recovery	
  
•  Datacenter	
...
replace(“key”, data)	
  replace(“key”, data)	
  
(conflict)	
  
Bi-­‐Direc0onal	
  Conflicts	
  
What	
  happens	
  when	
  ...
Bi-­‐Direc0onal	
  Conflicts	
  
Resolu0on	
  Strategy	
  
•  XDCR	
  is	
  eventually	
  consistent;	
  checks	
  document...
Bi-­‐Direc0onal	
  Cau0on	
  
•  Avoid	
  upda0ng	
  the	
  same	
  document	
  in	
  mul0ple	
  clusters	
  with	
  
bi-­...
Opera0onal	
  Details	
  
XDCR	
  Opera0ons	
  
Configurable	
  Replica0on	
  
•  Documents	
  are	
  pushed	
  to	
  XDCR	
  a_er	
  disk	
  write	
...
XDCR	
  Opera0ons	
  
Advanced	
  Seangs	
  
Use	
  Version	
  2	
  (XMEM)	
  for	
  all	
  
except	
  Elas0csearch	
  int...
XDCR Ports-Connections	
  
Retrieve	
  Target	
  Cluster	
  Map	
  &	
  
Configura0on	
  
	
  
Data	
  Replica0on	
  
Mode	...
Replication Follows Cluster Map	
  
	
  
Different	
  Configura0on	
  OK	
  
XDCR	
  Planning	
  
Impact	
  on	
  Cluster	
  Sizing	
  
Your	
  clusters	
  need	
  to	
  be	
  sized	
  for	
  XDCR	
 ...
XDCR	
  Planning	
  
Cloud/Internet	
  Considera0ons	
  (EC2,	
  internet	
  WAN,	
  etc.)	
  
•  Make	
  sure	
  ports	
 ...
3	
  3	
   2	
  
Document	
  Lifecycle	
  Review	
  
Bringing	
  the	
  webinars	
  together	
  2	
  
Managed	
  Cache	
  ...
Interface	
  Demo	
  
Elas0csearch	
  
Integrated via XDCR	
  
Architecture	
  
Using Elastic Search	
  
Application Workflow	
  
Applica%on	
  
Search Query	
  
result	
  set	
  
	

Retrieve Docs	
  doc...
Addi0onal	
  Resources	
  
•  Elas%csearch	
  Install	
  &	
  Configure	
  Gist	
  
•  Elas%csearch	
  Transport	
  Plugin	...
Interface	
  Demo	
  
Q&A	
  
email:	
  don.stacy@couchbase.com	
  
Upcoming SlideShare
Loading in...5
×

Couchbase 105: XDCR and Elasticsearch

1,477

Published on

How-To NoSQL with Couchbase: Couchbase 105 - Cross Datacenter Replication and Elasticsearch

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,477
On Slideshare
0
From Embeds
0
Number of Embeds
5
Actions
Shares
0
Downloads
53
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Different from intra-cluster replication
    Replicate your Couchbase data across clusters
    Clusters may be spread across geographies
    Configured on a per-bucket basis
    Allows for:
    Test and Development Cluster Creation (Real Production Data)
    Active Backup (Uni-Directional)
    Geographically local data for better performance (Bi-Directional)
    Many to one scenarios (Aggregation, Propagation)
  • More details on conflict management here: http://www.couchbase.com/docs/couchbase-manual-2.0/couchbase-admin-tasks-xdcr-functionality.html
  • Replication occurs over port 8092
    Cluster configuration is loaded on port 8091
  • Replication occurs between every node in each cluster
    Communication is direct between vBuckets on each node
    Every node must communicate on 8091 & 8092 (or the SSL ports) to remote cluster
  • This slide has an click-by-click animation
    1.  (click) A set request comes in from the application .
    2.  Couchbase Server responses back that they key is written
    3. (click)Couchbase Server then Replicates the data out to memory in the other nodes
    At the same time it is put the data into a write queue to be persisted to disk
    (click)Once it is on disk, the item is processed by the view engine and sent out any configured XDCR link to one or more clusters
  • Couchbase 105: XDCR and Elasticsearch

    1. 1. Solu%ons  Architect   email:  don.stacy@couchbase.com   Don  Stacy   Couchbase  105:     XDCR  and  Elas%csearch  
    2. 2. Cross  Data  Center  Replica0on   (XDCR)  
    3. 3. •  Deliver  highly  performant,  asynchronous  data  replica0on   •  Provide  for  disaster  recovery  and  high  availability   •  Allow  for  varying  topologies  and  replica0on  schemes   •  Support  data  locality   •  Support  load  separa0on   •  Reduce  opera0ons  effort  and  cost  
    4. 4. Intracluster  Replica0on  
    5. 5. Intercluster  Replica0on  
    6. 6. Key  Features   • Replicates existing and modified data FROM source cluster • Compares document revisions before transfer -  De-duplicates writes to disk -  Only the last version is written to disk and thus passed to XDCR • Simplified Administration via console, REST, and CLI • Automatically handles node addition and removal • Automatically resumes after network disruption • Clusters may be different sizes and configurations -  Clusters must use the same platform (supported/tested)
    7. 7. Storage  to  XDCR  Replica0on   Remote     Cluster  
    8. 8. XDCR:  Eventually  Consistent   Remote     Cluster   Time  2   Time  3   Time  1  
    9. 9. Bucket  A   Bucket  B   Bucket  C   Cluster  1   Bucket  A   Bucket  B   Bucket  C   Cluster  2   Cluster  Mechanics   Replica0on  between  clusters  
    10. 10. Topology  and  Use  Cases  
    11. 11. Uni-­‐Direc0onal   •  Hot  spare  /  Disaster  recovery   •  Development/Tes0ng  copies   •  Heavy  repor0ng   •  Integrate  to  Elas0csearch   •  Integrate  to  custom  consumer  
    12. 12. Chain  
    13. 13. Data  aggrega0on  
    14. 14. Propaga0on  
    15. 15. Bi-­‐Direc0onal  (aka  Ac0ve/Ac0ve)   •  Mul0ple  Ac0ve  Masters   •  Disaster  Recovery   •  Datacenter  Locality  
    16. 16. replace(“key”, data)  replace(“key”, data)   (conflict)   Bi-­‐Direc0onal  Conflicts   What  happens  when  you  write  the  same  key  in  mul0ple  clusters?  
    17. 17. Bi-­‐Direc0onal  Conflicts   Resolu0on  Strategy   •  XDCR  is  eventually  consistent;  checks  document  metadata   to  resolve  conflicts:   1.  Numerical  sequence  (incremented  on  each  muta%on)     2.  CAS  value     3.  Expira%on  (TTL)  value   •  All  clusters  will  pick  the  same  “winner”     {  …  }     3   3   {  …  }   Doc  1  on  DC1   Doc  1  on  DC2   Winner  
    18. 18. Bi-­‐Direc0onal  Cau0on   •  Avoid  upda0ng  the  same  document  in  mul0ple  clusters  with   bi-­‐direc0onal  XDCR   ­  Be  sure  to  understand  the  conflict  resolu%on  rules   •  Best  Prac0ces   ­  Data  Center  s%ckiness   •  Keep  users/transac%ons  isolated  to  a  DC   •  Only  redirect  to  another  DC  in  case  of  major  outage   ­  Use  separate  key  spaces  (e.g.,  DC  prefix)  to  avoid  conflicts  on  individual   documents.    Example:   •  DC1::user:a9838-­‐s92-­‐s00   •  DC2::user:293ba-­‐293-­‐922  
    19. 19. Opera0onal  Details  
    20. 20. XDCR  Opera0ons   Configurable  Replica0on   •  Documents  are  pushed  to  XDCR  a_er  disk  write  **   ­  Mul%ple  document  changes  are  combined  to  reduce  network  bandwidth   ­  32  parallel  data  streams  work  across  vBuckets   •  Checkpoints  are  used  to  maintain  progress  per  vBucket   ­  Allows  for  stop/restart  without  transferring  all  data   ­  Op%mizes  data  exchange   ­  Transient  failures  cause  a  pause  and  restart   •  Op0mis0c  XDCR   ­  Bandwidth  saving  feature   ­  <256  bytes  (default,  configurable)  sent  without  checking  first   ­  >=256  bytes:  des%na%on  checked  first  before  sending  
    21. 21. XDCR  Opera0ons   Advanced  Seangs   Use  Version  2  (XMEM)  for  all   except  Elas0csearch  integra0on   8-­‐256;  Increase  on  high-­‐end  HW   60  to  14,400  (seconds);  Interval   between  checkpoints   500-­‐1,000;  Doc  batching  count;   Increase  2-­‐3x  for  unidirec0onal   10-­‐100,000  KB;  Doc  batching   size;  Increase  2-­‐3x  for   unidirec0onal     1-­‐300  seconds;  Lower  when   expec0ng  network  failures   0  to  2,097,152  Bytes  (20MB)   compressed;  Op0mis0c  below   this  size  
    22. 22. XDCR Ports-Connections   Retrieve  Target  Cluster  Map  &   Configura0on     Data  Replica0on   Mode   Required  Open  Ports   Non-­‐SSL   8091,  8092   SSL  w/  Version2/XMEM   11214,  11215,  18092   SSL  w/  Version1/CAPI   11214,  11215,  18091  
    23. 23. Replication Follows Cluster Map     Different  Configura0on  OK  
    24. 24. XDCR  Planning   Impact  on  Cluster  Sizing   Your  clusters  need  to  be  sized  for  XDCR   •  XDCR  is  CPU  intensive     ­  Configure  the  number  of  parallel  streams  based  on  your  CPU  capacity   (xdcrMaxConcurrentReps)   •  You  are  doubling  your  I/O  usage   ­  I/O  capacity  needs  to  be  sized  correctly   •  Network  bandwidth  will  likely  increase   •  You  will  need  more  memory,  par0cularly  if  bi-­‐direc0onal   ­  Memory  capacity  needs  to  be  sized  correctly  
    25. 25. XDCR  Planning   Cloud/Internet  Considera0ons  (EC2,  internet  WAN,  etc.)   •  Make  sure  ports  are  open  and  IP’s  are  reachable  on  both  sides   •  Use  the  public  IP  address  in  the  des0na0on  cluster   •  Use  a  DNS  to  avoid  individual  IP  address  changes   •  Traffic  is  not  encrypted  prior  to  Couchbase  2.5   ­  Leverage  op%onal  SSL  in  2.5+  
    26. 26. 3  3   2   Document  Lifecycle  Review   Bringing  the  webinars  together  2   Managed  Cache   Disk  Queue   Disk   Replica%on   Queue   App  Server   Couchbase  Server  Node   Doc  1  Doc  1   Doc  1   To  other  node   XDCR  Queue   Doc  1   To  other   cluster  or   Elas%csearch     View  engine     Doc  1  
    27. 27. Interface  Demo  
    28. 28. Elas0csearch  
    29. 29. Integrated via XDCR  
    30. 30. Architecture  
    31. 31. Using Elastic Search   Application Workflow   Applica%on   Search Query   result  set   Retrieve Docs  docs  
    32. 32. Addi0onal  Resources   •  Elas%csearch  Install  &  Configure  Gist   •  Elas%csearch  Transport  Plugin  Github  Repo   •  Elas%csearch  Head  Plugin  Github  Repo   •  Couchbase  /  Elas%c  Search  Docs  
    33. 33. Interface  Demo  
    34. 34. Q&A   email:  don.stacy@couchbase.com  
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×