I presented about Couchbase with .NET Core. I talked about scaling with Couchbase, Couchbase Data Service Architecture and How we implement Couchbase with .NET Core. Thanks Steve Yen :) Project: https://github.com/peacecwz/couchbase-net-core-meetup

1. Baris Ceviz | Software Developer @Trendyol
Twitter – Github – LinkedIn : @PeaceCwz
Mail: baris.ceviz@trendyol.com
Introduction to NoSQL
Solutions with
Couchbase and .NET
Core

2. A G E N D A
01/
02/
03/
0 4 /
What’s Couchbase
Clusters and nodes
Key-Value Database
Couchbase .NET SDK

3. A Cluster
Application
SDK
App Server
Application
SDK
App Server
3
CLUSTER of
COUCHBASE NODES

4. A Cluster and Sync Gateway
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
4
CLUSTER of
COUCHBASE NODES

5. App Server
SDK
Application Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
Sync
Gateway
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
5
A Cluster and Sync Gateway
CLUSTER of
COUCHBASE NODES

6. Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
Sync
Gateway
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
6
A Cluster and Sync Gateway
Kubernetes
CLUSTER of
COUCHBASE NODES

7. Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
Sync
Gateway
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
A Cluster and Sync Gateway
CLUSTER of
COUCHBASE NODES
7

8. Kubernetes
Autonomous
Operator
A Cluster and Sync Gateway and Kubernetes Autonomous
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
8
CLUSTER of
COUCHBASE NODES

9. Nodes
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager

10. Nodes
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager

11. Mix & match subsets of services (Multidimensional Scaling)
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE

12. Mix & match subsets of services (Multidimensional Scaling)
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE

13. Cluster
Manager
Cluster
Manager
Cluster
Manager
Cluster
Manager
Cluster
Manager
Cluster
Manager
Cluster
Manager
Cluster
Manager
Analytics SearchKey ValueQueryIndexing Key Value Key Value
Mix & match subsets of services (Multidimensional Scaling)
Analytics

14. The Cluster Manager
The control plane of the cluster:
• Admin APIs & User Interface
• Authentication / RBAC
• Master Services
• Rebalance and Auto-Failover
• Automatic shard placement
• Bucket Services
• Per-node Services
• Generic facilities
per-node-&-bucket services
Generic distributed facilities
Generic local facilities
Logging and other services
Distributed node discovery
Master Services
• Cluster-wide operations
• Rebalance
• Auto-failover
• Automatic shard placement
Global config (gossip replication)
Local config Store
Per-node Services
• Heartbeats
• Babysitter
Cluster Manager
Bucket services
• Replica stream management
• Stats collection & monitoring
14
Admin Console & REST API Authentication

15. Automatic Sharding
26

16. KEY VALUE DATA
SERVICE
App Server
SDK
Application
HASH("user_profile-1234")
==> 27
VB Node
0 A
1 B
2 C
…
27 C
1023 D
{
…
}
Automatic Sharding and Rebalancing
NODE A NODE B NODE C NODE D
CLUSTER MANAGER
16
user_profile-1234

17. App Server
SDK
Application
VB Node
0 A
1 B
2 C
…
27 C
1023 D
Automatic Sharding and Rebalancing
CLUSTER MANAGER
KEY VALUE DATA
SERVICE
NODE A NODE B NODE C NODE D
CLUSTER MANAGER
17
user_profile-1234
{
…
}

18. App Server
SDK
Application
VB Node
0 A
1 B
2 C
…
27 C
1023 D
Automatic Sharding and Rebalancing
KEY VALUE DATA
SERVICE
NODE A NODE B NODE C NODE D NODE E
CLUSTER MANAGER
18
user_profile-1234
{
…
}

19. CL
DATA
SERVICE
App Server
SDK
Application
VB Node
0 A
1 B
2 C
…
27 C
1023 D
{
…
}
NODE A NODE B NODE C NODE D
VB Node
0 A
1 B
2 C
…
27 E
1023 D
NODE E
Automatic Sharding and Rebalancing
CLUSTER MANAGER
KEY VALUE DATA
SERVICE
HASH(“user_profile-1234”)
==> 27
user_profile-1234
19

20. Efficient Data Propagation

21. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
21
The Database C h a n g e Protocol

22. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke
22
y Value D ata
Service
The Database C h a n g e Protocol

23. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke y Value Data
Service
The Database C h a n g e Protocol
D C P
23

24. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke y Value Data
Service
The Database C h a n g e Protocol
D C P
24

25. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke y Value D ata
Service
The Database C h a n g e Protocol
D C PD C P
25

26. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
The Database C h a n g e Protocol
D C P
Key Value Data
Service
D C P
26

27. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke y Value D ata
Service
The Database C h a n g e Protocol
OT H E R F EAT U R ES &S E RV I C ES
• XDCR & Backup
• Sync Gateway
D C PD C P
27

28. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Key Value Data
Service
D C P
The Database C h a n g e Protocol
INTEGRATIONS
• hadoop, kafka & more
OT H E R F EAT U R ES &S E RV I C ES
• XDCR & Backup
• Sync Gateway
D C P
28

29. Key Value
…
Networking I/O & handler threads
VBucket Manager
Item
Pager
Expiry
Pager
Checkpoint
Manager
Cache ..Partition
Hash Tables
Fetcher
Scheduler
Reader IO
Writer IO
Generic Tasks
Flusher
The Key Value Data Service
Authentication
29
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager

30. Key Value
…
Networking I/O & handler threads
VBucket Manager
Item
Pager
Expiry
Pager
Checkpoint
Manager
Cache ..
H
Partition
ash Tables
Fetcher
Scheduler
Reader IO
Writer IO
Generic Tasks
Flusher
The Key Value Data Service
The KV "Data Plane" of a Couchbase cluster:
● High performance memcached DNA
● With data partitioning ("vbuckets")
● Sub-Document operations
● Authentication / RBAC
And also...
● Asynch task workers and scheduler
● For I/O: persistence, backfill
● For expiry & cleanup
● For replication
● The source of Database Change Protocol
● A consumer of Database Change Protocol
30
Authentication

31. Multiple Clusters
in Different Regions

32. z oom ing out
32
ASIA CLUSTER
US WEST CLUSTER
US EAST CLUSTER

33. z oom ing out
33
ASIA CLUSTER
US WEST CLUSTER
US EAST CLUSTER
XDCR
XDCRXDCR

34. Couchbase SDK
with .NET Core

35. e 2017. All rights reserved.
May the sorce
be with you