I presented about Couchbase with .NET Core. I talked about scaling with Couchbase, Couchbase Data Service Architecture and How we implement Couchbase with .NET Core. Thanks Steve Yen :)
Project: https://github.com/peacecwz/couchbase-net-core-meetup
4. A Cluster and Sync Gateway
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
4
CLUSTER of
COUCHBASE NODES
5. App Server
SDK
Application Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
Sync
Gateway
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
5
A Cluster and Sync Gateway
CLUSTER of
COUCHBASE NODES
8. Kubernetes
Autonomous
Operator
A Cluster and Sync Gateway and Kubernetes Autonomous
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
Application
SDK
App Server
Application
SDK
App Server
Sync
Gateway
8
CLUSTER of
COUCHBASE NODES
9. Nodes
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
10. Nodes
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
11. Mix & match subsets of services (Multidimensional Scaling)
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE
12. Mix & match subsets of services (Multidimensional Scaling)
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE COUCHBASE SERVER NODE
16. KEY VALUE DATA
SERVICE
App Server
SDK
Application
HASH("user_profile-1234")
==> 27
VB Node
0 A
1 B
2 C
…
27 C
1023 D
{
…
}
Automatic Sharding and Rebalancing
NODE A NODE B NODE C NODE D
CLUSTER MANAGER
16
user_profile-1234
17. App Server
SDK
Application
VB Node
0 A
1 B
2 C
…
27 C
1023 D
Automatic Sharding and Rebalancing
CLUSTER MANAGER
KEY VALUE DATA
SERVICE
NODE A NODE B NODE C NODE D
CLUSTER MANAGER
17
user_profile-1234
{
…
}
18. App Server
SDK
Application
VB Node
0 A
1 B
2 C
…
27 C
1023 D
Automatic Sharding and Rebalancing
KEY VALUE DATA
SERVICE
NODE A NODE B NODE C NODE D NODE E
CLUSTER MANAGER
18
user_profile-1234
{
…
}
19. CL
DATA
SERVICE
App Server
SDK
Application
VB Node
0 A
1 B
2 C
…
27 C
1023 D
{
…
}
NODE A NODE B NODE C NODE D
VB Node
0 A
1 B
2 C
…
27 E
1023 D
NODE E
Automatic Sharding and Rebalancing
CLUSTER MANAGER
KEY VALUE DATA
SERVICE
HASH(“user_profile-1234”)
==> 27
user_profile-1234
19
21. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
21
The Database C h a n g e Protocol
22. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke
22
y Value D ata
Service
The Database C h a n g e Protocol
23. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke y Value Data
Service
The Database C h a n g e Protocol
D C P
23
24. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke y Value Data
Service
The Database C h a n g e Protocol
D C P
24
25. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke y Value D ata
Service
The Database C h a n g e Protocol
D C PD C P
25
26. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
The Database C h a n g e Protocol
D C P
Key Value Data
Service
D C P
26
27. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Ke y Value D ata
Service
The Database C h a n g e Protocol
OT H E R F EAT U R ES &S E RV I C ES
• XDCR & Backup
• Sync Gateway
D C PD C P
27
28. • High Performance / In-Memory
• De-Duplication
• Ordered, predictable and consistent
• Restartable
Key Value Data
Service
D C P
The Database C h a n g e Protocol
INTEGRATIONS
• hadoop, kafka & more
OT H E R F EAT U R ES &S E RV I C ES
• XDCR & Backup
• Sync Gateway
D C P
28
29. Key Value
…
Networking I/O & handler threads
VBucket Manager
Item
Pager
Expiry
Pager
Checkpoint
Manager
Cache ..Partition
Hash Tables
Fetcher
Scheduler
Reader IO
Writer IO
Generic Tasks
Flusher
The Key Value Data Service
Authentication
29
Analytics
Query
Indexing
Full Text
Eventing
Key Value
Cluster
Manager
30. Key Value
…
Networking I/O & handler threads
VBucket Manager
Item
Pager
Expiry
Pager
Checkpoint
Manager
Cache ..
H
Partition
ash Tables
Fetcher
Scheduler
Reader IO
Writer IO
Generic Tasks
Flusher
The Key Value Data Service
The KV "Data Plane" of a Couchbase cluster:
● High performance memcached DNA
● With data partitioning ("vbuckets")
● Sub-Document operations
● Authentication / RBAC
And also...
● Asynch task workers and scheduler
● For I/O: persistence, backfill
● For expiry & cleanup
● For replication
● The source of Database Change Protocol
● A consumer of Database Change Protocol
30
Authentication