Many organizations use Apache Kafka to facilitate the flow of data between multiple applications or data sources. Thanks to Kafka’s distributed architecture, it is easy to set up a scalable and reliable broker, but doing the same with producers or consumers is quite often a fine art. This session provides a quick overview of Couchbase, describes the Couchbase Kafka Connector, and showcases a demo of how it can be used as both a source and a sink for building real-time data processing pipelines for mission-critical applications.

1. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved.
Building Scalable & Real-Time Data
Pipelines with Couchbase Kafka Connector
Denis Rosa | Developer Advocate

2. https://connect.couchbase.com/

3. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 3
Kafka
• Data Replication
• Sharding
• Fault Tolerant
• High Performance
• Highly Scalable
Operational
Data

4. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved.
Why Kafka + Couchbase?
There are some many connectors out there already ...

5. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 5
Why Kafka + Couchbase?
• Sharding
• Data Replication
• Highly Scalable
• High Performance
• Fault Tolerant
• Sharding
• Data Replication
• Highly Scalable
• High Performance
• Fault Tolerant
• Easy to Manage

6. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 6
Auto-Sharding / Replication
Replication

7. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 7
Cross Data Center Replication (XDCR)

8. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 8
Highly Scalable - Couchbase Cluster

9. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 9
Highly Scalable - Couchbase Cluster

10. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 10
N1QL - Query Service

11. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 11
High Performance

12. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 12
Fault Tolerant

13. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 13
Fault Tolerant

14. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 14
Fault Tolerant

15. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved.
Easy to Manage

16. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 16
Easy to Manage
• Cluster Manager is embedded
• Online Upgrades, Failovers, Scale Up/Down/Out

17. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 17
Easy to Manage

18. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 18
Easy to Manage

19. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 19
Easy to Manage

20. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 20
Couchbase Autonomous Operator Architecture
POD 10
Kubernetes Cluster
API Server Scheduler
Controller Manager Server
Etcd DNS
Couchbase Admission Controller Mutating Configuration Validating Configuration
Couchbase Autonomous Operator
Master
Services Config Maps
Resources
Persistent Volume Claim
Bucket Resources Backup Resources User Resources Replication Resources
POD 1 POD 2 POD 3 POD 4 POD 5 POD 6 POD 7 POD 8 POD 9
PERSISTENT
STORAGE
PERSISTENT STORAGE
PV01 PV02 PV03 PV06 PV07
DATA SERVICE
QUERY SERVICE
-
INDEX SERVICE
PERSISTENT
STORAGE
PV08 PV09
SEARCH SERVICE
PERSISTENT
STORAGE
PV04 PV05
ANALYTICS
SERVICE
POD 10
Pods
EVENTING SERVICE
Controller
Webhook
Cluster Resources Secret
namespace – kube-system
Couchbase Cluster
namespace – user-defined
PERSISTENT
STORAGE – BACKUP
PV
PVPV
namespace – monitoring
Prometheus
Alert Manager

21. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 21
Cloud-Native Database Releases
• Automated Provisioning: Deploy a
self-driving Couchbase cluster with
just one kubectl command
• Auto Recovery: Automatically
recover from node failure and bring
cluster back to capacity
• On-demand Scale: Deliver and
elastic, on-demand Couchbase data
platform that can scale-up or down
dynamically
• Configuration Management: Ability
handle all the Couchbase data
platform configuration through
centralized repository
• Production Certification: Open
source Kubernetes & RedHat
OpenShift
• Automated Upgrade: Push button
online rolling zero downtime upgrade
• Public Connectivity for
Couchbase Clients: Connecting
clients to the Couchbase cluster over
the internet via public IP-based
addressing and dynamic DNS.
• Helm Chart: Official Couchbase
charts make installation simple and
repeatable
• Production Certification Public
Cloud Kubernetes Service: AWS
EKS, Azure AKS and Google GKE
CAO 1.0 2018 CAO 1.x 2019
• Automated Security: Provide Role-
Based Access Control
Authentication and Authorization
• Automated Backup: Backup
scheduling through Operator
• Centralized Monitoring
Prometheus: Monitoring entire
Couchbase deployment with
centralized dashboard
• Automated Cross Datacenter
Replication Management:
Automatically configure and
establish XDCR replication from
source to target Couchbase cluster
• Open Service Broker: Simplified
Service Broker architecture
• Certificate Authentication using
Mutual TLS Support
• Auto-Scaling Stateless: Scale-up
or down automatically based-on
predefined Thresholds
• Usage Metering: Usage tracking for
self-service provisioning and usage
reporting
• Sync Gateway on Kubernetes:
Support provisioning, configuration,
on-demand scaling, and recovery for
Couchbase Sync Gateway
• Support Istio/Envoy: Allows
Operator to work with a Istio Service
Mesh for creating a network of
deployed services
• Cluster Hibernation: Ability to
PAUSE and RESUME Couchbase
Instance without losing the data
CAO 2.0 Q1 2020 CAO 2.1 Q4 2020
Azure AKS
Google GKE

22. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved.
Couchbase & Kafka Demo

23. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 23
Sink and Source

24. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved.
Other Cool Stuff

25. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 25
Transactions

26. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved. 26
Couchbase Lite

27. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2021. All rights reserved.
THANK YOU
denis.rosa@couchbase.com
@deniswsrosa