Thousands of companies, from Uber and Netflix to Goldman Sachs and Cisco, use Apache Kafka to transform and reshape their data architectures. Kafka is frequently used as the bridge between legacy RDBMS and new NoSQL database systems, effectively transforming SQL table data into JSON documents and vice versa. Many companies also use Kafka for business-critical applications that drive real-time stream processing and analytics, intersystem messaging, high-volume data ingestion, and operational metrics collection. Couchbase and Kafka can be used together to address high throughput, distributed data management, and transformation challenges. In this webinar we’ll explore: Where Kafka fits into the big data ecosystem How companies are using Kafka for both real-time processing and as a bus for data exchange An example of how Kafka can bridge legacy RDBMS and new NoSQL database systems Several real-world use case architectures

1. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
COUCHBASE &
CONFLUENT
Bridging the Gap between RDBMS and NoSQL
Tyler Mitchell, Senior Product Manager, Couchbase
David Tucker, Director Partner Engineering and Alliances, Confluent

2. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
AGENDA
01/
02/
03/
04/
Big Data Architecture Evolution
Kafka as a Streaming Bus
Real World Use Cases
DEMO – Couchbase | Kafka | MySQL

3. BIG DATA
ARCHITECTURE
EVOLUTION1

4. Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2017. All rights reserved. 5
Big Data
Architecture
Evolution
Big Data version 1 focused on
• ingest
• archive

5. Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2017. All rights reserved. 6
Big Data
Architecture
Evolution
Big Data version 2 focused on
• collate
• analyze

6. Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2017. All rights reserved. 7
Big Data
Architecture
Evolution
Big Data version 3 ...
• stream
• remix

7. Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2017. All rights reserved. 8
Big Data
Architecture
Evolution
Big Data version 3 ...
• stream
• remix
• ... engagement

8. Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2017. All rights reserved. 9
Data Processing
Platform
• Critical connectivity
• Analysis
• Streaming
HDFS
DBMS
Mobile
Other
Platforms
Data
Processing
Platform

9. Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2017. All rights reserved. 10
Stream
Data
Platform
• Critical connectivity
• Analysis
• Streaming
• Distributed
• High throughput
• Wider engagement
HDFS
DBMS
Mobile
Other
Platforms
Stream
Data
Platform

10. KAFKA AS A
STREAMING BUS2

11. Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2017. All rights reserved. 12
Couchbase
& Kafka
• Source
• Sink
• Custom Filter
• Apache Kafka
• Confluent Platform
Kafka
?

12. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 13
How Organizations Handle Data Flow: A Giant Mess
Data
Warehouse
Hadoop
NoSQL
Oracle
SFDC
Logging
Bloomberg
… any sink/source
… and more
OLTP
ActiveMQ
App App
Caches
OLTP OLTPAppAppApp
Web Custom Apps Microservices Monitoring Analytics

13. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 14
Apache Kafka™: A Distributed Streaming Platform
Apache Kafka
Data
Warehouse
Hadoop
NoSQL
Oracle
SFDC
Twitter
Bloomberg
… any sink/source … any sink/source
… and more
Web Custom Apps Microservices Monitoring Analytics

14. REAL WORLD USE
CASE3

15. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 16
Applications Across Industries
Healthcare & Pharma
Patient Monitoring, Pharma Substance
control, Patient Relapse, Lab Results
Alerts
Banking & Capital Markets
Fraud Detection, Trade Data Capture,
Customer 360
Retail
Inventory Management, Product
Catalog, A/B Testing, Proactive Alerts
Telecommunications
Personalized Ads, Customer 360,
Network Integrity
Automotive
Connected Car, Manufacturing Data
Processing
Travel & Leisure
Visitor Segmentation,
Fraud Detection

16. Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2017. All rights reserved. 17
Confluent
Platform
Enterprise
Streaming based on
Apache Kafka™
Database
Changes
Log
Events
loT Data
Web
Events
…
CRM
Data
Warehouse
Database
Hadoop
Data
Integration
…
Monitoring
Analytics
Custom Apps
Transformations
Real-time
Applications
…
Apache Open Source Confluent Open Source Confluent Enterprise
Confluent Platform
Apache Kafka™
Data Compatibility
Monitoring & Administration
Operations
Clients Connectors
Complete Open Trusted Enterprise Grade

17. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 18
Over 35% of Fortune 500 are using Apache Kafka™
6 of top 10
Travel
7 of top 10
Global banks
8 of top 10
Insurance
9 of top 10
Telecom

18. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 19
Bank Reduces OpEx by $25M/year via Mainframe Offload
Date Amount
1/27/2017 $4.56
1/22/2017 $32.14
Transaction Data
Vendor Description
Starbucks Coffee
Walmart Blu-Ray
Transaction Description
Schema
Website
Microservices
Match data and
description
Client profiles
Lookup client
profiles
Mainframe MIPS = $$

19. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 20
Ingest, Process, Load, and Serve Data at a Global Scale
Engagement Database
…
Engagement Database
…
Kafka cluster
Applications
Other data
stores
Kafka cluster
Kafka Streams
(Data Enrichment and Transformation)
Kafka Connect
(Connectors to Extract and Load data)
Confluent
Replicator
Confluent
Replicator
Custom
Replication
Custom
Replication
Raw event
data

20. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 21
How do I get streams of data
into and out of my apps?
Connect Clients REST

21. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 22
Kafka Connect – Streaming Data Capture
JDBC
IRC /
Twitter
CDC
Couchbase
NoSQL
HDFS
Kafka Connect API
Kafka Pipeline
Connector
Connector
Connector
Connector
Connector
Connector
Sources Sinks
Fault tolerant
Manage hundreds of data
sources and sinks
Preserves data schema
Part of Apache Kafka project
Integrated within Confluent
Platform’s Control Center

22. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 23
Kafka Connect API Library of Connectors
* Denotes Connectors developed at Confluent and distributed by Confluent. Extensive validation and testing have been performed.
Databases
*
Analytics
*
Applications / Other
Datastore / File Store
*
*

23. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 24
Kafka Streams API: A Part of Apache Kafka
Kafka
Streams API
Producer
Kafka Cluster
Topic TopicTopic
Consumer Consumer
Key Benefits
• No additional cluster
• Easy to run as a service
• Supports large aggregations and
joins
• Security and permissions fully
integrated from Kafka
Example Use Cases
• Microservices
• Continuous queries
• Continuous transformations
• Event-triggered processes

24. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 25
Architecture Example
Before: Complexity for development and operations, heavy footprint
1 2 3
Capture business
events in Kafka
Must process events with separate,
special-
purpose clusters
Write results
back to Kafka
Your Processing Job

25. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 26
Architecture Example
With Kafka Streams: App-centric architecture that blends well into your existing
infrastructure
1 2 3a
Capture business
events in Kafka
Process events fast, reliably,
securely with standard
Java applications
Write results
back to Kafka
Your App
3b
External apps can directly query
the latest results
AppApp
Kafka
Streams API

26. DEMO
COUCHBASE KAFKA &
MYSQL4

27. Confidential and Proprietary. Do not distribute without
Couchbase consent. © Couchbase 2017. All rights reserved. 28
Demo
Scenario
Powerful
Streaming Data
Pipeline
Simulated transaction workload captured at scale to Couchbase
• Sysbench workload running against a MySQL database
• Debezium CDC Source Connector publishes change records to Kafka topic
• Sink connector saves Kafka topic to Couchbase - with optional filtering
• Key-value queries can aggregate RDBMS data or track explicit operations

28. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved. 29
Kafka Connect Demonstration
Kafka Connect
Apache Kafka Brokers
Optional K-Streams app(s)
Couchbase
SinkConnector
CDC
SourceConnector
52
3 4
1

29. Confidential and Proprietary. Do not distribute without Couchbase consent. © Couchbase 2017. All rights reserved.
THANK YOU