Can Apache Kafka Replace a Database?

@KaiWaehner - www.kai-waehner.de – Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?
Kafka’s Capabilities and Trade-Offs for Storage, Queries, Processing, Transactions, Connectivity
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
linkedin.com/in/kaiwaehner

This was answered [with ‘yes’] a long time ago…
2
https://www.confluent.io/kafka-summit-SF18/is-kafka-a-database/
… and many things changed [= improved] since then!

Yes.
Kafka is
a database!
Can
replace a
Database?

Agenda
4
1. What is a Database?
2. What is Apache Kafka?
3. Storage in Kafka
4. Queries and Processing in Kafka
5. Transactions in Kafka
6. Connectivity

Agenda
5
3. Storage in Kafka
6. Connectivity

What is a Database?
6
Database Concepts
1960s: Navigational DBMS
1970s, Relational DBMS
Late 1970s: SQL DBMS
1980s: On the desktop
1990s: Object-oriented
2000s: NoSQL / NewSQL
2010s: DBaaS
Database Features
Storage
Queries (CRUD)
Processing
Transactions
Backup
Replication
…

Database Theorems
7
Atomicity
Consistency
Isolation
Durability
Consistency
Availability
Partitioning

Database Examples
8

Database Examples
9
I thought
Kafka is for
data in motion?

Agenda
10
3. Storage in Kafka
6. Connectivity

Apache Kafka is a Platform for Data in Motion
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
11

The Rise of Data in Motion
2010
Apache Kafka
created at LinkedIn by
Confluent founders
2014
2020
80%
Fortune 100
Companies
trust and use
Apache Kafka

ETL/Data Integration Messaging
Highly Scalable
Durable
Persistent
Ordered
Real-time Difficult to Scale
No Persistence After
Consumption
No Replay
Batch
Expensive
Time Consuming

Highly Scalable
Persistent
Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Real-time
Highly Scalable
Durable
Persistent
Ordered
Real-time
Event Streaming

Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital replatforming/
Mainframe Offload
Connected Car: Navigation & improved in-
car experience: Audi
Customer 360
Simplifying Omni-channel Retail at Scale:
Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives; LinkedIn,
Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka Streams:
Funding Circle
Detect Fraud & Prevent Fraud in Real Time:
PayPal
Kafka as a Service - A Tale of Security and
Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)

Agenda
16
3. Storage in Kafka
6. Connectivity

Kafka’s Distributed Commit Log is the Storage
(and enables real decoupling and domain-driven design)
17
https://www.confluent.io/blog/microservices-apache-kafka-domain-driven-design/

Kafka Stores Your Data Durably.
https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1
Kafka is the leading system.
Account Activity Replay API to recover events
that weren’t delivered for various reasons
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2020/kafka-as-a-storage-system.html

Confluent Tiered Storage for Kafka
19
(Only available in Confluent Platform)
Store data forever
Hot and cold storage
Cheap object store
Easy scale up/down
No changes in clients

Tiered Storage for Apache Kafka
KIP-405 –
Add Tiered Storage Support to Kafka
Confluent is actively working on this with the
open source community -
Uber is leading this initiative
Confluent Tiered Storage is available today in
Confluent Platform and used under the hood in
Confluent Cloud
https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage

Log Compaction with Compacted Topics
21
Retain last known value
for each message key
No retention time

Stateful Client Applications
Kafka Streams and ksqlDB embed RocksDB
22
Do I really need
another database for
my microservice?
streams

Kafka as Single Source of Truth
23
The Leading System is
Real-Time and Scalable
Real Decoupling
Handling Slow Consumers

Agenda
24
3. Storage in Kafka
6. Connectivity

Kafka
Java Client
Kafka Cluster
Monolith
Kafka Decouples Storage and Compute
Cloud DWH
Snowflake
Connector
Schema Registry
Storage
Compute Compute
(+ non-Kafka Storage)
KSQL
App
KSQL
App
KSQL
App
Compute
Compute Compute

Query and Event Processing in Kafka
26
PUSH à Continuously process and forward events
PULL à Client requests events (like you know it from your favourite database)

ksqlDB - The Event Streaming Database
27
-- Continuously look up data in a table; query keeps running
SELECT * FROM myTable WHERE ... EMIT CHANGES
-- Continuously look up data in a stream; query keeps running
SELECT * FROM myStream WHERE ... EMIT CHANGES
-- Look up data in a table once; query then terminates
SELECT * FROM myTable WHERE ...
app
app

ksqlDB - The Event Streaming Database
• Project created by Confluent, source-available license: https://ksqldb.io/
• A ksqlDB cluster runs in a distributed manner across many server nodes
• Tightly integrates with Apache Kafka® as its persistent storage layer
• Has projections, transformations, aggregations, windowing, joins, etc.
• Distinguishes between event-time and processing-time
• Handles out-of-order and late data
• Streaming import-export for external data systems
• DDL and DML via SQL-like statements
• Security features like role-based access control
• Run it yourself or use SaaS offering in Confluent Cloud
28

Continuous Queries and Processing
29

Queries through the Kafka Consumer
30
• Continuous consumption of the latest events (in real time or batch)
• Just specific time frames or partitions
• All data from the beginning
connect
Cluster Linking
REST Proxy

Queries for Reprocessing Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• New consumer application
• Error-handling
• Compliance / regulatory processing
• Query and analyze existing events
• Schema changes in analytics platform
• Model training
Real-time Consumer
Consumer of Historical
Data

Interactive Queries
Query values from the client applications’ state store
Optional Proxy (e.g. HTTP or WebSockets)
Limitation: Only Key/Value, no complex queries
32
streams

ANSI SQL Queries against the Kafka Log
3rd Party Add-Ons help
Integration with any Business Intelligence Tool
33
https://www.confluent.io/blog/analytics-with-apache-kafka-and-rockset/

Agenda
34
3. Storage in Kafka
6. Connectivity

Exactly-Once Semantics (EOS) in Kafka
No Two-Phase-Commit (because that does not scale)
Idempotent Producer and Transactions API
Supported by the whole Kafka Ecosystem (not just Messaging)
35

Transaction API in Apache Kafka
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
https://www.confluent.io/kafka-summit-london18/dont-repeat-yourself-introducing-exactly-once-semantics-in-apache-kafka/

From the Mainframe to ksqlDB in the Cloud
Bi-Directional End-to-End Referential Integrity
ksqlDB
App
CICS
Mainframe
Transactions
Bi-Directional Integration
Secured Referential Integrity
End-to-End “Transactions”
Low Latency
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Kafka
Exactly-Once
Semantics
using librdkafka
IMS
DB
Cobol App
Kafka
Exactly-Once
Semantics
using ksqlDB

Agenda
38
3. Storage in Kafka
6. Connectivity

Kafka Connect
Integration between Databases, Applications, APIs, SaaS
Kafka-native (no other middleware required)
Sources and Sinks
Legacy and Modern
Real-Time and Batch
39

Turn the Database Inside Out!
Materialized Views
Integration with any Database
40
https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/

Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO=0
Stream Data Globally with
Replication and Cluster Linking
41

Yes.
But it does not replace
other databases!
Can
replace a
Database?

TL;DR
• Kafka can store data forever in a durable and high available manner providing ACID
guarantees
• Different options to query historical data are available in Kafka
• Kafka-native add-ons like ksqlDB or Tiered Storage make Kafka more powerful than ever
before for processing data in motion and event-based long-term storage
• Stateful applications can be built leveraging Kafka clients (microservices, business
applications) without the need for another external database
• Not a replacement for existing databases like MySQL, MongoDB, Elasticsearch or
Hadoop
• Other databases and Kafka complement each other; the right solution has to be
selected for a problem; often purpose-built materialized views are created and updated
in real time from the central event-based infrastructure
• Different options are available for bi-directional pull and push based
integration between Kafka and databases to complement each other

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Can Apache Kafka Replace a Database?

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Can Apache Kafka Replace a Database?

Similar to Can Apache Kafka Replace a Database? (20)

More from Kai Wähner

More from Kai Wähner (20)

Recently uploaded

Recently uploaded (20)

Can Apache Kafka Replace a Database?