Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent

@KaiWaehner - www.kai-waehner.de – Can Apache Kafka Replace a Database?
Can Apache Kafka Replace a Database?
Kafka’s Capabilities and Trade-Offs for Storage, Queries, Processing, Transactions, Connectivity
Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.confluent.io
www.kai-waehner.de
linkedin.com/in/kaiwaehner

Is Kafka a Database?
This was answered [with ‘yes’] a few years ago…
2
https://www.confluent.io/kafka-summit-SF18/is-kafka-a-database/
… and many things changed [= improved] since then!

Yes.
Kafka is
a database!
Can
replace
another
Database?

Agenda
4
1. What is a Database?
2. What is Apache Kafka?
3. Storage in Kafka
4. Queries and Processing in Kafka
5. Transactions in Kafka
6. Connectivity

Agenda
5
3. Storage in Kafka
6. Connectivity

What is a Database?
6
Database Concepts
1960s: Navigational DBMS
1970s, Relational DBMS
Late 1970s: SQL DBMS
1980s: On the desktop
1990s: Object-oriented
2000s: NoSQL / NewSQL
2010s: DBaaS
Database Features
Storage
Queries (CRUD)
Processing
Transactions
Backup
Replication
…

Database Theorems
7
Atomicity
Consistency
Isolation
Durability
Consistency
Availability
Partitioning

Database Examples
8

Database Examples
9
I thought
Kafka is for
data in motion?

Agenda
10
3. Storage in Kafka
6. Connectivity

Apache Kafka is an Event Streaming Platform
MES
ERP
Sensors
Mobile
Customer 360
Real-time
Alerting System
Data warehouse
Producers
Consumers
Streams and storage of real time events
Stream
processing
apps
Connectors
Connectors
Stream
processing
apps
Supplier
Alert
Forecast
Inventory Customer
Order
11

ETL/Data Integration Messaging
Highly Scalable
Durable
Persistent
Ordered
Real-time Difficult to Scale
No Persistence After
Consumption
No Replay
Batch
Expensive
Time Consuming

Highly Scalable
Persistent
Messaging
Batch
Expensive
Time Consuming
Difficult to Scale
No Persistence After
Consumption
No Replay
Real-time
Highly Scalable
Durable
Persistent
Ordered
Real-time
Event Streaming

Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital replatforming/
Mainframe Offload
Connected Car: Navigation & improved in-
car experience: Audi
Customer 360
Simplifying Omni-channel Retail at Scale:
Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives; LinkedIn,
Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka Streams:
Funding Circle
Detect Fraud & Prevent Fraud in Real Time:
PayPal
Kafka as a Service - A Tale of Security and
Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$↔
Example Case Studies
(of many)

Agenda
15
3. Storage in Kafka
6. Connectivity

Kafka’s Distributed Commit Log is the Storage
(and enables real decoupling and domain-driven design)
16
https://www.confluent.io/blog/microservices-apache-kafka-domain-driven-design/

Kafka Stores Your Data Durably.
https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/
Kafka is the source of truth.
Powers NYTimes.com, and stores
all articles ever published since 1851.
September 30, 1851, Page 1
Kafka is the leading system.
Account Activity Replay API to recover events
that weren’t delivered for various reasons
https://blog.twitter.com/engineering/en_us/topics/infrastructure/2020/kafka-as-a-storage-system.html

Confluent Tiered Storage for Kafka
18
(Only available in Confluent Platform)
Store data forever
Hot and cold storage
Cheap object store
Easy scale up/down
No changes in clients

Tiered Storage for Apache Kafka
KIP-405 –
Add Tiered Storage Support to Kafka
Confluent is actively working on this with the
open source community -
Uber is leading this initiative
Confluent Tiered Storage is available today in
Confluent Platform and used under the hood in
Confluent Cloud
https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage

Log Compaction with Compacted Topics
20
Retain last known value
for each message key
No retention time

Stateful Kafka Applications
Kafka Streams and ksqlDB embed RocksDB
21
Do I really need
another database for
my microservice?
streams

Kafka as Single Source of Truth
22
The Leading System is
Real-Time and Scalable
Real Decoupling
Handling Slow Consumers

Agenda
23
3. Storage in Kafka
6. Connectivity

Kafka
C++ Client
Kafka Cluster
Monolith
Kafka Decouples Storage and Compute
Cloud DWH
Snowflake
Connector
Schema Registry
Storage
Compute Compute
(+ non-Kafka Storage)
KSQL
App
KSQL
App
KSQL
App
Compute
Compute Compute

Confluent & The Future of SQL
ksqlDB’s model is
leading the way to the
new official standard.
What is it?
Confluent collaborates with other DB vendors
in the US Standards Body (INCITS DM32) on adding
Streaming SQL to the SQL standard.
Here, we discuss how to represent:
• Streams vs. Tables
• Traditional vs. Continuous queries
• Mutable vs. Immutable data
• Event-time vs. Processing-time
• Historical queries vs. Present-time queries
and more

Query and Event Processing in Kafka
26
PUSH à Continuously process and forward events
PULL à Client requests events (like you know it from your favourite database)

ksqlDB - The Event Streaming Database
27
-- Continuously look up data in a table; query keeps running
SELECT * FROM myTable WHERE ... EMIT CHANGES
-- Continuously look up data in a stream; query keeps running
SELECT * FROM myStream WHERE ... EMIT CHANGES
-- Look up data in a table once; query then terminates
SELECT * FROM myTable WHERE ...
app
app

ksqlDB - The Event Streaming Database
• Project created by Confluent, source-available license: https://ksqldb.io/
• A ksqlDB cluster runs in a distributed manner across many server nodes
• Tightly integrates with Apache Kafka® as its persistent storage layer
• Has projections, transformations, aggregations, windowing, joins, etc.
• Distinguishes between event-time and processing-time
• Handles out-of-order and late data
• Streaming import-export for external data systems
• DDL and DML via SQL-like statements
• Security features like role-based access control
• Run it yourself or use SaaS offering in Confluent Cloud
28

Continuous Queries and Processing
29

Queries through the Kafka Consumer
30
• Continuous consumption of the latest events (in real time or batch)
• Just specific time frames or partitions
• All data from the beginning
connect
Cluster Linking
REST Proxy

Queries for Consuming Historical Events
Give me all events from time A to time B
Real-time Producer
Time
• New consumer application
• Error-handling
• Compliance / regulatory processing
• Query and analyze existing events
• Schema changes in analytics platform
• Model training
Real-time Consumer
Consumer of Historical Data

Interactive Queries
Query values from the client applications’ state store
Optional Proxy (e.g. HTTP or WebSockets)
Limitation: Only Key/Value or table scans,
but no complex queries or ANSI SQL
32
streams

Interactive Queries
Table Scan Example
33
streams

ANSI SQL Queries against the Kafka Log
3rd Party Add-Ons help
Integration with any Business Intelligence Tool
34
https://www.confluent.io/blog/analytics-with-apache-kafka-and-rockset/

Agenda
35
3. Storage in Kafka
6. Connectivity

Exactly-Once Semantics (EOS) in Kafka
No Two-Phase-Commit (because that does not scale)
Idempotent Producer and Transactions API
Supported by the whole Kafka Ecosystem (not just Messaging)
36

Transaction API in Apache Kafka
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
https://www.confluent.io/kafka-summit-london18/dont-repeat-yourself-introducing-exactly-once-semantics-in-apache-kafka/

From the Mainframe to ksqlDB in the Cloud
Bi-Directional End-to-End Referential Integrity
ksqlDB
App
CICS
Mainframe
Transactions
Bi-Directional Integration
Secured Referential Integrity
End-to-End “Transactions”
Low Latency
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Kafka
Exactly-Once
Semantics
using librdkafka
IMS
DB
Cobol App
Kafka
Exactly-Once
Semantics
using ksqlDB

Agenda
39
3. Storage in Kafka
6. Connectivity

Kafka Connect
Integration between Databases, Applications, APIs, SaaS
Kafka-native (no other middleware required)
Sources and Sinks
Legacy and Modern
Real-Time and Batch
40

Turn the Database Inside Out!
Materialized Views
The heart is real-time, scalable and durable
Integration with any Database for specific use cases and queries
41
https://www.confluent.io/blog/turning-the-database-inside-out-with-apache-samza/

Global Event Streaming
Streaming Replication between Kafka Clusters
Bridge to Databases, Data Lakes, Apps, APIs, SaaS
Aggregate Small Footprint
Edge Deployments with
Replication (Aggregation)
Simplify Disaster Recovery
Operations with
Multi-Region Clusters
with RPO=0 and RTO=0
Stream Data Globally with
Replication and Cluster Linking
42

Yes.
But it does not replace
all other databases!
Can
replace a
Database?

TL;DR
• Kafka can store data forever in a durable and high available manner providing ACID
guarantees
• Different options to query historical data are available in Kafka
• Kafka-native add-ons like ksqlDB or Tiered Storage make Kafka more powerful than ever
before for processing data in motion and event-based long-term storage
• Stateful applications can be built leveraging Kafka clients (microservices, business
applications) without the need for another external database
• Not a replacement for existing databases like MySQL, MongoDB, Elasticsearch or
Hadoop
• Other databases and Kafka complement each other; the right solution has to be
selected for a problem; often purpose-built materialized views are created and updated
in real time from the central event-based infrastructure
• Different options are available for bi-directional pull and push based
integration between Kafka and databases to complement each other

Kai Waehner
Field CTO
contact@kai-waehner.de
@KaiWaehner
www.kai-waehner.de
www.confluent.io
linkedin.com/in/kaiwaehner
Questions? Feedback?
Let’s connect!

Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent

Similar to Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Can Apache Kafka Replace a Database? – The 2021 Update | Kai Waehner, Confluent