Apache Kafka is an open-source event streaming platform used to complement or replace existing middleware, integrate applications, and build microservice architectures. Used at almost every large company today, it's understood, battled-tested, highly scalable, and reliable.
Blockchain is a different story. Being related to cryptocurrencies like Bitcoin, it's often in the news. But what is the value of software architecture? And how is it related to an integration architecture and event streaming platform?
This session explores blockchain use cases and different alternatives such as Hyperledger, Ethereum, and Kafka-native blockchain implementation. We discuss the value blockchain brings for different architectures, and how it can be integrated with the Kafka ecosystem to build a highly scalable and reliable event streaming infrastructure.
This talk discusses the concepts, use cases, and architectures behind Event Streaming, Apache Kafka, Distributed Ledger (DLT), and Blockchain. A comparison of different technologies such as Confluent, AIBlockchain, Hyperledger, Ethereum, Ripple, IOTA, and Libra explores when to use Kafka, a Kafka-native blockchain, a dedicated blockchain, or Kafka in conjunction with another blockchain.
Microsoft 365 Copilot; An AI tool changing the world of work _PDF.pdf
Apache Kafka and Blockchain - Comparison and a Kafka-native Implementation
1. Building a Secure, Tamper-Proof & Scalable
Blockchain on Top of Apache Kafka
Kai Waehner
Technology Evangelist
Confluent
kai.waehner@confluent.io
LinkedIn
@KaiWaehner
confluent.io
kai-waehner.de
Stephen Reed
CEO, Co-founder
AiB
stephen@ai-blockchain.com
LinkedIn
ai-blockchain.com
2. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda
3. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain for Supply Chain Management
(Manufacturing, Logistics, …)
3
https://coletiv.com/blog/blockchain-supply-chain-management-system/
https://www.disruptordaily.com/blockchain-use-cases-supply-chain-management/
4. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain in Healthcare
4
• Use Cases
• Management of patient data, especially
access control
• RVU Agreements with Ai-Blockchain
• Management and use of academic research
data
• Logistics / Supply chain for drugs, medical
technology, and other products
• Often no need for Blockchain /
DLT
• Only for Identity
Management
• Processing and (big) data storage outside
blockchain
https://digiforest.io/en/blog/blockchain-examples-in-healthcare
5. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain for
Financial Services
5
https://www.accenture.com/t20170120T074124Z__w__/us-en/_acnmedia/Accenture/Conversion-Assets/DotCom/Documents/Global/PDF/Consulting/Accenture-Banking-on-Blockchain.pdf%23zoom=50
6. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda
7. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Bitcoin – The Tip of the Iceberg!
7
‘Bitcoin’ compared to
‘blockchain’, ‘distributed
ledger’, etc.
8. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain vs. Distributed Ledger Technology (DLT)
8
https://www.da.ventures/post/gaining-clarity-on-key-terminology-bitcoin-versus-blockchain-versus-distributed-ledger-technology
9. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Blockchain Concepts
• Permissionless or permissioned (or consortium)
• Consensus algorithm (Proof of work, Proof of Stake, or Voting systems)
• Blockchain: Global Consensus across all nodes
• DLT: Consensus without having to validate across the entire Blockchain
• Blockchain
• Growing list of records, called blocks, linked using cryptography
• Each block contains a cryptographic hash of the previous block, a timestamp, and transaction data
11
10. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Challenges and Concerns of Blockchains
• Technical and organizational
complexity
• Transaction speed and
scalability
• Energy consumption
• Security
• Data tenancy
• Lifecycle costs
• Ease of integration
11. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Purpose-built solutions
• Ripple: Payment settling, currency exchange and remittance system
• Libra: Global payment system and financial infrastructure
• IOTA: IoT transactions
Non-Blockchain DLTs
to solve the challenges to provide better performance and scalability
13
Open and flexible solution for many use cases
• Apache Kafka: Open, scalable, real time event
streaming
Technology Partners
• Ai-Blockchain: “AiB”
12. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda
13. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Event Streaming in the Finance Industry
Check past Kafka Summit videos for details about the use cases:
https://kafka-summit.org/past-events/
14. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Tamper-Proof Event Streaming for
(some of the) Use Cases in the Finance Industry
Real Time Processing Digital Transformation Strategic Goals
Short-Sale Risk Calculation
/ Trade Approval
Mainframe Offloading
and Replacement
Instant Payment
Next-Best
Offer
Robot Process Automation
(e.g. Know Your Customer, KYC)
Customer Service
(e.g. Chat Bots)
IT
ModernizationRegulatory
Reporting
Fraud Detection
Anomaly Detection
Across Assets and Locations
Derivatives
Pricing Compliance
Trading Post-
Processing
Strategic
Planning and
Simulations
15. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
STREAM
PROCESSING
Create and store
materialized views
Filter
Analyze in-flight
Time
C CC
Event Streaming
16. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Event Streaming Platform –
The Immutable Commit Log
Time
P
C1 C2
C3
17. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Event Streaming Platform –
A Distributed System for 24/7 and Zero Data Loss
Broker 1
Topic1
partition1
Broker 2 Broker 3 Broker 4
Topic1
partition1
Topic1
partition1
Leader Follower
Topic1
partition2
Topic1
partition2
Topic1
partition2
Topic1
partition3
Topic1
partition4
Topic1
partition3
Topic1
partition3
Topic1
partition4
Topic1
partition4
18. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
An Event Streaming Platform
is the Underpinning of an Event-driven Architecture
Microservices
Mainframes
SaaS apps
Mobile
Customer 360
Real-time fraud
detection
Data warehouse
Producers
Consumers
Database
change
Microservices
events
SaaS
data
Customer
experiences
Streams of real time events
Stream processing apps
Connectors
Connectors
Stream processing apps
19. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Kafka Connect
with Ethereum Connector
Kafka Cluster
Blockchain
Integration
Domain-Driven Design and Decoupled Applications
Legacy
Integration
Custom
Application
ETL Tool
with Mainframe Connector
Java / C++ /
Go / Python / KSQL
Schema
Registry
Event Streaming Platform
Blockchain Domain Payment Domain Fraud Domain
Audit Logs,
RBAC, etc.
20. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Reasons for Event Streaming with Apache Kafka
Real Time
Scalable
High Throughput
Cost Reduction
24/7 – Zero downtime, zero data loss
Decoupling – Storage, Domain-driven Design
Data (re-)processing and stateful client applications
Integration – Connectivity to IoT, legacy, big data, everything
Hybrid Architecture – On Premises, multi cloud, edge computing
Fully managed cloud
No vendor locking
24
21. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Kafka Clusters
can stretch over
Regions
Zero Downtime + Zero Data loss
(RPO=0 and RTO=0)
e.g. Stretched over US East + Mid +
West
Automate Disaster Recovery
Sync or Async Replication per Topic
Offset Preserving
Automated Client Failover without
Custom Code
Multi-Region Cluster
(Only available in Confluent Platform)
26
22. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Example of a Multi-Region Cluster in a Bank
Large FinServ Customer
Payment
Log
Payment
Log
Location Location
synchronous
asynchronous
● ‘Payment’ transactions enter
from us-east and us-west with
fully synchronous replication
● ‘Log’ and ‘Location’ information
in the same cluster use async -
optimized for latency
● Automated disaster recovery
(zero downtime, zero data loss)
Result: Clearing time from ‘deposit’ to
‘available’ goes from 5 days to 5 seconds
(including security checks)
(Only available in Confluent Platform)
23. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda
24. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Real-time
High Throughput
Decentralized database
Distributed log of records
Immutable log
Replication
High availability
Decoupling of applications / clients
Role-based access control to data
Tamper-Proof
Encrypted payloads
Deployment over independent organizations
is not a Blockchain!
X
25. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Kafka AND Blockchain
32
Kafka
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk,
etc.)
Blockchain
(Bitcoin)
Blockchain
(Ethereum)
Kafka-native
Blockchain
Kafka AS Blockchain
Real Time
Instant Payment
App
(Java, C++,
Python, etc.)
Batch
Analytics Platform
(Spark, Splunk,
etc.)
26. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Kafka AND Blockchain:
Financial services platform
combining Kafka and Blockchains
• Provides the speed and convenience of
traditional exchanges and the security of
non-custodial approaches.
• Invest in, make payments with and trade
Bitcoin, Ethereum, NEO, and other digital
assets.
• The exchange is the first of its kind, offering
non-custodial cross-chain trading with the
full power of a real order book
• Deterministic replayability in its exact order
at any time
33
https://community.nash.io/t/first-of-its-kind-non-custodial-exchange-using-confluent-cloud/8254
27. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Kafka AS Blockchain:
Consensus algorithm
in Hyperledger Fabric
using Kafka to be fault tolerant,
highly scalable and performant
34
https://hyperledger-fabric.readthedocs.io/en/release-2.0/kafka.html
https://www.meetup.com/Dubai-Kafka/events/268001616/ (presented by Farrukh Munir)
28. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda
30. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
PRODUCTS
Built FIRST DocuSign Blockchain
product.
Allows for the writing of contracts on
the blockchain with proof of trust
Software agents keep the blockchain
tamper proof and immutable
Current POC’s across eCommerce,
Healthcare and Supply Chain
SOLUTIONS
• Artificial Intelligence, Blockchain, Ecommerce,
Healthcare, Media, Payments and Supply Chain
• Securely store commercial and non-commercial
transactions
• Energy efficient and ESG
GROWTH OPPORTUNITIES
How do we expand globally with new customers
using AiB products?
AiB offers services for scalability to meet all of the
demands of global businesses offering architecture
assessment diagrams to clients.
TECHNOLOGY
We employ one of a kind Artificial Agents on
the Blockchain to give world class security
and to ensure all data is immutable.
PLATFORM
Its most visible success is in the unique
approach to financial transactions through its
implementation of its Artificial Intelligent
Blockchain Cloud Payment Platform.
Enterprise
Grade
Platform
COMPANY
Since its founding in 2014, Ai-
Blockchain has built a
combination of innovative IP
and blockchain products
that have been recognized in
the marketplace and USPTO.
Potential
Customer
Use Cases
and
Problems
Ai-Blockchain
Products
Ai-Blockchain
Solutions and
advantages
Industry
Expansion
Ai-Blockchain – Patented Private-Permissioned Blockchain
for Smart Contracts
31. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
AiB’s BRICK Messaging Platform
For security, privacy and scalability, the AiB’s BRICK
platform is a platform agnostic layer that is built for
enterprise and micro payments in the cloud.
Third-Party Audited
AiB’s technology stack has been
security audited and stress tested by
third-party global cloud providers.
Full-Stack Capability
Deploying nodes and writing smart contracts
up to front end development and deployment.
Thought Leaders
Consistently asked to
speak at international
events to share our
Blockchain expertise.
Blockchain Experts
Established in 2014
Ai Blockchain’s team of over
100 Blockchain developers &
experts have been building
solutions for over six years
and are ready today for any
and all market
opportunities.
Ai-Blockchain – Proprietary Blockchain Technology Stack
32. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
39
KafkaBlockchain Architecture
Producers reuse the KafkaBlockchain
library and Tamper-Evident Records.
Consumers reuse the KafkaBlockchain library and Tamper-Evident Records.
33. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
KafkaBlockchain – Producer, Consumer & Verifier
Open Source Apache
Licensed on GitHub
35. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
1. Use Cases for Secure and Tamper-Proof Data Processing
2. Distributed Ledger and Blockchain Technologies
3. Event Streaming with Apache Kafka
4. Relation between Kafka and Blockchain
5. A Tamper-Proof Blockchain implemented with Kafka
6. Live Demo
Agenda
36. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Demonstration:
run-kafka-blockchain.demo.sh
• Demonstrate putting messages into a Kafka blockchain with one partition, one
producer and one consumer.
• Describe the single blockchain partition
• Demonstrate verifying messages with one consumer
Demonstration:
run-kafka-blockchain-multiple-partition.demo.sh
• Demonstrate putting messages into a Kafka blockchain with five partitions, one
producer and one consumer.
• Describe the five blockchain partitions
• Demonstrate verifying messages with five cooperating consumers - one for each
partition.
43
KafkaBlockchain – Live Demo
https://github.com/ai-coin/KafkaBlockchain
37. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Next steps…
38. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
When to use
Kafka vs. a “real blockchain framework”?
45
Use Apache Kafka
with AIBlockchain for
• Enterprise infrastructure
• Open, scalable, real-time
requirements
• Flexible architectures for many
use cases
Use Hyperledger,
Ethereum, et al. for
• Deployment over various
independent organizations
• Participants verify the
distributed ledger contents
themselves.
• Specific use cases
• Server-side managed and
controlled by multiple
organizations
• Scenarios where the
business value overturns the
added complexity and
project risk
Use Kafka and Blockchain
together to combine the
benefits of both
(where this makes sense!)
39. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
What about long-term storage in Kafka?
46
40. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Today, Kafka works well for
recent events, short
horizon storage, and
manual data balancing
Kafka’s present-day design offers
extraordinarily low messaging latency by
storing topic data on fast disks that are
collocated with brokers. This is usually good.
But sometimes, you need to store a huge
amount of data for a long time.
Blockchain is such a use case!
Kafka
Processing
App
Storage
Transactions, auth,
quota
enforcement,
compaction, ...
41. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Confluent Tiered Storage for Kafka
Object Store
Processing Storage
Transactions,
auth, quota
enforcement,
compaction, ...
Local
Remote
Kafka
Apps
Store Forever – AiB’s Tamper-Proof Blockchain
with KafkaBlockchain
Older data is offloaded to inexpensive object storage,
permitting it to be consumed at any time. Using AiB, storage
can be made tamper-proof and immutable
Save $$$
Storage limitations, like capacity and duration, are effectively
uncapped.
Instantaneously scale up and down
Your Kafka clusters will be able to automatically self-balance
load and hence elastically scale
(Only available in Confluent Platform)
42. kai-waehner.de | @KaiWaehner | Building a Secure, Tamper-Proof & Scalable Blockchain on Top of Apache Kafka
Tamper-proof, encrypted event streaming
Time
P
C1
C2
Payment
Real-time data processing and historical analytics with one scalable infrastructure
Compliance check on historical data
Real-time
fraud detection