Apache Kafka and the Data Mesh | Michael Noll, Confluent

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Apache Kafka and the Data Mesh
Ben Stopford, Michael G. Noll
Office of the CTO, Confluent
Kafka Summit Americas, September...

Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express...

Data ownership by
domain
Data as a product Data governed
wherever it is
Data available
everywhere, self
serve
1 2 3 4
The ...

4
A First Look
Domain
Inventory
Orders
Shipments
...
Data Product
Data Mesh

Domain-driven
Decentralization
Local Autonomy
(Organizational Concerns)
Data as a
First-class Product
Product thinking,
“M...

Principle 1: Domain-driven Decentralization
Pattern: Ownership of a data asset given to
the “local” team that is most fami...

Principle 1: Domain-driven Decentralization
Pattern: Ownership of a data asset given to
the “local” team that is most fami...

Data Mesh is about Connectivity
8
“Instead of collecting, you want to come up with a
model that allows connectivity of the...

9
Shipping Data
Joe
Practical example
1. Joe in Inventory has a problem with
Order data.
2. Inventory items are going nega...

10
Shipping Data
Joe
Practical example
1. Joe in Inventory has a problem with
Order data.
2. Inventory items are going neg...

Recommendations: Domain-driven Decentralization
11
Learn from DDD:
• Use a standard language and nomenclature for data.
• ...

1 2 3 4
Domain-driven
Decentralization
Local Autonomy
(Organizational Concerns)
Data as a
First-class Product
Product thin...

Principle 2: Data as a First-Class Product
13
• Objective: Make shared data discoverable, addressable, trustworthy, secure...

Infra
Code
Data product, a “microservice for the data world”
14
• Data product is a node on the data mesh, situated within...

15
Connectivity within the mesh lends itself...
Domain
Data Product
Data Mesh

16
...naturally to Event Streaming with Kafka
Domain
Data Product
Mesh is a logical view, not physical!
Data Mesh

Copyright 2021, Confluent, Inc. All rights reserved. This document may not be reproduced in any manner without the express...

Data Product
Data Product
Why is Event Streaming a good fit for meshing?
18
0 1 2 3 4 5 6 1
7
Streams are real-time, low l...

How to get data into & out of a data product
19
Data Product
Input
Data
Ports
Output
Data
Ports
Snapshot via
Nightly ETL
S...

Onboarding existing data
20
Data
Product
Input
Data
Ports
Source
Connectors
Use Kafka connectors to stream data from cloud...

Data product: what’s happening inside
21
Input
Data
Ports
Output
Data
Ports
…pick your favorites...
Data on the Inside: HO...

Event Streaming inside a data product
22
Input
Data
Ports
Output
Data
Ports
ksqlDB to filter,
process, join,
aggregate, an...

Use Kafka connectors and CDC to “streamify” classic databases.
Event Streaming inside a data product
23
Input
Data
Ports
O...

Dealing with data change: schemas & versioning
24
Data
Product
Output
Data
Ports
V1 - user, product, quantity
V2 - userAno...

Find Data Products
25

Recommendations: Data as a First-class Product
26
1. Data-on-the-Outside is harder to change, but it has more value in a h...

Recommendations: Data as a First-class Product
27
1. Data-on-the-Outside is harder to change, but it has more value in a h...

Recommendations: Data as a First-class Product
28
1. Data-on-the-Outside is harder to change, but it has more value in a h...

1 2 3 4
Domain-driven
Decentralization
Local Autonomy
(Organizational Concerns)
Data as a
First-class Product
Product thin...

Why Self-service Matters
30
Trade Surveillance System
● Data from 13 sources
● Some sources publish events
● Needed both h...

Why Self-service Matters
31
Trade Surveillance System
● Data from 13 sources
● Some sources publish events
● Needed both h...

Principle 3: Self-serve Data Platform
32
Central infrastructure that provides real-time and historical data on demand
Obje...

Consuming real-time & historical data from the mesh
33
1) Separate Systems for Real-time and Historical Data (Lambda Archi...

What this can look like in practice
34
Browse Schemas

35
With ksqlDB the data mesh is queryable and
decentralized.
Destination
Data Port
STREAM
PROCESSOR
ksqlDB
Query is the in...

36
Mesh is one logical cluster. Data product has another.
Data
Product
Data Product has its own
cluster for internal use

1 2 3 4
Domain-driven
Decentralization
Local Autonomy
(Organizational Concerns)
Data as a
First-class Product
Product thin...

Principle 4: Federated Governance
38
• Objective: Independent data products can interoperate and create network effects.
•...

Example standard: Identifying customers globally
• Define how data is represented, so you can join and correlate data acro...

Example standard: Detect errors and recover with Streams
40
• Use strategies like logging, data profiling, data lineage, e...

Example standard: Tracking data lineage with Streams
41
• Lineage must work across domains and data products—and systems, ...

Recommendations: Federated Governance
42
1. Be pragmatic: Don’t expect governance systems to be perfect.
a. They are a map...

Implementing a Data Mesh

Data Mesh Journey
44
Principle 1
Data should have one owner:
the team that creates it.
Principle 2
Data is your product:
A...

Data Mesh Journey
45
Principle 1
Data should have one owner:
the team that creates it.
Principle 2
Data is your product:
A...

Data Mesh Journey
46
Principle 1
Data should have one owner:
the team that creates it.
Principle 2
Data is your product:
A...

Implement a Data Mesh: Cheat Sheet
- Secure event streams
Access to event streams is
permissioned by a central body.
- Con...

developer.confluent.io
• Free Courses on all things Kafka
and Event Streaming
• 50+ Design Patterns for Event
Streaming
• ...

Data Mesh Event Sourcing Kafka 101
Recommended free courses

Thank you!
@benstopford
@miguno

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1035460 views
Storytelling For The Web: Integrate... by Chiara Aliotta 947335 views
Artificial Intelligence, Data and C... by OECD Directorate ... 869118 views
How to Leverage AI to Boost Employe... by SocialHRCamp 364901 views
2024 State of Marketing Report – by... by Marius Sescu 207323 views
Everything You Need To Know About C... by Expeed Software 227648 views

View on Slideshare

1 of 50 Ad

View on Slideshare

1 of 50 Ad