Studying the ""how"" of Kafka makes you better at using Kafka, but studying its ""whys"" makes you better at so much more. In looking at the tradeoffs behind a system like Kafka, we learn to reason more clearly about distributed systems and to make high-stakes technology adoption decisions more effectively. These are skills we all want to improve! In this talk, we'll examine trade-offs on which our favorite distributed messaging system takes opinionated positions: - Whether to store data contiguously or using an index - How many storage tiers are best? - Where should metadata live? - And more. It's always useful to dissect a modern distributed system with the goal of understanding it better, and it's even better to learn to deeper architectural principles in the process. Come to this talk for a generous helping of both.

1. Kafka Summit
Europe 2021
1
The Physics of Streaming

2. Data in motion
2

3. physics of streaming
3

4. 4
time

5. 5
space

6. 6
mass

7. The Paradigm for Data in Motion: Event Streams
7
Rich front-end
customer experiences
Real-time
Data
Real-time
Stream Processing
Real-time backend
operations
QUERY
A Sale
A shipment
A Trade
A Customer
Experience

8. Zhamak Dehghani : @zhamakd : 2021
Zhamak Dehghani : @zhamakd : 2021
Data Mesh
Data Infrastructure Plane
Providing the underlying infrastructure
required to build, run, and monitor data
products
CI/CD
Networking
Polyglot
big data
storage
Access
control
Data
transformation
orchestration
© ThoughtWorks 2021
Declaratively
create DP
Read
DP
Version
DP
Secure
DP
Build, deploy,
monitor DP
Data Product Developer Experience Plane
The higher level abstraction of data infrastructure
designed to support the common data product
developer journey
Mesh Experience Plane
Capabilities that are accessible
more conveniently at mesh level
Manage emergent
graphs of DPs
Manage security
policies of DPs
Discovery and
explore DPs

9. 9

10. Physical analogies
10

11. Why the Cloud?
11

12. Cloud Native Data Systems
Are The Future
12

13. Thank you.
13