Kafka is widely positioned as the proverbial "central nervous system" of the enterprise. In this session, we explore how the central nervous system can be used to build a mesh topology & unified catalog of enterprise wide events, enabling development teams to build event driven architectures faster & better. The central theme of this topic is also aligned to seeking idioms from API Management, Service Meshes, Workflow management and Service orchestration. We compare how these approaches can be harmonized with Kafka. We will also touch upon the topic of how this relates to Domain Driven Design, CQRS & other patterns in microservices. Some potential takeaways for the discerning audience: 1. Opportunities in a platform approach to Event Driven Architecture in the enterprise 2. Adopting a product mindset around Data & Event Streams 3. Seeking harmony with allied enterprise applications

1. Data & Eventing Mesh
The next frontier of modernization
Pavan Keshavamurthy
Kafka Summit APAC 2021

2. Whoami?
● Founder
- OpenDevX → www.opendevx.io
- Platformatory → www.platformatory.io
● Technologist, Programmer
- Distributed Systems
- Modernization
● OSS Contributor
● Hobby Genomics
● Classical music
● Amateur Astronomy
● ..and a lot of other things

3. Layers & Silos
Big Data (Warehouse, Lake, Mart)
Transactional DB Transactional DB Transactional DB
{API Management}
no|new-sql
<Service/> <Service/> <Service/> <Service/>
<Service/> <Service/> <Service/> <Service/>
ETL / Data
Engg
Analytics & BI Advanced
Analytics (ML)
Data
Governance
…... …... …...
no|new-sql no|new-sql no|new-sql
ESB | MoM | BPM

4. The Reality.
That Middleware... That Warehouse... That Lake...
Some things about that didn’t scale very well in practice.

5. Layers & Silos
Big Data (Warehouse, Lake, Mart)
Transactional DB Transactional DB Transactional DB
no|new-sql
ETL / Data
Engg
Analytics & BI Advanced
Analytics (ML)
Data
Governance
…... …... …...
no|new-sql no|new-sql no|new-sql
The biggest impediment = Hyperspecialization Silos

6. “A database is a giant global
variable that pervades your
code. How, exactly, is that a
good thing?”
“We pride ourselves on
creating the biggest monolith
of them all, the big data
platform”
https://twitter.com/allenholub/status/140379691815
3531393
https://martinfowler.com/articles/data-monolith-to-mesh.html

7. - Decompose data (just like
services) by product ←→
domain & context bounds
- Evolve a Reactive Core
(Event Sourcing & CQRS)
Break the Monolith: Let the data (& events) flow
- Mesh Topology over
layered architecture

8. Real-Time Event
Store
Data Flow &
Observability
ProductC
atalog
Data
Security &
Governance
Service Mesh
Observability
(OpenTracing)
Product
Catalog
Security &
Governance
Data & Event Mesh Control Plane
API Mgmt & Service Mesh Control Plane
Traffic &
Routing
Bounded
Context
Bounded
Context
Bounded
Context

9. The Data Product
{{Data}}
Bulk /Raw Data Sets
(REST / File Transfer)
Querying, Aggregation
(OData, GraphQL..)
Real Time Events
(AsyncAPI, CloudEvents, *RPC
Polyglot Pipeline Composition
(Beam, DAG…)
X X X
X X X
Developer
BI / Analyst
X X X
X X
… {as a service}
Stakeholder Use-cases

10. Closing Thoughts
- Culture eats strategy for breakfast
- Small, distributed & self contained → resilient in
the long haul
- Not about tooling alone. People & process
transformation crucial
- Data Ops is the new DevOps - Embed Domain
Data Product ownership into cross functional
teams

11. Thank you
pavan@platformatory.com
www.platformatory.io