This document discusses streaming app changes to event stores. It covers change data capture (CDC) which involves identifying, capturing, and delivering changes made to data. CDC can be done by capturing events from app code or by tailing database transaction logs. Capturing from app code provides flexibility but requires extra code, while capturing from databases is easier to control but depends on database log formats. The document also discusses using event stores for data warehouses, data lakes, CQRS patterns, and hybrid transactional/analytical processing (HTAP) databases.

1. Streaming
App Changes
to
Event Store
Shivji Kumar Jha
SME,
Stream Platform,
Nutanix

2. About Me
 Platform Engineer
 Specialties:
 DBs, Stream, SOA, infra etc
 Love
 Distributed systems
 Open source contributions
 Apache Pulsar, MySQL
 Communities & conferences!
 Talks
 https://www.slideshare.net/shiv4289/
 https://www.youtube.com/watch?v=Bx4csRi1b8Y&list=PL
A7KYGkuAD071myyg4X5ShsDHsOaIpHOq
@ShivjiJha in/shivjijha

3. The days
of
Monolith
One application Data in one place
Easy to query and
fetch information

4. Microservices
are hard!
Move ALL transactional data to one store – event store
Can’t query all data at once
1.Aggregate 1.Load 1.Index 1.Query
Data is fragmented.
Application split into modular microservices

5. Event Store – use cases
Data warehouses
Data Lakes
Data Lakehouse
CQRS

6. Change
Data
Capture
(CDC)
What
Identify
Capture
Deliver

7. Change
Data
Capture
(CDC)
How
Send Events from
App Code
Tail database

8. CDC: from App Code
 Capture
 Bad
 Write code for each event
 Difficult to control from platform teams.

9. CDC: from App Code
 Capture
 Bad
 Write code for each event
 Difficult to control from platform teams.
 Good
 Can leverage Flexibility in code
 App has more data than database.
 Eg : Data aggregated from other apps

10. CDC: from App Code
 Capture
 Bad
 Write code for each event
 Difficult to control from platform teams.
 Good
 Can leverage Flexibility in code
 App has more data than database.
 Eg : Data aggregated from other apps
 Delivery:
 Mostly asynchronously
 Synchronous if critical transaction, extra latency is ok.

11. CDC: from database
 Capture:
 Tail transaction logs

12. CDC: from database
 Capture:
 Tail transaction logs
 Bad
 Different structure in transactional logs of each DB
 Deal with log format changes in upgrades.
 Deal with DDL changes
 Maybe DB reverted transaction while recovering..

13. CDC: from database
 Capture:
 Tail transaction logs
 Bad
 Different structure in transactional logs of each DB
 Deal with log format changes in upgrades.
 Deal with DDL changes
 Maybe DB reverted transaction while recovering..
 Good
 Easy control from platform team.
 No extra code. No extra latency.
 App and CDC decoupled via queue and offset.
 Standard tools available : Debezium, maxwell

14. CDC: Outbox Pattern
https://medium.com/engineering-varo/event-driven-architecture-and-the-outbox-pattern-569e6fba7216

15. Hybrid Databases (HTAP)
 HTAP: Hybrid Transactional/Analytical Processing
 Evolving Landscape
 One database to serve
 OLTP and
 OLAP
 Examples:
 SingleStore DB
 TiDB
 MySQL Heatwave

16. References
 Change Data Capture
 https://en.wikipedia.org/wiki/Change_data_capture
 Debezium
 https://github.com/debezium
 Maxwell
 http://maxwells-daemon.io/
 https://github.com/zendesk/maxwell
 https://medium.com/@purbon/patterns-to-track-changes-in-data-
8d239734dc32
 Outbox pattern
 https://medium.com/engineering-varo/event-driven-architecture-
and-the-outbox-pattern-569e6fba7216

17. @ShivjiJha in/shivjijha/
ShivjiKumarJha
shiv4289
QUESTIONS?

18. THANK YOU!
QUESTIONS?