Fully embracing the “one database per microservice” principle can present challenges for the management of data across the whole ecosystem. Ideas like event sourcing, CQRS and applying Saga patterns approach the problem from different perspectives. Change Data Capture (CDC) is another method that can help stream changes made to a database into an event medium such as Kafka. When applied successfully, CDC keeps all other microservices and their databases informed. At Brolly, we have implemented a log-based Change Data Capture (CDC) solution using Kafka Connect and Debezium. Having built and productionised the solution, we know what we have done well, what we could have improved and what challenges you may face designing your own. We share the reasons behind our decision to use CDC, some insights about what to watch out for and lessons learned along the journey. We touch on topics such as how to guarantee transactional messaging by applying the famous outbox pattern, the implications of having badly designed aggregates, how Avro and schema registry can assist, what to do when processing a message on the receiving end fails for whatever reason, and how to mitigate the risk of processing an event multiple times or out of order.

1. How Change Data Capture with
Kafka Connect Came to Our rescue
Ali Nazemian
Milad Vadood
Sharing Data
Among Microservices

2. Ali Nazemian
Chief Technology Officer,
Brolly
Milad Vadood
Senior Data Engineer,
Brolly
Who we are

3. What is Brolly?

4. What is Brolly?

5. Who uses
Brolly?

6. Agenda
Takeaways
01
02
03
04
Background
Things to keep an eye on
What we did at Brolly

7. One Database
per Microservice
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

8. Problem with multiple databases
How to support
distributed write
How to support
distributed read
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

9. Known solutions
Saga Pattern
01
https://www.youtube.com/watch?v=YPbGW3Fnmbc
Chris Richardson
> Background > What we did at Brolly > Things to keep an eye on > Takeaways
Event sourcing
& CQRS
02

10. What if we don’t
have event
sourcing in place?

11. Streaming database
changes
03
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

12. Streaming database
changes
03
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

13. How can we
stream changes?
● Native db-backed methods
● Data integration tools
● Streaming changes through a medium
such as Kafka
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

14. What we did

15. Overall
solution
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

16. How to
address
distributed
read/write
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

17. How can we guarantee transactional messaging?
1- Create a channel
2- Store
channel
3- publish
“channel
created”
4- Return success
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

18. Kafka Connect to our rescue
● Sink Connectors
● Source Connectors
● JDBC connectors
● Debezium connectors
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

19. How to stream changes
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

20. The Outbox Pattern
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

21. The Outbox Pattern
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

22. Things to
keep an eye
on
● Complications with ordering in Kafka
● Error management
● Optimising the outbox
● More on Debezium

23. Complications
with ordering
in Kafka

24. Complications
with
ordering
in
Kafka
Entities to sync:
● Alert
● Channel
● Channel Alert
Kafka message key: pk
? Problem
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

25. 2
Lionel Messi
leomessi
5
26
taxes
10
5
2
Temp
Temp
Temporary placeholders (Scenario 1)
1. “Add alert” arrives
2. “Add channel_alert” arrives -> Create placeholder
3. “Add channel” arrives -> Update channel
i
Complications
with
ordering
in
Kafka
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

26. 2
Lionel Messi
leomessi
5
26
taxes
10
5
2
Temp
Temp
1. “Add alert” arrives
4. “Add channel_alert” arrives -> Create placeholder in channel
5. “Delete channel_alert” -> Delete channel Alert
2. “Add channel” arrives
3. “Delete channel” arrives
Temporary placeholders (Scenario 2)
i
Complications
with
ordering
in
Kafka
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

27. Soft deletes
Stashing changes
https://twitter.com/gunnarmorling
Having one Kafka
partition!
https://www.youtube.com/watch?v=fWU8ZK0Dmxs&t=1758s
Alternative solutions
Complications
with
ordering
in
Kafka
01
03
02
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

28. It shouldn’t be
this difficult

29. On DDD
Bounded context
Aggregate
- Cluster of objects treated as a unit
- Graph consisting of a root object and
one or more others
Domain Object
- https://www.eventstorming.com
- Alberto Brandolini
Complications
with
ordering
in
Kafka
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

30. Rules when Defining Aggregates
1. Reference other Aggregate roots via identity
2. Transaction = processing one command by one
aggregate
● Each object containing full information
● Remove the connecting entity
○ Channels could be repeated for multiple alerts
Complications
with
ordering
in
Kafka
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

31. Error
Management

32. Specify error type
Determine error management
strategy
- Non retryable
- Stateless retryable
- Stateful blocking
retryable
- Stateful non-blocking
retryable
- Ignore / prevent
- Delay (Automatic)
- Replay
- Itself
- Itself and the following
- Block and intervene manually
Error
Management
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

33. Optimising
the outbox

34. - Too many events
- long time to sync (new services/components)
- Rolling up events
t1
Insert ‘foo’
t2
Update ‘foo’
t3
Delete ‘foo’
- Snapshots (recreate
outbox)
Challenge
?
Solution
i
Optimising
the
Outbox
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

35. - Outbox table can become huge over time
- Slower writes over time
- Performance drop searching for events
- Retain only for a period of time
- Delete change row as soon as inserted
Challenge
?
Solution
i
Optimising
the
Outbox
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

36. More on
Debezium

37. Use Kafka
connector
Use
embedded
client
> Background > What we did at Brolly > Things to keep an eye on > Takeaways
More
on
Debezium

38. Takeaways
01
02
03
CDC for sharing data among microservices
Transactional messaging and Kafka
connect to the rescue
Flexibly defined domain objects using the
outbox pattern
> Background > What we did at Brolly > Things to keep an eye on > Takeaways

39. Takeaways
04
05
06
07
> Background > What we did at Brolly > Things to keep an eye on > Takeaways
Define your aggregates with extra-care
Plan for optimising the outbox table and
the events topics
Pick up the best error management
strategy for each type of error
Use the most suitable version of Debezium

40. Ali Nazemian
www.linkedin.com/in/ali-nazemian
Milad Vadood
www.linkedin.com/in/mvadood
Thank you
www.brolly.com.au