Building systems around an event-driven architecture is a powerful pattern for creating awesome data intensive applications. Apache Kafka simplifies scalability and provides an event-driven backbone for service architectures. But what can go wrong? Let me share some of my own blunders and lessons learnt in building event driven architecture so you don’t have to repeat my mistakes.

1. @SimonAubury
Event Driven Architecture
Mistakes – I’ve made a few …
A tale of confessions from Simon Aubury

2. @SimonAubury
Why am I here?
2
I am Simon Aubury
Principal Data Engineer @ ThoughtWorks

3. @SimonAubury
Let’s help a younger
me
3

4. @SimonAubury
“
Event-driven architecture (EDA) is a software
architecture paradigm promoting the production ..
consumption and reaction to events.
https://en.wikipedia.org/wiki/Event-driven_architecture

5. @SimonAubury
Mistake #1
The “right size” for a
Microservice ..
5

6. @SimonAubury
Thinking of events and boundaries
6
BBQ
Scout
Cashier
Stock
Room
Server

7. @SimonAubury
Written in Scala
Too obsessed with microservices
7
Single repository
Team fed by 2 pizzas
Written in Java
Dev-ops for 6 services
Cashier
Stock
Room
Server
BBQ
Scout

8. @SimonAubury
O
rder 26
Boundary between services
8
Cashier
Stock
Room
Server
2 Sausages with sauce
Sausage for order 26
Sausages
2 Sausages with
sauce & onion
BBQ
Scout

9. @SimonAubury
Understand event boundaries
9
Scout Server Cashier
Event Stream
Order for #26
Stock
Room
2 Sausages
2 Sausages for
customer #26
2 Sausages 2 Sausages for
customer #26
BBQ

10. @SimonAubury
Event Granularity
◉ Context boundaries
○ How big is a microservice?
◉ Record events as received
○ Avoid early splitting
◉ Capture broad categories
○ Producers shouldn’t discard
10

11. @SimonAubury
Lesson: the rules are different
within vs across boundaries
Favour asynchrony and eventual
consistency at context boundaries,
embrace the productive coupling of
synchronous within the boundary
11

12. @SimonAubury
Mistake #2
Messages != Events
12

13. @SimonAubury
Messages are not events
13
“Charge Jane’s credit card $20”
“Send an email to bob@example.com”
Message
“Jane ordered a burger”
“Bob viewed the new customer web page”
Event

14. @SimonAubury
Why events?
14
Messages
◉ Persisted until consumed
◉ Targeted to the entity
◉ Couple producers and
consumers
Events
◉ Re-playable stream history
◉ Consume historic events
◉ Loose coupling consumers
& producer
Messages in Job Queue
Events in an Event Log

15. @SimonAubury
Lesson: Beware the passive aggressive events
An event shouldn’t be used as a passive-aggressive
command.
It’s a “bad smell” if a source system expects the
recipient to carry out an action yet styles the
message as an event instead.
15

16. @SimonAubury
Mistake #3
Event notification != Event sourcing
16

17. @SimonAubury
Choreography vs. orchestration
Which system decides that an action
should be taken?
◉ Orchestration – system that will perform
once told to do it by another system
◉ Choreography (event driven) - a system
takes independent action
17

18. @SimonAubury
Many Meanings of Event Driven
◉ Event Notification
◉ Event-Carried State Transfer
◉ Event-Sourcing
◉ Command Query Responsibility
Segregation (CQRS)
18
https://martinfowler.com/articles/201701-event-driven.html

19. @SimonAubury
Lesson: don’t over engineer
Pick an event driven approach – and
be consistent and simple.
Don’t over-engineer an eventing
solution using Event-Sourcing/CQRS
19

20. @SimonAubury
Mistake #4
Building a message bus in
Kafka
20

21. @SimonAubury
History vs State
21
OCR Scan Start
Document found
Customer contact section identified
Name extracted with 98% confidence
Property land size found on page 4
OCR jam while scanning page 5
{"Cust": "Bob", "Property": "House"}
Document Scanner

22. @SimonAubury
History vs State
22
Projection of events
into local state store
Event notification:
Document scanned
OCR Scan Start
Document found
Customer contact section identified
Name extracted with 98% confidence
Property land size found on page 4
OCR jam while scanning page 5
{"Cust": "Bob", "Property": "House"}
Document Scanner

23. @SimonAubury
Lesson: Don’t build a message bus in Kafka
It’s a slippery slope – remember to produce & be
reactive to events … and not messages.
23

24. @SimonAubury
Mistake #5
Producers are not children
24

25. @SimonAubury
Producers are not children
◉ Basic rules for events
○ EDA guard-rails
○ Data Governance & Security
○ Good citizen rules
○ Naming scheme for topics
◉ Support & training
○ Opinioned framework approach
25

26. @SimonAubury
Cross bounded context …
26
Direct API Access Event Streaming
Coupled
API’s
Schema
required

27. @SimonAubury
Plan for schema evolution
Support change - data domains
need to evolve at their own
rate … without breaking
consumers.
TL;DR - Use schema registry
27

28. @SimonAubury
There is a process to find the
events that matter
◉ Use everyone to identify the
events that matter
◉ Understand the systems
◉ Start with broad categories
28

29. @SimonAubury
Lesson: EDA guardrails
Event must have’s
◉ Name – past tense
◉ Correlation ID
◉ Event production time
◉ Originating system
◉ Event creation system (may be different)
◉ A payload of stuff
29

30. @SimonAubury
Mistake #6
Don’t reinvent the EDA wheel
30

31. @SimonAubury
Event first thinking
◉ Capture facts & behaviour
◉ Represent the real world
◉ Model use cases of how we think
◉ Repeatability & scaling
◉ Common language
31
https://www.confluent.io/blog/journey-to-event-
driven-part-1-why-event-first-thinking-changes-
everything/

32. @SimonAubury
DDD - existing practices
◉ Problem modelling
○ Contexts - delineate boundary of consistency
◉ Separate our business logic from other
application concerns
◉ Reduce complexity
○ More effective software delivery
◉ Communicate better / A common language
32

33. @SimonAubury
Lesson: know your events
Modelling - discovery & integration …
Use simple language; solicit everyone's
input.
Develop your system inside out, focus
on the domain
33

34. @SimonAubury
Lesson: it’s hard work without DDD
Domain Driven Design gives us the tools to define
our bounded contexts, which give us our services.
Modelling the domain helps us identify the events
that are important to the domain.
34

35. @SimonAubury 35
What did I learn?
◉ Messages != Events
◉ Rules are different within vs across boundaries
◉ Don’t over engineer
◉ Don’t build a message bus in Kafka
◉ EDA guardrails
◉ Know your events

36. @SimonAubury
Lesson: start .. now
Event Driven Architecture adoption should start now.
Starting the first leads to the next transformational
opportunity. Shift towards EDA is driven by increasing
demands and heightened expectations and system
modernisation
Make (and share) your own mistakes …
36

37. @SimonAubury
Any questions ?
Thanks!
37
@SimonAubury
linkedin.com/in/simonaubury
Presentation template by SlidesCarnival