Patterns and Practices for Event Design With Adam Bellemare | Current 2022

You Put WHAT in Your Stream?!
Adam Bellemare - Staff Technologist
Ofﬁce of the CTO

@AdamBellemare | developer.confluent.io
My name is Adam
● Staff Technologist at Conﬂuent
● Previously a Data Platform Engineer
● Written a few Books
● Check out the Practical Data Mesh!
Released 2020
AVAILABLE NOW!

So you’re at an event-driven architectures talk…

“But what did you actually put in your event?”

The hand-waving event design technique
Source: NHLGIFs, apparently

Okay it didn’t work.
And now my arms are tired.

And no, not by going to 7th Normal Form

Schema-on-Read Doesn’t Work for Streams
A typical
“Schema-On-Read”
big data ﬁle system
Source: https://commons.wikimedia.org/wiki/File:Messy_storage_room_with_boxes.jpg
“I swear that data is around here
somewhere.
I put it right next to the tuna
sandwich…”

Per-Consumer Sensitivity to Bad Data
Stream
Stream

Schemaless events
Source: Dumpster Fires as a Service

Bloopers
promotedContent:
True, False

Bloopers
promotedContent:
True, False,
“Yes”, “No”

Bloopers
promotedContent:
True, False,
“Yes”, “No”,
4291, null

Bloopers
promotedContent:
True, False,
4291, null,
“null”

Bloopers
promotedContent:
Source: Personal Experience
True, False,
4291, null,
“null”

Use a Schema Registry
Conﬂuent Schema Registry
● Supports:
○ Avro
○ Protobuf
○ JSON Schema (note! It has a schema!)
● Supports Schema Evolution
● Enables Code Generators
● Enables Schema Veriﬁcation and Testing

Factor 1:
Fact or Delta Event Types

Fact Events vs. Delta Events
Deltas Facts

When do you use Deltas, and when do you use Facts?

TIP: Ask yourself:
Are you transferring data?
OR
Are you notifying of an occurrence?

A lot of it depends on boundaries
Stream
Stream
Internal World
(Data on the Inside)
- Relational DB
- Document DB
- Event Sourcing
- e.g.
External World
(Data on the Outside)
External World
Bounded Context

Fact Events as Data Transfer Objects
Event Stream
Tables
Stream Stream
Database Backed
Event Sourcing
Create External Event
from Internal Model

Cart Fact Event (a DTO)
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
"shipping": "express"
}
item_added_to_cart
{
"cart_id": 3125123,
"item_id": 521,
"quantity": 1
}
discount_code_applied
{
"cart_id": 3125123,
"discount_code": "SAVE-20-2022",
"discount_percent": "20"
}
Facts Model State – Deltas Model Change
Delta
Events

Deltas Are Used in Event Sourcing
(Data on the Inside)
Customer Product Quantity
Robert
Pants 1
T-Shirts 1
Hats 15
11:37 2 Pants added to cart
11:39 1 T-Shirt added to cart
11:41 1 Pants removed from cart
11:42 15 Hats added to cart
11:42 Apply Discount Code
Event
Time Events
Read
Apply in order to build state
Current Consumer State
Stream Table

Deltas Not Suitable for Building External State
11:37 2 Pants added to cart
11:39 1 T-Shirt added to cart
11:41 1 Pants removed from cart
11:42 15 Hats added to cart
11:42 Apply Discount Code
Event
Time Events
Read
Duplicate and
Tightly Coupled
Cart Building
Logic
Independent Consumer
Services
Apply in order to build state
Apply in order
to build state
Stream

Use Case: General Alerting of Changes
item_added_to_cart
item_removed_from_cart
Consumers

What about communicating changes with facts?

Inferring Changes from Fact Events
{
"cart_id": 3125123,
"item_map": [ (521,1) ],
"shipping": "standard"
}
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
}
Cart Fact 1 Cart Fact 2

Facts with Before/After Fields
{
"before": {
"cart_id": 3125123,
"item_map": [ (521,1) ],
"shipping": "standard"
},
"after": {
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
}
}
Cart Fact Event

Fact Event Considerations
Event Size

Event Size Update Frequency

Event Size Update Frequency
Intent
?

Facts are good for state transfer and ﬂexibility
Deltas are good for event sourcing
and notiﬁcations

Factor 2:
How normalized should your events be?

Highly Normalized Streams
Item Stream
Brand Stream
Rating Stream
Item
Brand
Rating
Connectors

Streaming Joins Can Be Costly
Item Stream
Brand Stream
Rating Stream
Same Joins!
Item
Brand
Rating
Consumers

Denormalizing Streams - Everybody’s doing it
Source: The 60s. Maybe the early 70s. That’s like 50 years ago…

Events are best served denormalized

Option 1: Outbox Table Pattern
Item
Brand
Connector
Outbox
Table
Enriched Item Stream
Transaction
Rating

Item
Brand
Rating
Connector
Outbox
Table
Transaction
Join
Triggers?

Item
Brand
Rating
Connector
Outbox
Table
Transaction
Extra Load
Join
Triggers?

Item
Brand
Rating
Connector
Outbox
Table
Transaction
Code
Extra Load
Join
Triggers?

Item
Brand
Rating
Connector
Outbox
Table
Transaction
Deletions
Extra Load
Join
Triggers?
Code

Option 2: Denormalize the Streams
Item Facts Stream
Brand Facts Stream
Rating Facts Stream
Enriched Item Facts Stream
Join events together
using ksqlDB

Option 2: Denormalize the Streams
Normalized
SELECT
I.ID,
I.PRICE,
I.NAME,
B.NAME
FROM ITEM I
INNER JOIN BRAND B
ON I.BRAND_ID = B.ID
EMIT CHANGES;
ksqlDB
Item Facts Stream
Brand Facts Stream
Enriched Item Facts Stream
Denormalized

Denormalization Considerations
Event Size Join Trigger
Frequency

Factor 3:
Single vs. Multiple
Deﬁnitions per Stream

Use Case 1: Transferring State with Facts
Cart Fact Event
{
"cart_id": 3125123,
"item_map": [ (521,1), (923,3) ],
}
Single Type per Stream
Cart Facts
Consumers

Use Case: Transferring State with Facts
Cart Facts
Item Facts
Consumers

Single Fact Type per Stream
CREATE TABLE Items(
id BIGINT PRIMARY KEY,
price DECIMAL(10,2),
name STRING,
description STRING,
brand_id BIGINT
) WITH (
kafka_topic = 'items',
key_format = 'KAFKA',
value_format = 'AVRO');
Item Facts
ksqlDB

Best Practice: Record the Entire State in one Fact
Order
Cart
User
ksqlDB
Order_Cart
Cart
Order_User
User

Use Case 2: General Alerting of Changes
item_added_to_cart
Consumers

Remove
item
from
cart
Add
item
to
cart
Apply
discount
code
Remove
item
from
cart
Apply
discount
code
Remove
item
from
cart
Cart Events Stream
Use Case 3: Strict Order Processing

Multiple Types per Stream
Cart Events Stream
Consumer
item_added_to_cart
Use Case 3: Strict Order Processing

Facts:
Single Deﬁnition per Stream
Deltas:
Single or Multiple Deﬁnitions per Stream

Evolving your Data Model?

Evolving your Data Model?
Source: Personal Experience

Versioned Streams for Breaking Changes
Cart Version 2
Cart Version 1
Producer
(Writes both for a period)

Versioned Streams for Breaking Changes
Cart Version 2
Producer

Deltas
Naming so far?
item_added_to_cart
Facts
Cart
Item
User

Streams Across the Business
Inventory
Orders
Shipments
Finance

Event Stream Naming
1) <domain>.<event-type>.<version>

Event Stream Naming
1) <domain>.<event-type>.<version>
Old Fact Stream Name:
Customers.Advertisement
New Fact Stream Names:
Customers.Advertisement.v2

2) <domain>.<service>.<event-type>.<version>
Event Stream Naming Including Service Name

2) <domain>.<service>.<event-type>.<version>
Event Stream Naming Including Service Name
Orders.Order_Placer_Service.Order.v1
Customers.Ad_Manager.Advertisement.v2
Customers.Ad_Budget_Manager.Ad_Promo_Budget.v2

Check out my
Data Mesh Book! Free full courses at:
developer.conﬂuent.io
AVAILABLE NOW!

Patterns and Practices for Event Design With Adam Bellemare | Current 2022

Recommended

Recommended

More Related Content

Similar to Patterns and Practices for Event Design With Adam Bellemare | Current 2022

Similar to Patterns and Practices for Event Design With Adam Bellemare | Current 2022 (20)

More from HostedbyConfluent

More from HostedbyConfluent (20)

Recently uploaded

Recently uploaded (20)

Patterns and Practices for Event Design With Adam Bellemare | Current 2022