Discuss the core tradeoffs and considerations involved in order-free and ordered stream processing. Brian Taylor walks through the pros and cons of three different approaches: no data dependency, deferred inter-event data dependency, and streaming inter-event data dependency.

1. A Tale of 3 Pipelines
Brian Taylor

2. Reference Architectures
2
Stream to Stream
Stream to State
Stateful Stream to
Stream
Write: Money = Performance
Read: Data dependency
limited
Data dependency
limited
Money = Performance
ODP “Analytic
segments”
ODP “Real-Time
Segments”
Webhook system
What it is: How it scales:

3. Inter-Event Data Dependency
3
A property of the stream and the problem. Measures the way that events impact
the processing of following events.
For example:
A stream of record mutations that must be applied one after another within a
single record id
● A stream with many records and no sequential mutations for any given record
id has no data dependency
● A stream with a single record id and only sequential mutations has maximum
data dependency
● A topic with a single partition has maximum data dependency (in some sense)

4. Inter-Event Data Dependency
The average length of the data-dependent chains in your stream
decide your average throughput at any scale.
This is equivalent to the way “the sequential portion” of a problem
constrains the maximum parallel speedup in Amdhal’s Law.
S: max speedup fraction, s: parallelism, p: “data dependency fraction”

5. Big Idea
5
● “It’s all about the data-dependency, baby”
● No data dependency: Smooth scaling
● Data dependency: Navigating hell

6. No Data Dependency
∅DD
6

7. Reference Architecture
7
Stream to Stream Money = Performance
Webhook system
What it is: How it scales:
Subscription information
Change
Notifications
Delivery
Requests

8. What you can do with ∅DD
8
Abstractly
■ Data reshaping
■ Order-independent enrichment
■ Non-self Joins
Concrete Use Cases
■ Adapters
■ Sentiment detectors
■ Geo-IP mappers
■ Redaction
If no external data
access is required:
Redpanda transforms
FTW!

9. Performance Tradespace
9
More money = More Throughput
Tactics: Add shards and partitions until you have enough capacity

10. Deferred Inter-Event
Data Dependency
DEDD
10

11. Reference Architecture
11
Stream to State
Write: Money = Performance
Read: Data dependency
limited
ODP “Analytic
segments”
Optimizely
Experimentation
What it is: How it scales:

12. What you can do with DEDD
12
Abstractly
● Use it when Write Performance is more important than Read
Performance
Concrete Use Cases
● Reporting: Especially when users read less than they write
● Nightly model training

13. Performance Tradespace
13
Write side: More money = More Speed
Read side: Data-dependency limited
Tactics: Reduce data dependency with finer grained partitioning

14. Streaming Inter-Event
Data Dependency
SEDD
14

15. Reference Architecture
15
Stateful Stream to
Stream
Data dependency
limited
ODP “Real-Time
Segments”
What it is: How it scales:

16. What you can do with SEDD
16
Abstractly
■ Streaming aggregates
■ Pattern detectors
Concrete Use Cases
■ Segmentation
■ Real time model training

17. Performance Tradespace
17
Throughput: Data-dependency limited
Tactics for reducing data-dependency:
■ Finer grained partitioning
■ Accept eventual consistency with CRDTs

18. Fundamental Tradeoff
Inter-event data dependency Max throughput
If you need SEDD and throughput, welcome to hell.
18

19. But…
19

20. Query Latency Data Latency
Query Latency: Time it takes to
respond to a request
Driven by: DD work remaining
to resolve the request
Impact: The places where it’s
suitable to use your query API
20
Data Latency: How long it
takes for new information to
impact a query
Driven by: How you cheated to
hide from your data
dependency
Impact: How actionable the
results from your API are

21. “Cheating” out of Hell
21
Stream to State
Introduces a data
latency / cost tradeoff
Min-data latency is
now data dependency
limited
Everyone else’s
“Real-Time
Segments”
What it is:
How it scales:
Periodic State to Stream

22. Wrapping it Up

23. Data Dependency Decides Everything
∅DD - Oddly common in example code and marketing materials. Very
rarely happens in real life.
DEDD - Practical workaround most of the time. Became truly effective
in the last decade as data warehouses have matured.
SEDD - Sounds like “sad” for a reason. A difficult place to be. Hopefully
the next decade will bring some meaningful breakthroughs here.
23

24. Keep in touch!
Brian Taylor
Director of Engineering
Optimizely
brian.taylor@optimizely.com
@netguy204