Have you ever imagined what it would be like to build a massively scalable streaming application on Kafka, the challenges, the patterns and the thought process involved? How much of the application can be reused? What patterns will you discover? How does it all fit together? Depending upon your use case and business, this can mean many things. Starting out with a data pipeline is one thing, but evolving into a company-wide real-time application that is business critical and entirely dependent upon a streaming platform is a giant leap. Large-scale streaming applications are also called event streaming applications. They are classically different from other data systems; event streaming applications are viewed as a series of interconnected streams that are topologically defined using stream processors; they hold state that models your use case as events. Almost like a deconstructed real-time database. In this talk, I step through the origins of event streaming systems, understanding how they are developed from raw events to evolve into something that can be adopted at an organizational scale. I start with event-first thinking, Domain Driven Design to build data models that work with the fundamentals of Streams, Kafka Streams, KSQL and Serverless (FaaS). Building upon this, I explain how to build common business functionality by stepping through the patterns for: – Scalable payment processing – Run it on rails: Instrumentation and monitoring – Control flow patterns Finally, all of these concepts are combined in a solution architecture that can be used at an enterprise scale. I will introduce enterprise patterns such as events-as-a-backbone, events as APIs and methods for governance and self-service. You will leave talk with an understanding of how to model events with event-first thinking, how to work towards reusable streaming patterns and most importantly, how it all fits together at scale.

1. 1
The art of the event streaming application.
streams, stream processors and scale
Neil Avery,
Office of the CTO,
@avery_neil

2. 44

3. 55
Out of the tar pit, 2006

4. 66
“We believe that the major
contributor to this complexity in
many systems is the handling of
state and the burden that this adds
when trying to analyse and reason
about the system.”
Out of the tar pit, 2006

5. 77
What about Microservices?

6. 88
What are microservices?
Microservices are a software development
technique - a variant of the service-oriented
architecture (SOA) architectural style that
structures an application as a collection of
loosely coupled services.
https://en.wikipedia.org/wiki/Microservices

7. 99
structures an application as a collection of
loosely coupled services.
this is new!

8. 1010
So what went wrong?

9. 11
Making changes is risky

10. 12
Handling state is hard Cache?
Embedded?
Route to right instance?

11. 13
● Scaling is hard
● Handling state is hard
● Sharing, coordinating is hard
● Run a database in each microservice - is hard
What have we learned about microservices?

12. 14
We had it all wrong

13. 1515
We actually had some of it right

14. 1616
Immutability

15. 17
What’s the big idea?

16. 1818
Event driven architectures

17. 1919
aren’t new…
...but...

18. 2020
the world has changed

19. 21
New technology, requirements and expectations

20. 2222
Events
FACT!
SOMETHING
HAPPENED!

21. Ad placement
Examples...
User signed up Item was sold Payment

22. Events
Why do you care?
Loose coupling, autonomy, evolvability, scalability, resilience, traceability, replayability
EVENT-FIRST CHANGES HOW YOU
THINK ABOUT WHAT YOU ARE BUILDING
...more importantly...

23. 25
Store events in
..a stream..

24. 26
Different types of event models
● Change Data Capture - CDC (database txn log)
● Time series (IoT, metrics)
● Microservices (domain events)

25. 27
Capture behavior

26. 28
Time travel user experience?
how many users
affected?has it happened
before?
Ask many questions of the same data, again and again
time

27. 3131
old world : event-driven architectures
new world: event-streaming architectures

28. 32
Stream processing
Kafka
Streams
processor
input events
output events
...temporal reasoning...
event-driven microservice

29. {
user: 100
type: bid
item: 389
cat: bikes/mtb
region: dc-east
}
Partitions give you horizontal scale
/bikes/ by item-id
key#
Key
space
{...}
{...}
{...}
ConsumerTopic
Partition
Partition
assignment

30. 3737
Stream processors are uniquely
convergent.
Data + Processing
(sorry dba’s)

31. 3838
All of your data
is
a stream of events

32. 3939
stop...where is my database?
(you said scaling data was hard)

33. 4040
Streams are your persistence model
They are also
your local
database

34. 4141
The atomic unit for tackling complexity
Stream
processor
input events
output events
...or microservice or whatever...

35. 42
It’s pretty powerful
Stream
processor
Stream
processor
Stream
processor
Topic: click-stream
Interactive query
CDC events from KTable
CDC Stream
partition
partition
partition
CQRS
Elastic

36. 4343
Stream processor == Single atomic unit
It does one thing
Like

37. 4444
We think in terms of function
“Bounded Context”
(dataflow - choreography)

38. 4545
Let’s build something….
A simple dataflow series of processors
“Payment processing”

39. 4646
KPay looks like this:
https://github.com/confluentinc/demo-scene/tree/master/scalable-payment-processing

40. 4747
Bounded context
“Payments”
1. Payments inflight
2. Account processing [debit/credit]
3. Payments confirmed

41. 48
Payments bounded context
choreography

42. 49
Payments system: bounded context
[1] How much is being processed?
Expressed as:
- Count of payments inflight
- Total $ value processed
[2&3] Update the account balance
Expressed as:
- Debit
- Credit [4] Confirm successful payment
Expressed as:
- Total volume today
- Total $ amount today

43. 50
Payments system: AccountProcessor
accountBalanceKTable = inflight.groupByKey()
.aggregate(
AccountBalance::new,
(key, value, aggregate) -> aggregate.handle(key, value), accountStore);
KStream<String, Payment>[] branch = inflight
.map((KeyValueMapper<String, Payment, KeyValue<String, Payment>>) (key, value) -> {
if (value.getState() == Payment.State.debit) {
value.setStateAndId(Payment.State.credit);
} else if (value.getState() == Payment.State.credit) {
value.setStateAndId(Payment.State.complete);
}
return new KeyValue<>(value.getId(), value);
})
.branch(isCreditRecord, isCompleteRecord);
branch[0].to(paymentsInflightTopic);
branch[1].to(paymentsCompleteTopic);
https://github.com/confluentinc/demo-scene/blob/master/scalable-payment-processing/.../AccountProcessor.java
KTable state
(Kafka Streams)

44. 51
Payments system: AccountBalance
public AccountBalance handle(String key, Payment value) {
this.name = value.getId();
if (value.getState() == Payment.State.debit) {
this.amount = this.amount.subtract(value.getAmount());
} else if (value.getState() == Payment.State.credit) {
this.amount = this.amount.add(value.getAmount());
} else {
// report to dead letter queue via exception handler
throw new RuntimeException("Invalid payment received:" + value);
}
this.lastPayment = value;
return this;
}
https://github.com/confluentinc/demo-scene/.../scalable-payment-processing/.../model/AccountBalance.java

45. 52
Payments system: event model
https://github.com/confluentinc/demo-scene/.../scalable-payment-processing/.../io/confluent/kpay/payments

46. 5353
Bounded context
“Payments”
Is it enough?
no

47. 5454
“It’s asynchronous, I don’t trust it”
(some developer, 2018)

48. 5555
We only have one part of the picture
○ What about failures?
○ Upgrades?
○ How fast is it going?
○ What is happening - is it working?

49. 5656
Event-streaming provides
● Evolution
● Decoupling
● Bounded context modelling
● Composition
(because of SoC)

50. 5757
Composition

51. 5858
Event-streaming pillars:
1. Business function (payment)
2. Instrumentation plane (trust)
3. …
4. ...

52. 59
Instrumentation Plane (trust)
Goal: Prove the application is meeting business requirements
Metrics:
- Payments Inflight, Count and Dollar value
- Payment Complete, Count and Dollar value

53. 60
Instrumentation Plane
KStream<String, Payment> complete = builder.stream(paymentsCompleteTopic);
statsKTable = complete
.groupBy((key, value) -> "all-payments")
.windowedBy(TimeWindows.of(ONE_MINUTE))
.aggregate(
ThroughputStats::new,
(key, value, aggregate) -> aggregate.update(value),
completeWindowStore
);

54. 61
Instrumentation Plane
public ThroughputStats update(Payment payment) {
totalPayments++;
totalDollarAmount = totalDollarAmount.add(payment.getAmount());
maxLatency = Math.max(maxLatency, payment.getElapsedMillis());
minLatency = Math.min(minLatency, payment.getElapsedMillis());
if payment.getAmount().doubleValue() > largestPayment.getAmount().doubleValue()) {
largestPayment = payment;
}
timestamp = System.currentTimeMillis();
return this;
}

55. 62
Instrumentation Plane: Using IQ
https://github.com/confluentinc/demo-scene/blob/master/scalable-payment-processing/../ThroughputStats.java

56. 6363
Event-streaming pillars:
1. Business function (payment)
2. Instrumentation plane (trust)
3. Control plane (coordinate)
4. ...

57. 64
Control Plane
Goal: Provide mechanisms to coordinate system behavior
Why? Recover from outage, DR, overload etc
Applied: Flow control, start, pause, bootstrap, scale, gate and rate limit
Model:
- Status [pause, resume)
- Gate processor [Status]
- etc

58. 6565
Event-streaming pillars:
1. Business function (payment)
2. Instrumentation plane (trust)
3. Control plane (coordinate)
4. Operational plane (run)

59. 6666
Dependent on Control and Instrumentation planes
Dataflow patterns
● Application logs
● Error/Warning logs
● Audit logs
● Lineage
● Dead-letter-queues
/dead-letter/bid/region/processor
/ops/logs/ca
/ops/metric
Stream
processor
Operational Plane

60. 67
Architectural pillars
/payments/incoming
PAY
/payments/confirmed
Core dataflow
Control plane
/control/state
START
STOP
/control/status
stream.filter()
Instrumentation plane
/payments/confirmed
BIZ
METRIC
IQ
IQ
IQ
/payments/dlq
ERROR
WARN
IQ
Operational plane

61. 6868
Payment system

62. 6969
Composition Patterns

63. Bounded context (dataflow)
Choreography:
- Capture business function as a bounded context
- Events as API
2.
Accounts
[from]
payment.incoming
3.
Accounts
[to]
4.
Payment
Conf’d
1.
Payment
Inflight
payment.confirmed
payment.inflight
payment.inflight
payment.complete
payment.complete

64. Multiple Bounded contexts
Choreography:
- Chaining
- Layering
2.
Logistics
payment.incoming 1.
Payment
payment.complete

65. Multiple Bounded contexts
Orchestration
○ Captures workflow
○ Controls bounded context interaction
○ Business Process Model and Notation 2.0 (BPMN)
(Zeebe, Apache Airflow)
Source: https://docs.zeebe.io/bpmn-workflows/README.html

66. 7373
Composition patterns at scale
Flickr: Dave DeGobbi

67. {faas}
events as a backbone
appappappapp
Payments Department 2
{faas}appappappapp
Department 3 Department 4
Pattern: Events as a backbone

68. {faas}
What is going on here?
appappappapp
Payments Department 2
Patterns: Topic naming
bikeshedding (uncountable)
1. Futile investment of time and energy in
discussion of marginal technical issues.
2. Procrastination.
https://en.wiktionary.org/wiki/bikeshedding
Parkinson observed that a committee whose
job is to approve plans for a nuclear power
plant may spend the majority of its time on
relatively unimportant but easy-to-grasp
issues, such as what materials to use for the
staff bikeshed, while neglecting the design
of the power plant itself, which is far more
important but also far more difficult to
criticize constructively.

69. Patterns: Topic conventions
Don’t
1. Use fields that change
2. Use fields if data is available elsewhere
3. Tie topic names to consumers or producers
Do
<message type>.<dataset name>.<data name>
<app-context>.<message type>.<dataset name>.<data name>
Source: Chris Riccomini
https://riccomini.name/how-paint-bike-shed-kafka-topic-naming-conventions
● Logging
● Queuing
● Tracking
● etl/db
● Streaming
● Push
● user

70. 7777
What about that software crisis that started in
1968?
“We believe that the major contributor to this
complexity in many systems is the handling of state
and the burden that this adds when trying to analyse
and reason about the system.”
Out of the tar pit, 2006

71. Our mental model: Abstraction as an Art
Chained/Orchestrated
Bounded contexts
Stream processor
Stream
Event
Pillars
Business function Control plane Instrumentation Operations
Bounded context

72. Key takeaway (state)
Event streamingdriven microservices are the new atomic unit:
1. Provide simplicity (and time travel)
2. Handle state (via Kafka Streams)
3. Provide a new paradigm: convergent data and logic processing
Stream
processor

73. Key takeaway (complexity)
● Event-Streaming apps: model as bounded-context dataflows, handle
state & scaling
● Patterns: Build reusable dataflow patterns (instrumentation)
● Composition: Bounded contexts chaining and layering
● Composition: Choreography and Orchestration

74. 81
This is just the beginning

75. 82
Questions?
@avery_neil
“Journey to event driven” blog
1. Event-first thinking
2. Programming models
3. Serverless
4. Pillars of event-streaming ms’s
https://bit.ly/2tFfU84
or @avery_neil twitter profile

76. 83
@avery_neil