Event Sourcing, Stream Processing and Serverless (Ben Stopford, Con...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Event Sourcing, Stream Processing & Serverless
Ben Stopford
Office of the CTO, Confluent

What we’re going to talk about
• Event Sourcing
• What it is and how does it relate to Event Streaming?
• Stream Processin...

Can you do event sourcing
with Kafka?

Traditional Event
Sourcing

Popular example: Shopping Cart
DB
Apps
Search
Apps Apps
Database Table matches
what the user sees.

12.42
12.44
12.49
12.50
12.59
Event Sourcing stores events, then derives the
‘current state view’
Apps Apps
DERIVE
Chronol...

Traditional Event Sourcing
(Store immutable events in a database in time order)
Apps
Search
NoSQL
Monitoring
Security
Apps...

Traditional Event Sourcing (Read)
Apps
Search
NoSQL
Monitoring
Security
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Se...

3 Benefits

Evidentiary
Accountants don’t use erasers
(e.g. audit, ledger, git)

Replayability
Recover corrupted data after a
programmatic bug

Analytics
Keep the data needed to
extract trends and behaviors
i.e. non-lossy
(e.g. insight, metrics, ML)

Traditional Event Sourcing
• Use a database (any one will do)
• Create a table and insert events as they occur
• Query all...

Traditional Event Sourcing with Kafka
• Use a database Kafka
• Create a table topic insert events as they occur
• Query al...

Confusion: You can’t query Kafka by say Customer Id*
*Aggregate ID in DDD parlance

If we can’t query by Customer ID
then what do we do?

CQRS is a tonic: Cache the projection in a ‘View’
Apps
Search Monitoring
Apps Apps
S T R E A M I N G P L AT F O R M
Query ...

CQRS provides the
benefits of event
sourcing using a
“Materialized View”

Even with CQRS, Event Sourcing is Hard
CQRS helps, but it’s still quite hard if you’re a CRUD app

What’s the problem?
Harder:
• Eventually Consistent
• Multi-model (Complexity ∝ #Schemas in the log)
• More moving parts
A...

New York Times Website
Source of Truth
Every article since
1851
https://www.confluent.io/blog/publishing-apache-kafka-new-...

If CRUD makes sense there are other ways:
audit tables, CDC, etc.
Trigger
Evidentiary
Replayable N/A to web app
Analytics
...

More advanced: Use a Bi-Temporal Database

Events make most sense
where data has to move

This is where CQRS comes
into its own!

Online Transaction Processing: e.g. a Flight Booking System
- Flight price served 10,000 x #bookings
- Consistency require...

CQRS with event movement
Apps
Search Monitoring
Apps Apps
S T R E A M I N G P L AT F O R M
Apps
Search
NoSQL
Apps Apps
DWH...

The exact same logic applies
to microservices

Event Sourcing for Microservices
Basket Service
Fraud Service
Billing Service
Email ServiceBasket Events

Event Sourcing for Microservices
Basket Service
Fraud Service
Billing Service
Email ServiceBasket Events
Events are the
st...

Event Sourcing “with a DB”
for monoliths.
Event Streaming for
Microservices & Scale.
(Often via. CQRS)

Event Streaming

Event Streaming is a more general form of Event Sourcing/CQRS
Event Streaming
• Events as shared data model
• Many microse...

Event Streams is about many event sources
(Join, Filter, Transform and Summarize)
Fraud Service
Orders
Service
Payment
Ser...

KStreams & KSQL have different positioning
•KStreams is a library for Dataflow programming:
• App Logic & Stream Processor...

This difference makes most
sense if we we look to the
future.

Cloud & Serverless

Thesis
• Serverless provides event-driven infrastructure
• KSQL is the corollary: an event-driven database

Serverless Functions (FaaS)
• Write a function
• Upload
• Configure a trigger (HTTP, Messaging, Object Store, Database, Ti...

FaaS in a Nutshell
• Fully managed (Runs in a container pool)
• Pay for execution time (not resources used)
• Auto-scales ...

Where is FaaS useful?
• Spikey workloads and ‘occasional’ use cases
• Use cases that don’t typically warrant massive paral...

But there are open questions

Serverless Developer Ecosystem
• Runtime diagnostics
• Monitoring
• Deploy loop
• Testing
• IDE integration
Currently quit...

Harder than current approaches Easier than current approaches
Amazon
Google
Microsoft
Serverless programming will likely b...

In the future it seems
unlikely we’ll manage our
own infrastructure.
But where will we manage
our data?

Event-Streaming approaches this
from a different angle

FaaS is event-driven
But it isn’t streaming

Complex, Timing issues, Scaling limits
Customers
Event Source
Orders
Event Source
Payments
Event Source
Serverless functio...

A slightly more complex
example:
Send email only to
platinum customers

Payments
Event Source
Event is received by serverless function
FaaS/μS

Payments
Event Source
Block and calls the database to get customer+order
FaaS/μS
Get customer
Get order

Payments
Event Source
Is it a ‘Platinum’ customer?
FaaS/μS
Get customer
Get order
Is the customer
platinum?

Payments
Event Source
Send email if ‘Platinum’
FaaS/μS
Get customer
Get order
Maybe send email

Payments
Event Source
Increase Load: 100 concurrant functions doing IO.
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
Fa...

Payments
Event Source
Only send 2 emails.
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
FaaS/μS
...

Send SQL
Process
boundary
Orders
Payments
KSQL
Customers
Table
Customers
KSQL simplifies:
App
Logic
CREATE STREAM foo AS
S...

Functions have no
additional data
dependencies:
Everything is in the event!

Queries filter out the
events you need
(much like you filter rows in a
database query)

FaaSFaaSFaaSKSQL
Customers
Table
KSQL as a “Database” for Event-Driven Infrastructure
FaaSFaaS
Stateless,
elastic compute
...

FaaS
Traditional
Application
Event-Driven
Application
Application
Database
KSQL
Stateful
Data Layer
FaaS
FaaS
FaaS
FaaS
Fa...

Event-Driven vs. Event Streaming
Event Driven Event Streaming
Multiple Event Sources Use Database + ETL + Code Handles aut...

Event Streaming Platform

Summary
• Event Streaming provides the benefits of Event Sourcing to
microservices and data pipelines.
• Events are the da...

FaaSFaaSFaaSKSQL
Can I Build This?
FaaSFaaS
AWS Lambda /
Azure Functions Connectors
(in Preview)
Hosted KSQL In Preview
Co...

Things I didn’t tell you
• Tools like KSQL provide data provisioning, not state mutation.
• Use single writers. Try KSQL D...

Find out More
• Peeking Behind the Curtains of Serverless Platforms, Wang et al.
• Cloud Programming Simplified: A Berkele...

Thank you
@benstopford
Book:
https://www.confluent.io/designing-event-driven-systems

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1028076 views
Storytelling For The Web: Integrate... by Chiara Aliotta 938210 views
Artificial Intelligence, Data and C... by OECD Directorate ... 855306 views
How to Leverage AI to Boost Employe... by SocialHRCamp 358105 views
2024 State of Marketing Report – by... by Marius Sescu 203706 views
Everything You Need To Know About C... by Expeed Software 224725 views

View on Slideshare

1 of 69 Ad

View on Slideshare

1 of 69 Ad