Stream processing affects a wide range of industries today: capturing sensor data, connecting microservices, processing the workloads of internet giants and giving us a real-time alternative to batch analytics. While these use cases are exciting and valuable they are only a taste of what is to come. In this talk we look at three areas that are likely to become more prominent: Global Apps, Event Stores and Serverless Stream Processing

1. The Future of Streaming: Global Apps, Event
Stores and Serverless
Ben Stopford
Office of the CTO, Confluent

2. Streaming sits at the intersection of
how we deal with data and how we
write programs

3. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute

4. Apps Apps Apps
Apps
Search Monitoring
Apps Apps
Apps Apps Apps
Apps
Search Monitoring
Apps Apps
Apps
Search
NoSQL
Apps
Apps
DWH
Hado
STREAM
ING
PLATFORM
Apps
Search
NoSQL
Apps
DWH
STREAMING
PLATFORM
PRODUCERCONSUMER
Streaming Platform

5. Event Storage
Kafka stores
petabytes of data
Stream Processing
Real-time processing
over streams and tables
Scalability
Clusters of hundreds
of machines. Global.
+ + +
Roots in big data messaging

6. > 2 trillion messages per day

7. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute

8. Events change our thinking
Monolithic Approach
-A database
-a variable
-a singleton
-a RPC
Event-First Approach
- An event
- A stream
- A log
- A stream processor

9. Event-driven programs have location transparency

10. They take us on journeys

11. Events let us run anywhere
Interconnecting these separate worlds as real-time ecosystems

12. The future lies in integrated global streaming

13. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute

14. Events change the way we
observe the world around us

15. Events:
A fact. An observation of the world.
An payment
A page view
A log line
A sensor reading

16. Events come in streams
Apps
M
onitoring
Security
Apps
Apps
L
A
T
F
O
R
M
Event
Stream
Order of events is important
Apps
Monitoring
Apps
Apps
O
R
M

17. Events record what
happened.
Streams record how it
happened

18. Traditional systems use mutable state
DB
This isn’t wrong, it’s just lossy
Apps
Search Mon
Apps Apps
S T R E A M I N G P L A T F O R M

19. Events record the user’s journey
Shopping Cart Events
2 Trousers added
1 Jumper added
1 Trousers removed
1 Hat added
Checkout
Shopping Cart
Event
User
Journey
12.42
12.44
12.49
12.50
12.59

20. Stored as a stream Stored statefully (think DB)
12.42
12.44
12.49
12.50
12.59 Information lost!
Event
User
Journey

21. 12.42
12.44
12.49
12.50
12.59
We can derive the current state
(but not the other way around)
Apps Apps
DERIVE
Stream Processor

22. Streaming is a form of Event Sourcing
The current state is a projection of the recording
Familiar
Stateful
View
LOSSY
PROJECTION
Stream = Exactly
what happened

23. Streams let us “observe the
game” one event at a time

24. The End State
Often the game is more important than the
end state
The Game

25. A stock price: observe the game, not just the current state

26. A customer journey: observe everything

27. Formula 1

28. Formula 1: Observe the game, optimize the end state
now and in the future
End state

29. Formulae 1 – High-Level Architecture
• 400 Sensors on car
• 70,000 derivative
measures
• Events streamed back to
base
• Analyzed in real time
• Tire modelling
• Racing line
• Aerodynamics
• Machine Learning and
Physics Models.
• Replayed later for post
race analysis.
Race Track HQ
e.g. Tire modelling:
- Temp
- Pressure
- Suspension compression
Stream Processing

30. Post race analysis

31. ML
SourceofTruth
Retain events, rewind and replay the stream processor

32. Another form of “Event Sourcing”
- Record what happened
- Rewind, replay and rederive (View, App, ML, Physics Model etc.)

33. New York Times
Store of Every
article since 1851
(Source of Truth)
https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/
Normalized assets
(images, articles, bylines, tags
all separate messages)
Denormalized into
“Content View”

34. Billing Shipping
Fraud Fraud
CUSTOMER
ANALYSIS
EVENT STORE
Rich, real-time recordings of customers and companies

35. Event Streams
Orders
Payments
Customers
Distinct Visits
Destination
Spark
Postgres
KSQL
Other Kafka
Select Organizational Events
Stream Processing
SELECT *
FROM ORDERS O, CUSTOMERS C
WHERE O.REGION = ‘EU’
AND C.TYPE = ‘Platinum’
Msgs/Day
Customers
Stream Processing
Spark
KSQL
Orders
History
1w
All
Event stores make data self service (real time & historical)

36. Rich recordings of customers and companies
Real-time
Historical
Self Service

37. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute

38. A future of
Streaming changes how we
observe the game.
Cloud changes how we play it.

39. Apps Apps Apps
Apps
Search Monitoring
Apps Apps
Apps Apps Apps
Apps
Search Monitoring
Apps Apps
Apps
Search
NoSQL
Apps
Apps
DWH
Hado
STREAM
ING
PLATFORM
Apps
Search
NoSQL
Apps
DWH
STREAMING
PLATFORM
PRODUCERCONSUMER
Confluent Cloud
2019
2019

40. Serverless and
Stream Processing are closely related

41. Using FaaS
• Write a function
• Upload
• Configure a trigger (HTTP, Event, Object Store, Database, Timer etc.)

42. FaaS in a Nutshell
• Short lived (max ~5 mins)
• Pay as you use
• 0-1000 concurrent functions, autoscales with load
• Interesting for spikey compute
• Interesting for low priority use cases e.g. CI systems.

43. But there are open questions

44. Serverless Developer Ecosystem
• Runtime diagnostics
• Monitoring
• Deploy loop
• Testing
• IDE integration
Currently quite poor

45. Harder than current approaches Easier than current approaches
Amazon
Google
Microsoft

47. FaaS is event-driven
But it isn’t streaming

48. Serverless Way: event driven but not streaming
Orders
Customers
Payments
FaaS
FaaS
FaaS

49. STREAMING:
Event-first - how we think
Event-sourced - how we store
Event-driven - how we combine data and interact

50. Transaction
Orders
Payments
KSQL
Customers
Streaming is Event-First, Event-Sourced & Event-Driven
Stateful or Stateless

51. FaaSFaaSFaaS
Transaction
KSQL
Stream processors can act as a “data layer” for FaaS ?
FaaSFaaS
StatelessStateful
(slower elasticity)
Orders
Payments
Customers

52. FaaSFaaSFaaS
Transaction
Orders
Payments
KSQL
Customers
StatelessStateful
Inherit Kafka’s Rich Feature Set?
FaaSFaaS

53. FaaS
Traditional
Application
Event-Driven
Application
Application
Database
KSQL
Stateful
Data Layer
FaaS
FaaS
FaaS
FaaS
FaaS
Streaming
Event-first
Event-sourced
Event-driven
Stateless
Stateless
Stateless
Compute Layer
Auto-scaling, correctness,
pluggability

54. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute

55. GLOBAL SYSTEMS, STORED EVENTS,
CLOUD NATIVE STREAM PROCESSING
Data Layer
FaaS
FaaS
FaaS
FaaS
FaaS

56. Thank you
@benstopford
Book:
https://www.confluent.io/designing-event-driven-systems