Stream processing affects a wide range of industries today: capturing sensor data, connecting microservices, processing the workloads of internet giants and giving us a real-time alternative to batch analytics.
While these use cases are exciting and valuable they are only a taste of what is to come. In this talk we look at three areas that are likely to become more prominent: Global Apps, Event Stores and Serverless Stream Processing
The Future of Streaming: Global Apps, Event Stores and Serverless
1. The Future of Streaming: Global Apps, Event
Stores and Serverless
Ben Stopford
Office of the CTO, Confluent
2. Streaming sits at the intersection of
how we deal with data and how we
write programs
3. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute
5. Event Storage
Kafka stores
petabytes of data
Stream Processing
Real-time processing
over streams and tables
Scalability
Clusters of hundreds
of machines. Global.
+ + +
Roots in big data messaging
7. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute
8. Events change our thinking
Monolithic Approach
-A database
-a variable
-a singleton
-a RPC
Event-First Approach
- An event
- A stream
- A log
- A stream processor
13. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute
22. Streaming is a form of Event Sourcing
The current state is a projection of the recording
Familiar
Stateful
View
LOSSY
PROJECTION
Stream = Exactly
what happened
28. Formula 1: Observe the game, optimize the end state
now and in the future
End state
29. Formulae 1 – High-Level Architecture
• 400 Sensors on car
• 70,000 derivative
measures
• Events streamed back to
base
• Analyzed in real time
• Tire modelling
• Racing line
• Aerodynamics
• Machine Learning and
Physics Models.
• Replayed later for post
race analysis.
Race Track HQ
e.g. Tire modelling:
- Temp
- Pressure
- Suspension compression
Stream Processing
32. Another form of “Event Sourcing”
- Record what happened
- Rewind, replay and rederive (View, App, ML, Physics Model etc.)
33. New York Times
Store of Every
article since 1851
(Source of Truth)
https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/
Normalized assets
(images, articles, bylines, tags
all separate messages)
Denormalized into
“Content View”
36. Rich recordings of customers and companies
Real-time
Historical
Self Service
37. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute
38. A future of
Streaming changes how we
observe the game.
Cloud changes how we play it.
41. Using FaaS
• Write a function
• Upload
• Configure a trigger (HTTP, Event, Object Store, Database, Timer etc.)
42. FaaS in a Nutshell
• Short lived (max ~5 mins)
• Pay as you use
• 0-1000 concurrent functions, autoscales with load
• Interesting for spikey compute
• Interesting for low priority use cases e.g. CI systems.
54. THREE TASTES OF THE FUTURE
Global Apps: Location independent applications
Event Stores: Rich recordings of customers and companies
Serverless Stream Processing: Melding real-time, elastic data
and compute
55. GLOBAL SYSTEMS, STORED EVENTS,
CLOUD NATIVE STREAM PROCESSING
Data Layer
FaaS
FaaS
FaaS
FaaS
FaaS