Unbundling the Modern Streaming Stack With Dunith Dhanushka | Current 2022 Event-first thinking and streaming help organizations transition from followers to leaders in the market. A reliable, scalable, and economical streaming architecture helps them get there. This talk first explores the ""classic streaming stack,"" based on the Lambda architecture, its origin, and why it didn't pick up amongst data-driven organizations. The modern streaming stack (MSS) is a lean, cloud-native, and economical alternative to classic streaming architectures, where it aims to make event-driven real-time applications viable for organizations. The second half of the talk explores the MSS in detail, including its core components, their purposes, and how Kappa architecture has influenced it. Moreover, the talk lays out a few considerations before planning a new streaming application within an organization. The talk concludes by discussing the challenges in the streaming world and how vendors are trying to overcome them in the future.

1. Unbundling the Modern
Streaming Stack
Dunith Dhanushka - 05/10/2022
Navigating the Real-time Analytics Landscape

2. About Me
twitter.com/dunithd medium.com/event-driven-utopia linkedin.com/in/dunithd/
• I’m Dunith Dhanushka
• Big data solution architect -> DevRel
• Blogs at eventdrivenutopia.com

3. Background
• This talk is based on my blog
that I published in April, 2022.
• This talk has been updated
with a few new things since
then.
• Enjoy!

4. Goal of the Talk
What Are We Going To Talk About Today?
Introduce you to the things
required to build real-time applications
that harness value from streaming data

5. The Plan
The Order of Things
1. A refresher on streaming data
2. The classic streaming stack
3. The modern streaming stack
4. Current trends and the future outlook

6. What is a Streaming Stack?

7. Streaming Data
What Is a stream?
A stream is a continuous, never-ending data
f
low with no beginning or
end. The data is incrementally made available over time, enabling you to
act upon it without needing to be downloaded
f
irst.

8. Events
Streams are made of events
A data stream consists of a series of data points ordered in time.
Each data point represents an “event” or a change in the state of the
business.
T4 T3 T2 T1 T0
Event source
Event stream
Time
Events

9. Event First Thinking
Modelling State Changes in Systems
A user with ID 1234 purchased item 567 for $3.99 on 2022/06/12 at Austin, TX
Fact Value
User ID 1234
Item ID 567
Price Paid $3.99
Date 2022/06/12
Place Austin, TX
• Events represents facts.
• Events are immutable.
• Events belong to the past.

10. Making Sense of Streaming Data

11. Events Have A Shelf Life
Act Fast Before You Lose Their Value
Image credit - https://d3i71xaburhd42.cloudfront.net/8cb6c2711afd3e504400ee12d3b582cc06348b08/7-Figure2-1.png

12. Real-time Analytics
Extracting Value From Events As Soon as They Are Made Available
REAL-TIME
ANALYTICS
Insights
React
Streams of Events

13. What is a Streaming Stack?

14. A streaming stack is the processes, tools, and technologies
you use to derive insights from unbounded data.

15. The Classic Streaming Stack

16. The Beginning
• Real-time analytics dates back to decades, existed in the forms of
Complex Event Processing (CEP) and Event Stream Processing (ESP).
• Most of the work has been academic. But few vendors like Progress
Apama, Esper, Tibco, and Streambase tried bringing it to the mass
market.

17. Then Came Big Data…

19. Lambda Architecture
Promotes A Uni
f
ied Serving Layer
Image credit - https://www.databricks.com/glossary/lambda-architecture

20. Why Didn’t It Pick Up?

22. • Overly complicated technology:
Specialised skillset of distributed
systems and performance
engineering.
• Limited only to the JVM: Non-
JVM developers had no option
rather than adapting.
• Higher footprint on
infrastructure: Stream
processors tax heavily on the CPU
and RAM.
• Maintenance overhead: Having
to maintain both speed and batch
layers.

23. The Modern Streaming Stack

24. Modern Streaming Stack
Modern Cloud-native tools
Managed and Serverless platforms
Rich tooling and developer experience
Expressive programming model

25. MSS is the classic streaming stack reimagined with
self-service cloud-native tools
providing a simpli
f
ied yet powerful developer experience
to build real-time analytics applications.

26. Modern Streaming Stack
STREAMING DATA
PLATFORM
STREAM PROCESSING
EVENT
PRODUCERS
TIERED
STORAGE
DATA API,
METADATA &
GOVERNANCE
Data-driven
Applications
Operational
Systems
Real-time
Analytics
SERVING LAYER

27. The Unbundling

28. Event Production/Enablement
The Origins of Events
STREAMING DATA
PLATFORM
Language Speci
f
ic SDK Clients

29. Streaming Data Platform

30. • Ingest events from sources in a
scalable manner, and store
them durably until they are
processed.
• Based on an immutable,
distributed log
f
ile. Events are
appended to the log and
partitioned across multiple
servers for durability and
scalability.
EVENT
PRODUCERS
Streaming Data Platform
TOPIC
TOPIC
TOPIC
TOPIC
TOPIC
TOPIC

31. Technology Choices

32. Stream Processors

33. STREAM PROCESSING
Event-driven Microservices
Streaming ETL
• Stream joins for enrichment
• Filtering/routing/transforming streams
• Data integration
• Repartitioning streams (re-keying)
Streaming Analytics
• Stateful aggregations
• Window operations
• Materialising streams, stream-table duality
• Actors
• Reactive logic execution
• Event-by-event processing, triggering side e
ff
ects

34. Technology Choices

35. Serving Layer

36. INPUT TOPIC OUTPUT TOPIC
Event Streaming Platform
STREAM PROCESSING
Serving Layer
Events Streaming ingestion
Real-time Insights Consumption
Internal/user-facing
Analytics
Data
Applications Recommendation
Ad-hoc
Exploration

37. Serving Layer
Expectations
• Serve queries with sub-second latency to provide a better user experience.
• Support a throughput of hundreds of thousands of queries per second to
serve an Internet-scale user base.
• Ensure data freshness — serve analytics from data ingested a few seconds
ago.
• Run complex OLAP queries, supporting joins, aggregations, and
f
iltering on
large data sets.

38. Serving Layer
Technology Choices
Key-value stores,
NoSQL databases Real-time OLAP Databases

39. Tiered Storage

40. Serving Layer
STREAMING DATA
PLATFORM
New Events
Older Events
Tiered Storage
• Back
f
illing
• Hydrating new applications
• Experimentation (ad-hoc querying)
• Archival/regulatory compliance
• Training ML models
O
ff
line Use Cases

41. Data APIs, Metadata, and Governance

42. Analytics must be democratised
and accessible across the board…
Image credits - https://www.datanami.com/2022/01/21/data-meshes-set-to-spread-in-2022/, https://www.con
f
luent.io/blog/how-to-build-a-data-mesh-using-
event-streams/

43. Event Mesh
EVENT CATALOG SCHEMA REGISTRY
STREAMING API GRAPHQL API
Serving Layer
STREAM PROCESSOR
EVENT STREAMING
PLATFORM
Decision makers Data applications Regulatory bodies Business partners
Real-time Insights

44. Technology Choices
Standards Schema Registries

45. Observations &
Future Outlook

46. Convergence of Stream Processing and Serving Layer
Streaming databases takes the stateful stream processing to the next level.
SaaS o
ff
errings Integrated serving layer Write logic with SQL
Pluggable integrations
A
ff
ordable Developer friendly
Pay-as-you-go
Less components to manage
Integrated tooling
Caters to non-JVM developers
Self-serve

47. Rise of The Lakehouse Architecture
A Lakehouse combines a data warehouse, data lake, and an event streaming platform
together.
High-throughput
streaming ingestion
Change Data Capture
Upserts
Transactions
Table formats

48. Takeaways

49. Takeaways
There’s No Silver Bullet
• Start small, build the critical path, and iterate.
• Pick components based on the need and know their limitations.
• Experiment, fail fast, and fail cheap.
• Go for managed services, if the team is small and new to streaming
technologies.
• Learn from mistakes, establish processes, and share wisdom!!

50. Book Announcement!

51. Thank you!
twitter.com/dunithd medium.com/event-driven-utopia linkedin.com/in/dunithd/
Find me at: