We start by setting up a common ground introducing why relational databases fall short, addressing common EDA characteristics such as the need for real-time response times and schemaless approaches to address recurring changes to adapt and on-board new use cases. Next, interact with a sample Rust-based application: a social network app demonstrating an integration of both ScyllaDB and Redpanda.

1. Integrating Distributed Data
Stores Across Event Streams
Felipe Cardeneti Mendes

2. 2
● Solution Architect at ScyllaDB
● Published Author
● Linux and Open Source enthusiast
Felipe Cardeneti Mendes

3. DDD? EDA? Event Sourcing?
Event Streaming? SAGA?
3

4. Complexity Exist in Both Sides… But for Different Reasons
4
■ Highly Encouraged: Martin Kleppmann — Event Sourcing and Stream Processing at Scale

5. PageView Event Example
Track whenever a page gets viewed… So what?
■ Who viewed your profile?
■ People also viewed …
■ Reporting
■ Relevance Models (ie: Result Ranking)
■ Metrics
5
From Martin Kleppmann — Event Sourcing and Stream Processing at Scale

6. Ok… so why do we need ScyllaDB then?
This is where we (slightly) diverge from Martin:
6
From Martin Kleppmann — Event Sourcing and Stream Processing at Scale
Super slow – But doesn't have to be!

7. So you're telling me to ditch Stream Processing?
Well… No. Although you could.
■ Data and Domain dependent
○ Full-text searches;
○ Ad-hoc querying;
○ Joins;
■ Stateless vs Stateful Processing
○ Keep doing Stateless (or semi) as you know;
○ Re-think your Stateful Processing strategy
■ We are (almost) a decade past 2016
7

8. Let's Get to Practice
8

9. (Very) High-level overview
(Very) High Level Overview
9
GitHub Project: fee-mendes/eda-socialnetwork

10. Tracking Events
■ Each event is uniquely identified
○ Great for even distribution and performance
■ Auditing, history of interactions, power Batch Analytics
○ Bad for aggregations (including sliding windows) – Blocker for Real-time
analytics
■ Potential workaround: Map event to entity+type, cluster by timestamp
■ Actual data model:
10
CREATE TABLE IF NOT EXISTS ks.events (
id uuid,
ts timestamp,
event_type text,
src_page text,
PRIMARY KEY(id, event_type, ts)
)

11. Counter Tables (Post Likes, Profile Views, Page Hits)
■ Well… Used for counting things :-)
○ Goods:
■ Simple
■ Highly performant (no aggregations needed, hooray!)
○ Problem: Misses context
■ Who viewed a profile?
■ Which pages were popular for users within a given region?
■ Was a post liked by similar users?
○ Hence the importance of defining your events table upfront
■ Actual data model:
11
CREATE TABLE IF NOT EXISTS ks.post_likes (
post_id uuid PRIMARY KEY,
count counter
)

12. Relationships (Follow and Followers)
12
■ Easy done (and consistent):
○ If Y is followed by X, then
○ X follows Y
○ Simply materialize keys in the opposite direction
■ Though admittedly takes a while to get the gist of it
■ Actual (base) data model:
CREATE TABLE IF NOT EXISTS ks.follows (
id uuid,
follower uuid,
ts timestamp,
PRIMARY KEY(id, follower)
)
Swap the two

13. Improvement Thoughts
13
■ Consistency guarantees – Apply common patterns such as:
○ Outbox – For non-idempotent operations (DON'T overuse it)
○ Listen to Yourself
■ Publish database events (CDC):
○ Strongly consistent Push notifications!
○ Feed other downstream app/services (ie: Support full-text, ad-hoc, etc)
■ WebSockets for real-time communication / async callbacks?
■ You name it! ;-)

14. User Extreme (and Inspiring) Stories
14
How Numberly Replaced Kafka with Rust + ScyllaDB How Palo Alto Networks Replaced Kafka with ScyllaDB
From 1M to 1B Features Per Second: Scaling ShareChat’s
ML Feature Store
Elasticity vs. State? Exploring Kafka Streams Cassandra
State Store

15. Keep in touch!
Felipe Cardeneti Mendes
Solution Architect
ScyllaDB
felipemendes@scylladb.com
Find me on LinkedIn!