The Possibilities and Pitfalls of Writing Your Own State Stores wit...

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

The Possibilities And Pitfalls of Writing
Your Own State Stores
Daan Gerits

Setting The Scene
KOR Financial
Regulatory Reporting
Event Driven Organization
Event Driven Foundation
Based on Kafka
Long...

3
Common API approach

4
CQRS API Approach

5
Focus Of This Talk

What is a state store
Part of Kafka Streams
Embedded “cache”
Internal to the application
Local vs Global Statestores
Fault...

The
Challenges

Challenge 1
Key Value Only

KEY
9
value

Challenge 1: Key Value Only
Fast!
GET, SCAN
RocksDB
In-Memory, overflow to disk
Tweak the default memory settings!
https:/...

Challenge 1: Key Value Only
But what if we had:
- Search capabilities
- Filter-like API
11

Challenge 1: Key Value Only
But what if we had:
- Search capabilities
- Filter-like API
Custom Statestores!
12

Challenge 1: Key Value Only
But what if we had:
- Search capabilities
- Filter-like API
Embedded databases:
- H2, HSQLDB, ...

Challenge 1: Key Value Only
NitriteDB
Document Store
Natural Filtering API
Supports Indexes
14
Cursor cursor = collection....

NO2 keys are Integers and values are Documents
→ Use NO2 Documents as envelopes
→ No relation between NO2 keys and kafka k...

What about fault-tolerance?
→ Changelog topic (compacted)
→ On application Start
Data on FS → Load from FS
Data not on FS ...

Challenge 1: Key Value Only
17
Topology topology = new Topology()
// read commands from the movie-events topic
.addSource(...

Challenge 1: Key Value Only
18
// get the statestore from the processor context
DocumentStore<String, Movie, ObjectFilter>...

Challenge 2
IQ Querying Limitations

Challenge 2: IQ Querying Limitations
Access data in statestores
Bring-your-own-api
IQv2
IQ Metadata API
Keeps track of whi...

22
GET people?firstname=john
A
API
B
API
n
API

23
GET people?firstname=john
A
API
B
API
n
API
GET
GET

24
GET people?firstname=john
A
API
B
API
n
API
[...]

25
GET people?firstname=john
A
API
B
API
n
API
[...]

26
[...]
A
API
B
API
n
API

Challenge 2: IQ Querying Limitations
Skewed Data
Time Query(B) < Time Query(C)
A cannot return until all results are in
Sh...

28
GET people?firstname=john&offset=50&size=25
A
API
B
API
n
API

29
GET people?firstname=john&offset=50&size=25
A
API
B
API
n
API

30
GET people?firstname=john&offset=50&size=25
A
API
B
API
n
API
[A.results, B.results, …].
sort().subset(50, 25)

Challenge 2: IQ Querying Limitations
Paging
Get parts of the result
Requires sorting for consistent results
Does not scale...

Challenge 3
Cloud Native Issues

33
“Kubernetes is great for many things.
Storage is not one of them.”

Challenge 3: Cloud Native Issues
Rebalances will get you
→ require statestores to be built again from scratch
→ Persistent...

Challenge 3: Cloud Native Issues
Rebalances will get you
→ require statestores to be built again from scratch
→ Persistent...

Conclusion

37
Are these challenges solvable?

38
YES

39
Did you solve all of them?

40
NO

Conclusion
Why?
Doing it right requires time and effort
Not our KOR (pun intended) business
Economical Trade-off
41

Conclusion
APIs can be hosted directly from statestores
→ Your Mileage May Vary
Alternatively
→ Use external storage (Elas...

SHONO
daan@shono.io
Twitter: @daangerits
Github: @calmera

Share Slideshare

LinkedIn
Facebook
Twitter

Embed

Size (px)

Show related Slideshows at end

WordPress Shortcode

Link

Share
Email

2024 Trend Updates: What Really Wor... by Search Engine Jou... 1049688 views
Storytelling For The Web: Integrate... by Chiara Aliotta 964375 views
Artificial Intelligence, Data and C... by OECD Directorate ... 895413 views
How to Leverage AI to Boost Employe... by SocialHRCamp 377996 views
2024 State of Marketing Report – by... by Marius Sescu 214368 views
Everything You Need To Know About C... by Expeed Software 232744 views

View on Slideshare

1 of 43 Ad

View on Slideshare

1 of 43 Ad