Stream Based Architecture

@_tommichiels_#Devoxx #Streambased
Stream Based Architecture
punching and crunching
tom.michiels@archimiddle.com

During this talk I will try to convince you that we need to

evolve to stream based architectures

Setting the stage
Our actors
Punchers
Crunchers

Birth, First Encounter,The LoveStory

Crunchers life

1950 1970 1990
Machine Code
Imperative
Crunchers life
Declarative

Punchers live

Data models
Flat
Hierarchical
Network
RelationalNavigational

Normalisation
Isolate data so that additions, deletions, and modiﬁcations can be made in just
one table and then propagated through the rest of the
database using the deﬁned foreign keys

1950 1970 1990
Flat
Navigational
Punchers life
Relational

The marriage, thx Edgar

Normalisation
Wedding vows
SQL +

Crunch(calculate)
Punch(store)
Logic
One happy family

And then it started to get complicate
Performance
Complexity

Performance
Concurrent users/operations

Performance

Complexity
(Re)act to changing world

Complexity
Historical reasons

Complexity
Legacy

Puncher problems
Performance
•Persistence / IO ?
•Model ?

Cruncher problems
Complexity
•Abstractions ?
•Reuse?

Crunch(calculate)
Punch(store)
Logic
Scaling up(centralised)

Crunch(calculate)
Punch(store)
Logic
Scaling out(distributed)

Propagating Through
•File
•DB
•RPC
•Message

Centralised vs distributed
Performance
Complexity
Distributed
Centralised

Layers
Crunch(calculate)
Punch(store)
Logic
Punch(store)
UI

2 vs N tier
Performance
Complexity
Distributed
Centralised
2 tier N tier

DataDriven
Performance
Complexity
Distributed
Centralised
DataDriven
2 tier N tier

Query
OLTP
ETL
Extract
Transform
Load
OLAP

Abstraction
Reuse
Crunchers midlife crisis

1950 1970 1990
Machine Code
Procedural
Object Oriented
Crunchers midlife crisis

AppDriven
Performance
Complexity
Distributed
Centralised
AppDriven
2 tier N tier

ORM Hell
Impedance Mismatch

Monolith Hell

Spaghetti integration

Service Oriented
Performance
Complexity
Distributed
Centralised
Service Oriented
2 tier N tier

Satellite Services
client
Service interface
Data aggregates
O/R mapping
RDBMS

Punchers midlife crisis
©

•Partitions
•Eventual Consistence
First Class Citizens

MicroServices
client
NOSql
Service

What about reports?

DataLake

Distributed Data Driven
Performance
Complexity
Distributed
Centralised
2 tier N tier
Distributed Data 
Driven

Distributed Data Driven
Performance
Complexity
Distributed
Centralised
2 tier N tier
Distributed Data 
Driven
DataDriven
AppDriven
Service Oriented

How did we end up here!!!!

How did we end up here!!!!
Shared DB
Messaging
RPC
File

The Divorce(ETL)

Remember our wedding vows

Get us out of this mess!

Immutable
an immutable object is an object whose state
cannot be modiﬁed after it is created
10

Stream
10 10 - 10 108vs
is a (ordered) sequence of immutable objects

Stream Computing
10 + 10+ (- 10) = 10
data
state is derivedintent

Back to Maths
•Associativity
•Commutative

Back to Maths
10 + 10+ (- 10) = 10
Type
Binary Operation

Back to Maths
A ∪ B
Type
Binary Operation

Birth of a commutative monoid
(10 + 10)+ (- 10) = ((-10) +10) + 10
Calculation can be partitioned
How we combine intermediate results is not important

Programming with solid base in maths

Examples of streams
•Logﬁles
•Journals/Ledgers
•Atom Feeds

Propagating New Style
•Make stream available
•Consumers keep track
of consumed events
•Producers Schema on
Write

Wedding vows renewed
Propagate immutable events by using streams
and use stream crunching as local as possible.

Wedding vows renewed
Not Only ETL

Streams on Micro(Service) Level
•EventSourcing
•CQRS

Streams on Micro Level

Streams on Macro level

Streams on Macro level
kappa architecture

Reports vs Dashboards

Fast Data

• BloomFilter
• HyperLogLog
• Count-MinSketch
• MinHash
Stream processing

Smack Stack
https://github.com/killrweather/killrweather

Happily Ever After

Stream Based Architecture

More Related Content

Similar to Stream Based Architecture

Recently uploaded

Stream Based Architecture