This keynote describes the 3 waves of the stream processing, starting from the Lambda architecture to stateful stream processing. We show that the rise of Stateful stream processing, Event-driven architecture, kappa architecture and micro-service architecture lead to rethink the way we can implement data architecture and micro-service architecture.

2. 2
THE PROGRAM COMMITTEE
The brains behind the workshop

3. 3
TWO KEYNOTES
The Workshop content
Victor GAMOV Solution Architect
at CONFLUENT
KSQL On Kafka Stream - Data Management on
Stream 30min
Sabri SKHIRI R&D Director EURA
NOVA
The rise of Stream Processing for data
management & micro service Architecture
15min

4. 4
THE PAPERS
The Workshop content
Stream Mining
Stream Architecture
Credits: https://www.confluent.io/product/confluent-platform/

5. KEYNOTE 1
The rise of Stream Processing for data management &
micro service Architecture
Sabri Skhiri, R&D Director @EURANOVA

6. Architecture evolution
The 3rd wave of Stream Processing & µ services

7. Evolution of Stream Processing
From Lambda to Kappa to microservices
1. WAVE 1: Lambda Architecture: integrates the real time and the batch processing.
Mainly for linear calculations that can be aggregated.
2. WAVE 2: Real time analytics: integrate BAM, real time analytics, stream mining,
etc.
3. WAVE 3: Converged Event Driven Architecture with Stateful Stream processing.
This provides µ services new capabilities for data sharing & consumption with new
architectural patterns.

8. WAVE 1 Lambda architecture
A first step toward Stream processing & data management

9. WAVE 1 Lambda architecture
A first step toward Stream processing & data management
Limitations
1. Only valid for “linear” calculations
that can be aggregated
2. Complexity to manage the serving
layer consistency
3. 1 serving layer per type of job
4. Still a silo wrt to data management
& data architecture (EDW, Data
lake, etc.)

10. WAVE 2 Real-time analytics
Driving value from a stream of event
Stream Mining
CEP & CER
Real-time insights
Cloud of events

11. WAVE 2 Real-time analytics
Driving value from a stream of event
Cloud of events
Stream Mining
CEP & CER
Real-time insights
Limitationsin industrialisation
1. No integration with service organisation:
where it should be implemented & integrated
in the Service Architecture or in the Data
Management Organisation (EDW, Data Lake,
Data lab ?) Data enrichment with referential
data ?
2. Still a silo wrt to data management & data
architecture (EDW, Data lake, etc.)

12. WAVE 3 Stateful stream & µ-services
An evolution of the Event-Driven Architecture
Convergence of 5 areas
1. Event-driven architecture & Event-based design patterns
2. µ-services architecture & design patterns
3. Stateful Stream Processing with state checkpointing & Exactly once semantics
4. Stream/Table duality
5. Kappa architecture & new design patterns for data sharing (cache, views, locks, etc.)
& academic papers
from 2004

13. WAVE 3 Stateful stream & µ-services
An evolution of the Event-Driven Architecture
Convergence of 5 areas
1. Event-driven architecture & Event-based design patterns
2. µ-services architecture & design patterns
3. Stateful Stream Processing with state checkpointing & Exactly once semantics
4. Stream/Table duality
5. Kappa architecture & new design patterns for data sharing (cache, views, locks, etc.)
& academic papers
from 2004

14. STATEFUL STREAM
PROCESSING MATTERS
OPENING NEW DOORS IN DATA MANAGEMENT

15. WAVE 3 Stateful stream & µ-service
Why stateful Stream processing matters ?
Stateful Stream processing allows
1. To manipulate Stream to dynamic table to stream (Stream/Table duality)
2. Exactly once semantics (required for data management)
3. Replay events from last checkpoint

16. Finally, what’s a state and a data ?
Is the database that really matter?
16
Transaction logs record all the changes made to the database. […].
From this perspective, the contents of the database hold a caching
of the latest record values in the logs. The truth is the log. The
database is a cache of a subset of the log.
The Cat enter in
the
Schrodinger's
box
Transaction log The DB

17. DATA & TABLE DUALITY
TWO ASPECTS OF THE SAME REALITY
17
Arasu and al., STREAM: The Stanford Data Stream
Management System. 2004
This is possible today with the
modern distributed Stream
Processor allowing STATEFUL
PROCESSING
(Required for Continuous query and
maintaining Dynamic tables).

18. WAVE 3 Stateful stream & µ-service
The new wave of architecture
1
2 3
4
5
We can use these patterns in
1. DATA ARCHITECTURE
2. SERVICE ARCHITECTURE

19. WAVE 3 Stateful stream & µ-service
An evolution of the Event-Driven Architecture back in 2008 with ED-SOA
EVENT Collector
App 1 App2 App3
CEP
A typical EDA
1. The service communicate via Event Request/response
2. The service are stateless and can scale by listening events on the bus
3. A CEP track event correlations for e2e response time, context-awareness, etc.
4. Enables CQRS like pattern by rebuilding state with event listening
5. A special service must listen all events on the bus (Event sourcing pattern)

20. WAVE 3 Stateful stream & µ-service
An evolution of the Event-Driven Architecture back in 2008 with ED-SOA
EVENT Collector
App 1 App2 App3
CEP
A typical EDA
1. The service communicate via Event Request/response
2. The service are stateless and can scale by listening events on the bus
3. A CEP track event correlations for e2e response time, context-awareness, etc.
4. Enables CQRS like pattern by rebuilding state with event listening
5. A special service must listen all events on the bus (Event sourcing pattern)
ETL

21. WAVE 3 Stateful stream & µ-service
The new wave of architecture
EVENT Collector
µ-service 1 µ-service 2 µ-service 3
CEP
A typical EDA
1. Services are State-machine based, stream analytics or stateful stream processing
service
2. The data integration between services is made through kappa architecture & CQRS for
materialized derived view
3. Scalable architecture, works for batch &/or RT
Stream Analytic
app
Statefull Stream
Processing
µ-service

22. That’s one of the 8 service patterns
Described by Confluent
22
Ben Stopford, Putting the Micro into Microservices with Stateful
Stream Processing. April 2017
https://fr.slideshare.net/ConfluentInc/putting-the-micro-into-micro
services-with-stateful-stream-processing
Today a µ-service can be written
as stream processing application !
State Management
Asynch
Data sharing

23. Summary
Key takeaways

24. 24
CONCLUSION
Key takeaways
1. STREAM TECHNOLOGIES ARE EVOLVING & STATEFUL STREAMING OPENS NEW
DOORS
2. DATA CAN BE SHARED THROUGH STREAMS APPLYING KAPPA ARCHITECTURE
PATTERNS EITHER FOR APPLICATIONS OR DATA MANAGEMENT/ARCHITECTURE
3. EVOLUTION OF THE MICROSERVICES TOWARDS EDA AND THEIR NEEDS IN TERM
OF SCALABLE DATA SHARING MAKE STATEFUL STREAM PROCESSING THE
PERFECT TARGET TO IMPLEMENT µ-SERVICE ARCHITECTURES

25. @sskhiri
@euranova
euranova.eu
research.euranova.eu
CONTACT