Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Keynote 1 the rise of stream processing for data management & micro service architecture 2017

149 views

Published on

This keynote describes the 3 waves of the stream processing, starting from the Lambda architecture to stateful stream processing. We show that the rise of Stateful stream processing, Event-driven architecture, kappa architecture and micro-service architecture lead to rethink the way we can implement data architecture and micro-service architecture.

Published in: Technology
  • Be the first to comment

Keynote 1 the rise of stream processing for data management & micro service architecture 2017

  1. 1. 2 THE PROGRAM COMMITTEE The brains behind the workshop
  2. 2. 3 TWO KEYNOTES The Workshop content Victor GAMOV Solution Architect at CONFLUENT KSQL On Kafka Stream - Data Management on Stream 30min Sabri SKHIRI R&D Director EURA NOVA The rise of Stream Processing for data management & micro service Architecture 15min
  3. 3. 4 THE PAPERS The Workshop content Stream Mining Stream Architecture Credits: https://www.confluent.io/product/confluent-platform/
  4. 4. KEYNOTE 1 The rise of Stream Processing for data management & micro service Architecture Sabri Skhiri, R&D Director @EURANOVA
  5. 5. Architecture evolution The 3rd wave of Stream Processing & µ services
  6. 6. Evolution of Stream Processing From Lambda to Kappa to microservices 1. WAVE 1: Lambda Architecture: integrates the real time and the batch processing. Mainly for linear calculations that can be aggregated. 2. WAVE 2: Real time analytics: integrate BAM, real time analytics, stream mining, etc. 3. WAVE 3: Converged Event Driven Architecture with Stateful Stream processing. This provides µ services new capabilities for data sharing & consumption with new architectural patterns.
  7. 7. WAVE 1 Lambda architecture A first step toward Stream processing & data management
  8. 8. WAVE 1 Lambda architecture A first step toward Stream processing & data management Limitations 1. Only valid for “linear” calculations that can be aggregated 2. Complexity to manage the serving layer consistency 3. 1 serving layer per type of job 4. Still a silo wrt to data management & data architecture (EDW, Data lake, etc.)
  9. 9. WAVE 2 Real-time analytics Driving value from a stream of event Stream Mining CEP & CER Real-time insights Cloud of events
  10. 10. WAVE 2 Real-time analytics Driving value from a stream of event Cloud of events Stream Mining CEP & CER Real-time insights Limitationsin industrialisation 1. No integration with service organisation: where it should be implemented & integrated in the Service Architecture or in the Data Management Organisation (EDW, Data Lake, Data lab ?) Data enrichment with referential data ? 2. Still a silo wrt to data management & data architecture (EDW, Data lake, etc.)
  11. 11. WAVE 3 Stateful stream & µ-services An evolution of the Event-Driven Architecture Convergence of 5 areas 1. Event-driven architecture & Event-based design patterns 2. µ-services architecture & design patterns 3. Stateful Stream Processing with state checkpointing & Exactly once semantics 4. Stream/Table duality 5. Kappa architecture & new design patterns for data sharing (cache, views, locks, etc.) & academic papers from 2004
  12. 12. WAVE 3 Stateful stream & µ-services An evolution of the Event-Driven Architecture Convergence of 5 areas 1. Event-driven architecture & Event-based design patterns 2. µ-services architecture & design patterns 3. Stateful Stream Processing with state checkpointing & Exactly once semantics 4. Stream/Table duality 5. Kappa architecture & new design patterns for data sharing (cache, views, locks, etc.) & academic papers from 2004
  13. 13. STATEFUL STREAM PROCESSING MATTERS OPENING NEW DOORS IN DATA MANAGEMENT
  14. 14. WAVE 3 Stateful stream & µ-service Why stateful Stream processing matters ? Stateful Stream processing allows 1. To manipulate Stream to dynamic table to stream (Stream/Table duality) 2. Exactly once semantics (required for data management) 3. Replay events from last checkpoint
  15. 15. Finally, what’s a state and a data ? Is the database that really matter? 16 Transaction logs record all the changes made to the database. […]. From this perspective, the contents of the database hold a caching of the latest record values in the logs. The truth is the log. The database is a cache of a subset of the log. The Cat enter in the Schrodinger's box Transaction log The DB
  16. 16. DATA & TABLE DUALITY TWO ASPECTS OF THE SAME REALITY 17 Arasu and al., STREAM: The Stanford Data Stream Management System. 2004 This is possible today with the modern distributed Stream Processor allowing STATEFUL PROCESSING (Required for Continuous query and maintaining Dynamic tables).
  17. 17. WAVE 3 Stateful stream & µ-service The new wave of architecture 1 2 3 4 5 We can use these patterns in 1. DATA ARCHITECTURE 2. SERVICE ARCHITECTURE
  18. 18. WAVE 3 Stateful stream & µ-service An evolution of the Event-Driven Architecture back in 2008 with ED-SOA EVENT Collector App 1 App2 App3 CEP A typical EDA 1. The service communicate via Event Request/response 2. The service are stateless and can scale by listening events on the bus 3. A CEP track event correlations for e2e response time, context-awareness, etc. 4. Enables CQRS like pattern by rebuilding state with event listening 5. A special service must listen all events on the bus (Event sourcing pattern)
  19. 19. WAVE 3 Stateful stream & µ-service An evolution of the Event-Driven Architecture back in 2008 with ED-SOA EVENT Collector App 1 App2 App3 CEP A typical EDA 1. The service communicate via Event Request/response 2. The service are stateless and can scale by listening events on the bus 3. A CEP track event correlations for e2e response time, context-awareness, etc. 4. Enables CQRS like pattern by rebuilding state with event listening 5. A special service must listen all events on the bus (Event sourcing pattern) ETL
  20. 20. WAVE 3 Stateful stream & µ-service The new wave of architecture EVENT Collector µ-service 1 µ-service 2 µ-service 3 CEP A typical EDA 1. Services are State-machine based, stream analytics or stateful stream processing service 2. The data integration between services is made through kappa architecture & CQRS for materialized derived view 3. Scalable architecture, works for batch &/or RT Stream Analytic app Statefull Stream Processing µ-service
  21. 21. That’s one of the 8 service patterns Described by Confluent 22 Ben Stopford, Putting the Micro into Microservices with Stateful Stream Processing. April 2017 https://fr.slideshare.net/ConfluentInc/putting-the-micro-into-micro services-with-stateful-stream-processing Today a µ-service can be written as stream processing application ! State Management Asynch Data sharing
  22. 22. Summary Key takeaways
  23. 23. 24 CONCLUSION Key takeaways 1. STREAM TECHNOLOGIES ARE EVOLVING & STATEFUL STREAMING OPENS NEW DOORS 2. DATA CAN BE SHARED THROUGH STREAMS APPLYING KAPPA ARCHITECTURE PATTERNS EITHER FOR APPLICATIONS OR DATA MANAGEMENT/ARCHITECTURE 3. EVOLUTION OF THE MICROSERVICES TOWARDS EDA AND THEIR NEEDS IN TERM OF SCALABLE DATA SHARING MAKE STATEFUL STREAM PROCESSING THE PERFECT TARGET TO IMPLEMENT µ-SERVICE ARCHITECTURES
  24. 24. @sskhiri @euranova euranova.eu research.euranova.eu CONTACT

×