Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"

551 views

Published on

Two of the main software architectural trends in software development this decade has been the move to streaming data processing, and the move to microservice architecture.

Both of these architectures are driven by the needs of managing and mining knowledge from ever increasing volumes of data in a close to real-time fashion—all while being reactive: responsive under failure, and responsive under load. I'm here to tell you that these two trends are converging, and a fusion of the two is both logical and inevitable. In this session we will talk about what a fused approach to stream processing and microservices could look like, what opportunities exist—what software development for business software can look like in the following decade.

Published in: Technology
  • Be the first to comment

Flink Forward Berlin 2018: Viktor Klang - Keynote "The convergence of stream processing and microservice architecture"

  1. 1. The convergence of stream processing & microservice architecture √Deputy CTO, Lightbend @viktorklang / v@lightbend.com / viktorklang.com
  2. 2. Overview
  3. 3. History of Data Processing 4 Offline Batch processing 4 MapReduce 4 Lambda Architecture
  4. 4. History of Services 4 Client-Server 4 CORBA & DCOM 4 EJB & N-Tier 4 SOA & ESB 4 HTTP-API & REST
  5. 5. Today
  6. 6. Catch of the day Live1 Stream Processing & Microservice Architecture 1  Near real-time
  7. 7. Shared requirements: 4 Available 4 Elastically scalable 4 Resilient & self-healing 4 Loosely coupled & upgradeable
  8. 8. Reactive Systems2 Available (responsive), under failure (resilience) and under load (elasticity), by being message-driven. 2  https://www.reactivemanifesto.org/
  9. 9. Reactive Streams3 Provides a standard for asynchronous stream processing with non-blocking back pressure. 3  www.reactive-streams.com & java.util.concurrent.Flow (JDK9+)
  10. 10. Ingredients 4 Stream processing stages: Stateless & Stateful 4 Microservices: RPC-style & Evented 4 Domain Events with Schema/Codec
  11. 11. #define Events4 4 Immutable recordings of observations 4 Can be disregarded but not be retracted once accepted 4 Can be invalidated but (typically) not be deleted 4 Knowledge accrues with every processed event 4  Learn more: https://speakerdeck.com/jboner/designing-events-first-microservices
  12. 12. Services (evented)
  13. 13. Realizationevented services are stream processing stages
  14. 14. The near future: Convergence
  15. 15. Is this the solution? 4 Alice: So we we lift evented services into the stream processing pipeline … 4 Bob: … and then we're done! 4 <record scratch> 4 Narrator: They're forgetting the request-reply use- case which RPC services excel at.
  16. 16. Services (rpc)
  17. 17. The improved solution 4 lift the service into a stream stage 4 lift input Events into Commands and submit 4 capture responses & convert into output Events
  18. 18. What if we could derive microservices from data streams?
  19. 19. Ingestion endpoints
  20. 20. Simple Projection endpoints
  21. 21. Complex Projection endpoints
  22. 22. Unification 4 Microservices are a part of a streaming pipeline 4 A pipeline can now be exposed as Microservices 4 We can independently upgrade parts of the pipeline
  23. 23. Developing converged applications 4 Deployment Orchestration 4 Events-First DDD & EventStorming 4 (Meta)Data-driven Continuous Deployment 4 Use of the right tools for each component
  24. 24. Beyond
  25. 25. Extend to the Edges
  26. 26. End of stream
  27. 27. Explore what we do at Lightbend: 4 Lightbend Fast Data Platform 4 https://www.lightbend.com/products/fast-data- platform 4 Lightbend Reactive Platform 4 https://www.lightbend.com/products/reactive-platform 4 Lightbend Enterprise Suite 4 https://www.lightbend.com/products/enterprise-suite
  28. 28. Thank you for attending! The convergence of stream processing & microservice architecture √ Deputy CTO, Lightbend @viktorklang / v@lightbend.com / viktorklang.com

×