VMware Tanzu, profile picture
Uploaded byVMware Tanzu
814 views

Cloud-Native Streaming and Event-Driven Microservices

This document discusses cloud-native streaming and event-driven microservices, emphasizing their advantages in processing large amounts of data rapidly and efficiently using frameworks like Spring Cloud Stream. It outlines the operational complexities associated with distributed systems and advocates for microservices to enhance development agility and failure isolation. Additionally, it details the capabilities of Spring Cloud Data Flow for orchestration and deployment of stream topologies.

Embed presentation

Downloaded 31 times
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud Native Streaming and Event-Driven Microservices Marius Bogoevici, Pivotal @mariusbogoevici
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Who am I ? • Software Engineer with Pivotal • Spring ecosystem contributor since 2008: – Spring Integration, Spring XD, Spring Integration Kafka, – Spring Cloud Stream, Spring Cloud Data Flow • Co-author, “Spring Integration in Action”, Manning, 2012
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Streaming • Processing large quantities of data as fast as possible (near real-time) • Use cases: • real-time data movement (ETL) • predictive maintenance, fraud detection, IoS, QoS • enterprise integration meets Big Data • High throughput/low latency • Specific operations: grouping, ordering, windowing • Transition in message middleware design: • Traditional (JMS-based, RabbitMQ) • Log-style (Kafka, Amazon Kinesis, Azure Event Hubs) • Cloud-native: Google Pub Sub
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why microservices for data processing? • Cohesiveness around business capability (“do one thing and do it well”) • Organizational alignment (Conway’s Law), cross-team collaboration • Development agility • Optimized for replacement • Enable continuous delivery • Failure isolation • Granular resource tuning: • scaling out the critical parts of the pipeline • per-process: memory, CPU, network 4
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Event-Driven/Messaging Microservices • Messaging via broker vs HTTP • Solving communication complexity for data processing • Decoupling: • Physical: discovery • Temporal: availability • Eventual consistency vs. shared stores/distributed transactions • especially over heterogenous resources • Pub-sub makes it easy to add new elements to the topology
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data and Event Streaming: Conceptually Similar Data Streaming: ingestion, analytics Async interaction, event sourcing
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How about operational complexity? • Distributed systems are inherently complex, operating them even more so • Operational prerequisites: • Self-servicing and provisioning • elastic infrastructure • Monitoring • Self-healing • Rapid delivery • CI/CD, deployment pipeline • automation • a DevOps culture is required https://martinfowler.com/bliki/MicroservicePrerequisites.html 7
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 8
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud native applications and platforms • Target a platform that makes running apps reliable, transparent and boring • In-built resource management • Memory, CPU, networking • Elastic scaling • Monitoring and failover • Health, logging, metrics • Routing and load balancing • Rolling upgrades 9 Apache YARN Apache Mesos Kubernetes
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Monolith, the Platform and the Microservice(s) 10 Spring Cloud Stream2015 Spring XD Spring Cloud Data Flow Spring Cloud Task
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 11 file jms http Messaging middleware Platform cassandra count-words Cloud-native event-driven microservices Apache Kafka
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream • Event-driven microservices framework • Goal: simplify writing messaging applications • Bias towards event/data streaming • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Opinionated primitives • Pluggable middleware abstractions 12
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream in a nutshell 13 Application Core Spring Integration Spring Messaging Reactive APIs Kafka Streams inputs outputs
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 14
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 15
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Binder Abstraction 16 Application Messaging Middleware Binderchannel can be input or output adapts channel to target middleware created by framework configured via Spring Boot properties
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 17 @EnableBinding + Binder Implementation Apache Kafka JMS Google PubSub Production-ready: Experimental
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 18 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(“input”)
 @SendTo(“output”)
 public String process(String s) {
 return s.toUpperCase();
 }
 }
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model - imperative/procedural 19 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(“input”)
 @SendTo(“output”)
 public String process(String s) {
 return s.toUpperCase();
 }
 }
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reactive Programming via Reactor (since 1.1) 20 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener
 @Output(“output”)
 public Flux<String> process(@Input(“input”) Flux<String> s) {
 return f.map(s -> s.toUppercase()); }
 }
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Stateful operators: grouping, windowing 21 SpringBootApplication
 @EnableBinding(Processor.class)
 public class WordCountApplication {
 
 @StreamListener
 @Output("output")
 public Flux<?> countWords(@Input("input") Flux<String> words) {
 return words.window(ofSeconds(5), ofSeconds(1))
 .flatMap(window -> window.groupBy(word -> word)
 .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1)
 .map(count -> new WordCount(group.key(), count))));
 } }
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ [dumb] Pipes and [smart] filters 22 http averages topN ingest data average value over last 5s hottest sensors over last 10 seconds
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 23 http raw-sensor-data averages averages top-n Calculator
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 24 http raw-sensor-data averages averages top-n Calculator Failure detector in Spring Cloud Stream, all destinations are pub-sub
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 25
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 26 raw-sensor-data averages HDFS http pubsub is great for interconnecting logical apps what if we add more instances ?
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 27 raw-sensor-data averages HDFS http we want these two to compete averages HDFS
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 27 raw-sensor-data averages HDFS http we want these two to compete averages HDFS and these two !
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 28 Averages HDDSHDFS HDFS raw-sensor-data groups are in a pub-sub relationship to each other AveragesAverages consumers are competing within a group
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 29
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 30 Average calculator Average calculator http raw-sensor-data
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 31 Average calculator Average calculator http partition 1 partition 2 raw-sensor-data
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 32 Average calculator Average calculator source partition 1 partition 2 raw-sensor-data Spring Cloud Stream: Partitioning
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Simple topologies: [relatively] easy to deploy … 33 http hdfs spring.cloud.stream.bindings.output.destination=httphdfs.1 spring.cloud.stream.bindings.input.destination=httphdfs.1 spring.cloud.stream.bindings.input.group=httphdfs httphdfs.1
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ … but how about complex topologies ? 34 http raw-sensor-data averages top-n Calculator Failure detector averagesaverages HDFSHDFSHDFS
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow • Orchestration: • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use-cases 35
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow - Stream DSL 36 Stream definition Spring Boot Apps built with Spring Cloud Stream httpfile = http | file |
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 38 cassandra gpfdist http stream1 = http | count-words | file stream2 = jms | cassandra Kafka Data Flow Server DB jms file jms http Message System Platform cassandra count-words
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http LoadBalancer
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work LoadBalancer
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer stream create s1 --definition “http | work | hdfs” stream deploy s1 --propertiesFile ingest.properties
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer stream create s1 --definition “http | work | hdfs” stream deploy s1 --propertiesFile ingest.properties app.http.count=2 app.work.count=3 app.hdfs.count=4 app.http.producer.partitionKeyExpression=payload.id
Unless otherwise indicated, these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Resource Management 40 http http work work work app.work.spring.cloud.deployer.cloudfoundry.memory=2048
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Some links … http://cloud.spring.io/spring-cloud-stream http://cloud.spring.io/spring-cloud-dataflow https://github.com/spring-cloud/spring-cloud-stream-samples
Unless otherwise indicated, these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Questions ?

More Related Content

PDF
TDD for Microservices
byVMware Tanzu
 
PDF
Microservices pattern language (microxchg microxchg2016)
byChris Richardson
 
PPTX
What's new in Spring Boot 2.0
byVMware Tanzu
 
PDF
4Developers 2015: Do you think you're doing microservice architecture? - Marc...
byPROIDEA
 
PDF
Serverless Spring
byVMware Tanzu
 
PDF
Cloud Native Java with Spring Cloud Services
byChris Sterling
 
PDF
Walking Through Spring Cloud Data Flow
byVMware Tanzu
 
PDF
Spring IO 2016 - Spring Cloud Microservices, a journey inside a financial entity
byToni Jara
 
TDD for Microservices
byVMware Tanzu
 
Microservices pattern language (microxchg microxchg2016)
byChris Richardson
 
What's new in Spring Boot 2.0
byVMware Tanzu
 
4Developers 2015: Do you think you're doing microservice architecture? - Marc...
byPROIDEA
 
Serverless Spring
byVMware Tanzu
 
Cloud Native Java with Spring Cloud Services
byChris Sterling
 
Walking Through Spring Cloud Data Flow
byVMware Tanzu
 
Spring IO 2016 - Spring Cloud Microservices, a journey inside a financial entity
byToni Jara
 

What's hot

PPTX
If Hemingway Wrote JavaDocs
byVMware Tanzu
 
PPTX
Core Spring + Reactive 김민석
byVMware Tanzu Korea
 
PDF
Going Serverless Using the Spring Framework Ecosystem
byVMware Tanzu
 
PDF
Java Microservices with Spring Boot and Spring Cloud - Denver JUG 2019
byMatt Raible
 
PDF
Ingress? That’s So 2020! Introducing the Kubernetes Gateway API
byVMware Tanzu
 
PPTX
Spring Framework 5.0: Hidden Gems
byVMware Tanzu
 
PDF
Introduction to Spring Cloud
byVMware Tanzu
 
PDF
Welcome to the Metrics
byVMware Tanzu
 
PPTX
Bulletproof Microservices with Spring and Kubernetes
byVMware Tanzu
 
PDF
What Is Spring?
byVMware Tanzu
 
PDF
Peering Inside the Black Box: A Case for Observability
byVMware Tanzu
 
PDF
Spring Boot Loves K8s
byVMware Tanzu
 
PDF
Resilient and Adaptable Systems with Cloud Native APIs
byVMware Tanzu
 
PDF
Full Steam Ahead, R2DBC!
byVMware Tanzu
 
PDF
Not Just Initializing
byVMware Tanzu
 
PDF
SpringOne Platform recap 정윤진
byVMware Tanzu Korea
 
PPTX
Developer Week - Reacting to an event-driven world
byGrace Jansen
 
PDF
Full-Stack Development with Spring Boot and VueJS
byVMware Tanzu
 
PPTX
State of Steeltoe 2020
byVMware Tanzu
 
PDF
Lock That Shit Down! Auth Security Patterns for Apps, APIs, and Infra - Sprin...
byMatt Raible
 
If Hemingway Wrote JavaDocs
byVMware Tanzu
 
Core Spring + Reactive 김민석
byVMware Tanzu Korea
 
Going Serverless Using the Spring Framework Ecosystem
byVMware Tanzu
 
Java Microservices with Spring Boot and Spring Cloud - Denver JUG 2019
byMatt Raible
 
Ingress? That’s So 2020! Introducing the Kubernetes Gateway API
byVMware Tanzu
 
Spring Framework 5.0: Hidden Gems
byVMware Tanzu
 
Introduction to Spring Cloud
byVMware Tanzu
 
Welcome to the Metrics
byVMware Tanzu
 
Bulletproof Microservices with Spring and Kubernetes
byVMware Tanzu
 
What Is Spring?
byVMware Tanzu
 
Peering Inside the Black Box: A Case for Observability
byVMware Tanzu
 
Spring Boot Loves K8s
byVMware Tanzu
 
Resilient and Adaptable Systems with Cloud Native APIs
byVMware Tanzu
 
Full Steam Ahead, R2DBC!
byVMware Tanzu
 
Not Just Initializing
byVMware Tanzu
 
SpringOne Platform recap 정윤진
byVMware Tanzu Korea
 
Developer Week - Reacting to an event-driven world
byGrace Jansen
 
Full-Stack Development with Spring Boot and VueJS
byVMware Tanzu
 
State of Steeltoe 2020
byVMware Tanzu
 
Lock That Shit Down! Auth Security Patterns for Apps, APIs, and Infra - Sprin...
byMatt Raible
 

Similar to Cloud-Native Streaming and Event-Driven Microservices

PDF
Cloud Event Driven Architectures with Spring Cloud Stream 2.0 - Oleg Zhurakousky
byVMware Tanzu
 
PDF
Dataservices - Processing Big Data The Microservice Way
byJosef Adersberger
 
PDF
Spring Cloud Stream: What's New in 2.x—and What's Next?
byVMware Tanzu
 
PDF
Dataservices: Processing (Big) Data the Microservice Way
byQAware GmbH
 
PDF
Pivoting Spring XD to Spring Cloud Data Flow with Sabby Anandan
byPivotalOpenSourceHub
 
PPTX
Cloud Event Driven Architectures with Spring Cloud Stream 2.0
byVMware Tanzu
 
PPTX
Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...
byconfluent
 
PDF
Stream Processing in the Cloud With Data Microservices
bymarius_bogoevici
 
PDF
Building a Data Exchange with Spring Cloud Data Flow
byVMware Tanzu
 
PPTX
Using Event Streams in Serverless Applications
byJonathan Dee
 
PDF
Event Driven Streaming Analytics - Demostration on Architecture of IoT
byLei Xu
 
PDF
Pragmatic Event Driven Microservices
byAllard Buijze
 
PPTX
Event Bus as Backbone for Decoupled Microservice Choreography (JFall 2017)
byLucas Jellema
 
PPTX
Event Bus as Backbone for Decoupled Microservice Choreography (Oracle Code, A...
byLucas Jellema
 
PPTX
Event Bus as Backbone for Decoupled Microservice Choreography - Lecture and W...
byLucas Jellema
 
PDF
Patterns of Distributed Application Design
byGlobalLogic Ukraine
 
PPTX
Patterns of Distributed Application Design
byOrkhan Gasimov
 
PPTX
Sweet Streams (Are made of this)
byCorneil du Plessis
 
PDF
Reactive Microservices with Spring 5: WebFlux
byTrayan Iliev
 
PPT
The Next Generation Application Server – How Event Based Processing yields s...
byGuy Korland
 
Cloud Event Driven Architectures with Spring Cloud Stream 2.0 - Oleg Zhurakousky
byVMware Tanzu
 
Dataservices - Processing Big Data The Microservice Way
byJosef Adersberger
 
Spring Cloud Stream: What's New in 2.x—and What's Next?
byVMware Tanzu
 
Dataservices: Processing (Big) Data the Microservice Way
byQAware GmbH
 
Pivoting Spring XD to Spring Cloud Data Flow with Sabby Anandan
byPivotalOpenSourceHub
 
Cloud Event Driven Architectures with Spring Cloud Stream 2.0
byVMware Tanzu
 
Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...
byconfluent
 
Stream Processing in the Cloud With Data Microservices
bymarius_bogoevici
 
Building a Data Exchange with Spring Cloud Data Flow
byVMware Tanzu
 
Using Event Streams in Serverless Applications
byJonathan Dee
 
Event Driven Streaming Analytics - Demostration on Architecture of IoT
byLei Xu
 
Pragmatic Event Driven Microservices
byAllard Buijze
 
Event Bus as Backbone for Decoupled Microservice Choreography (JFall 2017)
byLucas Jellema
 
Event Bus as Backbone for Decoupled Microservice Choreography (Oracle Code, A...
byLucas Jellema
 
Event Bus as Backbone for Decoupled Microservice Choreography - Lecture and W...
byLucas Jellema
 
Patterns of Distributed Application Design
byGlobalLogic Ukraine
 
Patterns of Distributed Application Design
byOrkhan Gasimov
 
Sweet Streams (Are made of this)
byCorneil du Plessis
 
Reactive Microservices with Spring 5: WebFlux
byTrayan Iliev
 
The Next Generation Application Server – How Event Based Processing yields s...
byGuy Korland
 

More from VMware Tanzu

PDF
Spring into AI presented by Dan Vega 5/14
byVMware Tanzu
 
PDF
What AI Means For Your Product Strategy And What To Do About It
byVMware Tanzu
 
PDF
Make the Right Thing the Obvious Thing at Cardinal Health 2023
byVMware Tanzu
 
PPTX
Enhancing DevEx and Simplifying Operations at Scale
byVMware Tanzu
 
PDF
Spring Update | July 2023
byVMware Tanzu
 
PPTX
Platforms, Platform Engineering, & Platform as a Product
byVMware Tanzu
 
PPTX
Building Cloud Ready Apps
byVMware Tanzu
 
PDF
Spring Boot 3 And Beyond
byVMware Tanzu
 
PDF
Spring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdf
byVMware Tanzu
 
PDF
Simplify and Scale Enterprise Apps in the Cloud | Boston 2023
byVMware Tanzu
 
PDF
Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023
byVMware Tanzu
 
PPTX
tanzu_developer_connect.pptx
byVMware Tanzu
 
PDF
Tanzu Virtual Developer Connect Workshop - French
byVMware Tanzu
 
PDF
Tanzu Developer Connect Workshop - English
byVMware Tanzu
 
PDF
Virtual Developer Connect Workshop - English
byVMware Tanzu
 
PDF
Tanzu Developer Connect - French
byVMware Tanzu
 
PDF
Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023
byVMware Tanzu
 
PDF
SpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring Boot
byVMware Tanzu
 
PDF
SpringOne Tour: The Influential Software Engineer
byVMware Tanzu
 
PDF
SpringOne Tour: Domain-Driven Design: Theory vs Practice
byVMware Tanzu
 
Spring into AI presented by Dan Vega 5/14
byVMware Tanzu
 
What AI Means For Your Product Strategy And What To Do About It
byVMware Tanzu
 
Make the Right Thing the Obvious Thing at Cardinal Health 2023
byVMware Tanzu
 
Enhancing DevEx and Simplifying Operations at Scale
byVMware Tanzu
 
Spring Update | July 2023
byVMware Tanzu
 
Platforms, Platform Engineering, & Platform as a Product
byVMware Tanzu
 
Building Cloud Ready Apps
byVMware Tanzu
 
Spring Boot 3 And Beyond
byVMware Tanzu
 
Spring Cloud Gateway - SpringOne Tour 2023 Charles Schwab.pdf
byVMware Tanzu
 
Simplify and Scale Enterprise Apps in the Cloud | Boston 2023
byVMware Tanzu
 
Simplify and Scale Enterprise Apps in the Cloud | Seattle 2023
byVMware Tanzu
 
tanzu_developer_connect.pptx
byVMware Tanzu
 
Tanzu Virtual Developer Connect Workshop - French
byVMware Tanzu
 
Tanzu Developer Connect Workshop - English
byVMware Tanzu
 
Virtual Developer Connect Workshop - English
byVMware Tanzu
 
Tanzu Developer Connect - French
byVMware Tanzu
 
Simplify and Scale Enterprise Apps in the Cloud | Dallas 2023
byVMware Tanzu
 
SpringOne Tour: Deliver 15-Factor Applications on Kubernetes with Spring Boot
byVMware Tanzu
 
SpringOne Tour: The Influential Software Engineer
byVMware Tanzu
 
SpringOne Tour: Domain-Driven Design: Theory vs Practice
byVMware Tanzu
 

Recently uploaded

PPTX
AI Clinic Management Software for Pulmonology Clinics Bringing Clarity, Contr...
byEasy Clinic
 
PDF
Revolutionising-SQL-Data-Modelling-with-SqlDBM.pdf
bySQL DBM
 
PDF
Solved First Semester B.C.A. C Programming Question Paper – Jan/Feb 2025
byKuvempu University
 
PDF
From Experiment to Enterprise: Scaling AI Coding Assistants Across Engineerin...
byMaxim Salnikov
 
PDF
One Agent to Rule Them All - The Fellowship of AI Agents
byIvo Andreev
 
PPTX
Fast-tracking quality for hundreds applications with automated testing layers
byBogdan Plieshka
 
PDF
Cloud-Based Underwriting Software for Insurance
byInsurance Tech Services
 
PPTX
Computer_Virus_Presentation_With_Slide7.pptx
byHEVISLIVE
 
PPTX
NSF Converter Software to Convert NSF to PST, EML, MSG
byNSF Converter Software
 
PPTX
Week 7 Introduction to HTML PowerPoint Notes.pptx
bylesandawelgens
 
PPTX
Custom chat gpt integration services.pptx
byChetu
 
PDF
DSD-INT 2025 Stochastic flood event generation using wflow and a diffusion do...
byDeltares
 
PDF
Virtual Study Circles Innovative Ways to Collaborate Online.pdf
byExplain Learning
 
PDF
Design and Analysis of Algorithms(DAA): Unit-II Asymptotic Notations and Basi...
byKuvempu University
 
PDF
DSD-INT 2025 Latest Developments for HydroMT and wflow - Meshgi
byDeltares
 
PDF
BCS-FACS Peter Landin Semantics Seminar - Formal Methods: Whence and Whither?
byJonathan Bowen
 
PPTX
SNG460-CNG489_Week6_3-7Nov25_Chp6-Part1.pptx
byberayseray382
 
PDF
How Modern Custom Software is Revolutionizing Mortgage Lending Processes -Ma...
byMatellio
 
PPTX
Code Assessment Tools .pptx
byCodexpro
 
PDF
DSD-INT 2025 Managing low flows in the International Meuse Basin with Nature-...
byDeltares
 
AI Clinic Management Software for Pulmonology Clinics Bringing Clarity, Contr...
byEasy Clinic
 
Revolutionising-SQL-Data-Modelling-with-SqlDBM.pdf
bySQL DBM
 
Solved First Semester B.C.A. C Programming Question Paper – Jan/Feb 2025
byKuvempu University
 
From Experiment to Enterprise: Scaling AI Coding Assistants Across Engineerin...
byMaxim Salnikov
 
One Agent to Rule Them All - The Fellowship of AI Agents
byIvo Andreev
 
Fast-tracking quality for hundreds applications with automated testing layers
byBogdan Plieshka
 
Cloud-Based Underwriting Software for Insurance
byInsurance Tech Services
 
Computer_Virus_Presentation_With_Slide7.pptx
byHEVISLIVE
 
NSF Converter Software to Convert NSF to PST, EML, MSG
byNSF Converter Software
 
Week 7 Introduction to HTML PowerPoint Notes.pptx
bylesandawelgens
 
Custom chat gpt integration services.pptx
byChetu
 
DSD-INT 2025 Stochastic flood event generation using wflow and a diffusion do...
byDeltares
 
Virtual Study Circles Innovative Ways to Collaborate Online.pdf
byExplain Learning
 
Design and Analysis of Algorithms(DAA): Unit-II Asymptotic Notations and Basi...
byKuvempu University
 
DSD-INT 2025 Latest Developments for HydroMT and wflow - Meshgi
byDeltares
 
BCS-FACS Peter Landin Semantics Seminar - Formal Methods: Whence and Whither?
byJonathan Bowen
 
SNG460-CNG489_Week6_3-7Nov25_Chp6-Part1.pptx
byberayseray382
 
How Modern Custom Software is Revolutionizing Mortgage Lending Processes -Ma...
byMatellio
 
Code Assessment Tools .pptx
byCodexpro
 
DSD-INT 2025 Managing low flows in the International Meuse Basin with Nature-...
byDeltares
 

Cloud-Native Streaming and Event-Driven Microservices

  • 1.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud Native Streaming and Event-Driven Microservices Marius Bogoevici, Pivotal @mariusbogoevici
  • 2.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Who am I ? • Software Engineer with Pivotal • Spring ecosystem contributor since 2008: – Spring Integration, Spring XD, Spring Integration Kafka, – Spring Cloud Stream, Spring Cloud Data Flow • Co-author, “Spring Integration in Action”, Manning, 2012
  • 3.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Streaming • Processing large quantities of data as fast as possible (near real-time) • Use cases: • real-time data movement (ETL) • predictive maintenance, fraud detection, IoS, QoS • enterprise integration meets Big Data • High throughput/low latency • Specific operations: grouping, ordering, windowing • Transition in message middleware design: • Traditional (JMS-based, RabbitMQ) • Log-style (Kafka, Amazon Kinesis, Azure Event Hubs) • Cloud-native: Google Pub Sub
  • 4.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Why microservices for data processing? • Cohesiveness around business capability (“do one thing and do it well”) • Organizational alignment (Conway’s Law), cross-team collaboration • Development agility • Optimized for replacement • Enable continuous delivery • Failure isolation • Granular resource tuning: • scaling out the critical parts of the pipeline • per-process: memory, CPU, network 4
  • 5.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Event-Driven/Messaging Microservices • Messaging via broker vs HTTP • Solving communication complexity for data processing • Decoupling: • Physical: discovery • Temporal: availability • Eventual consistency vs. shared stores/distributed transactions • especially over heterogenous resources • Pub-sub makes it easy to add new elements to the topology
  • 6.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Data and Event Streaming: Conceptually Similar Data Streaming: ingestion, analytics Async interaction, event sourcing
  • 7.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ How about operational complexity? • Distributed systems are inherently complex, operating them even more so • Operational prerequisites: • Self-servicing and provisioning • elastic infrastructure • Monitoring • Self-healing • Rapid delivery • CI/CD, deployment pipeline • automation • a DevOps culture is required https://martinfowler.com/bliki/MicroservicePrerequisites.html 7
  • 8.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 8
  • 9.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Cloud native applications and platforms • Target a platform that makes running apps reliable, transparent and boring • In-built resource management • Memory, CPU, networking • Elastic scaling • Monitoring and failover • Health, logging, metrics • Routing and load balancing • Rolling upgrades 9 Apache YARN Apache Mesos Kubernetes
  • 10.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Monolith, the Platform and the Microservice(s) 10 Spring Cloud Stream2015 Spring XD Spring Cloud Data Flow Spring Cloud Task
  • 11.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 11 file jms http Messaging middleware Platform cassandra count-words Cloud-native event-driven microservices Apache Kafka
  • 12.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream • Event-driven microservices framework • Goal: simplify writing messaging applications • Bias towards event/data streaming • Built on battle-tested components • Spring Boot: full-stack standalone applications • Spring Integration: Messaging, EIP patterns, connectors • Opinionated primitives • Pluggable middleware abstractions 12
  • 13.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream in a nutshell 13 Application Core Spring Integration Spring Messaging Reactive APIs Kafka Streams inputs outputs
  • 14.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 14
  • 15.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 15
  • 16.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ The Binder Abstraction 16 Application Messaging Middleware Binderchannel can be input or output adapts channel to target middleware created by framework configured via Spring Boot properties
  • 17.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 17 @EnableBinding + Binder Implementation Apache Kafka JMS Google PubSub Production-ready: Experimental
  • 18.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model 18 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(“input”)
 @SendTo(“output”)
 public String process(String s) {
 return s.toUpperCase();
 }
 }
  • 19.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Programming model - imperative/procedural 19 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener(“input”)
 @SendTo(“output”)
 public String process(String s) {
 return s.toUpperCase();
 }
 }
  • 20.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Reactive Programming via Reactor (since 1.1) 20 @SpringBootApplication
 @EnableBinding(Processor.class)
 public class UppercaseProcessor {
 
 @StreamListener
 @Output(“output”)
 public Flux<String> process(@Input(“input”) Flux<String> s) {
 return f.map(s -> s.toUppercase()); }
 }
  • 21.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Stateful operators: grouping, windowing 21 SpringBootApplication
 @EnableBinding(Processor.class)
 public class WordCountApplication {
 
 @StreamListener
 @Output("output")
 public Flux<?> countWords(@Input("input") Flux<String> words) {
 return words.window(ofSeconds(5), ofSeconds(1))
 .flatMap(window -> window.groupBy(word -> word)
 .flatMap(group -> group.reduce(0, (counter, word) -> counter + 1)
 .map(count -> new WordCount(group.key(), count))));
 } }
  • 22.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ [dumb] Pipes and [smart] filters 22 http averages topN ingest data average value over last 5s hottest sensors over last 10 seconds
  • 23.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 23 http raw-sensor-data averages averages top-n Calculator
  • 24.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Durable Publish Subscribe 24 http raw-sensor-data averages averages top-n Calculator Failure detector in Spring Cloud Stream, all destinations are pub-sub
  • 25.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 25
  • 26.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 26 raw-sensor-data averages HDFS http pubsub is great for interconnecting logical apps what if we add more instances ?
  • 27.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 27 raw-sensor-data averages HDFS http we want these two to compete averages HDFS
  • 28.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Competing consumers 27 raw-sensor-data averages HDFS http we want these two to compete averages HDFS and these two !
  • 29.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Consumer groups 28 Averages HDDSHDFS HDFS raw-sensor-data groups are in a pub-sub relationship to each other AveragesAverages consumers are competing within a group
  • 30.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 29
  • 31.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 30 Average calculator Average calculator http raw-sensor-data
  • 32.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Stream: Partitioning 31 Average calculator Average calculator http partition 1 partition 2 raw-sensor-data
  • 33.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 32 Average calculator Average calculator source partition 1 partition 2 raw-sensor-data Spring Cloud Stream: Partitioning
  • 34.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Simple topologies: [relatively] easy to deploy … 33 http hdfs spring.cloud.stream.bindings.output.destination=httphdfs.1 spring.cloud.stream.bindings.input.destination=httphdfs.1 spring.cloud.stream.bindings.input.group=httphdfs httphdfs.1
  • 35.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ … but how about complex topologies ? 34 http raw-sensor-data averages top-n Calculator Failure detector averagesaverages HDFSHDFSHDFS
  • 36.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow • Orchestration: • DSL for Stream topologies • REST API • Shell • UI • Portable Deployment SPI • OOTB apps for common integration use-cases 35
  • 37.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow - Stream DSL 36 Stream definition Spring Boot Apps built with Spring Cloud Stream httpfile = http | file |
  • 38.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
  • 39.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
  • 40.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
  • 41.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Spring Cloud Data Flow Deployment Platforms 37 Data Flow Server REST API Deployer SPI SCDF FloSCDF Shell
  • 42.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 38 cassandra gpfdist http stream1 = http | count-words | file stream2 = jms | cassandra Kafka Data Flow Server DB jms file jms http Message System Platform cassandra count-words
  • 43.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39
  • 44.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http
  • 45.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http LoadBalancer
  • 46.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work LoadBalancer
  • 47.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  • 48.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  • 49.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  • 50.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  • 51.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  • 52.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  • 53.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer
  • 54.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer stream create s1 --definition “http | work | hdfs” stream deploy s1 --propertiesFile ingest.properties
  • 55.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Partitioning and Instance Count 39 http http work work work hdfs hdfs hdfs hdfs LoadBalancer stream create s1 --definition “http | work | hdfs” stream deploy s1 --propertiesFile ingest.properties app.http.count=2 app.work.count=3 app.hdfs.count=4 app.http.producer.partitionKeyExpression=payload.id
  • 56.
    Unless otherwise indicated,these slides are © 2013-2016 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Deployment: Resource Management 40 http http work work work app.work.spring.cloud.deployer.cloudfoundry.memory=2048
  • 57.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Some links … http://cloud.spring.io/spring-cloud-stream http://cloud.spring.io/spring-cloud-dataflow https://github.com/spring-cloud/spring-cloud-stream-samples
  • 58.
    Unless otherwise indicated,these slides are © 2013-2017, Pivotal Software, Inc. and licensed under a Creative Commons Attribution- NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ Questions ?