Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spring Cloud Data Flow Overview

153 views

Published on

Spring Connect Charlotte
Spring Cloud Data Flow Overview
Speaker: Abhinav Rau

Published in: Software
  • Be the first to comment

  • Be the first to like this

Spring Cloud Data Flow Overview

  1. 1. © Copyright 2018 Pivotal Software, Inc. All rights Reserved. Abhinav Rau Senior Platform Architect arau@pivotal.io @abhinavrau https://github.com/abhinavrau/scdf-demo Spring Cloud Data Flow Overview
  2. 2. Sadly, we add Complexity to what’s complicated
  3. 3. Lack of Cohesion Hard to Deploy Excessive Configuration
  4. 4. What Complexity?
  5. 5. Avoid Tight Coupling What Complexity?
  6. 6. Continuous Delivery with Zero Downtime Avoid Tight Coupling What Complexity?
  7. 7. Scaling Avoid Tight Coupling Continuous Delivery with Zero Downtime What Complexity?
  8. 8. Test Surface Area What Complexity?
  9. 9. (Loosely) Story based on True Events..
  10. 10. ??
  11. 11. Let’s Dive In…
  12. 12. Test Surface Area • Unit Tests: 60+ Pre-built and Unit Tested Sources, Sinks and Processors • Integration Tests: Applications have a defined input/output • Component Tests: Application is the Component boundary • Contract Tests: External services are clearly visible and defined • End to End Tests: This is where testing is needed.
  13. 13. Integration Problem
  14. 14. Spring Integration
  15. 15. Spring Cloud Stream
  16. 16. Spring Boot
  17. 17. Spring Boot Data Pipeline
  18. 18. Microservice Data Pipeline
  19. 19. Microservice Data Pipeline
  20. 20. Function Data Pipeline
  21. 21. Spring Cloud Data Flow
  22. 22. Spring Cloud Data Flow
  23. 23. Spring Cloud Data Flow Tools to create complex topologies for streaming and batch data pipelines
  24. 24. Spring Cloud Data Flow Spring Cloud Stream Spring Cloud Task
  25. 25. framework for building highly scalable event-driven microservices connected with shared messaging systems Spring Cloud Stream
  26. 26. Spring Cloud Stream Rabbit MQ Apache Kafka Google PubSub Amazon Kinesis Azure Event Hubs Solace Pluggable Binder Implementations
  27. 27. Spring Cloud Stream Pluggable Binder Implementations Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }
  28. 28. Spring Cloud Stream @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public String replaceStringMsgHandler(String payload) { return StringUtils.replace(payload, "foo", "bar"); } } B I N D I N G E V E N T S foo channel B I N D I N G channel bar C O N S U M E R S Message Channel Abstraction Programming Model
  29. 29. Spring Cloud Stream @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public Flux<Average> sensorAverage(Flux<Sensor> data) { return . .; } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Native Reactor Flux’s Programming Model
  30. 30. Spring Cloud Stream Programming Model @EnableBinding(Processor.class) public class Application { @StreamListener("foo") @SendTo("bar") public KStream<Object, Foo> handler(KStream<Object, Event> input){ return . .; } } Native Kafka Streams E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S
  31. 31. Spring Cloud Stream Programming Model @EnableBinding(Processor.class) public class Application { @Bean public Function<String, String> toUpperCase() { return s -> s.toUpperCase(); } } E V E N T S B I N D I N G foo channel bar B I N D I N G channel C O N S U M E R S Plain Old Java Functions
  32. 32. Spring Cloud Stream Pluggable Binder Implementations Stream Partitions Consumer Groups Message Headers Testing Framework Content-type Negotiation Imperative + Functional Programming Model public class TransferServiceImpl implements TransferService { public TransferServiceImpl(AccountRepository ar) { this.accountRepository = ar; }
  33. 33. Spring Cloud Data Flow Spring Cloud Stream Spring Cloud Task
  34. 34. Spring Cloud Task makes it easy to orchestrate short-lived microservices.
  35. 35. @EnableTask public class TimestampTask { @Bean public TimestampTask timeStampTask() { return new TimestampTask(); } public static class TimestampTask implements CommandLineRunner { @Override public void run(String... strings) throws Exception { DateFormat dateFormat = . . logger.info(dateFormat.format(new Date())); } } } Database R E P O S I T O R Y An arbitrary business-logic as short-lived Application @EnableTask Spring Cloud Task
  36. 36. @EnableTask @EnableBatchProcessing public class BatchJobApplication { @Bean public Step extractStep() { // extract business logic } @Bean public Step transformStep() { // transformation logic } @Bean public Step loadStep() { // persistence logic } @Bean public Job etlJob() { return this.jobBuilderFactory.get("etlJob") .start(extractStep()) .next(transformStep()) .next(loadStep()) .build(); } } Database R E P O S I T O R Y Spring Batch Job as short-lived Application Spring Cloud Task
  37. 37. Spring Cloud Task Lifecycle Management Transactions Bookkeeping for Restarts/Replay Historical Representation Remote Partitions
  38. 38. Spring Cloud Data Flow Spring Cloud Stream Spring Cloud Task
  39. 39. Spring Cloud Stream Spring Cloud Task Dem o
  40. 40. Spring Cloud Stream -> Task SFTP Source TaskLauncher ETL Job/Task Database Orchestrated by Spring Cloud Data Flow SFTP Server poll for new files publish each file launch task for each file persist parsed data
  41. 41. Spring Cloud Task -> Stream SFTP Source TaskLauncher Batch Job/Task Database SFTP Server poll for new files publish each file launch task for each file persist parsed data :task-eventsTaskLauncher
  42. 42. Spring Cloud Data Flow Integration Problems Event Streaming Orchestrating Short Lived Tasks
  43. 43. References • Software developers are now more valuable to companies than money - CNBC • Testing Strategies in a Microservice Architecture by Martin Fowler and Toby Clemson • Presentation Patterns by Neal Ford, Matthew McCullough, and Nate Schutta  • Don't Walk Away from Complexity, Run • Venkat Subramaniam • Distributed Sagas: A Protocol for Coordinating Microservices - Caitie McCaffrey • Dave Syer, Pivotal—Choosing Software Abstractions, SpringOne Platform 2018
  44. 44. References (con’t) • Code for Demos: https://github.com/abhinavrau/scdf-demo • Spring Cloud Data Flow Microsite Reference Doc • Spring Cloud Stream Samples | Gitter | StackOverflow • Spring Cloud Task Samples | Gitter | StackOverflow • Spring Cloud Skipper Samples | Gitter | StackOverflow
  45. 45. Transforming How The World Builds Software © Copyright 2019 Pivotal Software, Inc. All rights Reserved.

×