Building cloud native data microservice with Spring Cloud Data Flow

1. Building Cloud Native
Data Microservices
With Spring Cloud
Data Flow
Nilanjan Roy

2. Motivation

3. Spring XD Architecture

4. Spring XD Limitations
• How to scale up/down instances at runtime?
• How to upgrade/downgrade module instances at runtime?
• How to specify resources unique to each module, e.g. memory?
• Container architecture lead to parent/child class loader issues
• Too many libraries in root classpath

5. Stream and Batch Processing in the
Cloud with Spring Cloud Data Flow
Spring Cloud Data Flow is a unified, distributed, and
extensible system for data ingestion, real time analytics, batch
processing, and data export. The project's goal is to simplify
the development of big data applications.

6. Stream and Batch Processing in the Cloud
with Spring Cloud Data Flow
• Microservice pattern applied to data processing
• Typical benefits:
• scalability, isolation, agility, continuous deployment,
operational control
• Tuning process specific resources
• Instance Count
• Memory
• CPU
• Event Driven
• Short Lived processes

7. Spring Cloud Data Flow

8. SCDF deployment Platform

9. Spring Cloud Stream
• Event-driven microservice framework
• Built on battle-tested components (Spring Boot / Spring
• Integration)
• Opinionated primitives for streaming applications
• Persistent Pub/Sub
• Consumer Groups
• Partitioning Support
• Pluggable messaging middleware bindings

10. Spring Cloud Stream

11. Register your own apps

12. Deploy with Instance Count

13. Deploy with Resource Count

14. Spring Cloud Task
• Spring Boot based framework for short lived processes
• Auto-configuration provides a task repository and pluggable data
source
• Result of each process persists beyond the life of the task for future
reporting
• Tasks can deployed executed and removed on demand
• Well integrated with Spring Batch

15. Spring Cloud Task
• Spring Boot based framework for short lived processes
• Auto-configuration provides a task repository and pluggable data source
• Result of each process persists beyond the life of the task for future
reporting
• Well integrated with Spring Batch

16. REST API for SCDF
Data Flow Template
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dataflow-rest-client</artifactId>
<version>1.2.0.BUILD-SNAPSHOT</version>
</dependency>
DataFlowTemplate dataFlowTemplate = new DataFlowTemplate(
new URI("http://localhost:9393/"), restTemplate);

17. Spring Flo Dashboard

18. References
• http://cloud.spring.io/spring-cloud-stream
• https://github.com/spring-cloud/spring-cloud-stream
• http://cloud.spring.io/spring-cloud-task
• https://github.com/spring-cloud/spring-cloud-task
• http://cloud.spring.io/spring-cloud-dataflow
• https://github.com/spring-cloud/spring-cloud-
dataflow

19. Thanks !