The document outlines the implementation of batch processing using Spring Cloud Data Flow Server in Cloud Foundry, detailing the solution design and process flow. It covers aspects such as deploying new applications, passing data between web and batch processes, and configuration management for tasks in the cloud environment. Key lessons learned and additional sessions for further learning are also mentioned.

1. Batch Processing With Spring Cloud
Data Flow Server in Cloud Foundry
By Bruce Thelen
@brucethelen
1

2. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Introduction
2

3. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RiskMeter Order Flow
3

4. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Problem Statement
4

5. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Solution Design
5
Choose Products Upload Input Verify Input Process Batch Download
Output

6. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RiskMeter Batch Order Flow
6

7. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Solution Overview
7

8. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Process To Implement Solution
8
http://www.appcontinuum.io/

9. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Process To Implement Solution
9
SingleApplicationwith
Namespaces
• Monolith
SingleApplicationand
Components
• Monolith
• Web
• Batch
• Shared
MultipleApplicationsand
Components
• Web
• Web
• Shared
• Data Flow Server
• Batch
• Batch
• Shared

10. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Solution Diagram
10

11. Lessons Learned

12. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Register Tasks With SCDF
12

13. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Deploy
New Web
Smoke Test
New Web
Reroute to
New Web
Stop Old
Web
Deploy
New Web
Register
New Batch
Smoke Test
New Web
Smoke Test
New Batch
Reroute to
New Web
Stop Old
Web
Cleanup
Any
Completed
Batch
Blue Green Deployment Process
13

14. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Cleaning Up CF Tasks
14

15. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Passing Data Between Web And Batch
15
@Service
public class BatchRunner {
…
public JobExecution runBatch() {
…
return jobLauncher.run(batchJob, batchJobParameters);
}
}

16. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Passing Data Between Web And Batch
16
public void execute(String batchJobId) {
…
MultiValueMap<String, String> params = new …
params.add("name", taskName);
params.add("arguments", "--batchJobId=" + batchJobId);
…
HttpEntity<MultiValueMap<String, String>> requestEntity =
new HttpEntity<>(params, null);
…
asyncRestTemplate.postForLocation(uri, requestEntity);
}

17. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Passing Data Between Web And Batch
17
$ curl -s -X POST -u scdfuser:password
http://localhost:9393/tasks/executions -d 'name=riskmeter-batch-
app-527b9c8-task&arguments=--batchJobId%3D1'

18. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Injecting Configuration Into SCDF and TASK
18
spring:
cloud:
deployer:
cloudfoundry:
domain: clgxlabs.com
org: Corelogic
password: "{cipher}ThisIZAFakePazzW0Rd"
space: rmeter-prod
url: "https://api.sys.clgxlabs.io"
username: ciprodadmin
task:
buildpack: java_buildpack_offline
services: "config-server,riskmeter-rabbit,splunk-syslog"
memory: 2048

19. Demo

20. Conclusion

21. Learn More. Stay Connected.
Orchestrating Data Microservices with Spring Cloud Data Flow
Tomorrow 10:30am Room 2005
Latency and Event Tracing with Spring Cloud Data Flow
Tomorrow 11:50am Room 2005
21
#springone@s1p