SpringOne Platform 2017 Bruce Thelen, Corelogic "Problem: CoreLogic RiskMeter is a platform for insurance underwriters to assess natural hazard risk to insured properties. RiskMeter is going through a replatforming as part of CoreLogic’s cloud native transformation with help from Pivotal Labs. The replatformed Riskmeter is implemented in Spring Boot and deployed on Cloud Foundry as a cloud native 12 factor app. While executing this replatforming, the team discovered the need for recreating the old application’s batch processing features. In this new cloud native version of the application, there was a strong desire to be able to blue green deploy the application without disrupting running batches. Solution: In order to accomplish this, we chose to use Spring Cloud Data Flow Server with Spring Cloud Task to execute a Spring Batch Job. The batch job updates its status via a StepExecutionListener sending messages to a RabbitMQ message queue which is read via a websocket connection from the Angular frontend thus providing a real time progress bar. Specific Items To Emphasize As Lessons Learned: Decomposing the system to extract the short lived microservice that composes the batch run Injecting configuration into Spring Cloud Data Flow Server Injecting configuration into Spring Tasks Techniques for blue/green deployment of Spring Tasks Realtime batch status progress bar via websockets to Angular front end Conclusion: Previous CoreLogic projects which implemented homegrown batch processing relied heavily on complex infrastructure, deployment processes (i.e. allocating dedicated VM instances, configuring app servers), and long development cycles. In one case, it took approximately a year to implement a similar system. Using Cloud Foundry, Spring Cloud Data Flow Server, Spring Task, and Spring Batch, we were able to decompose our app and deploy the batch solution in about two months without having to write custom and complex batch management tooling."

1. Batch Processing With Spring Cloud
Data Flow Server in Cloud Foundry
By Bruce Thelen
@brucethelen
1

2. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Introduction
2

3. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RiskMeter Order Flow
3

4. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Problem Statement
4

5. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Solution Design
5
Choose Products Upload Input Verify Input Process Batch Download
Output

6. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
RiskMeter Batch Order Flow
6

7. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Solution Overview
7

8. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Process To Implement Solution
8
http://www.appcontinuum.io/

9. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Process To Implement Solution
9
SingleApplicationwith
Namespaces
• Monolith
SingleApplicationand
Components
• Monolith
• Web
• Batch
• Shared
MultipleApplicationsand
Components
• Web
• Web
• Shared
• Data Flow Server
• Batch
• Batch
• Shared

10. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Solution Diagram
10

11. Lessons Learned

12. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Register Tasks With SCDF
12

13. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Deploy
New Web
Smoke Test
New Web
Reroute to
New Web
Stop Old
Web
Deploy
New Web
Register
New Batch
Smoke Test
New Web
Smoke Test
New Batch
Reroute to
New Web
Stop Old
Web
Cleanup
Any
Completed
Batch
Blue Green Deployment Process
13

14. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Cleaning Up CF Tasks
14

15. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Passing Data Between Web And Batch
15
@Service
public class BatchRunner {
…
public JobExecution runBatch() {
…
return jobLauncher.run(batchJob, batchJobParameters);
}
}

16. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Passing Data Between Web And Batch
16
public void execute(String batchJobId) {
…
MultiValueMap<String, String> params = new …
params.add("name", taskName);
params.add("arguments", "--batchJobId=" + batchJobId);
…
HttpEntity<MultiValueMap<String, String>> requestEntity =
new HttpEntity<>(params, null);
…
asyncRestTemplate.postForLocation(uri, requestEntity);
}

17. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Passing Data Between Web And Batch
17
$ curl -s -X POST -u scdfuser:password
http://localhost:9393/tasks/executions -d 'name=riskmeter-batch-
app-527b9c8-task&arguments=--batchJobId%3D1'

18. Unless otherwise indicated, these slides are © 2013 -2016 Piv otal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Injecting Configuration Into SCDF and TASK
18
spring:
cloud:
deployer:
cloudfoundry:
domain: clgxlabs.com
org: Corelogic
password: "{cipher}ThisIZAFakePazzW0Rd"
space: rmeter-prod
url: "https://api.sys.clgxlabs.io"
username: ciprodadmin
task:
buildpack: java_buildpack_offline
services: "config-server,riskmeter-rabbit,splunk-syslog"
memory: 2048

19. Demo

20. Conclusion

21. Learn More. Stay Connected.
Orchestrating Data Microservices with Spring Cloud Data Flow
Tomorrow 10:30am Room 2005
Latency and Event Tracing with Spring Cloud Data Flow
Tomorrow 11:50am Room 2005
21
#springone@s1p