Spring Batch - Lessons Learned out of a real life banking system.

6,939 views
6,698 views

Published on

Talk at Java User Group Switzerland in Zürich

Published in: Career, Technology, Business
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
6,939
On SlideShare
0
From Embeds
0
Number of Embeds
11
Actions
Shares
0
Downloads
157
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Spring Batch - Lessons Learned out of a real life banking system.

  1. 1. Spring Batch Lessons Learned out of a real life banking system. Michael Beer Senior Consultant Raffael Schmid Consultant BASEL 1 BERN BRUGG LAUSANNE ZUERICH DUESSELDORF 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA
  2. 2. About us Michael Beer Senior Consultant  Trivadis since 2001  Trivadis since 2010  design and development of web based applications  design and development of web based applications  part of the Trivadis APM team 2 Raffael Schmid Consultant  interested in performance related things on the JVM 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  3. 3. AGENDA Initial position 3 Requirements and Topics 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Lessons learned Conclusion
  4. 4. Initial position Initial position 4 Requirements and Topics 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Lessons learned Conclusion
  5. 5. Initial and target system context Web Interface  Automatic load out of thirdparty systems Rule Engine  multiple times per day  export / import as CSV file  Manual load by user common usage  at irregular time intervals  Migration load due to thirdparty system decommissioning Thirdparty system File Interface Thirdparty system 5 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013  run once  different volumes
  6. 6. Why we used Spring Batch?  It was already part of the technology stack within the customers environment.  There were no real (free) alternatives at that time (or at least we didn‘t know any).  Exceptional permit was needed for Spring Integration because it was shipped with Spring Batch Admin. 6 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  7. 7. Terminology of Spring Batch jobs * Job Step * Job Instance * Job Parameter * * Job Execution * Job Execution Context 7 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Step Execution Step Execution Context * Tasklet
  8. 8. Requirements and Topics Initial position 8 Requirements and Topics 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Lessons learned Conclusion
  9. 9. Requirement: Performance Performance Reprocess failed items Summary mail Gather detailed job information Trigger jobs Control load Deactivate jobs Inter-job dependencies  parallelized on multiple threads  order of execution matters  number of threads can be changed before or after job execution 9 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  10. 10. Requirement: Reprocess failed items Performance Reprocess failed items Summary mail Gather detailed job information Trigger jobs Control load Deactivate jobs Inter-job dependencies  rerunning a job processes failed items (only)  process state of item therefore needs to be maintained 10 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  11. 11. Requirement: Summary mail Performance Reprocess failed items Summary mail Gather detailed job information Trigger jobs Control load Deactivate jobs Inter-job dependencies  write a detailed execution log that contains  Exceptions  messages out of the Rule Engine  aggregate to summary  Number of errors, warnings, etc.  worst status level  send summary to submitter and operator 11 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  12. 12. Requirement: Gather detailed job information Performance Reprocess failed items Summary mail Gather detailed job information Trigger jobs Control load Deactivate jobs Inter-job dependencies  rerun failed single item in trace mode  collect diagnostic information  e.g. out of the Rule Engine 12 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  13. 13. Requirement: Trigger jobs Performance Reprocess failed items Summary mail Gather detailed job information Trigger jobs Control load Deactivate jobs Inter-job dependencies  periodically  either fixed delay or fixed rate  on event  e.g. new data arrived in database 13 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  14. 14. Requirement: Control load Performance Reprocess failed items Summary mail Gather detailed job information Trigger jobs Control load Deactivate jobs Inter-job dependencies  prevent too many jobs in parallel  conditions might prevent job execution  requests should prevent system shutdown 14 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  15. 15. Requirement: Deactivate jobs Performance Reprocess failed items Summary mail Gather detailed job information Trigger jobs Control load Deactivate jobs Inter-job dependencies  deactivate job execution  set job execution on hold 15 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  16. 16. Requirement: Inter-job dependencies Performance Reprocess failed items Summary mail Gather detailed job information Trigger jobs Control load Deactivate jobs Inter-job dependencies  finished jobs might trigger dependencies 16 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  17. 17. Requirements grouped into five different topics Performance Partitioning Error Reprocess failed items Handling Trigger jobs Control load Summary mail Monitoring Job Control Deactivate jobs Sourcing 17 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Gather detailed & Tracing job information Inter-job dependencies
  18. 18. Lessons learned Initial position 18 Requirements and Topics 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Lessons learned Conclusion
  19. 19. Topics we will cover Monitoring & Tracing depends on Partitioning depends on supervising Sourcing firing trigger depends on Error Handling 19 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Job Control
  20. 20. Sourcing Monitoring & Tracing depends on Partitioning depends on supervising Sourcing firing trigger depends on Error Handling 20 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Job Control
  21. 21. Sourcing Load data into staging table CAT_ID 310 310 311 | | | | PERSON_ID 2908 1608 1608 | | | | AMOUNT 200000 100000 100000 | | | | CURRENCY USD CHF CHF | | | | VALID_FORM 03.12.2013 03.12.2013 03.12.2013 | | | | LAST_CHANGE| 02.02.2013 | 02.01.2013 | 02.01.2013 | PARTITION LINE TYPE STATE FIELD01 FIELD02 FIELD03 FIELD04 PT_000122 171 D 02 441 1804 400000 EUR PT_000122 172 D 02 441 1002 221000 EUR PT_000123 1 H 01 CAT_ID PERSON_ID AMOUNT CURRENCY PT_000123 2 D 01 310 2908 200000 USD PT_000123 3 D 01 310 1608 100000 CHF PT_000123 4 D 01 311 1608 100000 CHF 21 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  22. 22. Allows partitioning, single record execution Sourcing PARTITION LINE TYPE STATE FIELD01 FIELD02 FIELD03 FIELD04 PT_000123 1 H 01 CAT_ID PERSON_ID AMOUNT CURRENCY PT_000123 2 D 01 310 2908 200000 USD PT_000123 3 D 01 310 1410 100000 CHF  Partitioning out of the box SELECT * FROM LOAD WHERE PARTITION =‘PT_000123’ AND LINE BETWEEN 0 AND 99;  Reexcution of failed records SELECT * FROM LOAD WHERE PARTITION =‘PT_000123’ AND STATE = ‘01’  Execution of single record SELECT * FROM LOAD WHERE PARTITION =‘PT_000123’ AND LINE = 8 22 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  23. 23. Job Control Monitoring & Tracing depends on Partitioning depends on supervising Sourcing firing trigger depends on Error Handling 23 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Job Control
  24. 24. Job Control Simplest way of triggering a job 0 0/5 * * * ? Trigger job-requests Spring Batch (SimpleJobLauncher) Request Factory  Define polling channel adapter (CronTrigger) <int:inbound-channel-adapter method="create" channel="job-requests" ref="requestFactory"> <int:poller cron="0 0/5 * * * ?" /> </int:inbound-channel-adapter>  Or even simpler (PeriodicTrigger) <int:inbound-channel-adapter method="create" channel="job-requests" ref="requestFactory"> <int:poller fixed-rate="300000"/> </int:inbound-channel-adapter> 24 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  25. 25. Job Control Persist job launch requests into database Data import job (database) job queue Periodic jobs *JobQueueMessageHandler 0 0/5 * * * ? persist *JobLaunchingMessageHandler job launch request jobrequests Guard verification update status save job dependencies 25 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 *JobLauncher
  26. 26. Partitioning Monitoring & Tracing depends on Partitioning depends on supervising Sourcing firing trigger depends on Error Handling 26 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Job Control
  27. 27. Partitioning Performance Type Description Multi-threaded Step Local A step is multithreaded (TaskExecutor) Parallel Steps Local Executes steps in parallel using multithreading Partitioning Step Local Remote Partitions data and splits up processing Remote Chunking 27 Local/Remote Remote Distributed chunk processing to remote nodes 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  28. 28. Partitioning Partitioning overview Job Step Master Step Slave Slave Slave Slave Slave 28 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  29. 29. Partitioning Partitioning detail StepScopes Execution Context [0] Execution Context [1] 29 Slave [2] Execution Context [3] Slave [3] Execution Context [4] rowId % 5 Slave [1] Execution Context [2] Master Slave [0] Slave [4] 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  30. 30. Partitioning Characteristics of data There might be dependencies between records. The order of execution matters at some point. CAT_ID 310 310 311 312 313 310 390 310 | | | | | | | | | PERSON_ID 2908 1608 1608 1608 1608 1410 1108 2908 | | | | | | | | | AMOUNT 200000 100000 100000 100000 100000 100000 100000 500000 | CURRENCY | Insert USD | CHF | CHF | CHF | CHF | CHF | CHF |Update USD Read 30 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 | | | | | | | | | VALID_FORM 03.12.2013 03.12.2013 03.12.2013 03.12.2013 03.12.2013 03.12.2013 03.12.2013 04.12.2013 Write Process / Write | | | | | | | | | LAST_CHANGE| 02.02.2013 | 02.01.2013 | 02.01.2013 | 02.01.2013 | 02.01.2013 | 02.01.2013 | 02.01.2013 | 03.02.2013 |
  31. 31. Partitioning detail – Spring Batch Admin 31 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Partitioning
  32. 32. Performance – Read / Write / Process Partitioning Begin Step Open Transaction For each item For each item ItemReader Item read() ItemProcessor Process(Item) RuntimeException RuntimeException ItemWriter Write(List<Item>) Commit Transaction RuntimeException Rollback Transaction Finish Step 32 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  33. 33. Partitioning Performance No error Error on each item • ~ 22 sec • 0 Rollbacks • • • • 33 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 ~ 50 sec 1‘100 Rollbacks 5‘500 Filter 1‘000 Write Skips
  34. 34. Error Handling Monitoring & Tracing depends on Partitioning depends on supervising Sourcing firing trigger depends on Error Handling 34 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Job Control
  35. 35. Error Handling Restartability regular start restart Job 1 Instance 1 Execution 1 35 read new/error 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 read new/error Job 1 Instance 1 Execution 2
  36. 36. Error Handling Restartability configuration initial (failed) execution Job restarted execution serialized context Step initialization Step process Step initialization Execution Context Step finalize 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Step process Step finalize cleanup 36 Job
  37. 37. Error Handling Item based error handling Feature What? Where? Skip For nonfatal exceptions Keeps processing for an incorrect item Chunk-oriented step Retry 37 When? For transient exceptions Makes new attemps on an operation Chunk-oriented step, application code 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  38. 38. Monitoring Monitoring & Tracing depends on Partitioning depends on supervising Sourcing firing trigger depends on Error Handling 38 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Job Control
  39. 39. Monitoring – Spring Batch Admin 39 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Monitoring & Tracing
  40. 40. Monitoring – logging to a database Step Monitoring & Tracing Listener Before Read OnSkip [WARN] Process LoggerService @Transactional(propagation = Propagation.REQUIRES_NEW) OnError [ERROR] Write After [INFO] 2 calls on error 40 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  41. 41. Monitoring & Tracing Monitoring - Mail Job Step ReadProcessWrite FAILED OK Mail 41 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Error Mail
  42. 42. Monitoring & Tracing Tracing – Explain plan explicit job start Operator Step ReadProcessWrite Partitioner Read Process Writer 1 Record 1 Record Debug mode does nothing Step … 42 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  43. 43. Monitoring & Tracing Monitoring / Tracing summary Mail OK/ERROR Mail ERROR UI Log UI Explain plan User Spring Batch Admin 43 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Operator
  44. 44. Conclusion Initial position 44 Requirements and Topics 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Lessons learned Conclusion
  45. 45. Challenges – Error Handling  Fire and forget  Commit until first error - Skip - Retry  Depending on the characteristic of data  Rollback - Whole job - Partition - Chunk  Job flow  Deciders  Operator takes care about the errors 45 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013
  46. 46. Why we still use Spring Batch Out of 155 SE tests in the JSR-352 TCK, Spring Batch 3.0 (milestone 1) passes 70. The terminology stays more or less the same: Job, Step, Chunk, Item, ItemProcessor, JobInstance, JobExecution. Spring Batch JSR 352 Tasklet Batchlet ItemReader / ItemStream ItemReader JSR-352‘s ItemReader includes Spring Batchs ItemStream capabilities ItemWriter / ItemStream ItemWriter JSR-352‘s ItemReader includes Spring Batchs ItemStream capabilities JobExecutionListener JobListener StepExecutionListener StepListener 46 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 Comments
  47. 47. Suitable for different applications Spring Batch supports implementing reliable batch jobs with small effort. In few cases even no Java code is necessary. It’s very easy to customize existing parts at your needs. Different level of customization is possible: configuration 47 partial implementation 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 job control
  48. 48. Questions and answers ... Michael Beer michael.beer@trivadis.com +41 58 459 51 90 Raffael Schmid raffael.schmid@trivadis.com +41 58 459 52 34 BASEL 48 BERN BRUGG LAUSANNE ZUERICH DUESSELDORF 2013 © Trivadis Lessons Learned out of a real life banking system December 3rd, 2013 FRANKFURT A.M. FREIBURG I.BR. HAMBURG MUNICH STUTTGART VIENNA

×