Batch Processing - A&BP CC
•
0 likes•375 views
A lot of projects sooner or later encounter the challenge of processing huge data sets. You might want to choose a system where you receive the requests, but process them later on to avoid a system overload, eg. during the night. During this workshop we will teach you the principles of batch processing. We will try to explain the matter in a framework independant way, but we will concretize the theory with code samples using the Spring Batch Framework.
More Related Content
Viewers also liked
Viewers also liked (10)
Similar to Batch Processing - A&BP CC
Similar to Batch Processing - A&BP CC (15)
More from JWORKS powered by Ordina
More from JWORKS powered by Ordina (17)
Recently uploaded
Recently uploaded (20)
Batch Processing - A&BP CC
- 5. Batch processing ≠ Scheduling Batch processing Scheduling Process large volumes of data Scheduling tasks eg. Spring Batch, JEE eg. Quartz, Obsidian
- 8. Categorize based on responsibility ▪ ▪ ▪ ▪ ▪ ▪
- 9. Categorize based on input source ▪ ▪ ▪
- 13. Transactions are handled at the step level ▪ ▪
- 14. Transaction management in steps (tasklets) Step 1 Step 2 Step 3 Transaction Transaction Transaction x nn
- 15. Example: chunk-oriented step <step id="importProductsStep"> <tasklet> <chunk reader="reader" writer="writer" commit-interval="100" /> <transaction-attributes isolation="READ_UNCOMMITTED" /> </tasklet> </step> Transaction
- 16. Transaction management in listeners ▪ ▪ ▪ ▪ ChunkListener beforeChunk afterChunk
- 17. Transactional reader <step id="importProductsStep"> <tasklet> <chunk reader="reader" writer="writer" commit-interval="100" reader-transactional-queue="true"/> <transaction-attributes isolation="READ_UNCOMMITTED" /> </tasklet> </step>
- 20. Transaction management patterns ▪ … ▪ ▪
- 22. Drawbacks to global transactions ▪ ▪ ▪ ▪ ▪ ▪ ▪
- 23. Shared resource transaction pattern ▪
- 24. Best effort pattern with JMS
- 25. What can go wrong with JMS? ▪ ▪
- 26. Best effort pattern with JMS commit start commit start Chunk local transaction JMS local transaction timeline Commits are synchronized <bean id="jmsTemplate" class="org.springframework.jms.core.JmsTemplate"> <property name="connectionFactory" ref="connectionFactory" /> <property name="defaultDestination" ref="orderQueue" /> <property name="receiveTimeout" value="100" /> <property name="sessionTransacted" value="true" /> </bean>
- 27. JMS failure will result in a duplicate message commit start commit start Chunk local transaction JMS local transaction timeline Commits are synchronized failed
- 28. Handling duplicate messages with manual detection ▪ ▪
- 29. Handling duplicate messages with manual detection
- 30. Handling duplicate messages with idempotency
- 32. Execution flows
- 34. Standard flow
- 35. Conditional flow - example
- 36. Conditional flow
- 43. Parallel execution - types ▪ ▪ ▪
- 44. Parallel execution - Multithreaded ▪ ▪ ▪
- 45. Parallel execution - Remote chunking ▪ ▪ ▪ ▪ ▪ ▪
- 46. Parallel execution - Step partitioning ▪ ▪ ▪ ▪
- 47. Parallel execution - Step partitioning
- 48. Parallel execution - Step partitioning example
- 51. Introduction ▪ ▪
- 57. Processing the lines we just read ▪ ▪ KmPerHourCalculator ▪ null
- 58. Writing the data
- 59. Writing the Fines ▪ ▪ BeanWrapperFieldExtractor DelimitedLineAggregator
- 60. Adding it all together ▪ ▪ BatchConfiguration ▪ ▪ ▪ ▪ Step ▪ ▪
- 64. Spring Batch and JEE
- 65. JSR-352 ▪ ▪ ▪ ▪
- 68. org.springframework.batch.core. SkipListener#onSkipInWrite(S item, Throwable t) javax.batch.api.chunk.listener. SkipWriteListener#onSkipWriteItem(List<Object> items, Exception ex) JSR-352 Spring vs. JEE
- 69. Questions?