Design & Develop
Batch Applications
in
Java/JEE
-Naresh Chintalcheru
What are Batch Applications ?
● Execution of a series of programs ("jobs")
without manual intervention
● Conducting a grou...
The input data is pre-selected through
scripts, command-line parameters or Job-
control language
Batch Applications
● No user intervention
● No UI or minimal UI
● Need not be high
available
● Single machine based
processing
● Larger Input...
Benefits of Batch Processing
● Shift the time of job processing to when the
computing resources are less busy. (e.g., Nightly,
Weekly or Monthly Jobs)
...
● Payroll Processing
● Bank monthly statements
● Credit card companies processing bills
● Fraud detection system
● Extract...
● Data intensive B2B Applications
● Conversion Applications
● Low Response Time Application
● Extract/Update Application
●...
● File-driven applications are driven by
records or values retrieved from a file
● Database-driven applications are driven...
● Normal processing in a batch window
during off-line
● Concurrent batch & on-line processing
● Parallel processing of man...
● Simple Batch Repeat
● Automatic Retry After Failure
● Commit Batch Process Periodically: chunk processing
● Asynchronous...
Terminology
● Job Control, Job Launch
● Job Execution, Job Operator
● Step, Step Scope
● Skip Task
● Repeat Jobs
● Transac...
Programming Model to Develop Batch
Application in Java/JEE
Batch Programming Model
Developing Batch Application Java/JEE
● Java Batch (JSR-352)
● Spring Batch
● Websphere App Server Batch Support
Batch Pro...
● Java Batch (JSR 352)
● First time addresses developing Batch
Applications in Java
● Part of JEE 7, Scheduled to release ...
● Spring Batch is an open source module
from Spring Framework
● Spring Batch framework started in 2007
● JSR-352 is influe...
Spring Batch Architecture
JAVA
Spring Core
Spring
Batch
Spring
MVC
Spring
Integration
Spring
Web Services
Batch Processing Architecture
A Job is an instance that encapsulates an entire batch process. A job is
typically put together using a Job Specification ...
JobRepository holds information about jobs current
running and jobs that run in the past. JobOperator
provides access to t...
<job id="myJob" xmlns="http://batch.jsr/jsl">
<step id="Step1" >
<chunk reader="ItemReader1"
writer="ItemWriter1"
processo...
@ItemReader
public class ItemReader1 {
private static int id;
MyCheckPoint checkpoint = null;
@Open
void open(MyCheckPoint...
@ItemProcessor
public class ItemProcessor1 {
@ProcessItem
MyBatchRecord process(MyBatchRecord record) {
return (record.get...
@ItemWriter
public class ItemWriter1 {
MyCheckPoint checkpoint = null;
@Open
void open(MyCheckPoint checkpoint) {
this.che...
Item Programming
Batch App Sequence Diagram
JVM GC Policy considerations for Batch
Applications
● Optimal Throughput
● Generational Concurrency (gencon)
JVM Considera...
The Big Data and Apache Hadoop
ecosystem the future for the Batch
Application looks good.
Integration of Spring Batch to H...
Feedback Appreciated!!!
Thank You
Upcoming SlideShare
Loading in...5
×

Design & Develop Batch Applications in Java/JEE

8,313

Published on

Design & Develop Batch Applications in Java/JEE

Published in: Technology

Design & Develop Batch Applications in Java/JEE

  1. 1. Design & Develop Batch Applications in Java/JEE -Naresh Chintalcheru
  2. 2. What are Batch Applications ? ● Execution of a series of programs ("jobs") without manual intervention ● Conducting a group of computer tasks at one time Batch Applications
  3. 3. The input data is pre-selected through scripts, command-line parameters or Job- control language Batch Applications
  4. 4. ● No user intervention ● No UI or minimal UI ● Need not be high available ● Single machine based processing ● Larger Input data ● System latency is not an issue ● Transactions run for long time Batch Processing vs Online Processing ● Interacting with System ● Command-line based UI or GUI interface ● High availability ● Distributed processing ● Smaller Input data ● System response time is critical ● Short transaction time
  5. 5. Benefits of Batch Processing
  6. 6. ● Shift the time of job processing to when the computing resources are less busy. (e.g., Nightly, Weekly or Monthly Jobs) ● Avoids idling the computing resources with minute- by-minute manual intervention and supervision. ● Manage large volume of transactions ● Long running application Benefits of Batch Processing
  7. 7. ● Payroll Processing ● Bank monthly statements ● Credit card companies processing bills ● Fraud detection system ● Extract Transform Load Data processing ● Business Intelligence or Social Intelligence Reporting ● Predictive or Risk analysis systems Examples Of Batch Processing
  8. 8. ● Data intensive B2B Applications ● Conversion Applications ● Low Response Time Application ● Extract/Update Application ● Mostly Sequential ● Image Processing Applications ● Bulk processing Batch Processing Strategies
  9. 9. ● File-driven applications are driven by records or values retrieved from a file ● Database-driven applications are driven by rows or values retrieved from the database ● Message-driven applications are driven by messages retrieved from a message queue Batch App Categories
  10. 10. ● Normal processing in a batch window during off-line ● Concurrent batch & on-line processing ● Parallel processing of many different batch runs or jobs at the same time ● Partitioning (i.e. processing of many instances of the same job at the same time) Processing Options
  11. 11. ● Simple Batch Repeat ● Automatic Retry After Failure ● Commit Batch Process Periodically: chunk processing ● Asynchronous Chunk Processing: parallel processing within a chunk. ● Copy File to File in a Batch ● Massively Parallel Batch Processing ● Manual Restart After Failure ● Sequential Processing of Dependent Steps ● Partial Processing: skip records (e.g. on rollback). ● Whole-Batch Transaction ● Scheduled Processing ● Non-Sequential Processing of Steps (Conditional Branching) Batch App Use Cases
  12. 12. Terminology ● Job Control, Job Launch ● Job Execution, Job Operator ● Step, Step Scope ● Skip Task ● Repeat Jobs ● Transaction Management ● Item Processor, Item Reader Batch Programming Terminology
  13. 13. Programming Model to Develop Batch Application in Java/JEE Batch Programming Model
  14. 14. Developing Batch Application Java/JEE ● Java Batch (JSR-352) ● Spring Batch ● Websphere App Server Batch Support Batch Programming Model
  15. 15. ● Java Batch (JSR 352) ● First time addresses developing Batch Applications in Java ● Part of JEE 7, Scheduled to release third quarter of 2013 ● IBM & VMware are the Spec lead JSR-352
  16. 16. ● Spring Batch is an open source module from Spring Framework ● Spring Batch framework started in 2007 ● JSR-352 is influenced by Spring Batch Spring Batch
  17. 17. Spring Batch Architecture JAVA Spring Core Spring Batch Spring MVC Spring Integration Spring Web Services
  18. 18. Batch Processing Architecture
  19. 19. A Job is an instance that encapsulates an entire batch process. A job is typically put together using a Job Specification Language and consists of multiple steps. The Job Specification Language is implemented with XML and is referred as "Job XML". A Step is a domain object that encapsulates an independent, sequential phase of a job. A step contains all of the information necessary to define and control the actual batch processing. JobOperator provides an interface to manage all aspects of job processing, including operational commands, such as start, restart, and stop, as well as job repository commands, such as retrieval of job and step executions. Batch Processing
  20. 20. JobRepository holds information about jobs current running and jobs that run in the past. JobOperator provides access to this repository. Reader-Processor-Writer pattern is the primary pattern and is called as Chunk-oriented processing. In this, ItemReader reads one item at a time, ItemProcessor processes the item based upon the business logic, such as calculate account balance and hands it to ItemWriter for aggregation. Once the 'chunk' number of items are aggregated, they are written out, and the transaction is committed. Batch Processing
  21. 21. <job id="myJob" xmlns="http://batch.jsr/jsl"> <step id="Step1" > <chunk reader="ItemReader1" writer="ItemWriter1" processor="ItemProcessor1" buffer-size="5" checkpoint-policy="item" commit-interval="10" /> </step> <step id="Step2" > .... </step> <step id="Step3" > .... </step> </job> Job XML
  22. 22. @ItemReader public class ItemReader1 { private static int id; MyCheckPoint checkpoint = null; @Open void open(MyCheckPoint checkpoint) { this.checkpoint = checkpoint; System.out.println(getClass().getName() + ".open: " + checkpoint. getItemCount()); } @ReadItem ItemReader: MyBatchRecord read() { FlatFileItemReader checkpoint.incrementByOne(); JdbcCursorItemReader return new MyBatchRecord(++id); JdbcPagingItemReader }} HibernateCursorItemReader Item Reader
  23. 23. @ItemProcessor public class ItemProcessor1 { @ProcessItem MyBatchRecord process(MyBatchRecord record) { return (record.getId() % 2 == 0) ? record : null; } DataSource FlatFile XML DataBase Message Item Processor
  24. 24. @ItemWriter public class ItemWriter1 { MyCheckPoint checkpoint = null; @Open void open(MyCheckPoint checkpoint) { this.checkpoint = checkpoint; System.out.println(getClass().getName() + ".open: " + checkpoint.getItemCount()); } @WriteItems void write(List<MyBatchRecord> list) { System.out.println("Writing the chunk..."); for (MyBatchRecord record : list) { System.out.println(record.getId()); ItemWriter: } FlatFileItemWriter HibernateItemWriter checkpoint.increment(list.size()); JdbcBatchItemWriter System.out.println("... done."); JpaItemWriter }} Item Writer
  25. 25. Item Programming
  26. 26. Batch App Sequence Diagram
  27. 27. JVM GC Policy considerations for Batch Applications ● Optimal Throughput ● Generational Concurrency (gencon) JVM Considerations
  28. 28. The Big Data and Apache Hadoop ecosystem the future for the Batch Application looks good. Integration of Spring Batch to Hadoop framework Batch Application Future
  29. 29. Feedback Appreciated!!! Thank You

×