Spring Batch in Code - simple DB to DB batch applicaiton

Spring Batch in Action
May 2014

Copyright © 2014 Accenture All rights reserved. 2
Batch Applications provide
• Large Data Volume
• Automation
• Robustness (retry / skip)
• Reliability (restart)
• Performance
• Long Running Processes
Objectives
• Configurable error tolerance
• Optimized transaction
management
• Iteration through input data
• Throughput
• Manageability
Why Batch Applications?
Spring Batch = batch architecture framework
• Highly scalable
• Easy-to-use
• Customizable
• Industry-accepted

Spring Batch Architecture
Job
Launcher
Job Step
Reader
Processor
Writer
Job Repository

Spring Batch Application: DB  DB

DB to DB Batch Application
<Value Object>
Input
<Value Object>
Output
<Processor>
Transformation
<Writer><Reader>
<SQL>
Driving Query
<Row Mapper>
<Prepared Statement Setter>
SQL Property Setter
<SQL>
Insert / Update
Configuration
Java Code

Skeleton generation and
configuration

Install SpringSource Tool Suite (current version is 3.5.1)
• http://spring.io/tools/sts
Install Maven-3 (current version is 3.2.1)
• http://maven.apache.org/download.cgi
Install MySQL in WampServer distribution
• WampServer 2.4 (it needs also MS Visual C++
redistributable binaries - see dialog during download of
WampServer)
• http://www.wampserver.com/en/
Preparing developer workstation

Create new project:
• New > Other > Spring > Spring Project
• Batch > Simple Spring Batch Project
• Set the project namespace: com.sample.bratislava.db2db
• Generated application reads from private String field text
"Hello world!", and writes to Log.info.
Fix the namespace
• STS does not generate complete namespace, if deeper
than 3 elements (!)
Run test of skeleton application
• ExampleJobConfigurationTests.java
Creating a new batch program

Edit schama in launch-context.xml and module-
context.xml
• in <beans xsi:schemaLocation="..."> remove the version
numbers for all spring and spring-batch XSD schemas
• it takes the version defined in dependencies of pom.xml
Edit pom.xml
Set current version
of Spring Batch and Spring Framework
<properties>
...
<spring.framework.version>4.0.2.RELEASE</spring.framework.version>
<spring.batch.version>2.2.6.RELEASE</spring.batch.version>
</properties>

Replace body of the method testLaunchJob
Create unique job parameters
@Test
public void testLaunchJob() throws Exception {
// jobLauncher.run(job, new JobParameters());
final DateFormatter dateFormatter = new DateFormatter("yyyy-MM-dd
HH:mm:ss.SSS");
final Map<String, JobParameter> params = new LinkedHashMap<>();
params.put("test", new JobParameter(this.getClass().getSimpleName()));
params.put("timestamp", new JobParameter(dateFormatter.print(new Date(),
Locale.US)));
final JobExecution run = jobLauncher.run(job, new JobParameters(params));
assertEquals(ExitStatus.COMPLETED, run.getExitStatus());
}

Create schema for batch application(s)
• Run DDL to create spring batch tables
– in spring-batch-core JAR
– in org.springframework.batch.core directory
– file schema-mysql.sql
Create Spring Batch tables in database

Edit pom.xml - add dependency to JDBC driver
Add dependency to JDBC driver
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<version>5.1.29</version>
</dependency>

Edit src/main/resources/launch-context.xml
• select MySQL properties:
– <context:property-placeholder location="classpath:batch-
mysql.properties" />
• Set isolation level for job repository:
– <batch:job-repository id="jobRepository" isolation-level-for-
create="READ_COMMITTED" />
• Remove database initialization
(creating the Spring Batch tables):
– delete or comment out element jdbc:initialize-database
Copy batch-mysql.properties from batch samples
• Update schema, user name and password, ...
Switch to MySQL
<context:property-placeholder location="classpath:batch-mysql.properties" />
<batch:job-repository id="jobRepository“
isolation-level-for-create="READ_COMMITTED" />

Run Junit
• Unit test should run without errors
• In the Spring Batch tables should appear operational data
and statistics
• In the BATCH_JOB_PARAMS should appear 2 new
records for the same job instance ID:
– key_name(test) = ExampleJobConfigurationTest
– key_name(timestamp) = date and time of the JUnit test execution
Test the Persistent State in Database

Debug Reader, Processor, Writer

Reader
• IteratorItemReader
• Reads preconfigured records defined in the bean configuration
Row Mapper
• ColumnMapRowMapper
• Translates incoming RowSet into a HashMap, where key is
column name
Processor
• PassThroughItemProcessor
• Sends the value object from input to output without any change
Writer
• FlatFileItemWriter
• Writes incoming value objects to a text file
Testing
Mock Spring Beans

Create spring debug context
• New > Spring Bean Configuration File
• META-INF/spring/module-debug-context.xml
Add debug reader
Spring debug context
<bean id="debugReader" class="... IteratorItemReader" scope="step">
<constructor-arg>
<list>
<value>foo</value>
<value>bar</value>
<value>baz</value>
</list>
</constructor-arg>
</bean>

Add debug processor and writer
Debug reader, processor and writer
<bean id="debugProcessor" class="... PassThroughItemProcessor"/>
<bean id="debugWriter" class="... FlatFileItemWriter" scope="step">
<property name="resource">
<bean class="... FileSystemResource">
<constructor-arg value="test-output.txt" />
</bean>
</property>
<property name="lineAggregator">
<bean class="... PassThroughLineAggregator"/>
</property>
</bean>

Edit spring context META-INF/spring/module-
context.xml
• Include debug beans
• Redefine the Job configuration with debug components
Include Debug Beans into the Application
<import resource="module-debug-context.xml"/>
<batch:job id="[module-name]">
<batch:step id="processing">
<batch:tasklet transaction-manager="transactionManager" start-limit="100">
<batch:chunk reader="debugReader" processor="debugProcessor"
writer="debugWriter" commit-interval="1" />
</batch:tasklet>
</batch:step>
</batch:job>

Delete example reader and writer and example unit tests
• ExampleItemReader, ExampleItemWriter
• ExampleItemReaderTests, ExampleItemWriterTests
Rename
• ExampleConfiguration.java
to [module-name]BatchApp.java
• and its unit test to [module-name]BatchAppTests
Run test [module-name]BatchAppTests
• Check the result in test-output.txt file
Cleanup and Test

Database Reader with Driving Query

Add Reader Bean with ColumnMapRowMapper
Wire the new reader into the job, and run test
Create DB Reader in Application Context
<bean id="reader" class="... JdbcCursorItemReader" scope="step">
<property name="dataSource" ref="dataSource"/>
<property name="sql">
<value><![CDATA[
SELECT STATUS, START_TIME, END_TIME FROM BATCH_JOB_EXECUTION
]]></value>
</property>
<property name="rowMapper">
<bean class="... ColumnMapRowMapper"/>
</property>
</bean>
<batch:chunk reader="reader" processor="debugProcessor" writer="debugWriter"
commit-interval="1" />

Create class (value object) for input with name InputData
• Add fields for each column
– private String status;
– private String startTime;
– private String endTime;
• Generate getters and setters methods
• Generate toString() method
Replace hash map with value object

Add factory bean for the input value object
Replace row mapper in the reader
Add Input Value Object to Application Context
<bean id="inputDataPrototype" class="...InputData"/>
<property name="rowMapper">
<bean class="... BeanPropertyRowMapper">
<property name="mappedClass">
<bean factory-bean="inputDataPrototype" factory-method="getClass"/>
</property>
</bean>
</property>

Prepared Statement Setter
Parameterization of Driving Querry
<value><![CDATA[
SELECT JOB_EXECUTION_ID, STATUS, START_TIME, END_TIME
FROM BATCH_JOB_EXECUTION
WHERE STATUS LIKE ?
ORDER BY JOB_EXECUTION_ID DESC
]]></value>
</property>
<property name="preparedStatementSetter">
<bean class="... ArgumentPreparedStatementSetter">
<constructor-arg>
<array>
<value>COMPLETED</value>
</array>
</constructor-arg>
</bean>
</property>

Output Value Object
Value Object Transformation

Run in database the DDL
Create value object for output - class BatchDuration
• Add fields
– private Date startTime;
– private long duration;
• Generate getters and setters methods
• Generate toString() method
Create a Table for Results and Output Value Object
CREATE TABLE `BATCH_DURATION` (
`START_TIME` datetime DEFAULT NULL,
`DURATION` bigint(20) NOT NULL);

Transformer from InputData to BatchDuration
public class InputDataToBatchDuration
implements ItemProcessor<InputData, BatchDuration> {
public BatchDuration process(InputData item) throws Exception {
DateFormat formatter = new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
Date start = formatter.parse(item.getStartTime());
Date end = formatter.parse(item.getEndTime());
BatchDuration duration = new BatchDuration();
duration.setStartTime(start);
duration.setDuration( (end.getTime() - start.getTime()) / 1000);
return duration;
}
}

Edit module-context.xml
Add transformer bean
Configure job with new transformer
Run the test, check the output file.
Replace debug processor with the new transformer
<bean id="transformer" class=“... InputDataToBatchDuration"/>
<batch:chunk reader="reader" processor="transformer" writer="debugWriter"
commit-interval="1" />

Create writer in module context
Connect the new writer into the batch step
Configure Writer to Database
<bean id="writer" class="... JdbcBatchItemWriter" scope="step">
<property name="dataSource" ref="dataSource" />
<value><![CDATA[
INSERT INTO BATCH_DURATION ( START_TIME, DURATION )
VALUES ( :startTime, :duration )
]]></value>
</property>
<property name="itemSqlParameterSourceProvider">
<bean class="... BeanPropertyItemSqlParameterSourceProvider"/>
</property>
</bean>
<batch:chunk reader="reader" processor="transformer" writer="writer" commit-
interval="1" />

Create JdbcTasklet that implements Tasklet interface
from Spring Batch framework
Add fields with getters and setters
• private DataSource dataSource;
• private String sql;
Implement method form interface
Create JdbcTasklet class
public RepeatStatus execute(StepContribution contribution, ChunkContext
chunkContext) throws Exception {
JdbcTemplate jdbcTemplate = new JdbcTemplate(getDataSource());
jdbcTemplate.execute(sql);
return RepeatStatus.FINISHED;
}

Add cleanup step before the processing step
Add Cleanup Step
<batch:step id="cleanup" next="processing">
<batch:tasklet transaction-manager="transactionManager">
<bean class="... JdbcTasklet" scope="step">
<property name="sql"><value><![CDATA[
DELETE FROM BATCH_DURATION ]]></value>
</property>
<property name="dataSource" ref="dataSource" />
</bean>
</batch:tasklet>
</batch:step>

Spring Batch in Code - simple DB to DB batch applicaiton

More Related Content

What's hot

Viewers also liked

Similar to Spring Batch in Code - simple DB to DB batch applicaiton

Spring Batch in Code - simple DB to DB batch applicaiton