Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Java 
EE 
7 
Batch 
Processing 
in 
the 
Real 
World 
Roberto 
Cortez 
& 
Ivan 
St. 
Ivanov 
JavaOne 
2014 
#CON2818
Who 
are 
we? 
Roberto 
Cortez 
@radcortez 
h/p://www.radcortez.com 
Freelancer, 
Speaker, 
RebelLabs 
Author, 
Blogger 
I...
QuesHons? 
As 
soon 
as 
you 
have 
them!
What 
is 
Batch? 
“A 
group 
of 
records 
processed 
as 
a 
single 
unit, 
usually 
without 
input 
from 
a 
user.” 
(from...
Why 
Batch? 
(Computers) 
• Make 
use 
of 
idle 
resources 
• ShiT 
the 
Hme 
of 
processing 
• Manage 
large 
repeated 
w...
Batch 
is 
in 
our 
lives 
• By 
paying 
bills 
• On 
the 
grocery 
• In 
traffic 
• Cooking 
food
Why 
Batch? 
(General) 
• Efficiency 
• Focus 
on 
a 
single 
problem 
• Avoid 
context 
switch 
• Reduce 
costs
State 
of 
the 
Art 
• Spring 
Batch 
• IBM 
Websphere 
• Hadoop 
• Java 
SE, 
Scheduler, 
in-­‐house 
frameworks
The 
JSR-­‐352 
• Batch 
ApplicaHons 
for 
the 
Java 
pla]orm 
• Heavily 
inspired 
by 
Spring 
Batch 
• Available 
since ...
The 
JSR-­‐352 
Features 
• Task 
and 
Chunk 
oriented 
processing 
• SequenHal 
and/or 
Parallel 
execuHon 
• CheckpoinHn...
Batch 
Domain 
Language 
Job 
Operator 
Job 
Step 
Job 
Repository 
Item 
Reader 
Item 
Processor 
Item 
Writer 
1 
*
Job 
composiYon 
* 
Job 
Step 
* 
JobInstance 
* 
* 
* 
JobExecuYon 
StepExecuYon
Java 
EE 
Batch 
API
Batchlet 
DefiniYon 
@Named 
public 
class 
MyBatchlet 
extends 
AbstractBatchlet 
{ 
@Override 
public 
String 
process()...
Job 
SpecificaYon 
Language 
<job 
id="myFirstBatch"> 
<step 
id="myStep" 
> 
<batchlet 
ref="myBatchlet"/> 
</step> 
</jo...
Start 
the 
Job 
JobOperator 
jobOperator 
= 
BatchRunYme.getJobOperator(); 
Long 
execuYonId 
= 
jobOperator.start("myJob...
Batchlet 
• Task 
oriented 
Batch 
Step 
• Suitable 
for 
• Short 
execuHons 
• Handle 
system 
resources 
• IniHalizaHon ...
Chunk 
• ETL 
style 
(Reader, 
Processor, 
Writer) 
• Suitable 
for 
• Handle 
large 
amounts 
of 
data 
• Failover 
safet...
Chunk 
Processing 
Step 
ItemReader 
ItemProcessor 
ItemWriter 
read() 
read() 
process(item) 
process(item) 
write(items)...
Defining 
Chunk 
Step 
<step 
id="myChunk"> 
<chunk 
checkpoint-­‐policy="item" 
item-­‐count="3"> 
<reader 
ref=“myItemRe...
Item 
Reader 
@Named 
public 
class 
MyItemReader 
extends 
AbstractItemReader 
{ 
public 
Object 
readItem() 
throws 
Exc...
Item 
Processor 
@Named 
public 
class 
MyItemProcessor 
implements 
ItemProcessor 
{ 
public 
Object 
processItem(Object ...
Item 
Writer 
@Named 
public 
class 
MyItemWriter 
extends 
AbstractItemWriter 
{ 
public 
void 
writeItems(List<Object> 
...
ExcepYon 
Handling 
• Job 
ExecuHon 
Fails 
on 
ExcepHon 
• Possible 
to 
Skip 
or 
Retry 
ExcepHons 
• Applied 
to 
the 
...
ExcepYon 
Handling 
DefiniYon 
<chunk 
checkpoint-­‐policy="item” 
item-­‐count="3" 
skip-­‐limit="3" 
retry-­‐limit="3"> ...
ParYYon 
• Steps 
run 
on 
mulHple 
Threads 
• One 
ParHHon 
per 
Thread 
• Define 
ParHHon 
Plan 
• CheckpoinHng 
is 
ind...
ParYYon 
DefiniYon 
(In 
Step) 
<parYYon> 
<plan 
parYYons="2"> 
<properYes 
parYYon="0"> 
<property 
name="start" 
value=...
ParYYon 
DefiniYon 
(In 
Step) 
<chunk 
item-­‐count="3"> 
<reader 
ref="myItemReader"> 
<properYes> 
<property 
name="sta...
Flow 
• Sequence 
of 
ExecuHon 
elements 
• Execute 
together 
as 
a 
unit 
• Flows, 
Steps, 
Decision 
and 
Split 
• Used...
Split 
• Concurrent 
ExecuHon 
of 
Flows 
• Each 
Flow 
run 
on 
a 
separate 
Thread 
• Is 
not 
the 
same 
as 
a 
ParHHon...
Split 
and 
Flow 
DefiniYon 
<split 
id="mySplit"> 
<flow 
id="flow1"> 
<step 
id="myChunk" 
next="myBatchlet”>…</step> 
<...
ParYYon 
vs 
Split 
• ParHHon 
is 
used 
at 
the 
data 
level 
• Split 
is 
used 
at 
the 
business 
level
Decision 
• Decide 
the 
next 
TransiHon 
• Applied 
to 
Steps, 
Flows 
and 
Splits 
• It’s 
almost 
like 
an 
if 
/ 
else...
Decision 
DefiniYon 
<step 
id=”myStep" 
next="myDecider"> 
<batchlet 
ref="myBatchlet”/> 
</step> 
<decision 
id="myDecid...
Decision 
@Named 
public 
class 
MyDecider 
implements 
Decider 
{ 
public 
String 
decide(StepExecuYon[] 
execuYons) 
thr...
TransiYons 
• TransiHon 
Elements 
define 
the 
Workflow 
• Applied 
to 
Steps, 
Flows 
and 
Decision 
• Next, 
Fail, 
End...
Batch 
Exit 
Status 
• A 
Job 
and 
Steps 
has 
an 
Exit 
Status 
• Used 
to 
control 
the 
Workflow 
• STARTING, 
STARTED...
Schedule 
a 
Job 
(By 
@Schedule) 
@Singleton 
public 
class 
BatchJobRunner 
{ 
@Schedule(dayOfWeek 
= 
"Sun") 
public 
v...
Schedule 
a 
Job 
(By 
ManagedScheduledExecutorService) 
@Resource(lookup="java:comp/DefaultManagedScheduledExecutorServic...
ImplementaYons 
• JBatch 
IBM 
(Glassfish, 
JEUS) 
• JBeret 
(Wildfly) 
• Spring 
Batch
A 
Few 
Cons 
• Lacks 
standard 
Readers 
/ 
Writers 
• No 
Generics 
• ParHHon 
only 
supports 
a 
single 
Step 
• No 
Sy...
Demo 
Time
WoW 
AucYon 
House 
• World 
of 
WacraT 
is 
MMORPG 
• More 
than 
500 
servers 
in 
US 
and 
EU 
• Each 
server 
has 
2 
...
WoW 
AucYon 
House 
• Data 
available 
to 
download 
• Let’s 
process 
the 
data 
• Extract 
metrics 
• Share 
the 
knowle...
Live 
Code
Resources 
• JSR-­‐352 
SpecificaHon 
hips://jcp.org/en/jsr/detail?id=352 
• Java 
EE 
Samples 
hips://github.com/javaee-­...
Thank 
you 
for 
A/ending! 
Roberto 
Cortez 
@radcortez 
h/p://www.radcortez.com 
Ivan 
St. 
Ivanov 
@ivan_stefanov 
h/p:/...
Upcoming SlideShare
Loading in …5
×

Java EE 7 Batch processing in the Real World

20,544 views

Published on

This talk will explore one of the newest API for Java EE 7, the JSR 352, Batch Applications for the Java Platform. Batch processing is found in nearly every industry when you need to execute a non-interactive, bulk-oriented and long running operation task. A few examples are: financial transactions, billing, inventory management, report generation and so on. The JSR 352 specifies a common set of requirements that every batch application usually needs like: checkpointing, parallelization, splitting and logging. It also provides you with a job specification language and several interfaces that allow you to implement your business logic and interact with the batch container. We are going to live code a real life example batch application, starting with a simple task and then evolve it using the advanced API's until we have a full parallel and checkpointing reader-processor-writer batch. By the end of the session, attendees should be able to understand the use cases of the JSR 352, when to apply it and how to develop a full Java EE Batch Application.

Published in: Technology

Java EE 7 Batch processing in the Real World

  1. 1. Java EE 7 Batch Processing in the Real World Roberto Cortez & Ivan St. Ivanov JavaOne 2014 #CON2818
  2. 2. Who are we? Roberto Cortez @radcortez h/p://www.radcortez.com Freelancer, Speaker, RebelLabs Author, Blogger Ivan St. Ivanov @ivan_stefanov h/p://nosoGskills.com Architect, SAP Labs Bulgaria, JBoss Forge contributor
  3. 3. QuesHons? As soon as you have them!
  4. 4. What is Batch? “A group of records processed as a single unit, usually without input from a user.” (from Google)
  5. 5. Why Batch? (Computers) • Make use of idle resources • ShiT the Hme of processing • Manage large repeated work easily • Shared by mulHple users
  6. 6. Batch is in our lives • By paying bills • On the grocery • In traffic • Cooking food
  7. 7. Why Batch? (General) • Efficiency • Focus on a single problem • Avoid context switch • Reduce costs
  8. 8. State of the Art • Spring Batch • IBM Websphere • Hadoop • Java SE, Scheduler, in-­‐house frameworks
  9. 9. The JSR-­‐352 • Batch ApplicaHons for the Java pla]orm • Heavily inspired by Spring Batch • Available since Java EE 7 • Also designed for Java SE
  10. 10. The JSR-­‐352 Features • Task and Chunk oriented processing • SequenHal and/or Parallel execuHon • CheckpoinHng • Workflow • Stop and restart • ExcepHon handling
  11. 11. Batch Domain Language Job Operator Job Step Job Repository Item Reader Item Processor Item Writer 1 *
  12. 12. Job composiYon * Job Step * JobInstance * * * JobExecuYon StepExecuYon
  13. 13. Java EE Batch API
  14. 14. Batchlet DefiniYon @Named public class MyBatchlet extends AbstractBatchlet { @Override public String process() { System.out.println("Running inside a batchlet"); return BatchStatus.COMPLETED.toString(); } }
  15. 15. Job SpecificaYon Language <job id="myFirstBatch"> <step id="myStep" > <batchlet ref="myBatchlet"/> </step> </job>
  16. 16. Start the Job JobOperator jobOperator = BatchRunYme.getJobOperator(); Long execuYonId = jobOperator.start("myJob", new ProperYes()); JobExecuYon jobExecuYon = jobOperator.getJobExecuYon(execuYonId);
  17. 17. Batchlet • Task oriented Batch Step • Suitable for • Short execuHons • Handle system resources • IniHalizaHon and Cleanup
  18. 18. Chunk • ETL style (Reader, Processor, Writer) • Suitable for • Handle large amounts of data • Failover safety (checkpoint) • Long running applicaHons
  19. 19. Chunk Processing Step ItemReader ItemProcessor ItemWriter read() read() process(item) process(item) write(items) item item item item execute() ExitStatus
  20. 20. Defining Chunk Step <step id="myChunk"> <chunk checkpoint-­‐policy="item" item-­‐count="3"> <reader ref=“myItemReader"/> <processor ref=“myItemProcessor"/> <writer ref=“myItemWriter"/> </chunk> </step>
  21. 21. Item Reader @Named public class MyItemReader extends AbstractItemReader { public Object readItem() throws ExcepYon {…} }
  22. 22. Item Processor @Named public class MyItemProcessor implements ItemProcessor { public Object processItem(Object item) throws ExcepYon {…} }
  23. 23. Item Writer @Named public class MyItemWriter extends AbstractItemWriter { public void writeItems(List<Object> items) throws ExcepYon {…} }
  24. 24. ExcepYon Handling • Job ExecuHon Fails on ExcepHon • Possible to Skip or Retry ExcepHons • Applied to the Chunk • Include or Exclude ExcepHons
  25. 25. ExcepYon Handling DefiniYon <chunk checkpoint-­‐policy="item” item-­‐count="3" skip-­‐limit="3" retry-­‐limit="3"> <reader ref="myItemReader"/> <processor ref="myItemProcessor"/> <writer ref="myItemWriter"/> <skippable-­‐excepYon-­‐classes> <include class="java.lang.RunYmeExcepYon"/> </skippable-­‐excepYon-­‐classes> <retryable-­‐excepYon-­‐classes> <exclude class="java.lang.IllegalArgumentExcepYon"/> </retryable-­‐excepYon-­‐classes> </chunk>
  26. 26. ParYYon • Steps run on mulHple Threads • One ParHHon per Thread • Define ParHHon Plan • CheckpoinHng is independent per Thread
  27. 27. ParYYon DefiniYon (In Step) <parYYon> <plan parYYons="2"> <properYes parYYon="0"> <property name="start" value="1"/> </properYes> <properYes parYYon="1"> <property name="start" value="11"/> </properYes> </plan> </parYYon>
  28. 28. ParYYon DefiniYon (In Step) <chunk item-­‐count="3"> <reader ref="myItemReader"> <properYes> <property name="start” value="#{parYYonPlan['start']}" /> </properYes> </reader> <processor ref="myItemProcessor"/> <writer ref="myItemWriter"/> </chunk>
  29. 29. Flow • Sequence of ExecuHon elements • Execute together as a unit • Flows, Steps, Decision and Split • Used to aggregate logical business.
  30. 30. Split • Concurrent ExecuHon of Flows • Each Flow run on a separate Thread • Is not the same as a ParHHon • Can only use Flows
  31. 31. Split and Flow DefiniYon <split id="mySplit"> <flow id="flow1"> <step id="myChunk" next="myBatchlet”>…</step> <step id="myBatchlet">...</step> </flow> <flow id="flow2"> <step id=”otherChunk" next=”otherBatchlet”>…</step> <step id=”otherBatchlet">...</step> </flow> </split>
  32. 32. ParYYon vs Split • ParHHon is used at the data level • Split is used at the business level
  33. 33. Decision • Decide the next TransiHon • Applied to Steps, Flows and Splits • It’s almost like an if / else if • Requires an implementaHon
  34. 34. Decision DefiniYon <step id=”myStep" next="myDecider"> <batchlet ref="myBatchlet”/> </step> <decision id="myDecider" ref="MyDecider"> <next on="foo" to="myFooStep"/> <next on="bar" to="myBarStep"/> </decision>
  35. 35. Decision @Named public class MyDecider implements Decider { public String decide(StepExecuYon[] execuYons) throws ExcepYon {…} }
  36. 36. TransiYons • TransiHon Elements define the Workflow • Applied to Steps, Flows and Decision • Next, Fail, End, Stop • Fail, End and Stop terminate a Job
  37. 37. Batch Exit Status • A Job and Steps has an Exit Status • Used to control the Workflow • STARTING, STARTED, STOPPING, STOPPED, FAILED, COMPLETED, ABANDONED
  38. 38. Schedule a Job (By @Schedule) @Singleton public class BatchJobRunner { @Schedule(dayOfWeek = "Sun") public void scheduleJob() { JobOperator jobOperator = BatchRunYme.getJobOperator(); jobOperator.start("myJob", new ProperYes()); } }
  39. 39. Schedule a Job (By ManagedScheduledExecutorService) @Resource(lookup="java:comp/DefaultManagedScheduledExecutorService") private ManagedScheduledExecutorService executor; public void scheduleJob() { JobOperator jobOperator = BatchRunYme.getJobOperator(); executor.schedule( () -­‐> jobOperator.start("myJob", new ProperYes()), 7, TimeUnit.DAYS); }
  40. 40. ImplementaYons • JBatch IBM (Glassfish, JEUS) • JBeret (Wildfly) • Spring Batch
  41. 41. A Few Cons • Lacks standard Readers / Writers • No Generics • ParHHon only supports a single Step • No Sync mode
  42. 42. Demo Time
  43. 43. WoW AucYon House • World of WacraT is MMORPG • More than 500 servers in US and EU • Each server has 2 AucHon Houses • Trades around 70k items / server / hour
  44. 44. WoW AucYon House • Data available to download • Let’s process the data • Extract metrics • Share the knowledge
  45. 45. Live Code
  46. 46. Resources • JSR-­‐352 SpecificaHon hips://jcp.org/en/jsr/detail?id=352 • Java EE Samples hips://github.com/javaee-­‐samples • Wow AucHon House hips://github.com/radcortez/wow-­‐aucHons
  47. 47. Thank you for A/ending! Roberto Cortez @radcortez h/p://www.radcortez.com Ivan St. Ivanov @ivan_stefanov h/p://nosoGskills.com

×