Timothy C. Fanelli - Senior IT Specialist
23 September 2013

Three Key Concepts for
Understanding JSR-352:
Batch Applicati...
Important Disclaimers
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY.
WHILST E...
About me
 Timothy C. Fanelli
 Senior IT Specialist, IBM - Mainframe
Workload Modernization
 Instructor of Software Engi...
Agenda
 Background
 Three Key Concepts
– Implementation
– Orchestration
– Execution
 An Example JSR 352 Application
 A...
Background

© 2013 IBM Corporation

Sunday, September 22, 13
Batch Processing
 One of the oldest processing paradigms
 Historically associated with mainframe computing
 Still incre...
Batch Processing
 One of the oldest processing paradigms
 Historically associated with mainframe computing
 Still incre...
Java for Batch Processing?
 Mainframe developers have shied away from Java
– Performance concerns over native languages
–...
Java for Batch Processing?
 Mainframe developers have shied away from Java
– Performance concerns over native languages
–...
Java for Batch Processing?
 Mainframe developers have shied away from Java
– Performance concerns over native languages
–...
Java for Batch Processing?
 Mainframe developers have shied away from Java
– Performance concerns over native languages
–...
JSR 352: Batch Applications for the Java Platform
 Expert working group formed 29 November 2011
– IBM*, VMWare, RedHat, O...
JSR 352: Batch Applications for the Java Platform
 Expert working group formed 29 November 2011
– IBM*, VMWare, RedHat, O...
JSR 352: Batch Applications for the Java Platform
 Expert working group formed 29 November 2011
– IBM*, VMWare, RedHat, O...
Three Key Concepts...

© 2013 IBM Corporation

Sunday, September 22, 13
Three Key Concepts ...
 JSR 352 defines
– Implementation: A programming model for
implementing the artifacts
– Orchestrat...
Three Key Concepts ...
 JSR 352 defines
– Implementation: A programming model for
implementing the artifacts
– Orchestrat...
Anatomy of JSR352
 Those concepts define the anatomy of JSR 352: Batch
Applications for the Java Platform...

11

Sunday,...
Anatomy of JSR352
 Those concepts define the anatomy of JSR 352: Batch
Applications for the Java Platform...

Orchestrate...
Anatomy of JSR352
 Those concepts define the anatomy of JSR 352: Batch
Applications for the Java Platform...

Listeners
C...
Implementation: The programming model
 Chunk and Batchlet provide models for implementing
a step.

Listeners
Contexts

Li...
Implementation: The programming model
 Chunk and Batchlet provide models for implementing
a step.
 Contexts provide Job-...
Implementation: The programming model
 Chunk and Batchlet provide models for implementing
a step.
 Contexts provide Job-...
Implementation: The programming model
 Chunk and Batchlet provide models for implementing
a step.
 Contexts provide Job-...
Implementation: The programming model
Chunk vs Batchlet
Listeners
Contexts

Listeners

Partitioning

Batchlet
Reader
Proce...
Implementation: The programming model
Chunk vs Batchlet
 Both are implementations of a step within a batch job
Listeners
...
Implementation: The programming model
Chunk vs Batchlet
 Both are implementations of a step within a batch job
 The chun...
Implementation: The programming model
Chunk vs Batchlet
 Both are implementations of a step within a batch job
 The chun...
Orchestration: The Job Specification Language (JSL)
 The JSL defines a batch job as an XML document
Job

14

Sunday, Sept...
Orchestration: The Job Specification Language (JSL)
 The JSL defines a batch job as an XML document
 Describes a step as...
Orchestration: The Job Specification Language (JSL)
 The JSL defines a batch job as an XML document
 Describes a step as...
Execution: The JobOperator and Repository
 JobOperator is the runtime interface for job
management, including start, stop...
Execution: The JobOperator and Repository
 JobOperator is the runtime interface for job
management, including start, stop...
Execution: The JobOperator and Repository
 JobOperator is the runtime interface for job
management, including start, stop...
Execution: JobInstance, JobExecution, and StepExecution

Job

Step

*
JobInstance

*
JobExecution

*

*

StepExecution
Job...
Execution: JobInstance, JobExecution, and StepExecution
 The state of a job is broken down into various parts,
and persis...
An Example JSR 352
Application

© 2013 IBM Corporation

Sunday, September 22, 13
The Application

18

Sunday, September 22, 13

© 2013 IBM Corporation
The Application
‣ A typical “batch hello world”:

18

Sunday, September 22, 13

© 2013 IBM Corporation
The Application
‣ A typical “batch hello world”:
– Reads strings from an input file

18

Sunday, September 22, 13

© 2013 ...
The Application
‣ A typical “batch hello world”:
– Reads strings from an input file
– Performs some validation or transfor...
The Application
‣ A typical “batch hello world”:
– Reads strings from an input file
– Performs some validation or transfor...
The Application
‣ A typical “batch hello world”:
– Reads strings from an input file
– Performs some validation or transfor...
The Application
‣ A typical “batch hello world”:
– Reads strings from an input file
– Performs some validation or transfor...
The Application
‣ A typical “batch hello world”:
– Reads strings from an input file
– Performs some validation or transfor...
The Application
‣ A typical “batch hello world”:
– Reads strings from an input file
– Performs some validation or transfor...
The Application
‣ A typical “batch hello world”:
– Reads strings from an input file
– Performs some validation or transfor...
The Design
‣ Let’s implement a string-transform in an extract-transform-load pattern

19

Sunday, September 22, 13

© 2013...
The Design
‣ Let’s implement a string-transform in an extract-transform-load pattern
‣ We’ll use JSR352’s Chunk programmin...
The Design
‣ Let’s implement a string-transform in an extract-transform-load pattern
‣ We’ll use JSR352’s Chunk programmin...
The Design
‣ Let’s implement a string-transform in an extract-transform-load pattern
‣ We’ll use JSR352’s Chunk programmin...
The Design
‣ Let’s implement a string-transform in an extract-transform-load pattern
‣ We’ll use JSR352’s Chunk programmin...
The Design
‣ Let’s implement a string-transform in an extract-transform-load pattern
‣ We’ll use JSR352’s Chunk programmin...
The Design
‣ Let’s implement a string-transform in an extract-transform-load pattern
‣ We’ll use JSR352’s Chunk programmin...
The Code
Implement

20

Sunday, September 22, 13

© 2013 IBM Corporation
The Code
Implement

‣ An ItemReader encapsulates the data
access and deserialization of a record.

20

Sunday, September 2...
The Code
Implement

‣ An ItemReader encapsulates the data
access and deserialization of a record.
‣ No restriction on data...
The Code
Implement

‣ An ItemReader encapsulates the data
access and deserialization of a record.
‣ No restriction on data...
The Code
Implement

21

Sunday, September 22, 13

© 2013 IBM Corporation
The Code
Implement

 An ItemWriter is the output counterpart to
ItemReader

21

Sunday, September 22, 13

© 2013 IBM Corp...
The Code
Implement

 An ItemWriter is the output counterpart to
ItemReader
 Primary difference is that writeItems
accept...
The Code
Implement

 An ItemWriter is the output counterpart to
ItemReader
 Primary difference is that writeItems
accept...
The Code
Implement

22

Sunday, September 22, 13

© 2013 IBM Corporation
The Code
Implement

 An ItemProcessor encapsulates the
business logic applied to each record

22

Sunday, September 22, 1...
The Code
Implement

 An ItemProcessor encapsulates the
business logic applied to each record
 “main” here demonstrates t...
The Code
Implement

 An ItemProcessor encapsulates the
business logic applied to each record
 “main” here demonstrates t...
The Batch Descriptor and Job Specification
Orchestrate

23

Sunday, September 22, 13

© 2013 IBM Corporation
The Batch Descriptor and Job Specification
Orchestrate

 batch.xml defines and names
the batch artifacts in this
applicat...
The Batch Descriptor and Job Specification
Orchestrate

 batch.xml defines and names
the batch artifacts in this
applicat...
The Execution
Execute

24

Sunday, September 22, 13

© 2013 IBM Corporation
The Execution
Execute

 Package the application as a standard JAR or WAR
for deployment in JavaSE or EE environments
– ba...
The Execution
Execute

25

Sunday, September 22, 13

© 2013 IBM Corporation
The Execution
 See it live?

25

Sunday, September 22, 13

Execute

© 2013 IBM Corporation
Advanced Topics

© 2013 IBM Corporation

Sunday, September 22, 13
Job Management - Restart, Stop, Abandon
Execute

27

Sunday, September 22, 13

© 2013 IBM Corporation
Job Management - Restart, Stop, Abandon
 Had something gone wrong, what then?
– The “main” program shown was too simple.....
Job Management - Restart, Stop, Abandon
 Had something gone wrong, what then?
– The “main” program shown was too simple.....
Job Management - Restart, Stop, Abandon
 Had something gone wrong, what then?
– The “main” program shown was too simple.....
Java EE Integration
Execute

28

Sunday, September 22, 13

© 2013 IBM Corporation
Java EE Integration
 JSR-352: Java Batch is included in Java EE 7

28

Sunday, September 22, 13

Execute

© 2013 IBM Corp...
Java EE Integration
 JSR-352: Java Batch is included in Java EE 7

Execute

 Provides EE clustering, security, resource
...
Java EE Integration
 JSR-352: Java Batch is included in Java EE 7

Execute

 Provides EE clustering, security, resource
...
Parallel Job Processing
Orchestrate

29

Sunday, September 22, 13

© 2013 IBM Corporation
Parallel Job Processing
 Splits and Flows provide a mechanism for executing
job steps concurrently at the orchestration l...
Parallel Job Processing
 Splits and Flows provide a mechanism for executing
job steps concurrently at the orchestration l...
Parallel Job Processing
 Splits and Flows provide a mechanism for executing
job steps concurrently at the orchestration l...
Parallel Job Processing
 Splits and Flows provide a mechanism for executing
job steps concurrently at the orchestration l...
Parallel Job Processing
Implement

30

Sunday, September 22, 13

© 2013 IBM Corporation
Parallel Job Processing
 Step-level parallelism can be achieved programmatically using
step partitioning

30

Sunday, Sep...
Parallel Job Processing
 Step-level parallelism can be achieved programmatically using
step partitioning

Implement

 A ...
Parallel Job Processing
 Step-level parallelism can be achieved programmatically using
step partitioning

Implement

 A ...
Parallel Job Processing

31

Sunday, September 22, 13

© 2013 IBM Corporation
Wrap up...

© 2013 IBM Corporation

Sunday, September 22, 13
Batch Processing
 The oldest “new thing” in Java
 JSR 352 applies the modern thinking and abstraction of Java EE and SOA...
Questions?
Find this presentation and more!
http://ibm.co/JavaOne2013

© 2013 IBM Corporation

Sunday, September 22, 13
Upcoming SlideShare
Loading in …5
×

Three key concepts for java batch

876 views

Published on

An introduction to Java Batch (JSR-352), and example program.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
876
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
29
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Three key concepts for java batch

  1. 1. Timothy C. Fanelli - Senior IT Specialist 23 September 2013 Three Key Concepts for Understanding JSR-352: Batch Applications for the Java Platform © 2013 IBM Corporation Sunday, September 22, 13
  2. 2. Important Disclaimers THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILST EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. ALL PERFORMANCE DATA INCLUDED IN THIS PRESENTATION HAVE BEEN GATHERED IN A CONTROLLED ENVIRONMENT. YOUR OWN TEST RESULTS MAY VARY BASED ON HARDWARE, SOFTWARE OR INFRASTRUCTURE DIFFERENCES. ALL DATA INCLUDED IN THIS PRESENTATION ARE MEANT TO BE USED ONLY AS A GUIDE. IN ADDITION, THE INFORMATION CONTAINED IN THIS PRESENTATION IS BASED ON IBM’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM, WITHOUT NOTICE. IBM AND ITS AFFILIATED COMPANIES SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: - CREATING ANY WARRANT OR REPRESENTATION FROM IBM, ITS AFFILIATED COMPANIES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS 2 Sunday, September 22, 13 © 2013 IBM Corporation
  3. 3. About me  Timothy C. Fanelli  Senior IT Specialist, IBM - Mainframe Workload Modernization  Instructor of Software Engineering Clarkson University, Potsdam NY  tim@timfanelli.com  tfanelli@us.ibm.com, tfanelli@clarkson.edu  Visit the IBM booth #5112 and meet other IBM developers at JavaOne 2013 3 Sunday, September 22, 13 © 2013 IBM Corporation
  4. 4. Agenda  Background  Three Key Concepts – Implementation – Orchestration – Execution  An Example JSR 352 Application  Advanced Topics – Splits and Flows – Partitioning – Java EE  Conclusion and Thoughts on What’s Next... 4 Sunday, September 22, 13 © 2013 IBM Corporation
  5. 5. Background © 2013 IBM Corporation Sunday, September 22, 13
  6. 6. Batch Processing  One of the oldest processing paradigms  Historically associated with mainframe computing  Still incredibly relevant today, with fresh challenges in an OLTP driven world 6 Sunday, September 22, 13 © 2013 IBM Corporation
  7. 7. Batch Processing  One of the oldest processing paradigms  Historically associated with mainframe computing  Still incredibly relevant today, with fresh challenges in an OLTP driven world 6 Sunday, September 22, 13 © 2013 IBM Corporation
  8. 8. Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java 7 Sunday, September 22, 13 © 2013 IBM Corporation
  9. 9. Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java  Java and JavaEE have dominated the Online Transaction Processing world 7 Sunday, September 22, 13 © 2013 IBM Corporation
  10. 10. Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java  Java and JavaEE have dominated the Online Transaction Processing world  Time to bridge the two worlds together – IBM Java for zOS, IBM WebSphere, and Spring Batch paved new paths – Just-in-Time Compilation, Garbage Collection optimizations proved it out – Adoption is wide-spread! 7 Sunday, September 22, 13 © 2013 IBM Corporation
  11. 11. Java for Batch Processing?  Mainframe developers have shied away from Java – Performance concerns over native languages – Integration concerns for legacy data – Disparate developer skill set between System Z and Java  Java and JavaEE have dominated the Online Transaction Processing world  Time to bridge the two worlds together – IBM Java for zOS, IBM WebSphere, and Spring Batch paved new paths – Just-in-Time Compilation, Garbage Collection optimizations proved it out – Adoption is wide-spread!  Only remaining challenge was the lack of a standard – The need for JSR-352 was obvious 7 Sunday, September 22, 13 © 2013 IBM Corporation
  12. 12. JSR 352: Batch Applications for the Java Platform  Expert working group formed 29 November 2011 – IBM*, VMWare, RedHat, Oracle, Credit Suisse, Independent participants – Broad range of talent with deep batch experience 8 Sunday, September 22, 13 © 2013 IBM Corporation
  13. 13. JSR 352: Batch Applications for the Java Platform  Expert working group formed 29 November 2011 – IBM*, VMWare, RedHat, Oracle, Credit Suisse, Independent participants – Broad range of talent with deep batch experience  Final Release 24 May 2013 8 Sunday, September 22, 13 © 2013 IBM Corporation
  14. 14. JSR 352: Batch Applications for the Java Platform  Expert working group formed 29 November 2011 – IBM*, VMWare, RedHat, Oracle, Credit Suisse, Independent participants – Broad range of talent with deep batch experience  Final Release 24 May 2013  Included in Java EE 7! 8 Sunday, September 22, 13 © 2013 IBM Corporation
  15. 15. Three Key Concepts... © 2013 IBM Corporation Sunday, September 22, 13
  16. 16. Three Key Concepts ...  JSR 352 defines – Implementation: A programming model for implementing the artifacts – Orchestration: A Job Specification Language, which orchestrates the execution of a batch artifacts within a job. – Execution: A runtime environment for executing batch application, according to a defined lifecycle. Orchestrate Implement Execute 10 Sunday, September 22, 13 © 2013 IBM Corporation
  17. 17. Three Key Concepts ...  JSR 352 defines – Implementation: A programming model for implementing the artifacts – Orchestration: A Job Specification Language, which orchestrates the execution of a batch artifacts within a job. – Execution: A runtime environment for executing batch application, according to a defined lifecycle. Orchestrate Implement Execute  Note: “key” concepts, not “new” concepts! – Roles and abstractions should be familiar to SOA and JavaEE developers 10 Sunday, September 22, 13 © 2013 IBM Corporation
  18. 18. Anatomy of JSR352  Those concepts define the anatomy of JSR 352: Batch Applications for the Java Platform... 11 Sunday, September 22, 13 © 2013 IBM Corporation
  19. 19. Anatomy of JSR352  Those concepts define the anatomy of JSR 352: Batch Applications for the Java Platform... Orchestrate Implement Execute 11 Sunday, September 22, 13 © 2013 IBM Corporation
  20. 20. Anatomy of JSR352  Those concepts define the anatomy of JSR 352: Batch Applications for the Java Platform... Listeners Contexts Listeners Partitioning Batchlet Orchestrate Implement Job Operator Job Reader Step Processor Chunk Writer Execute Job Repository Chunk Chunk Chunk Listeners 11 Sunday, September 22, 13 © 2013 IBM Corporation
  21. 21. Implementation: The programming model  Chunk and Batchlet provide models for implementing a step. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
  22. 22. Implementation: The programming model  Chunk and Batchlet provide models for implementing a step.  Contexts provide Job- and Step- level runtime information, and provide interim data persistence. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
  23. 23. Implementation: The programming model  Chunk and Batchlet provide models for implementing a step.  Contexts provide Job- and Step- level runtime information, and provide interim data persistence.  Listeners provide callback hooks to respond to lifecycle events on batch artifacts. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
  24. 24. Implementation: The programming model  Chunk and Batchlet provide models for implementing a step.  Contexts provide Job- and Step- level runtime information, and provide interim data persistence.  Listeners provide callback hooks to respond to lifecycle events on batch artifacts. Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk  Partitioning provides a mechanism imposing parallel processing on jobs and steps Writer Chunk Chunk Chunk Listeners 12 Sunday, September 22, 13 © 2013 IBM Corporation
  25. 25. Implementation: The programming model Chunk vs Batchlet Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 13 Sunday, September 22, 13 © 2013 IBM Corporation
  26. 26. Implementation: The programming model Chunk vs Batchlet  Both are implementations of a step within a batch job Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 13 Sunday, September 22, 13 © 2013 IBM Corporation
  27. 27. Implementation: The programming model Chunk vs Batchlet  Both are implementations of a step within a batch job  The chunk model – Encapsulates a very common pattern: ETL – Single “reader”, “processor” and “writer” – Reader/Processor combination is invoked until an entire “chunk” of data is processed – Output “chunk” is written atomically Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners 13 Sunday, September 22, 13 © 2013 IBM Corporation
  28. 28. Implementation: The programming model Chunk vs Batchlet  Both are implementations of a step within a batch job  The chunk model – Encapsulates a very common pattern: ETL – Single “reader”, “processor” and “writer” – Reader/Processor combination is invoked until an entire “chunk” of data is processed – Output “chunk” is written atomically  Batchlet provides a “roll your own” step type – Invoked and runs to completion, producing a return code upon exit. 13 Sunday, September 22, 13 Listeners Contexts Listeners Partitioning Batchlet Reader Processor Chunk Writer Chunk Chunk Chunk Listeners © 2013 IBM Corporation
  29. 29. Orchestration: The Job Specification Language (JSL)  The JSL defines a batch job as an XML document Job 14 Sunday, September 22, 13 Step © 2013 IBM Corporation
  30. 30. Orchestration: The Job Specification Language (JSL)  The JSL defines a batch job as an XML document  Describes a step as an assemblage of batch artifacts 14 Sunday, September 22, 13 Job Step © 2013 IBM Corporation
  31. 31. Orchestration: The Job Specification Language (JSL)  The JSL defines a batch job as an XML document  Describes a step as an assemblage of batch artifacts Job Step  Provides for the description of steps, step groupings, and execution sequencing 14 Sunday, September 22, 13 © 2013 IBM Corporation
  32. 32. Execution: The JobOperator and Repository  JobOperator is the runtime interface for job management, including start, stop, restart and job repository related commands Job Operator Job Repository 15 Sunday, September 22, 13 © 2013 IBM Corporation
  33. 33. Execution: The JobOperator and Repository  JobOperator is the runtime interface for job management, including start, stop, restart and job repository related commands  The Job Repository holds information about completed and executing jobs 15 Sunday, September 22, 13 Job Operator Job Repository © 2013 IBM Corporation
  34. 34. Execution: The JobOperator and Repository  JobOperator is the runtime interface for job management, including start, stop, restart and job repository related commands  The Job Repository holds information about completed and executing jobs Job Operator Job Repository  To start a batch job, get a JobOperator instance use it to start a job described (described by JSL). 15 Sunday, September 22, 13 © 2013 IBM Corporation
  35. 35. Execution: JobInstance, JobExecution, and StepExecution Job Step * JobInstance * JobExecution * * StepExecution Job Operator 16 Sunday, September 22, 13 Job Repository © 2013 IBM Corporation
  36. 36. Execution: JobInstance, JobExecution, and StepExecution  The state of a job is broken down into various parts, and persisted in the repository – Submitting a job creates a JobInstance, a logical representation of a particular “run” of a job. – A JobExecution is a single attempt to run a JobInstance. A restart attempt creates another JobExecution – Similarly, a StepExecution is a single attempt to run a step within a job. It is created when a step starts execution. Job Operator 16 Sunday, September 22, 13 Job Step * JobInstance * JobExecution * * StepExecution Job Repository © 2013 IBM Corporation
  37. 37. An Example JSR 352 Application © 2013 IBM Corporation Sunday, September 22, 13
  38. 38. The Application 18 Sunday, September 22, 13 © 2013 IBM Corporation
  39. 39. The Application ‣ A typical “batch hello world”: 18 Sunday, September 22, 13 © 2013 IBM Corporation
  40. 40. The Application ‣ A typical “batch hello world”: – Reads strings from an input file 18 Sunday, September 22, 13 © 2013 IBM Corporation
  41. 41. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms 18 Sunday, September 22, 13 © 2013 IBM Corporation
  42. 42. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file 18 Sunday, September 22, 13 © 2013 IBM Corporation
  43. 43. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities 18 Sunday, September 22, 13 © 2013 IBM Corporation
  44. 44. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off 18 Sunday, September 22, 13 © 2013 IBM Corporation
  45. 45. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off – We want control over the transaction scoping so prevent lock contention in high volume periods 18 Sunday, September 22, 13 © 2013 IBM Corporation
  46. 46. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off – We want control over the transaction scoping so prevent lock contention in high volume periods – We want flexibility to “plug and play” where our records come from 18 Sunday, September 22, 13 © 2013 IBM Corporation
  47. 47. The Application ‣ A typical “batch hello world”: – Reads strings from an input file – Performs some validation or transforms – Writes validated or transformed string to an output file ‣ Key capabilities – If something goes wrong, we don’t want to discard all the prior work; and we want to pick up where we left off – We want control over the transaction scoping so prevent lock contention in high volume periods – We want flexibility to “plug and play” where our records come from – For unit testing, development testing, and QA testing records may come from a variety of sources 18 Sunday, September 22, 13 © 2013 IBM Corporation
  48. 48. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern 19 Sunday, September 22, 13 © 2013 IBM Corporation
  49. 49. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model 19 Sunday, September 22, 13 © 2013 IBM Corporation
  50. 50. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces 19 Sunday, September 22, 13 © 2013 IBM Corporation
  51. 51. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later 19 Sunday, September 22, 13 © 2013 IBM Corporation
  52. 52. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later – “Free” checkpoint/restart capability 19 Sunday, September 22, 13 © 2013 IBM Corporation
  53. 53. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later – “Free” checkpoint/restart capability – Transaction scoping imposed externally in the job descriptor 19 Sunday, September 22, 13 © 2013 IBM Corporation
  54. 54. The Design ‣ Let’s implement a string-transform in an extract-transform-load pattern ‣ We’ll use JSR352’s Chunk programming model – Encapsulates the ETL pattern components as Reader, Processor, and Writer interfaces – Loosely coupled artifacts will be orchestrated into a single-step job later – “Free” checkpoint/restart capability – Transaction scoping imposed externally in the job descriptor ‣ Job will be executed as a Java SE command line batch application 19 Sunday, September 22, 13 © 2013 IBM Corporation
  55. 55. The Code Implement 20 Sunday, September 22, 13 © 2013 IBM Corporation
  56. 56. The Code Implement ‣ An ItemReader encapsulates the data access and deserialization of a record. 20 Sunday, September 22, 13 © 2013 IBM Corporation
  57. 57. The Code Implement ‣ An ItemReader encapsulates the data access and deserialization of a record. ‣ No restriction on data access paradigm: use DAO patterns, JDBC, JPA, Hibernate, Spring Data, etc! 20 Sunday, September 22, 13 © 2013 IBM Corporation
  58. 58. The Code Implement ‣ An ItemReader encapsulates the data access and deserialization of a record. ‣ No restriction on data access paradigm: use DAO patterns, JDBC, JPA, Hibernate, Spring Data, etc! ‣ Checkpoint/Restart data provided as Serializable argument to “open” and from “checkpointInfo” methods. 20 Sunday, September 22, 13 © 2013 IBM Corporation
  59. 59. The Code Implement 21 Sunday, September 22, 13 © 2013 IBM Corporation
  60. 60. The Code Implement  An ItemWriter is the output counterpart to ItemReader 21 Sunday, September 22, 13 © 2013 IBM Corporation
  61. 61. The Code Implement  An ItemWriter is the output counterpart to ItemReader  Primary difference is that writeItems accepts a “chunk” of output objects (as a list) to serialize. 21 Sunday, September 22, 13 © 2013 IBM Corporation
  62. 62. The Code Implement  An ItemWriter is the output counterpart to ItemReader  Primary difference is that writeItems accepts a “chunk” of output objects (as a list) to serialize.  Again, no restriction on data access paradigm! 21 Sunday, September 22, 13 © 2013 IBM Corporation
  63. 63. The Code Implement 22 Sunday, September 22, 13 © 2013 IBM Corporation
  64. 64. The Code Implement  An ItemProcessor encapsulates the business logic applied to each record 22 Sunday, September 22, 13 © 2013 IBM Corporation
  65. 65. The Code Implement  An ItemProcessor encapsulates the business logic applied to each record  “main” here demonstrates the invocation of a batch job, using the JobOperator 22 Sunday, September 22, 13 © 2013 IBM Corporation
  66. 66. The Code Implement  An ItemProcessor encapsulates the business logic applied to each record  “main” here demonstrates the invocation of a batch job, using the JobOperator  Would typically not be in the processor implementation 22 Sunday, September 22, 13 © 2013 IBM Corporation
  67. 67. The Batch Descriptor and Job Specification Orchestrate 23 Sunday, September 22, 13 © 2013 IBM Corporation
  68. 68. The Batch Descriptor and Job Specification Orchestrate  batch.xml defines and names the batch artifacts in this application archive 23 Sunday, September 22, 13 © 2013 IBM Corporation
  69. 69. The Batch Descriptor and Job Specification Orchestrate  batch.xml defines and names the batch artifacts in this application archive  sample.xml is an example Job Specification Language document for SampleBatchApp 23 Sunday, September 22, 13 © 2013 IBM Corporation
  70. 70. The Execution Execute 24 Sunday, September 22, 13 © 2013 IBM Corporation
  71. 71. The Execution Execute  Package the application as a standard JAR or WAR for deployment in JavaSE or EE environments – batch.xml goes in META-INF or WEB-INF/ classes/META-INF – JSL may go in META-INF/batch-jobs, or submitted from an external source (up to the provider!) 24 Sunday, September 22, 13 © 2013 IBM Corporation
  72. 72. The Execution Execute 25 Sunday, September 22, 13 © 2013 IBM Corporation
  73. 73. The Execution  See it live? 25 Sunday, September 22, 13 Execute © 2013 IBM Corporation
  74. 74. Advanced Topics © 2013 IBM Corporation Sunday, September 22, 13
  75. 75. Job Management - Restart, Stop, Abandon Execute 27 Sunday, September 22, 13 © 2013 IBM Corporation
  76. 76. Job Management - Restart, Stop, Abandon  Had something gone wrong, what then? – The “main” program shown was too simple... only “started” the job 27 Sunday, September 22, 13 Execute © 2013 IBM Corporation
  77. 77. Job Management - Restart, Stop, Abandon  Had something gone wrong, what then? – The “main” program shown was too simple... only “started” the job Execute  JobOperator exposes APIs for a variety of job management tasks: start, stop, abandon, restart – Would have had to take advantage of these for advanced job management capabilities. 27 Sunday, September 22, 13 © 2013 IBM Corporation
  78. 78. Job Management - Restart, Stop, Abandon  Had something gone wrong, what then? – The “main” program shown was too simple... only “started” the job Execute  JobOperator exposes APIs for a variety of job management tasks: start, stop, abandon, restart – Would have had to take advantage of these for advanced job management capabilities.  The door is left open for more advanced batch job management systems to be built! – Integration into existing enterprise schedulers? – New Java EE batch scheduling standard? – Plenty of options, but currently left to the provider to implement 27 Sunday, September 22, 13 © 2013 IBM Corporation
  79. 79. Java EE Integration Execute 28 Sunday, September 22, 13 © 2013 IBM Corporation
  80. 80. Java EE Integration  JSR-352: Java Batch is included in Java EE 7 28 Sunday, September 22, 13 Execute © 2013 IBM Corporation
  81. 81. Java EE Integration  JSR-352: Java Batch is included in Java EE 7 Execute  Provides EE clustering, security, resource management, etc to Java Batch applications 28 Sunday, September 22, 13 © 2013 IBM Corporation
  82. 82. Java EE Integration  JSR-352: Java Batch is included in Java EE 7 Execute  Provides EE clustering, security, resource management, etc to Java Batch applications  Performance benefits to dispatching into longrunning, reusable container – JIT compilation through the first couple runs – Eliminates overhead of starting / stopping JVM 28 Sunday, September 22, 13 © 2013 IBM Corporation
  83. 83. Parallel Job Processing Orchestrate 29 Sunday, September 22, 13 © 2013 IBM Corporation
  84. 84. Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer 29 Sunday, September 22, 13 Orchestrate © 2013 IBM Corporation
  85. 85. Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer Orchestrate  A flow is a sequence of one or more steps which execute sequentially, but as a single unit. 29 Sunday, September 22, 13 © 2013 IBM Corporation
  86. 86. Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer Orchestrate  A flow is a sequence of one or more steps which execute sequentially, but as a single unit.  A Split is a collection of flows that may execute concurrently – A split may only contain “flows”; a step is not implicitly a flow 29 Sunday, September 22, 13 © 2013 IBM Corporation
  87. 87. Parallel Job Processing  Splits and Flows provide a mechanism for executing job steps concurrently at the orchestration layer Orchestrate  A flow is a sequence of one or more steps which execute sequentially, but as a single unit.  A Split is a collection of flows that may execute concurrently – A split may only contain “flows”; a step is not implicitly a flow  This is done entirely in the JSL descriptor – Imposed on the batch application with no code changes! 29 Sunday, September 22, 13 © 2013 IBM Corporation
  88. 88. Parallel Job Processing Implement 30 Sunday, September 22, 13 © 2013 IBM Corporation
  89. 89. Parallel Job Processing  Step-level parallelism can be achieved programmatically using step partitioning 30 Sunday, September 22, 13 Implement © 2013 IBM Corporation
  90. 90. Parallel Job Processing  Step-level parallelism can be achieved programmatically using step partitioning Implement  A partitioned step runs as multiple instances with distinct property sets 30 Sunday, September 22, 13 © 2013 IBM Corporation
  91. 91. Parallel Job Processing  Step-level parallelism can be achieved programmatically using step partitioning Implement  A partitioned step runs as multiple instances with distinct property sets  PartitionMapper defines the number of partitions, and property values for each partition – Can be a fixed set of partitions in JSL – Can be dynamic using a PartitionMapper implementation 30 Sunday, September 22, 13 © 2013 IBM Corporation
  92. 92. Parallel Job Processing 31 Sunday, September 22, 13 © 2013 IBM Corporation
  93. 93. Wrap up... © 2013 IBM Corporation Sunday, September 22, 13
  94. 94. Batch Processing  The oldest “new thing” in Java  JSR 352 applies the modern thinking and abstraction of Java EE and SOA and applies it to sequential batch processing  The standardized programming model provides application developers vendor portability  Inclusion in Java EE 7 ensures wide spread availability 33 Sunday, September 22, 13 © 2013 IBM Corporation
  95. 95. Questions? Find this presentation and more! http://ibm.co/JavaOne2013 © 2013 IBM Corporation Sunday, September 22, 13

×