Introduction to
Spring Integration and Spring Batch
© 2012 SpringOne 2GX. All rights reserved. Do not distribute without p...
What we will cover...
• Spring Integration
• Spring Batch
• Using Spring Batch and Spring Integration together
2
Tuesday, ...
Spring Integration
3
Tuesday, December 18, 12
Integration Styles
• Business to Business Integration (B2B)
• Inter Application Integration (EAI)
• Intra Application Inte...
Integration Styles
• File Transfer
• Shared Database
• Remoting
• Messaging
5
Tuesday, December 18, 12
Common Patterns
6
Retrieve Parse Transform Transmit
Tuesday, December 18, 12
Enterprise Integration Patterns
• By Gregor Hohpe & Bobby Woolf
• Published 2003
• Collection of well-known patterns
• htt...
8
“Spring Integration provides an extension
of the Spring programming model
to support the well-known enterprise
integrati...
What is Spring Integration?
• Light-weight messaging framework
• Provides an adapter-based platform for communicating with...
Advantages
• Provides building blocks to implement systems that are:
– are loosely Coupled (Logically or Physically)
– are...
Configuration
• XML Namespace Support
• Annotation Support (e.g. @Transformer, @Router, @ServiceActivator)
11
<beans xmlns...
Spring Integration DSLs
• Scala
• Groovy
12
val messageFlow =
filter {payload: String => payload == "World"} -->
transform...
What is in a Message?
• Unit of information
• Encapsulates data
• Passed between endpoints
• Consists of headers
– contain...
What is in a Message?
14
package org.springframework.integration;
public interface Message<T> {
MessageHeaders getHeaders(...
Message Headers
• Message ID (automatically generated UUID)
• Timestamp
• Correlation Id
• Reply Channel
• Error Channel
•...
Function of a Message
• Command Message
• Event Message
• Document Message
16
E
D
C
Tuesday, December 18, 12
What is a Channel?
• Channels connect producers and consumers (decoupling)
• MessageChannel Interface:
–PollableChannel (P...
What is an Endpoint?
• Polling or event-driven
• Inbound or outbound
• Unidirectional (Channel Adapter) or bidirectional (...
Router
• Message Router
• Content-based router
• Recipient list router (with selectors)
• Payload type router
• Header val...
Transformer
• Delegating via ref/method
• Spring Expression Language
• Groovy, JRuby, Jython, JavaScript
• Object-to-JSON ...
Spring Integration Components
21
• Claim Check (In/Out)
• Content Enricher
–Header Enricher
–Payload Enricher
• Control Bu...
Adapters
22
• AMQP/RabbitMQ
• AWS*
• File/Resource
• FTP/FTPS/SFTP
• GemFire
• HTTP (REST)
• JDBC
• JMS
• JMX
• JPA
• Mong...
Tooling - Spring Tool Suite (STS)
• Namespace Support
• Visualization
• 4 Spring Integration specific STS Templates
–Simpl...
Samples
• https://github.com/SpringSource/spring-integration-samples
• Contains 50 Samples and Applications
• Several Cate...
Books
• Just Spring Integration
• Pro Spring Integration
• Spring Integration in Action
25
Tuesday, December 18, 12
What’s new in Spring Integration 2.2
• JPA support
– http://blog.springsource.org/2012/10/05/whats-new-in-spring-integrati...
Source Code
• https://github.com/SpringSource/spring-integration
• https://github.com/SpringSource/spring-integration-samp...
Contribute
• Post Question and Answers the Forums
–http://forum.springsource.org/forumdisplay.php?42-Integration
• Create ...
Spring Batch
29
Tuesday, December 18, 12
30
Batch Jobs
Differ from online/real-time processing applications:
• Long-running
– Often outside office hours
• Non-inte...
Batch and offline processing
• Close of business processing
– Order processing
– Business reporting
– Account reconciliati...
Job and Step
32
Tuesday, December 18, 12
Chunk-Oriented Processing
• Input-output can be grouped together
• Input collects Items before outputting:Chunk-Oriented P...
JobLauncher
34
Tuesday, December 18, 12
Simple File Load Job
35
Tuesday, December 18, 12
More Complex Use Cases
• It's very common to use an off-the-shelf reader
and writer
• More complex jobs often require cust...
Job and Step in Context
37
Tuesday, December 18, 12
JobRepository and Batch Metadata
38
Tuesday, December 18, 12
ExecutionContext
• We need to know where a failure occurred to restart a batch
process
• Job Repository metadata is used t...
Common Batch Idioms
• Batch jobs typically process large amounts of homogeneous
input
• Makes iteration a common concern: ...
Spring Batch
• Spring Batch supports these common concerns
• Abstracts them in the framework
– Job business logic doesn't ...
Business Logic Delegation – Spring Application
42
Tuesday, December 18, 12
Spring Batch Admin
• Sub project of Spring Batch
• Provides Web UI and ReSTFul interface to manage batch
processes
• Manag...
Scaling and Parallel Processing
• First Rule:
– Use the simplest technique to get the job done in the required
time
– Do n...
Using Spring Batch
and
Spring Integration
Together
45
Tuesday, December 18, 12
Launching batch jobs through messages
• Event-Driven execution of the JobLauncher
• Spring Integration retrieves the data ...
Providing feedback with informational messages
• Spring Batch provides support for listeners:
– StepListener
– ChunkListen...
Externalizing batch process execution
• Use Spring Integration inside of Batch jobs, e.g.:
– ItemProcessor
– ItemWriter
• ...
Business Logic Delegation – Spring Integration
49
Tuesday, December 18, 12
•
Questions
50
Questions?
Thank You!!
Tuesday, December 18, 12
Upcoming SlideShare
Loading in …5
×

Russell 2012 introduction to spring integration and spring batch

2,658 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,658
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
72
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Russell 2012 introduction to spring integration and spring batch

  1. 1. Introduction to Spring Integration and Spring Batch © 2012 SpringOne 2GX. All rights reserved. Do not distribute without permission. Gary Russell, Staff Engineer, SpringSource; @gprussell Tuesday, December 18, 12
  2. 2. What we will cover... • Spring Integration • Spring Batch • Using Spring Batch and Spring Integration together 2 Tuesday, December 18, 12
  3. 3. Spring Integration 3 Tuesday, December 18, 12
  4. 4. Integration Styles • Business to Business Integration (B2B) • Inter Application Integration (EAI) • Intra Application Integration 4 JVM JVM EAI External Business Partner B2B Core Messaging Tuesday, December 18, 12
  5. 5. Integration Styles • File Transfer • Shared Database • Remoting • Messaging 5 Tuesday, December 18, 12
  6. 6. Common Patterns 6 Retrieve Parse Transform Transmit Tuesday, December 18, 12
  7. 7. Enterprise Integration Patterns • By Gregor Hohpe & Bobby Woolf • Published 2003 • Collection of well-known patterns • http://www.eaipatterns.com/eaipatterns.html 7 Tuesday, December 18, 12
  8. 8. 8 “Spring Integration provides an extension of the Spring programming model to support the well-known enterprise integration patterns. Tuesday, December 18, 12
  9. 9. What is Spring Integration? • Light-weight messaging framework • Provides an adapter-based platform for communicating with external systems • Pipes and Filters at the core of Spring Integration’s architecture –Endpoint (Filter) –Channel (Pipe) –Message 9 Tuesday, December 18, 12
  10. 10. Advantages • Provides building blocks to implement systems that are: – are loosely Coupled (Logically or Physically) – are Event Driven (EDA) – have a staged event-driven architecture (SEDA) • Sophisticated support for synchronous / asynchronous messaging 10 Tuesday, December 18, 12
  11. 11. Configuration • XML Namespace Support • Annotation Support (e.g. @Transformer, @Router, @ServiceActivator) 11 <beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:int="http://www.springframework.org/schema/integration" xsi:schemaLocation=".."> ... <int:channel id="orders"/> <int:splitter input-channel="orders" expression="payload.items" output-channel="drinks"/> <int:channel id="drinks"/> <int:router input-channel="drinks" expression="payload.iced ? 'coldDrinks' : 'hotDrinks'"/> ... </beans> Tuesday, December 18, 12
  12. 12. Spring Integration DSLs • Scala • Groovy 12 val messageFlow = filter {payload: String => payload == "World"} --> transform { payload: String => "Hello " + payload} --> handle { payload: String => println(payload) } messageFlow.send("World") def builder = new IntegrationBuilder() def flow = builder.messageFlow { transform {payload->payload.toUpperCase()} filter {payload-> payload =="HELLO"} handle {payload->payload} } assert flow.sendAndReceive("hello") == "HELLO" assert flow.sendAndReceive("world") == null Tuesday, December 18, 12
  13. 13. What is in a Message? • Unit of information • Encapsulates data • Passed between endpoints • Consists of headers – contains data relevant to the messaging system • and a payload – actual data for the receiver – depending on use-cases: POJO instances or serialized data 13 Tuesday, December 18, 12
  14. 14. What is in a Message? 14 package org.springframework.integration; public interface Message<T> { MessageHeaders getHeaders(); T getPayload(); } Tuesday, December 18, 12
  15. 15. Message Headers • Message ID (automatically generated UUID) • Timestamp • Correlation Id • Reply Channel • Error Channel • Expiration Date • Priority • ... • Add your own headers using a Header Enricher 15 Tuesday, December 18, 12
  16. 16. Function of a Message • Command Message • Event Message • Document Message 16 E D C Tuesday, December 18, 12
  17. 17. What is a Channel? • Channels connect producers and consumers (decoupling) • MessageChannel Interface: –PollableChannel (Polling Consumer) –SubscribableChannel (Event Driven) • Implementations: –DirectChannel –PublishSubscribeChannel –QueueChannel –PriorityChannel –RendezvousChannel –ExecutorChannel 17 <int:channel id="input"> <int:queue capacity="10"/> </int:channel> Tuesday, December 18, 12
  18. 18. What is an Endpoint? • Polling or event-driven • Inbound or outbound • Unidirectional (Channel Adapter) or bidirectional (Gateway) • Internal or external (application context) 18 <inbound-channel-adapter/> <outbound-channel-adapter/> <inbound-gateway/> <outbound-gateway/> <gateway/> <service-activator/> Tuesday, December 18, 12
  19. 19. Router • Message Router • Content-based router • Recipient list router (with selectors) • Payload type router • Header value router • Exception type router 19 Tuesday, December 18, 12
  20. 20. Transformer • Delegating via ref/method • Spring Expression Language • Groovy, JRuby, Jython, JavaScript • Object-to-JSON / JSON-to-Object • Payload serializing/deserializing • File-to-bytes, File-to-String • JAXB, JibX, Castor, XMLBeans, Xstream • XPath, XSLT • Object XML Marshalling/Unmarshalling (Spring OXM) • ... 20 Tuesday, December 18, 12
  21. 21. Spring Integration Components 21 • Claim Check (In/Out) • Content Enricher –Header Enricher –Payload Enricher • Control Bus • Delayer • JMX Support • Message Handler Chain • Messaging Bridge • Resequencer • Service Activator • Scripting support (JSR 223) – Ruby/JRuby, Javascript ... • Groovy • Message History • Message Store – JDBC, Redis, MongoDB, Gemfire • Wire Tap • ... Tuesday, December 18, 12
  22. 22. Adapters 22 • AMQP/RabbitMQ • AWS* • File/Resource • FTP/FTPS/SFTP • GemFire • HTTP (REST) • JDBC • JMS • JMX • JPA • MongoDB • POP3/IMAP/SMTP • Print* • Redis • RMI • RSS/Atom • SMB* • Splunk* • Spring Application Events • Stored Procedures • TCP/UDP • Twitter • Web Services (SOAP or POX) • XMPP • XPath • XQuery* • ... * Spring Integration Extensions Project Tuesday, December 18, 12
  23. 23. Tooling - Spring Tool Suite (STS) • Namespace Support • Visualization • 4 Spring Integration specific STS Templates –Simple Template (Core Components only) –File Polling Template (File Adapter) –War Template (Uses Twitter Adapter) –Adapter Template (Create your own components) 23 Tuesday, December 18, 12
  24. 24. Samples • https://github.com/SpringSource/spring-integration-samples • Contains 50 Samples and Applications • Several Categories: –Basic –Intermediate –Advanced –Applications 24 Tuesday, December 18, 12
  25. 25. Books • Just Spring Integration • Pro Spring Integration • Spring Integration in Action 25 Tuesday, December 18, 12
  26. 26. What’s new in Spring Integration 2.2 • JPA support – http://blog.springsource.org/2012/10/05/whats-new-in-spring-integration-2-2-part-3-jpa-support/ • MongoDB adapters – http://blog.springsource.org/2012/09/24/whats-new-in-spring-integration-2-2-rc1-part-1-mongodb/ • Transaction synchronization – http://blog.springsource.org/2012/09/26/whats-new-in-spring-integration-2-2-part-2-transaction- synchronization/ • Retry, Circuit Breaker and ExpressionEvaluatingRequestHandlerAdvice – http://blog.springsource.org/2012/10/09/spring-integration-2-2-retry-and-more/ • Redis Store Adapters 26 Tuesday, December 18, 12
  27. 27. Source Code • https://github.com/SpringSource/spring-integration • https://github.com/SpringSource/spring-integration-samples • https://github.com/SpringSource/spring-integration-extensions • https://github.com/SpringSource/spring-integration-templates • https://github.com/SpringSource/spring-integration-dsl-groovy • https://github.com/SpringSource/spring-integration-dsl-scala 27 Tuesday, December 18, 12
  28. 28. Contribute • Post Question and Answers the Forums –http://forum.springsource.org/forumdisplay.php?42-Integration • Create Jiras –https://jira.springsource.org/browse/INT • Submit Pull Requests - Contributor Guidelines: – github.com/SpringSource/spring-integration/wiki/Contributor-Guidelines • New Spring Integration Extensions Repository 28 Tuesday, December 18, 12
  29. 29. Spring Batch 29 Tuesday, December 18, 12
  30. 30. 30 Batch Jobs Differ from online/real-time processing applications: • Long-running – Often outside office hours • Non-interactive – Often include logic for handling errors or restarts • Process large volumes of data – More than fits in memory or a single transaction Tuesday, December 18, 12
  31. 31. Batch and offline processing • Close of business processing – Order processing – Business reporting – Account reconciliation • Import/export handling – a.k.a. ETL jobs (Extract-Transform-Load) – Instrument/position import – Data warehouse synchronization • Large-scale output jobs – Loyalty scheme emails – Bank statements • 31 Tuesday, December 18, 12
  32. 32. Job and Step 32 Tuesday, December 18, 12
  33. 33. Chunk-Oriented Processing • Input-output can be grouped together • Input collects Items before outputting:Chunk-Oriented Processing • Optional ItemProcessor 33 Tuesday, December 18, 12
  34. 34. JobLauncher 34 Tuesday, December 18, 12
  35. 35. Simple File Load Job 35 Tuesday, December 18, 12
  36. 36. More Complex Use Cases • It's very common to use an off-the-shelf reader and writer • More complex jobs often require custom readers or writers • ItemProcessor is often used if there's a need to delegate to existing business logic • Use a writer if it's more efficient to process a complete chunk 36 Tuesday, December 18, 12
  37. 37. Job and Step in Context 37 Tuesday, December 18, 12
  38. 38. JobRepository and Batch Metadata 38 Tuesday, December 18, 12
  39. 39. ExecutionContext • We need to know where a failure occurred to restart a batch process • Job Repository metadata is used to determine the step at which the failure occurred • Application Code (in reader/writer) needs to maintain state within a step (e.g. current chunk) • Spring Batch can supply that data during restart to facilitate repositioning 39 Tuesday, December 18, 12
  40. 40. Common Batch Idioms • Batch jobs typically process large amounts of homogeneous input • Makes iteration a common concern: Repeat • Transient errors during processing may require a Retry of an input item • Some input may not be valid, may want to Skip it without failing • Some errors should fail the job execution, allowing one to fix the problem and Restart the job instance where it left off 40 Tuesday, December 18, 12
  41. 41. Spring Batch • Spring Batch supports these common concerns • Abstracts them in the framework – Job business logic doesn't need to care about details • Allows for simple configuration with pluggable strategies 41 Tuesday, December 18, 12
  42. 42. Business Logic Delegation – Spring Application 42 Tuesday, December 18, 12
  43. 43. Spring Batch Admin • Sub project of Spring Batch • Provides Web UI and ReSTFul interface to manage batch processes • Manager, Resources, Sample WAR – Deployed with batch job(s) as single app to be able to control & monitor jobs – Or monitors external jobs only via shared database 43 Tuesday, December 18, 12
  44. 44. Scaling and Parallel Processing • First Rule: – Use the simplest technique to get the job done in the required time – Do not optimize/parallelize unnecessarily • Options: – Multi-threaded Step (single process) – Parallel Steps (single process) – Remote Chunking of Step (multi process) – Partitioning a Step (single or multi process) 44 Tuesday, December 18, 12
  45. 45. Using Spring Batch and Spring Integration Together 45 Tuesday, December 18, 12
  46. 46. Launching batch jobs through messages • Event-Driven execution of the JobLauncher • Spring Integration retrieves the data (e.g. file system, FTP, ...) • Easy to support separate input sources simultaneously 46 D C FTP Inbound Channel Adapter JobLauncher Transformer File JobLaunchRequest Tuesday, December 18, 12
  47. 47. Providing feedback with informational messages • Spring Batch provides support for listeners: – StepListener – ChunkListener – JobExecutionListener 47 <batch:job  id="importPayments">        ...        <batch:listeners>                <batch:listener  ref="notificationExecutionsListener"/>        </batch:listeners> </batch:job> <int:gateway  id="notificationExecutionsListener"        service-­‐interface="o.s.batch.core.JobExecutionListener"        default-­‐request-­‐channel="jobExecutions"/> Tuesday, December 18, 12
  48. 48. Externalizing batch process execution • Use Spring Integration inside of Batch jobs, e.g.: – ItemProcessor – ItemWriter • Offload complex processing • Asynchronous processing support: – AsyncItemProcessor – AsyncItemWriter • Externalize chunk processing using ChunkMessageChannelItemWriter 48 Tuesday, December 18, 12
  49. 49. Business Logic Delegation – Spring Integration 49 Tuesday, December 18, 12
  50. 50. • Questions 50 Questions? Thank You!! Tuesday, December 18, 12

×