SlideShare a Scribd company logo
Integration Pattern
Splitter and Aggregators
Introduction
• Messages passing through an integration solution consist of multiple elements.
• E.g. an order placed by a customer typically consists of more than just a single items like Address, Product
details , Mode Of Payment etc.
• We should process a complete order but treat each order item contained in the order
individually
• How can we process a message having multiple elements, each of which may have to
be processed in a different way?
• Publish Subscriber Model ????
Page 2
Publish Subscriber an alternative to Splitter ?
• Publish a message having Multiple Orders to All Subscribers
• Each Subscriber would process the required/supported Order and discard other orders
• Inefficient
• Each subscriber needs to implement logic to divide whole message and process supported one
• Increase Network Traffic
• Implement extra logic for lost / duplicate message handling
Page 3
Splitter
• Use a Splitter to break out the composite message into a series of individual messages, each containing
data related to one item.
• To avoid sending the complete message multiple times
• Split them into multiple
• Each message would then contain only one items
• The Splitter publishes one message for each single element
Page 4
Splitter -Handling Common Elements
• In many cases, we repeat some common elements in each resulting message
• These extra elements might be required to make the resulting child message self-contained and enable stateless
processing of each child message
• It also allows reconciliation of associated child messages later on
• Common Element can be Some ID or Timestamp
Page 5
Iterative Splitter
• When Data in Message is stored in a Data Structure like Lists , Tree Or Map
• Then Splitter iterates over each records/Node and constructs the output
• Generic Splitter can be worked with any number of child elements
• Also Called Sequencer Splitter
Page 6
Static Splitter
• Customized Splitter
• Used to split a large message into smaller messages based on some criteria e.g.
subtype
• Usually it splits into Fixed number of messages
• The strategy of splitting is fixed hence its called static Splitter
Page 7
Splitter -Input Message
• <order>
<date>7/18/2002</date>
<ordernumber>3825968</ordernumber>
<customer>
<id>12345</id>
<name>Joe Doe</name>
</customer>
<orderitems>
<item>
<quantity>3.0</quantity>
<itemno>W1234</itemno>
<description>A Widget</description>
</item>
<item>
<quantity>2.0</quantity>
<itemno>G2345</itemno>
<description>A Gadget</description>
</item>
</orderitems>
</order>
Page 8
Splitter – Output Messages
• <orderitem>
<date>7/18/2002</date>
<ordernumber>3825968</ordernumber>
<customerid>12345</customerid>
<quantity>3.0</quantity>
<itemno>W1234</itemno>
<description>A Widget</description>
</orderitem>
<orderitem>
<date>7/18/2002</date>
<ordernumber>3825968</ordernumber>
<customerid>12345</customerid>
<quantity>2.0</quantity>
<itemno>G2345</itemno>
<description>A Gadget</description>
</orderitem>
Page 9
Aggregator
• Splitter splits single message into sequence of individual messages
• How can we combine results of individual but related messages ?
• Asynchronous behavior
• Unknown number of recipients
• Response of individual message may be out of sequence
• How long should we wait ?
• Wait for too long ?
• Set timeout
• What should we do when response of few message is pending
• Should we send incomplete information ?
Page
10
Aggregator
• A separate component which sends a single message Once all individual message response is
ready
• Its called Aggregator because it collects individual messages until it receives a complete set and
then it combines and sends a single message
• Special filter
• which receives stream of messages It identifies the related messages . Once a complete set is arrived it sends
to output channel
Page
11
• Stateful component
• It needs to store all incoming messages(some information)
• Design
1. Correlation: Which incoming messages belong together
 ID, Timestamp etc
2. Completeness Condition: When are we ready to publish the result message?
3. Aggregation Algorithm: How do we combine the received messages into a single result
message ?
Page
12
Aggregator
Aggregator Implementation
Page
13
Message
First
Message
?
Aggregator
Yes
Message
Correlatio
n exists
No
Aggregator
No
Create
Aggregator and
add message
Add Message to
Existing
Aggregator
Create
Aggregator and
add message
Complete
Set
Received
Yes
Aggregate and
Construct
Single Message
No
Wait till all messages received
System
Send
message
References
• Enterprise Integration Patterns a Book by Gregor Hohpe

More Related Content

Similar to Integration Pattern Splitter and Aggregators

Mastery Webinar Series: Strategies to Simplify Email Data Management
Mastery Webinar Series: Strategies to Simplify Email Data ManagementMastery Webinar Series: Strategies to Simplify Email Data Management
Mastery Webinar Series: Strategies to Simplify Email Data Management
Vaultastic
 
FOSSASIA 2016 - 7 Tips to design web centric high-performance applications
FOSSASIA 2016 - 7 Tips to design web centric high-performance applicationsFOSSASIA 2016 - 7 Tips to design web centric high-performance applications
FOSSASIA 2016 - 7 Tips to design web centric high-performance applications
Ashnikbiz
 
Publish Subscriber messaging pattern
Publish Subscriber messaging patternPublish Subscriber messaging pattern
Publish Subscriber messaging pattern
Shirish Bari
 
SOA Pattern-Asynchronous Queuing
SOA Pattern-Asynchronous QueuingSOA Pattern-Asynchronous Queuing
SOA Pattern-Asynchronous Queuing
WSO2
 
Building an Event Bus at Scale
Building an Event Bus at ScaleBuilding an Event Bus at Scale
Building an Event Bus at Scale
jimriecken
 
M split
M splitM split
Object Oriented Design
Object Oriented Design Object Oriented Design
Object Oriented Design
DivyaSure
 
5 email data management strategies and choosing the right one
5 email data management strategies and choosing the right one5 email data management strategies and choosing the right one
5 email data management strategies and choosing the right one
Vaultastic
 
“Controlling of messages flow in Microservices architecture” by Andris Lubans...
“Controlling of messages flow in Microservices architecture” by Andris Lubans...“Controlling of messages flow in Microservices architecture” by Andris Lubans...
“Controlling of messages flow in Microservices architecture” by Andris Lubans...
DevClub_lv
 
test
testtest
test
techweb08
 
test
testtest
test
techweb08
 
test
testtest
test
techweb08
 
test
testtest
test
techweb08
 
test
testtest
test
techweb08
 
test
testtest
test
techweb08
 
test
testtest
test
techweb08
 
test
testtest
test
techweb08
 
Apache kafka- Onkar Kadam
Apache kafka- Onkar KadamApache kafka- Onkar Kadam
Apache kafka- Onkar Kadam
Onkar Kadam
 
IBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
IBM MQ: An Introduction to Using and Developing with MQ Publish/SubscribeIBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
IBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
David Ware
 
Redis Day TLV 2018 - My Other Car is a Redis
Redis Day TLV 2018 - My Other Car is a RedisRedis Day TLV 2018 - My Other Car is a Redis
Redis Day TLV 2018 - My Other Car is a Redis
Redis Labs
 

Similar to Integration Pattern Splitter and Aggregators (20)

Mastery Webinar Series: Strategies to Simplify Email Data Management
Mastery Webinar Series: Strategies to Simplify Email Data ManagementMastery Webinar Series: Strategies to Simplify Email Data Management
Mastery Webinar Series: Strategies to Simplify Email Data Management
 
FOSSASIA 2016 - 7 Tips to design web centric high-performance applications
FOSSASIA 2016 - 7 Tips to design web centric high-performance applicationsFOSSASIA 2016 - 7 Tips to design web centric high-performance applications
FOSSASIA 2016 - 7 Tips to design web centric high-performance applications
 
Publish Subscriber messaging pattern
Publish Subscriber messaging patternPublish Subscriber messaging pattern
Publish Subscriber messaging pattern
 
SOA Pattern-Asynchronous Queuing
SOA Pattern-Asynchronous QueuingSOA Pattern-Asynchronous Queuing
SOA Pattern-Asynchronous Queuing
 
Building an Event Bus at Scale
Building an Event Bus at ScaleBuilding an Event Bus at Scale
Building an Event Bus at Scale
 
M split
M splitM split
M split
 
Object Oriented Design
Object Oriented Design Object Oriented Design
Object Oriented Design
 
5 email data management strategies and choosing the right one
5 email data management strategies and choosing the right one5 email data management strategies and choosing the right one
5 email data management strategies and choosing the right one
 
“Controlling of messages flow in Microservices architecture” by Andris Lubans...
“Controlling of messages flow in Microservices architecture” by Andris Lubans...“Controlling of messages flow in Microservices architecture” by Andris Lubans...
“Controlling of messages flow in Microservices architecture” by Andris Lubans...
 
test
testtest
test
 
test
testtest
test
 
test
testtest
test
 
test
testtest
test
 
test
testtest
test
 
test
testtest
test
 
test
testtest
test
 
test
testtest
test
 
Apache kafka- Onkar Kadam
Apache kafka- Onkar KadamApache kafka- Onkar Kadam
Apache kafka- Onkar Kadam
 
IBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
IBM MQ: An Introduction to Using and Developing with MQ Publish/SubscribeIBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
IBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
 
Redis Day TLV 2018 - My Other Car is a Redis
Redis Day TLV 2018 - My Other Car is a RedisRedis Day TLV 2018 - My Other Car is a Redis
Redis Day TLV 2018 - My Other Car is a Redis
 

Recently uploaded

Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
Globus
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
wottaspaceseo
 
SOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FME
Jelle | Nordend
 
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Globus
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
Globus
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
Max Andersen
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
takuyayamamoto1800
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
AMB-Review
 
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Anthony Dahanne
 
Advanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should KnowAdvanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should Know
Peter Caitens
 
2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx
Georgi Kodinov
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus
 
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Hivelance Technology
 
Corporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMSCorporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMS
Tendenci - The Open Source AMS (Association Management Software)
 
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
informapgpstrackings
 
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.ILBeyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
Natan Silnitsky
 
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume MontevideoVitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke
 
Cyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdfCyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdf
Cyanic lab
 

Recently uploaded (20)

Enhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdfEnhancing Research Orchestration Capabilities at ORNL.pdf
Enhancing Research Orchestration Capabilities at ORNL.pdf
 
How Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptxHow Recreation Management Software Can Streamline Your Operations.pptx
How Recreation Management Software Can Streamline Your Operations.pptx
 
SOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar Research Team: Latest Activities of IntelBroker
SOCRadar Research Team: Latest Activities of IntelBroker
 
De mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FMEDe mooiste recreatieve routes ontdekken met RouteYou en FME
De mooiste recreatieve routes ontdekken met RouteYou en FME
 
Prosigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns: Transforming Business with Tailored Technology Solutions
Prosigns: Transforming Business with Tailored Technology Solutions
 
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
Exploring Innovations in Data Repository Solutions - Insights from the U.S. G...
 
GlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote sessionGlobusWorld 2024 Opening Keynote session
GlobusWorld 2024 Opening Keynote session
 
Quarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden ExtensionsQuarkus Hidden and Forbidden Extensions
Quarkus Hidden and Forbidden Extensions
 
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamOpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoam
 
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdfDominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
Dominate Social Media with TubeTrivia AI’s Addictive Quiz Videos.pdf
 
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...
 
Advanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should KnowAdvanced Flow Concepts Every Developer Should Know
Advanced Flow Concepts Every Developer Should Know
 
2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx2024 RoOUG Security model for the cloud.pptx
2024 RoOUG Security model for the cloud.pptx
 
Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus Compute wth IRI Workflows - GlobusWorld 2024
Globus Compute wth IRI Workflows - GlobusWorld 2024
 
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
Multiple Your Crypto Portfolio with the Innovative Features of Advanced Crypt...
 
Corporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMSCorporate Management | Session 3 of 3 | Tendenci AMS
Corporate Management | Session 3 of 3 | Tendenci AMS
 
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
Field Employee Tracking System| MiTrack App| Best Employee Tracking Solution|...
 
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.ILBeyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
Beyond Event Sourcing - Embracing CRUD for Wix Platform - Java.IL
 
Vitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume MontevideoVitthal Shirke Microservices Resume Montevideo
Vitthal Shirke Microservices Resume Montevideo
 
Cyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdfCyaniclab : Software Development Agency Portfolio.pdf
Cyaniclab : Software Development Agency Portfolio.pdf
 

Integration Pattern Splitter and Aggregators

  • 2. Introduction • Messages passing through an integration solution consist of multiple elements. • E.g. an order placed by a customer typically consists of more than just a single items like Address, Product details , Mode Of Payment etc. • We should process a complete order but treat each order item contained in the order individually • How can we process a message having multiple elements, each of which may have to be processed in a different way? • Publish Subscriber Model ???? Page 2
  • 3. Publish Subscriber an alternative to Splitter ? • Publish a message having Multiple Orders to All Subscribers • Each Subscriber would process the required/supported Order and discard other orders • Inefficient • Each subscriber needs to implement logic to divide whole message and process supported one • Increase Network Traffic • Implement extra logic for lost / duplicate message handling Page 3
  • 4. Splitter • Use a Splitter to break out the composite message into a series of individual messages, each containing data related to one item. • To avoid sending the complete message multiple times • Split them into multiple • Each message would then contain only one items • The Splitter publishes one message for each single element Page 4
  • 5. Splitter -Handling Common Elements • In many cases, we repeat some common elements in each resulting message • These extra elements might be required to make the resulting child message self-contained and enable stateless processing of each child message • It also allows reconciliation of associated child messages later on • Common Element can be Some ID or Timestamp Page 5
  • 6. Iterative Splitter • When Data in Message is stored in a Data Structure like Lists , Tree Or Map • Then Splitter iterates over each records/Node and constructs the output • Generic Splitter can be worked with any number of child elements • Also Called Sequencer Splitter Page 6
  • 7. Static Splitter • Customized Splitter • Used to split a large message into smaller messages based on some criteria e.g. subtype • Usually it splits into Fixed number of messages • The strategy of splitting is fixed hence its called static Splitter Page 7
  • 8. Splitter -Input Message • <order> <date>7/18/2002</date> <ordernumber>3825968</ordernumber> <customer> <id>12345</id> <name>Joe Doe</name> </customer> <orderitems> <item> <quantity>3.0</quantity> <itemno>W1234</itemno> <description>A Widget</description> </item> <item> <quantity>2.0</quantity> <itemno>G2345</itemno> <description>A Gadget</description> </item> </orderitems> </order> Page 8
  • 9. Splitter – Output Messages • <orderitem> <date>7/18/2002</date> <ordernumber>3825968</ordernumber> <customerid>12345</customerid> <quantity>3.0</quantity> <itemno>W1234</itemno> <description>A Widget</description> </orderitem> <orderitem> <date>7/18/2002</date> <ordernumber>3825968</ordernumber> <customerid>12345</customerid> <quantity>2.0</quantity> <itemno>G2345</itemno> <description>A Gadget</description> </orderitem> Page 9
  • 10. Aggregator • Splitter splits single message into sequence of individual messages • How can we combine results of individual but related messages ? • Asynchronous behavior • Unknown number of recipients • Response of individual message may be out of sequence • How long should we wait ? • Wait for too long ? • Set timeout • What should we do when response of few message is pending • Should we send incomplete information ? Page 10
  • 11. Aggregator • A separate component which sends a single message Once all individual message response is ready • Its called Aggregator because it collects individual messages until it receives a complete set and then it combines and sends a single message • Special filter • which receives stream of messages It identifies the related messages . Once a complete set is arrived it sends to output channel Page 11
  • 12. • Stateful component • It needs to store all incoming messages(some information) • Design 1. Correlation: Which incoming messages belong together  ID, Timestamp etc 2. Completeness Condition: When are we ready to publish the result message? 3. Aggregation Algorithm: How do we combine the received messages into a single result message ? Page 12 Aggregator
  • 13. Aggregator Implementation Page 13 Message First Message ? Aggregator Yes Message Correlatio n exists No Aggregator No Create Aggregator and add message Add Message to Existing Aggregator Create Aggregator and add message Complete Set Received Yes Aggregate and Construct Single Message No Wait till all messages received System Send message
  • 14. References • Enterprise Integration Patterns a Book by Gregor Hohpe