Impact 2013 2971 - Fundamental integration and service patterns

© 2013 IBM Corporation
Fundamental Integration and Service Patterns
Kim Clark (kim.clark@uk.ibm.com), IBM UK
Brian Petrini (petrini@us.ibm.com), IBM US
TSS-2971

IBM Software Group
© 2013 IBM Corporation2
Objective
Improve consistency of integration implementations
§  Do you have a common language for integration
characteristics and patterns shared between the
architects, designers, and implementers?
§  Can you mechanically assess the patterns required for
integration and SOA in a given solution?
§  Where does orchestration fit within the layers of an
enterprise architecture?

IBM Software Group
Contents
§ Capturing interface complexity
§ Basic integration pattern evolution
§ What differentiates the SOA patterns
§ Categorizing orchestration

IBM Software Group
Example integration challenges from projects
§  The service must appear event driven, but
actually is a batch process on the back end
§  We can only make 10 requests per hour to the
back end, but can contain as many events in
a single request as you like.
§  Connection pooling is required but we want to
preserve the identity of the caller
§  The data format is “similar” to Comma
Separated Values
§  We must never call the back end system
when it is down. The availability of the
backend system can be determined by the
xxx function call.
§  We don’t know what the request is until we
inspect various aspects of the message
header and body. In some cases even the
data format is unknown at this stage.
§  Due to optimistic locking, the backend system
requires the exact timestamp/rowid that was
returned in the retrieve at the beginning of the
business process.
§  For testing, we need to be able to wire in our
(proprietary) test harness mechanism
§  We have written our own security mechanism
§  Any changes to code require a lengthy
validation process. We want everything
possible to be configurable.
§  The backend system is often not available.
Retries should happen automatically and
should not be considered an error condition.
§  We used powerful features of XML Schema in
definition of our object model.
§  We have our own integration framework. Most
of the work is done by a few common classes
and applications. New services are added via
configuration rather than code.
§  Due to acquisitions, customer accounts
are spread across multiple systems. We
need to provide a unified customer view
§  The adapter was missing one feature that we
needed, so we’ve decided to integrate using
custom code instead.
§  We only want to send the changes/deltas to
the back end system, not the whole original
message

IBM Software Group
Characterising the interface
Requester Provider
Interface Characteristics
Integrity
Security
Reliability
Error handling
Data
Technical interface
Interaction type
Performance
Capturing integration complexity for BPM and SOA solutions
http://www.ibm.com/developerworks/websphere/techjournal/1112_clark/1112_clark.html

IBM Software Group
Data
Principal data objects
Operation/function
Read or change
Request/response objects
Technical Interface
Transport
Protocol
Data format
Interaction type
Request-response or fire-forget
Thread-blocking or asynchronous
Batch or individual
Performance
Response times
Throughput
Volumes
Concurrency
Message size
Integrity
Validation
Transactionality
Statefullness
Event Sequence
Idempotence
Security
Identity/Authentication
Authorisation
Data Ownership
Privacy
Reliability
Availability
Delivery assurance
Error Handling
Error Management capabilities
Known exception conditions

IBM Software Group
Solution Context Diagram with Integration Detail
Process
OrderBilling
System
Data
Warehouse
Credit
Checking
Shipping
Product
Catalogue Order
Management
Printing
Order
(JDBC)
Customer/Credit Rating
(Web Service/HTTP)
Order Confirmation
(Web service/JMS)
Order Summary
(JDBC)
Customer/Order
(Flat file, batch)
Customer/Order
(Flat file, XML)
Product Details
(RMI/IIOP)
Order Details
(WebSphere MQ)
Customer
Relationship
Managemen
t
Customer
Relationship
Management

IBM Software Group
Resolving differences in interface characteristics
Integrity
Security
Reliability
Error handling
Data
Technical interface
Interaction type
Performance
Integrity
Security
Reliability
Error handling
Data
Technical interface
Interaction type
Performance
Integration
is about handling
the differences between
requester and provider
Integration Patterns

IBM Software Group
Evolving Integration Design Patterns – a sample
§  Data handling
§  Data model translation
§  Data synchronisation
§  Publish subscribe
§  Routing
§  Content based routing
§  Data enrichment
§  Protocol and transport translation
§  Large object management
§  Correlation
§  Assured delivery
§  Canonical data model
§  Relationships/cross referencing
§  Once only delivery
§  Scatter gather
§  Aggregation
§  Store and forward
§  Flow control/throttling
§  Service exposure
§  Failed event management
§  Idempotence resolution
§  Retry
§  Health check
§  Prioritization
§  Optimistic/pessimistic locking
§  Event sequencing
§  Verb conversion
§  Existence checking
§  Compensation
§  Batch processing (various)
§  Command controller (various)
§  Error handling (various)
§  Composition (various)
§  Orchestration (various)
§  …many more
The “EAI Patterns” book (http://www.eaipatterns.com/toc.html) covers the initial set of
primarily message based integration patterns, many of which have now been productised.
This list extends into the next level of complexity in integration patterns.

IBM Software Group
Contents

IBM Software Group
Point to point integration – connection spaghetti
Operational Systems
(Applications & Data)
Requesters

IBM Software Group
Capabilities required in data point to point integration
Adaptation
Connection Management
Application Programming Interface
Protocol conversion
Transport communication
Data Handling
Requester
System
Provider
System
Adaptatio
n code
Adaptatio
n code
Interaction Framework

IBM Software Group
11/14/13 Service and Integration Maturity
Hub and Spoke architecture with basic one to one integration
Operational
Systems
Requesters
Integration
Hub
Adapter Adapter Adapter
Adapter AdapterAdapter
Adapter
Adapter
One-to-one

IBM Software Group
Patterns/capabilities for hub-based basic one to one integration
Canonical Integration
Integration
Hub
Adapter Adapter
Adaptation
Protocol conversion
Data Handling
Data Translation

IBM Software Group
Interface Characteristics driving interaction styles
Data
Operation/function
Read or change
Technical Interface
Transport
Protocol
Data format
Interaction type
Batch or individual
Performance
Response times
Throughput
Volumes
Concurrency
Message size
Integrity
Validation
Transactionality
Statefullness
Event Sequence
Idempotence
Security
Authorisation
Data Ownership
Privacy
Reliability
Availability
Delivery assurance
Error Handling

IBM Software Group
Integration Hub
Hub and spoke interaction styles – basic one to one
Integration Hub
AdapterAdapter
AdapterAdapter
1) Non-blocking (message-based) transport
(shown using an open arrowhead)
2) Blocking (synchronous) transport
(shown using a closed arrowhead)

IBM Software Group
Basic Interaction Types
ProviderRequester
getCustomer(CustomerID)
returns Customer
ProviderRequester
submitOrder(Order)a)  Fire and forget,
non-blocking transport
ProviderRequester
returns Customer
b) Request-response,
blocking transport
c) Request-response,
A messaging transport is used. The request is “one-way” sending data to the provider, but with no response.
The requester need only wait for the message to be received by the transport – i.e. it is non-blocking.
A synchronous transport is used. The requester requires response data, and must wait for the provider to
respond. The provider must be available at the time of the request and must process the request
immediately.
A messaging transport is used. The requester need only wait for the message to be received by the transport.
It is then freed up to do other work. It must however provide a way in which it can receive messages from a
response queue at a later time and be able to correlate this with the original request.

IBM Software Group
Advanced Interaction Types
ProviderRequester
returns Customer
e) Request-response
blocking caller API,
ProviderRequester
Callback with Customer
f) Request-response by
call-back, using blocking
transport
ProviderRequester
submitOrder(OrderID)d) Request with
acknowledgement,
blocking transport returns Acknowledgement
A synchronous transport is used. The provider must be present to receive the request, but it provides only an
acknowledgement that the action will be performed at a later time. This is similar to fire-forget as the actual
business task required by the requester is performed after the acknowledgement is returned, but this is more
tightly coupled.
This is a hybrid between transport types. A messaging transport is used to communicate with the provider, but
the requestor interacts with the messaging using a synchronous API. Many messaging transports provide this
type of API to simplify requester interaction with the messaging layer. The provider can process the message at
its convenience but the requester is blocked waiting.
A synchronous transport (such as HTTP) is used to make two separate interactions for request and call-back.
Each interaction individually is blocking, but releases the block as soon as an acknowledgement is received.
This is similar to request response non-blocking but the requester and provider interact more directly.
returns Acknowledgement
returns Acknowledgement

IBM Software Group
11/14/13 Service and Integration Maturity
Hub and Spoke architecture
typical of Enterprise Application Integration (EAI)
Operational
Systems
Requesters
Integration
Hub
Adapter Adapter Adapter
Adapter
Adapter
One-to-many

IBM Software Group
Patterns/capabilities for basic multi-provider integration
Integration
Hub
Adapter
Adapter
Adapter
Adaptation
Protocol conversion
Data Handling
Data Translation
Routing/Distribution

IBM Software Group
Integration Hub
Integration Hub
Hub and spoke interaction styles – brokerage
Adapter
Adapter
Adapter
Adapter
Adapter
Adapter
OR
AND
a) Routing
b) Distribution

IBM Software Group
Patterns/capabilities for advanced multi-provider integration
Integration
Hub
Adapter
Adapter
Adapter
Adaptation
Protocol conversion
Data Handling
Data Translation
Orchestration

IBM Software Group
Hub and spoke interaction styles – orchestration
A selection of the possible types
Adapter
Adapter
Adapter
Read-only orchestration
Parallel aggregation
Updating orchestration
Stateful orchestration
Adapter
Adapter
Adapter
Adapter
Adapter
Adapter
Adapter
Adapter
Adapter

IBM Software Group
The importance of orchestration types
§  Orchestration is not new. It has been occurring in
applications and middleware solutions for decades.
§  Orchestration is becoming a critical architectural decision
point in large enterprises due to the increased need to
automate functionality distributed across an increasingly
larger number of enterprise systems.
§  Categorizing the different types of orchestration is critical
to ensure it:
– occurs in the right layers within the architecture
– is performed by products with the right capabilities
– is visible and changeable by the right audience

IBM Software Group
A high level “interaction pattern” notation for integration?
§  Provides a way to describe the
high level intent of an
integration solution without
being bound to technology
§  This is a subset of the overall
notation based on what we’ve
used in the presentation so far.
§  It is not a standardised
notation, but it does borrow
from notations that have come
before it.
(e.g. www.eaipatterns.com)
§  The notation would need to be
specialised to a given
environment.
Read-only
orchestration
Parallel
aggregation
Multi-invocation
Brokerage
Route Distribute
Updating
orchestration
Stateful
orchestration
Synchronous “request/reply interaction
One-way “fire/forget” interaction
Direct

IBM Software Group
Technology agnostic high-level design notation for integration
§  Learnable sub-set of the patterns the
technology should be used for.
§  Common language between solution
design and service design.
§  Highlights when we step beyond
regularly used capabilities
§  Maps to specific, documented
implementation standards
§  Enables pattern based design and
development
§  Downstream benefits in monitoring,
diagnostics and dependency analysis

IBM Software Group
Orchestration/Composition in the integration layer
Operational Systems
Integration
Requesters
Data
synchronization
Microflow
Stored procedure
Straight-through
processing

IBM Software Group
Contents

IBM Software Group
Service Oriented Architecture (SOA) Reference Architecture
https://collaboration.opengroup.org/projects/soa-ref-arch

IBM Software Group
Silo Services
Composite
Services
Virtualized
Services
Dynamically
Re-Configurable
ServicesComponentizedIntegrated
Level 1 Level 4 Level 5 Level 6 Level 7Level 3Level 2
Applications
Methods
Organization
Infrastructure
Architecture
Business View
Modules Services
Process
Integration via
Services
Dynamic
Application
Assembly
ComponentsObjects
Structured
Analysis &
Design
Service
Oriented
Modeling
Service
Oriented
Modeling
Grammar
Oriented
Modeling
Component
Based
Development
Object
Oriented
Modeling
Ad hoc IT
Governance
Emerging SOA
Governance
SOA and IT
Governance
Alignment
SOA and IT
Governance
Alignment
Ad hoc IT
Governance
Ad hoc IT
Governance
SOA and IT
Governance
Alignment
Service
Oriented
Modeling
Process
Integration
via Services
Platform
Specific
Platform
Specific
Platform
Neutral
Dynamic
Sense &
Respond
Platform
Specific
Platform
Specific
Monolithic
Architecture
Emerging
SOA
Grid Enabled SOA
Dynamically Re-
Configurable
Architecture
Component
Architecture
Layered
Architecture
SOA
Platform
Specific
Function
Oriented
Service
Oriented
Service
Oriented
Service
Oriented
Function
Oriented
Function
Oriented
Service
Oriented
Open Service Integration Maturity Model (OSIMM)
https://collaboration.opengroup.org/projects/osimm

IBM Software Group
Service Oriented Landscape
Introducing standardised service exposure
Operational
Systems
Requesters
Integration Hub
Service Exposure Gateway

IBM Software Group
Capabilities to components – service exposure
Integration
Hub
Adapter
Adapter
Adaptation
Protocol conversion
Data Handling
Data Translation
Orchestration
Service
Exposure
Gateway
Service Exposure
Traffic Management
Security
Virtualisation
Visibility

IBM Software Group
Contents

IBM Software Group
BPM in a Service Oriented Architecture
Operational Systems
Requesters
Integration Hub
Business Process
Manager
Service Exposure Gateway
So where should
we do orchestration?

IBM Software Group
What highlights a “business process”
in a business process management solution?
§  Performing the process provides value to the business
§  The process follows a relatively structured path
§  The process contains individually business relevant steps
§  The steps within the process are performed by multiple roles/teams.
§  Business relevant data flows through the process
§  The process changes over time as a result of changes in the business
Create
Account
Capture
Details
Send
Documents
Sign
Documents
Activate
Account
Customer
Call centre
Back office
Notify
Customer
New
revenue!
Existing customer
New customer
-- ------------------------------- -
-- ------------------------------- -
-- ------------------------------- -
-- ------------------------------- -
-- ------------------------------- -
-- ------------------------------- -

IBM Software Group
Orchestration/Composition
Where could it occur / Where should it occur?
Service Exposure
Operational Systems
Integration
Consumers
Business Process
Data
synchronization
Microflow
In-app.
workflow
Screenflow/
page navigation
Stored procedure
Straight-through
processing
Business process

IBM Software Group
Single Transaction
for all invocations
Multiple Transactions
across invocations
Human Tasks
Terminology: Orchestration – “Composition” vs. “Process”?
§  Appear to the business as a single
step.
§  Do not include human interaction from
the business.
Business ProcessComposition
§  Take long enough that the business
need to be aware of their intermediate
in-progress states.
§  Include human interaction, or
interaction with slow responding
systems.
Orchestration
(A sequence of “invocations” of systems, or humans)
“Process Implementation Types: Patterns based design for process based solutions”
http://www.ibm.com/developerworks/websphere/library/techarticles/1004_clark/1004_clark.html

IBM Software Group
Types of orchestration in relation to transactionality
Do any of the requests perform critical changes*?
No
Aggregation Enriched update
Yes
Can all critical changes* be
performed in one transaction?
State-free
Orchestration
Stateful
Orchestration
Yes No
How many critical changes*?
=1 >1
*Definition: “Critical Change”
Any request that performs a change
(create, update, delete) that MUST be
resolved in the case of a failure, either with
a transactional rollback, or with
compensatory actions if the update has
already been committed.
Non-transactional Isolated transaction Global transaction Multi-transaction

IBM Software Group
Types of Orchestration: Process vs. Composition
§  Process
– Makes calls to mature high level services
– Often triggered (i.e. one way call) rather than invoked as a two way call
– Where it is invoked as a two way interaction, the caller is typically asynchronous
(i.e. not a user) and therefore the service level agreement is throughput based
rather than response time based
– Stateful persistence of the steps in the process
– Events can correlate with the running process
– Often involves human interaction to perform some tasks within the process
§  Composition
– Grouping of relatively granular interactions
– Response time is the primary driver for the service level agreement
– Common for aggregation functions
– Some or all the granular interactions may not themselves be exposed as re-usable
services
– Generally state free
– Never involves human interaction during the composition

IBM Software Group
Integration orchestration vs. business orchestration
A “litmus test”
§  Imagine an ideal enterprise information system.
– It is a single system of record for all business data.
– All the data is normalized.
– It has appropriate APIs for every type of data access required.
– It is infinitely fast and completely reliable.
§  In this ideal system, if you find yourself doing multiple
actions, you are performing business orchestration.
§  Any orchestration done to manage the fact that the real
systems are not as perfect as the ideal system above, is
integration orchestration.

IBM Software Group
Interface Characteristics that drive functional orchestration
Data
Operation/function
Read or change
Technical Interface
Transport
Protocol
Data format
Interaction type
Batch or individual
Performance
Response times
Throughput
Volumes
Concurrency
Message size
Integrity
Validation
Transactionality
Statefullness
Event Sequence
Idempotence
Security
Authorisation
Data Ownership
Privacy
Reliability
Availability
Delivery assurance
Error Handling

IBM Software Group
Common functional orchestration patterns
§  Data synchronization
§  Aggregation
§  Enriched updates
§  Straight through processing
§  Complex error handling
§  Compensation
§  …and an infinite number more

IBM Software Group
Interface Characteristics that drive infrastructural orchestration
Data
Operation/function
Read or change
Technical Interface
Transport
Protocol
Data format
Interaction type
Batch or individual
Performance
Response times
Throughput
Volumes
Concurrency
Message size
Integrity
Validation
Transactionality
Statefullness
Event Sequence
Idempotence
Security
Authorisation
Data Ownership
Privacy
Reliability
Availability
Delivery assurance
Error Handling

IBM Software Group
Common infrastructural orchestration patterns
§  Data Consistency
– Once only delivery
– Optimistic/pessimistic locking
– Event sequencing
– Idempotence resolution
§  Large object
– Claim check
– Move by command
– Response by reference
§  Reliability
– Failed event management
– Retry
– Health check
– Store and forward
§  Batch management
– Batch splitter
– Batch processing
– Batch aggregator
Infrastructural integration orchestration pattern are those that concern the completion of
one primary request to one system, and are therefore essentially functionally agnostic.
These patterns are more likely to be productized.

IBM Software Group
Summary of dimensions for categorising orchestration
§  Composition vs. process
–  Beware. Loose terms, used relatively inconsistently.
§  Human involvement vs. system only
–  Are humans involved in the happy path?
–  Workforce management capabilities are critical
§  Single transaction vs. multi-transaction
–  Consider the number of “critical changes” performed
–  Directs us to the correct implementation technology
§  Integration orchestration vs. business orchestration
–  Try the “litmus test” of whether you would need multiple requests on a perfect system?
–  Who owns the requirements? Who can change them?
§  Infrastructural orchestration vs. functional orchestration?
–  Is it only one primary function on one system?
–  Is there a product feature that provides this capability?

IBM Software Group
Interface Characteristics relating to cross cutting concerns
Data
Operation/function
Read or change
Technical Interface
Transport
Protocol
Data format
Interaction type
Batch or individual
Performance
Response times
Throughput
Volumes
Concurrency
Message size
Integrity
Validation
Transactionality
Statefullness
Event Sequence
Idempotence
Security
Authorisation
Data Ownership
Privacy
Reliability
Availability
Delivery assurance
Error Handling

IBM Software Group
Key messages
§  Interface characteristics provide a mechanical way to
assess the required integration patterns.
§  A standard integration vocabulary and notation improve
communication between architect, designer and
implementer.
§  Categorization of orchestration enables correct
placement within the architecture.
§  Formalization of integration improves consistency,
and velocity in implementation.

IBM Software Group
Thank You
Merci
Grazie
Gracias
Obrigado
Danke
Japanese
English
French
Russian
GermanItalian
Spanish
Brazilian Portuguese
Arabic
Traditional Chinese
Simplified Chinese
Thai

IBM Software Group
How would we resolve differences in Characteristics?
Data (Functionality)
<Provider><Requester>
Characteristic Example Conflict Resolution options
Principal data
objects
The interface does not respond with
all the required data
Data Aggregation, Data Enrichment: More than one
operation call may be required, and the resulting data
aggregated.
Request/ response
objects
The request/response objects are not
the same structure as the objects in
the requester
Mapping: The requesters data must be mapped to/from the
structure of the provider
Operation/ function The provider does not offer an
operation that performs the exact
action required by the requester.
Orchestration/Composition, or Command Pattern. A
combination of existing operations must be found that
provide the same functionality. If not, a new operation must
be made available by the provider, or else the interface is
untenable.
Differences in

IBM Software Group
Integrity (Functionality)
Validation The requester’s request, sometimes
provides data that is considered invalid to
the requester, or indeed whether the
response data is valid.
We will need to perform either Data Enrichment, or Data
Cleansing on the incoming data. Where the data cannot be
sufficiently enhanced, we should provide a meaningful data
validation error.
Transactionality The providers interface cannot be included
in a wider transaction, but the requester
needs to ensure consistency between this
invocation and others
Without transactionality, an option for consistency would be
Compensation. It may be sufficient to simply put in place
Delivery Assurance to ensure that the data is “eventually”
consistent.
Statefullness The request requires of data from a
previous call (such as a session ID, or lock
ID), but the requester does not have this
data.
The previous call could be combined with this one
(Orchestration/Command Pattern), or the required data could be
stored on the previous call and looked up for this one (Data
Enrichment). A locking strategy could be converted from ID
based to time based such that an id is no longer required.
Event
Sequence
The provide needs updates to be applied
in order, but the requester sometimes
sends the events out of sequence.
A grouping sequencer/re-sequencer Event Sequencing pattern
must be applied.
Idempotence The requester may send more than one
request through for the same event, but
this will produce duplicates in the provider.
The incoming requests must be de-duplicated such that
Idempotence is offered by the provider.
<Provider><Requester> Differences in

IBM Software Group
Security (Functionality)
Data Ownership The requester requesting an update, is not
allowed to make changes to all parts of
the incoming data model.
The incoming object should be thinned appropriately (Mapping)
such that only selective updates occur. It might be considered
an error if the data that the requester passes, but is not allowed
to change, is different to that found in the provider. In this case,
Orchestration/Composition would be required in order to perform
a retrieve or data prior to update.
Privacy Some of the data that the provider
responds with is restricted, and should not
be made available to the requester.
The restricted data must be removed from the response
(Mapping).

IBM Software Group
Reliability
Mean time
between failures
The provider’s availability is not sufficient
for the requesters needs
We could put in place Protocol and Transport Translation to an
asynchronous medium, and a/or a Re-Try mechanism to make
the provide appear more reliable.
Maintenance
windows
The provider is regularly shutdown during
a time when the requester is expected to
be making requests.
We will need a Store and Forward mechanism to ensure that the
requester can continue to make requests even when the system
is down. A Healthcheck would allow us to automatically know
when to re-submit stored events.
Delivery
Assurance
The requester needs to be confident that
once a request has been submitted, it will
be actioned as some point in the near
future, and it will never be lost.
On the simplest level, we need to ensure that the service is
provided over a reliable transport, which may therefore require
Protocol and Transport Translation.
However, it may be that the requester cannot talk over such a
transport, so a more complex conversational interaction may be
required, which may in include the use of Idempotence, Re-
Tries, Store and Forward, and even Publish/Subscribe to provide
acknowledgements/responses back to a decoupled requester.

IBM Software Group
Interaction Type (Usability)
Synchronous or
Asynchronous
Requester talks web services, but
provider exposes over MQ.
Simplistically this is just Protocol Translation. However, Synch-
>Async conversion comes with a number of complex side effects
in terms of Error Handling, and Idempotence, and Once Only
Delivery, to name but a few. See Transactionality lecture for
more details.
Request/Response
or Fire/Forget
The requester may want to receive a
specific acknowledgement confirming
receipt of the request. The provider
may be a one way interface.
This is the simplest possible form of Orchestration/Composition.
A more interesting Orchestration/Composition would be the
opposite case where the requester was fire/forget and the
provider Request/Response. Careful thought would be needed
to decide what Error Handling to do with failure responses and
expirations.
Batch or Real-time a)  A batch file is received, but
individual events are expected
by the backend system
b)  Individual events are received,
but a batch file must be
delivered to the provider
a)  Some form of Batch Splitter pattern is required.
b)  Some form of Batch Aggregator is required.
In either case, wider Batch Processing patterns may be required
to manage the batch level transactions.

IBM Software Group
Performance (Efficiency)
Response times Requester sends asynchronous requests
in clumps. Provider’s response times slow
exponentially if more than 10 requests are
sent in parallel
Perform Flow Control to ensure only 10 requests to the backend
are ever made concurrently, storing up the other requests in a
Store and Forward style.
Throughput Requesters average 20 concurrent,
occasionally peak at 200 concurrent
synchronous requests. More than 50
concurrent requests to the provider cause
system outages.
Perform Throttling to ensure that no more than 50 concurrent
requests are ever in progress, if necessary rejecting the access
requests. Optionally only allow high Priority requests through if
the concurrency is greater than 30.
Availability
windows
The provider is not available from 5pm to
8:30am due to the length of overnight
batch runs, but requesters make requests
24/7
Store and forward the requests, implementing Flow Control to
ensure that the backend system isn’t overloaded when it comes
online in the morning.
Volumes Arrival of a batch file results in 20,000
messages, swamping the providers CPU.
Implement Flow Control and Store and Forward to control the
load.
Message Size Messages contain a 2Mb blob who’s
internal data is of no interest to the
process.
Use the Claimcheck pattern to detach and re-attach the data
outside of the process.

IBM Software Group
Error Handling (Maintainability)
Known
exception
conditions
Provider can respond with hard to interpret
error responses, some of which might be
business meaningful to the requester.
Implement and Error Handling strategy that intelligently maps
the system errors into business errors.
Error
management
capabilities
The provider has regular unexpected
outages, but re-try later in the day will
usually work ok. However, some errors are
poison messages that will never be
processed.
Use Failed Event Management to perform appropriate re-tries,
then off-lining of messages for later resolution so other
messages can be tried. Could be combined with Healthcheck to
avoid unnecessary re-try processing, and additionally with Store
and Forward to provide automated re-tries when the system is
available again.
Deferred error
conditions
A fire/forget request has failed
downstream, but the calling system is
unaware.
Use a Publish/Subscribe mechanism and Correlation to enable
the calling system to listen for intermediate messages notifying
failure, or use an Offline Error Handling strategy

IBM Software Group
Technical Interface (Portability)
Transport /
Protocols
Service exposed to requester as HTTP/
REST, but back end system interaction is
with MQ/XML.
Perform the Protocol Translation but be wary of the possible
error handling challenges. See Transactionality lecture and sync/
async switching.
Data formats Service is exposed as a SOAP message,
but the back end system requires a
copybook.
Use the Data Handler pattern to resolve the difference.

IBM Software Group
Where do functional orchestration patterns typically come from?
Evolving to Straight Through Processing (STP)
§  What the business consider to be an important step in a process
today, once automated, may become irrelevant to them.
§  The granularity of relevant business steps changes over time.
§  Example: Tallying up receipts from multiple stores, processing
invoices, route planning (travel agents etc. now replaced by internet
booking, and GPS). These things were primary operational costs in
the past, but become almost irrelevant now.
§  STP may turn a business process into a single step. Difference is that
we used to do this automation in a single system. Now the functional
areas of the business covered by multiple systems so automation now
requires multiple system orchestration.

IBM Software Group
Progressive process optimisation – “The journey is the goal”
Flow automation
Task automation
Straight through
processing
Manual process
“Filling in the
whitespace
between systems”

IBM Software Group
What is STP?
§  STP = “Straight Through Processing” – Typically a re-design
– The typical primary goal of STP is to increase throughput. The primary way to do
this is removal of humans from a process. This requires both flow and task
automation and also significant re-engineering.
– If the end goal is STP, you must design for this from the start in every aspect from
the way you re-engineer the flow, the services you make available to automate the
tasks to the way you implement security and so on..
§  STP = “Fully automated?” – Rarely
– STPs are rarely completely automated for every path through the process, but
should be automated on the most commonly traversed paths. Most have rarely
occurring edge cases that are still handled manually.
– Exception cases typically become exponentially harder to automate as you cater to
more diverse and rare circumstances. It is critically important to evaluate the
business benefit of each additional automated path.
Request
Account
Create
Account
Customer
Notify
Customer
System Activate
Account
STP

Impact 2013 2971 - Fundamental integration and service patterns

More Related Content

What's hot

Similar to Impact 2013 2971 - Fundamental integration and service patterns

Recently uploaded

Impact 2013 2971 - Fundamental integration and service patterns