Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Placement of BPM runtime components in an SOA environment

1,163 views

Published on

The service oriented architecture (SOA) reference architecture is intentionally simplistic at a high level but it holds some surprises when you look closely at how components really interact. This is especially true in relation to the placement of business process management (BPM) componentry. We discuss the most common design questions including: Is BPM a consumer or provider of services? To what extent should a user interface, be decoupled from the BPM runtime? How do we retain agility in BPM while adhering to the architectural separation of SOA? These subtleties are critical when designing solutions to reap benefits of both SOA and BPM simultaneously.

Published in: Software

Placement of BPM runtime components in an SOA environment

  1. 1. © 2012 IBM Corporation IBM Software Group Placement of BPM runtime components in an SOA environment Presenter: Kim Clark Authors: Kim Clark (kim.clark@uk.ibm.com) Brian Petrini (petrini@us.ibm.com) Version 1.6
  2. 2. IBM Software Group © 2012 IBM Corporation Related article Please note, the following article has been published that covers the concepts in the first half of this presentation: “The placement of BPM runtime components in an SOA” http://www.ibm.com/developerworks/bpm/bpmjournal/1404_clark/1404_clark.html 25/01/14
  3. 3. IBM Software Group © 2012 IBM Corporation How complicated can a simple sequence of boxes be? Process
  4. 4. IBM Software Group © 2012 IBM Corporation Common questions arising when considering BPM and SOA https://collaboration.opengroup.org/projects/soa-ref-arch •  Should all requests go through business process management (BPM)? •  Is BPM above or below the enterprise service bus (ESB)? •  Are processes also services? •  Should BPM handle page navigation? •  Should BPM perform integration?
  5. 5. IBM Software Group © 2012 IBM Corporation Consumers Are business processes just another type of consumer? Services (Exposure) Operational Systems (Applications & Data) Process Runtime (BPM as a consumer) Other consumer ServiceProvidersServiceConsumers Other consumer Could/should we remove the business process layer altogether? Removes temptation to assert that all requests should go through business process layer.
  6. 6. IBM Software Group © 2012 IBM Corporation Sequences of steps Where could they occur / Where should they occur? Service Exposure Operational Systems (Applications & Data) Integration Consumers Business Process
  7. 7. IBM Software Group © 2012 IBM Corporation People based processes Service Exposure Operational Systems (Applications & Data) Integration Consumers Business Process
  8. 8. IBM Software Group © 2012 IBM Corporation What makes a business process suitable for BPM? Performing the process provides value to the business The process contains individually business relevant steps Business relevant data flows through the process The process follows a relatively structured path The steps within the process are performed by multiple roles/teams. The process changes over time as a result of changes in the business Create Account Capture Details Send Documents Sign Documents Activate Account Customer Call centre Back office Notify Customer New revenue! Existing customer New customer -- ----------- ------------- ------- - -- ----------- ------------- ------- - -- ----------- ------------- ------- - -- ----------- ------------- ------- --- ----------- ------------- ------- - -- ----------- ------------- ------- -
  9. 9. IBM Software Group © 2012 IBM Corporation Consumers BPM as a consumer of services Services (Exposure) Operational Systems (Applications & Data) Business Process (Orchestration) Process Runtime (as a consumer) ServiceProvidersServiceConsumers A business process orchestrates (consumes) services in order to automate steps in the process Human task System task Exposed service Back end system Misleading representation of an interaction Interaction Key
  10. 10. IBM Software Group © 2012 IBM Corporation Consumers Positioning of the integrated BPM UI Services (Exposure) Operational Systems (Applications & Data) Business Process (Orchestration) Process Runtime (consumer) ServiceProvidersServiceConsumers Typically an internal BPM UI is provided to enable users to view and work their tasks. Integrated BPM UI
  11. 11. IBM Software Group © 2012 IBM Corporation Consumers Introducing an external BPM user interface Services (Exposure) Operational Systems (Applications & Data) Business Process (Orchestration) Process Runtime (consumer) Integrated BPM UI ServiceProvidersServiceConsumers Often some, or all of the human tasks need to be served up by an external custom portal due to the specific requirements of the graphical user interface. retrieveTask() updateTask() completeTask() Independent BPM UI
  12. 12. IBM Software Group © 2012 IBM Corporation Systems acting as both providers and consumers simultaneously §  The layered architecture is a logical model, not a physical deployment §  Components may appear more than once if they perform different roles §  Many operational systems are both providers and consumers Services (Exposure) Operational Systems (Applications & Data) System A (as a provider) ServiceProvidersServiceConsumers Consumers System B (as a consumer) System A (as a consumer) System B (as a provider)
  13. 13. IBM Software Group © 2012 IBM Corporation Consumers The process runtime acting as a provider of services Services (Exposure) Operational Systems (Applications & Data) Business Process (Orchestration) Process Runtime (as a consumer) Integrated BPM UI Process Runtime (as a provider) Independent BPM UI ServiceProvidersServiceConsumers In reality, the business process runtime also becomes a service provider in this situation providing services for external systems to read and manipulate tasks. So the BPM component now lives in two places on the diagram, each with a specific role. BPM is both above and below the service layer retrieveTask() updateTask() completeTask() Internalinteractionwithin businessprocessruntime API
  14. 14. IBM Software Group © 2012 IBM Corporation Pros and Cons of fully decoupling the process API Pros: •  Enables hiding of routing, versioning, security models etc. •  Becomes a standardize process capability, independent of technology and implementation •  Standardized monitoring and controlling service level agreements, throttling traffic •  Standardized governance: Enforcing policies for things such as encryption, deprecation. •  Optionally enables standardization of data model, but see Cons below. Cons: •  Significant up front work before the first project can use the API •  Results in another layer to be maintained, more skillsets involved operationally. •  May render the standard API product documentation less effective, and new documentation may need to be created and maintained. Particularly true if a new data model is introduced. Recommendations •  Only consider decoupling where multiple highly independent consumers are present, each with strong reliance on well defined service level agreements. •  Focus on requirements around traffic management, and security. •  Avoid replacing/translating the APIs data model. This is a lot of work both up front and ongoing for very little gain.
  15. 15. IBM Software Group © 2012 IBM Corporation Page navigation Service Exposure Operational Systems (Applications & Data) Integration Consumers Business Process
  16. 16. IBM Software Group © 2012 IBM Corporation Separating Process, Page Navigation and Page Design Loan Requester Loan Approver System Approve Loan Process Loan Capture Loan Details Capture Personal Details Capture Financial Details First name Second name Date of birth Personal details etc. What’s wrong with this process model? Multiple steps in the same human swimlane of a BPMN diagram suggest there is an issue with the granularity of the process modeling. It implies they could have been performed by the same person in one go, so they are not really separate steps. We should not model this low level page navigation in the BPMN diagram.
  17. 17. IBM Software Group © 2012 IBM Corporation Separating Process, Page Navigation and Page Design Loan Requester Loan Approver Capture Loan Application System Approve Loan Process Loan Process Capture Loan Application Loan Details Personal Details Financial Details First name Second name Date of birth Personal details etc. Page navigation Page Page navigation should be modeled separately from the BPMN swimlane diagram.
  18. 18. IBM Software Group © 2012 IBM Corporation Separation of concerns when creating a custom UI for BPM Independent user interfaces for BPM should also take on the page navigation. Example issues •  Page navigation depends on the page data •  There are often data dependencies across pages •  Pages/data may need to be pre- loaded based on navigation •  The ideal UI data model will be different to the process model Process Layer Presentation Layer Page navigation Page Process
  19. 19. IBM Software Group © 2012 IBM Corporation Consumers Is a process also a service? Services (Exposure) Operational Systems (Applications & Data) Business Process (Orchestration) Process Runtime (consumer) Integrated BPM UI ServiceProvidersServiceConsumers Process Runtime (provider) submitLoanApplication() Service Components startProcess() User Interface Internalinteractionwithin businessprocessengine API “Is a process a service?” – No! But a service can result in the initiation of a process.
  20. 20. IBM Software Group © 2012 IBM Corporation Processes with integration Service Exposure Operational Systems (Applications & Data) Integration Consumers Business Process
  21. 21. IBM Software Group © 2012 IBM Corporation Single Transaction for all invocations Multiple Transactions across invocations Human Tasks Terminology: Orchestration vs. Process vs. Composition? Orchestration is a sequence of “invocations” of systems, or humans Appear to the business as a single step. Do not include human interaction from the business. Take long enough that the business need to be aware of their intermediate in- progress states. Include human interaction, or interaction with slow responding systems. Orchestration Composition Process Note: These are not industry standard terms, but derive from common usage observed in the field
  22. 22. IBM Software Group © 2012 IBM Corporation Core types of Orchestration: Process vs. Composition Process Makes calls to mature high level services Often triggered (i.e. one way call) rather than invoked as a two way call Where it is invoked as a two way interaction, the caller is typically asynchronous (i.e. not a user) and therefore the service level agreement is throughput based rather than response time based Stateful persistence of the steps in the process Events can correlate with the running process Often involves human interaction to perform some tasks within the process Composition Grouping of relatively fine grained interactions Typically called in real-time in a request/response (two way) interaction style. Response time is the primary driver for the service level agreement Common for aggregation functions Some or all the granular interactions may not themselves be exposed as re-usable services Generally state free Never involves human interaction during the composition
  23. 23. IBM Software Group © 2012 IBM Corporation Separating the business process from low level composition Loan Requester Loan Approver System Approve Loan Save Loan Details Capture Loan Application What’s wrong with this process model? Multiple steps in the same system swimlane of a BPMN diagram suggest there is an issue with the granularity of the process modeling. It is likely we are modeling the separate systems users have to update today. In the future process this multiple update will be automated, so it is no longer be relevant to model the individual steps on the business process diagram. We should not model this low level integration logic in BPMN diagram. Save Personal Details Save Financial Details
  24. 24. IBM Software Group © 2012 IBM Corporation Pros/cons of moving composition out of the process layer Orchestration modelled and implemented within the process layer. Good for visibility, but process much more complex. Granular services exposed for re-use. Interactions traverse many layers (poor performance) Orchestration modelled and implemented in the integration layer. Hides integration complexity from process. Course grained services exposed for re-use Interactions are close to systems (good performance) Service Exposure Operational Systems (Applications & Data) Integration Layer Business Process Layer Composition Business Process a b c Service Exposure Operational Systems (Applications & Data) Integration Layer Business Process Layer Business Process ba c
  25. 25. IBM Software Group © 2012 IBM Corporation Just a selection of the different patterns of orchestration Integrated workflow Stateful integration Aggregation Isolated transaction Composition Exceptions only Process Stateless* engine Stateful* engine * This refers to persistence of orchestration state Real time retrieval of data from multiple systems Real time updates to multiple systems that can be combined into a single update Straight through processing across systems that can’t be combined transactionally People based exception handling Processes integrating people and systems
  26. 26. IBM Software Group © 2012 IBM Corporation “Process” patterns Integrated workflow Stateful integration Exceptions only Process Stateful engine
  27. 27. IBM Software Group © 2012 IBM Corporation Pure workflow process People based process with no system integration All data is stored in the process. State is stored as each user works a task Model changes can be in more in the hands of the users. Examples • Paper based processes • Procedures involving multiple teams of people Primary benefits: • Reduction/removal of paper • Improved work distribution/prioritisation • Management visibility on process status • Common understanding of process model Implementation: Stateful orchestration engine with good task user interface capabilities. System based activity People-based activity Queued events State persistence
  28. 28. IBM Software Group © 2012 IBM Corporation Integrated workflow process People and systems are involved in the process Enables progressive automated of tasks Users directly involved in modelling of process. IT required for integration. Examples • Processes where elements of data are duplicated across multiple systems • Processes where system updates are unnecessarily complex for users Benefits: • Benefits of Pure Workflow • Reduce/re-assign workforce • Improve end-to-end time • Improve data consistency Implementation: Stateful orchestration engine with good integration capabilities. Significantly simplified if implemented in conjunction with a service oriented architecture. System based activity People-based activity Queued events State persistence
  29. 29. IBM Software Group © 2012 IBM Corporation Process with people for exceptions only Main path is straight through, people handle known exception paths Users can progressively refine exception path rules to increase automation. A step in the direction of Straight Through Processing (STP) Examples • Approval processes rules enable some auto-approval Benefits: • Benefits of Integrated Workflow Process • Manage proportionally higher volumes Implementation: Orchestration engine with integration and rules capabilities. System based activity People-based activity Queued events 80% 20% State persistence
  30. 30. IBM Software Group © 2012 IBM Corporation “Composition” patterns Stateful integration Aggregation Isolated transaction Composition Stateless engine
  31. 31. IBM Software Group © 2012 IBM Corporation Aggregation No transactions required, just gathering data Data gathered from multiple sources and merged into a single result. By far the most common requirement. Most solutions need convenient access to rich data. Examples • Gathering quotations from multiple insurance brokers (parallel) • Combining customer data from multiple systems (sequenced) Benefits • Relatively simple implementation since no data integrity/ transactionality requirements. • Failures part way through can be simply re-tried • Hides complexities of data translation/merge and connectivity to disparate systems Implementation: Ideally performed as close to the back end systems as possible. Integration engine preferable due to strong data merging capabilities, and more direct connectivity to back end systems. Challenges •  Should you wait for all responses •  How do you merge parallel requests •  Which layer in the architecture should this be performed. System based activity
  32. 32. IBM Software Group © 2012 IBM Corporation Isolated transaction Multiple transactional calls to the same system Provides a real-time response regarding the status of the update to the caller. Commonly used to commit changes requested by user interfaces providing immediate feedback of success. Examples • Updates to multiple tables in the same database. e.g to create an order and its order items. Benefits • Ensures data integrity across multiple calls to the same system • Failures will be fully rolled back, so requests can be simply re-tried. Implementation: Ideally done as close to the data as possible for performance, and simplicity of the transactional protocols. Highest performance will be within the back end system (e.g. a stored procedure). Next best is a integration engine that can handle the transactional protocol (such as JDBC). Challenges • Transactions typically hold resources (e.g. threads/memory) whilst in progress • The aggregate of all the steps may take longer than the transaction timeout. • Transactions containing multiple steps can cause deadlocks if coding guidelines are not adhered to. Transaction Boundary System based activity Single phase commit
  33. 33. IBM Software Group © 2012 IBM Corporation Distributed transaction Data integrity critical across multiple systems Requires a independent transaction controller to perform a global transaction across more than one system. This requires that the systems are enabled for two phase commit transactions. Examples • Committing changes to multiple databases • Committing changes to a queue and a database. In reality two phase commit is more commonly used “within” a system, rather than across multiple systems. For the reasons noted in “Challenges” Benefits • Ensures data integrity across calls multiple systems • Failures will be fully rolled back, so requests can be simply re- tried. Challenges • Few system owners will allow a two phase commit to be performed against their systems by external parties due concerns over resource locking. • Two phase commit require the maintaining of a transaction log so systems can re-align should a failure happen mid transaction. This adds interdependencies that complicate design of high availability and disaster recovery. • Deadlocks are much more difficult to diagnose than with single phase commit. • Two phase commit is much more heavyweight in CPU both for the end systems, and for the caller than single phase commit. Transaction Boundary System based activity Two phase commit
  34. 34. IBM Software Group © 2012 IBM Corporation Enriched update Only the final update matters, everything else is preparation Transaction Boundary System based activity read non-critical update critical update *Definition: “Critical Update” Any request that performs a change to data (create, update or delete) that MUST be resolved in the case of a failure, either with a transactional rollback, or with compensatory actions if the update has already been committed. A complex scenario is simplified to focus on the one key update, thereby removing the need for intermediate state. Examples • Checking for existence of a parent objects before creating a child object – e.g. checking customer exists before creating order. • Validating data before performing update – e.g. checking for stock inventory before submitting an order. Benefits • Enables a complex composition yet still removes the need for stateful orchestration. Challenges • Re-tries result in repeating all of the pre- critical steps • Invocations deemed to be “non-critcial” may result in confusion during diagnosis of issues.
  35. 35. IBM Software Group © 2012 IBM Corporation Event distribution/broadcast We only need to know it will be done, not that it has been done Messages are placed in a number of queues within a single transaction. Examples • Useful for updates to systems that are not expected to be instantaneously up to date. E.g. data warehouses, audit logs • Useful for notification type requests (SMS, email etc.) Benefits • Provides a swift answer to the caller • With persistent queues, the messages should never be lost, and will be applied to their destinations eventually. Challenges • Destination systems have not received the data when the response is given • The time taken for destinations to receive the update is not in the control of the composition • Should problems occur processing the messages, the composition, and therefore also the caller are unaware. Therefore reliant on external exception handling. Transaction Boundary System based activity Queued events
  36. 36. IBM Software Group © 2012 IBM Corporation Basic Interaction Types ProviderRequester getCustomer(CustomerID) returns Customer ProviderRequester submitOrder(Order) a)  Fire and forget b) Request-response A messaging transport is used. The request is “one-way” sending data to the provider, but with no response. The requester need only wait for the message to be received by the transport – i.e. it is non-blocking. This is the style typically used to initiate a process. i.e. we do not wait for the process to complete. A synchronous transport is used. The requester requires response data, and must wait for the provider to respond. The provider must be available at the time of the request and must process the request immediately. This is the style typically used with compositions. i.e. the caller requires a response from the composition.
  37. 37. IBM Software Group © 2012 IBM Corporation Stateful Integration – The pattern with two homes Integrated workflowAggregation Isolated transaction Composition Exceptions only Process Stateless engine Stateful engine Stateful integration
  38. 38. IBM Software Group © 2012 IBM Corporation Stateful Integration Process Zero or near zero human interaction All steps are system invocations. Multiple separate transactional interactions are required. State must be stored in between each step, ideally as part of the same transaction. Examples • Data synchronisation. e.g. “Change of Address” • “Straight Through Processing” (STP). e.g. payments processing Benefits • Enables orders of magnitude higher volumes. e.g. moving sales to the internet • Enables precise data integrity across disparate systems Implementation: Depends significantly on non- functional requirements – throughput/volumes. See next slide for options. System based activity Queued events State persistence Transaction Boundary
  39. 39. IBM Software Group © 2012 IBM Corporation Implementation options for Stateful Integration Process Orchestrated interactions Pros Mature products available (e.g. BPM) First class notion of “process” Modelled process visible in implementation Monitoring/reporting at process level Process version migration support Clearer modelling of exception paths Easier to introduce human interaction Cons Higher CPU/network due to persistence Solution will be split between business process integration layers Chained interactions Pros Higher performance due to minimal persistence Can be implemented entirely in the integration layer Cons Largely a custom solution No notion of “process”, only the individual steps Process “model” not visible in implementation Custom work for effective monitoring/reporting Exception paths harder to understand Custom work to migrate to new processes Custom work to introduce human interaction Queued eventsState persistence
  40. 40. IBM Software Group © 2012 IBM Corporation What orchestration patterns are required? Service Exposure Operational Systems (Applications & Data) Integration Consumers Business Process Integrated workflow Page Navigation Enriched Update Aggregation Stateful integration
  41. 41. IBM Software Group © 2012 IBM Corporation Core characteristics for assessing orchestration requirements Time-span Granularity Human interaction Principal objects Application integration Complexity Monitoring Flow ownership Dynamicity Transactionality Performance Volumes Business value State management Security Infrastructure
  42. 42. IBM Software Group © 2012 IBM Corporation Many patterns, many possible implementation options Integration flow pub/ sub Business process short running Business process briefly persisted Integration flow 1:n Integration flow 1:1 with translation Business Process long running system centric Business Process long running human centric Integration flow 1:1 routing only Many attempts at decision trees for implementation choices fail because they try to cover the whole many- to-many mapping from all possible requirements to all possible implementations. This is typically an impossibly hard decision tree to draw, and inconceivably complex to navigate. Implementation Options Requirements more…more…
  43. 43. IBM Software Group © 2012 IBM Corporation Narrowing the complexity using a two stage decision tree Patterns Implementation Types Use architectural principles to determine the relevant implementation choices for that pattern Use requirements to choose the pattern
  44. 44. IBM Software Group © 2012 IBM Corporation Types of orchestration in relation to transactionality Do any of the requests perform critical updates*? No Aggregation Enriched update Yes Can all critical updates* be performed in one transaction? Distributed Transaction Stateful Integration Yes No How many critical updates*? =1 >1 *Definition: “Critical Update” Any request that performs a change to data (create, update or delete) that MUST be resolved in the case of a failure, either with a transactional rollback, or with compensatory actions if the update has already been committed. Non-transactional Isolated transaction Global transaction Multi-transaction
  45. 45. IBM Software Group © 2012 IBM Corporation No single answer Process solutions are always a combination of orchestration patterns Service Exposure Operational Systems (Applications & Data) Integration Consumers Business Process Data synchronization Microflow In-app. workflow Screenflow/ page navigation Stored procedure Straight-through processing Business process
  46. 46. IBM Software Group © 2012 IBM Corporation References The placement of BPM runtime components in an SOA http://www.ibm.com/developerworks/bpm/bpmjournal/1404_clark/1404_clark.html Process implementation types: Patterns based design for process-based solutions http://www.ibm.com/developerworks/websphere/library/techarticles/1004_clark/1004_clark.html Process-oriented modeling for SOA http://www.ibm.com/developerworks/library/ar-procmod1

×