The purpose of this paper is to explain a few simple concepts and methods that facilitate agility in Enterprise Integration (EAI) and Service Oriented Architecture (SOA). This paper is written as a short tutorial. However, the reader should be familiar with basic object-oriented analysis and object-oriented design concepts and practices to fully understand the concepts described here and to put the methods into practice. Richard Green is an Enterprise Architect at DTE Energy – a gas and electric utility with headquarters in Detroit, Michigan. DTE Energy has nine billion dollars in revenues, twelve thousand employees, and three million customers. It is, obviously, a large complex operation with significant investments in legacy systems, major challenges, and ambitions for better integration across the real enterprise and the virtual enterprise. Richard is responsible for defining and articulating IT technical strategy and mentoring other IT roles. Working in this environment has taught Richard many lessons about implementing EAI and SOA in a large and venerable organization.
Why is it important to be agile in EAI and SOA? Enterprise Integration is the primary work of an enterprise IT group. Mature enterprises rarely need to develop software products anymore. Usually, they can buy whatever they need cheaper than they can build it. There are only a few situations where an enterprise in a mature industry can create any strategic advantage by creating a do-it-yourself application. I am, of course, talking about enterprises whose primary business is something other than creating and selling software. However, even in that case, there are still many situations where even a software development enterprise should buy rather than build software. For example, if an enterprise sells engineering design software, it probably makes no sense for it to be building accounting applications. So although an enterprise can generally buy any software application it needs at a fraction of the cost of building it, the enterprise IT group will need to integrate that purchase into the enterprise systems. What is the nature of integration? It is the work of designing and building interfaces . How can we do that in an agile fashion?
This is a table of contents. I will talk about achieving agility across the progression of development work going from concept all the way through deployment. In my role as an Enterprise Architect at DTE Energy, I have been called in to consult on a number of integration projects “to explain SOA.” I’ve found that the teams are often quite confused about how to organize their project and design. The problem was not that they did not understand SOA. The problem was that they did not know how to apply object-oriented analysis (OOA) and object-oriented design (OOD) concepts and practices at the level of the enterprise. In too many cases, 1980’s designs were being replicated in Java. So, I started writing tutorials. I needed to show analysts how to apply OOA to business process integration scenarios. Analysts and designers did not know how to recognize a service when they saw one. I needed to show designers how to organize component and service interfaces in a way that avoided unneeded coupling between client and service. In the process of writing those tutorials, I discovered a very useful simplification. In an enterprise integration context such as EAI and/or SOA, the messages that are exchanged between collaborating processes are the essential features. Once you realize that the “message is the feature,” many additional simplifications follow. And the message-features then provide the user-story unit-of-work that ties your agile project together.
Basically, SOA is an old wine in a new bottle. I admit that there is some WSDL spice in the wine now … with some BPEL cheese on the side. But it is still old wine in a new bottle. You can verify this by looking at the promises that SOA makes. They sound very familiar to old timers like me. They are the same promises that were made for object-oriented analysis, for object-oriented design, for re-use, and for message-oriented middleware. And that is an important lesson! What it means is that we need to apply OOA and OOD concepts and principles and use some kind of message-oriented infrastructure if we are going to make those promises come true. The epiphany comes when you realize that Enterprise Integration is all about improving the way that business processes talk to each other . So … here is the simple key to achieving agility: The message is the feature. The rest follows from that … by applying well-known object-oriented analysis and design ideas. The following pages show how to reduce this simple idea to a set of simple techniques that allow you to apply agile methods in EAI and SOA projects.
Enterprise Integration implements communications between business processes. If you think about Enterprise Integration as just application-to-application (A2A) data exchanges, you will miss the point. Those applications are proxies for business processes . Hence, A2A and B2B are essentially the same thing. The difference is just a question of whether the communicating processes are under the same enterprise or in two different enterprises. Hence a service oriented architecture involving the web interface of an out-sourced business process is simply an extension of Enterprise Integration. Since Enterprise Integration is about improving communication, old-fashioned analysis techniques still work very well. For example: A scenario written as a sequence of stimulus-response pairs is still a very usable technique. The technique works well at the enterprise level when it focuses on the messages exchanged between the business processes. You can very quickly characterize the interfaces between two or more business processes by writing a scenario with a series of “he says … she says” speeches – like the script for a stage play. To find the features, simply identify and analyze the messages exchanged between the business processes. The message is the feature. The features are sorted into releases. Each message pattern has a design pattern. The design pattern provides the work breakdown structure (WBS) for project management, performance metrics, and estimating.
If your enterprise wants to be service oriented, you have to recognize that each business process is a business service. It has clients in the form of other business processes that delegate work to it. The delegation of work means that there is a request-response scenario . It also means that you can expect many messages to be riding on work orders. In analysis and design terms, expect a typical business pattern, a typical design pattern, and a typical work breakdown structure (WBS) for implementing the request-response interfaces. Work orders are often turn-around documents. You can expect them to gather data while the workers are getting the work done and then to return the data to the requester as a by-product. This creates a hazard because the data-gathering function of a work order is often treated as an after thought, and there may be some data quality problems as a result. Typically, a specific business process is the natural data custodian for one or more business data entities . As a consequence, the database that provides the enterprise memory for that process is the natural system of record for certain data entities. Therefore, expect to find question-answer scenarios in the service interface. Also, expect the custodian business process to publish events relating to the system of record.
I work for a gas and electric utility, DTE Energy. The examples are from my own experience. To understand the examples, you can think of a utility as an enterprise that delivers a service at a location. The point-of-sale data comes from meters that measure how much service was delivered. The point of this slide is that messages are speech acts. You can look at existing business interfaces and almost always classify the information transmittal into one of these four categories. I find the style used here convenient and simple. Each message is identified with a complete sentence. The key nouns are associated with parameters indicated by capital letters inserted after the noun. You may have to break people of the bad habit of stating a business feature as “Move file F from application A1 to application A2.” That is not a business requirement; it is design advice. Furthermore, it is bad design advice if you want a real-time enterprise. You have to insist that the message be at a level that clearly describes the business intent and the real-world objects, not data objects.
Write each message as a complete English language sentence. You might be surprised at how hard it is to get professional people to do this! Jargon is not English . Some IT people have gone jargon crazy. Some business people have gone jargon crazy, too. If you want to reduce cost, waste, and delays, you need to get down to basics. All too frequently the jargon obscures the basic concept. By eliminating the jargon, you often find the key concept exposed. Table, file, and application references are not English . If a person outside of your company would not know what you are talking about, it is not English yet. It is still jargon. Write each message as a complete sentence. Analyze the nouns . Most of them will refer to real-world entities. These are candidate business objects. You probably already have data stores, data schemas, and classes representing these business objects. Translate the sentence into the notation above to make the parameters obvious.
Agile practices often revolve around use cases. To make your EAI/SOA project agile, treat each message-exchange as a feature or user story. This is a unit of work that is easily understood by business people and technical people. I recommend that you use the complete sentence that expresses the sender’s message as the name or title for the user story or feature. This will assure clarity in your release plans and other communications. Most business integration scenarios can be classified into three patterns, as shown above. The pattern associated with the delegation of work is “Request-Response.” The pattern associated with a question to a data custodian is “Question-Answer.” The pattern associated with data synchronization is “Event Notice.” Each of these message exchange patterns will have a corresponding design pattern, test plan, and work plan for planning and estimating. The resulting design pattern, test plan, and work-plan depend on your organization’s particular integration infrastructure, implementation frameworks, development tools, testing tools, and agile methods.
Here is an example to show you how the analysis concepts might be applied. We have a business process in our utility that is responsible for metering energy usage. To put this into a service oriented view, we change the name of the business process to reflect the fact that it is a service. This is a big step for some people. They tend to think of business processes as boxes on organization charts. Or, they think only in terms of databases and data transactions. Being service oriented might require re-orientation. In the example, the Metering Service receives a request from one of its clients (there are several) to “Please install a meter … .” In a typical utility, the work is delegated to a field work organization. So Metering Services starts a workflow to process the request and delegates some of the work by sending a work order to another organization. Since the Metering Service is the custodian of the work-in-progress that it manages, the workflow component needs to provide basic capabilities to answer questions about the work-in-progress. The Metering Service is also the data custodian of the “Meter” business object. So it is responsible for publishing changes to the installed inventory of meters. This is how replicated data are synchronized.
This is a scenario or use case found in an Enterprise Integration project. It shows the messages exchanged between collaborating business processes. The business domain objects (data entities) referenced by each message are indicated in the notation. This notation tends to clarify thinking about what a message means and what information it must carry. This scenario is what we call the “happy path” -- the normal, simple, and successful scenario. Each process may have several ways of failing (generating a business-process exception). The business analyst is responsible for identifying variations in the message that kicks off the scenario. The business analyst is also responsible for identifying the business-process failure modes – process exceptions . (Note: This is not about IT system failures. Those are invisible to the business process.) The tester is responsible for creating a test case for each variant and each exception. This scenario is written from the point-of-view of the Metering Service. Details that occur within the Field Service business process are not shown here. We don’t need to include them here because those details are hidden from the Metering Service. This is an example of the OOA, OOD principle of “hiding the implementation.” Each business process is only responsible for its own workflow.
The “Planning Game,” as it is called in Extreme Programming practice, can be roughly twice as complicated when you are planning Enterprise Integration. This is because there is a sender and a receiver for every message. The sender process and the receiver process each have a full complement of project roles. Thus, you have roughly twice the number of players at the planning table. I strongly recommend that you start thinking about how you do EAI and SOA planning if you have not already done so. The biggest obstacles to becoming agile in Enterprise Integration are associated with the planning process. In any case, the message is the feature: The message is the primary unit of work for release planning and iteration planning.
A release only makes sense if it contains a coherent set of features . That begs the question: What makes a set of features coherent? In the case of a release with a set of user interfaces, coherence means that the features are sufficient to support a specific user task or a specific business process with a connected set of user tasks. Until the task or the process is adequately supported, the release is incomplete from the user’s point of view. In the case of a release with a set of business-process interfaces, coherence becomes a more abstract concept. However, it is still possible to think of coherence across a business process or sub-process in most cases. Thus, a coherent release includes all of the messages needed to implement some business process or sub-process.
A typical business process is facilitated by a core application and some additional applications that act as helpers or cover some non-core aspect of the business process. To achieve clarity in thinking and consistency in interface design, employ a strategy of wrappers within wrappers . The content of the wrappers is clearly indicated by the messages exchanged between the business processes. You can design the interfaces by starting with the business-process interfaces and then moving to the wrappers on the applications. The essence of lean design and agile practice is that each interface signature, each method, is directly traceable to a business need. In this case, it is traceable to a specific message in the Enterprise Integration case. In Java coding, the interfaces and wrappers are defined by Java “interface” objects and documented in the associated JavaDoc's. Thus, for each business process, expect one or more Java interface declarations. If you don’t already have them, you need to define business domain objects in Java interfaces or classes prior to defining the business-process interfaces. Once the business-process interfaces are written, you can define Java wrappers for the applications by allocating the business-process interfaces to the appropriate application wrapper. Summary: Identify the inter-process messages. What are the business processes saying to each other? Write equivalent method signatures for the Business Service wrapper. Allocate the method signatures to the Legacy Application wrappers.
You transform a business process into a business service by building a well-defined API around it. Build a wrapper for each of your business processes. The interfaces to the business service are defined by the messages that it sends and receives. It is a simple matter to translate the message signatures into method signatures. In a request-response scenario, there business service will typically need to invoke a workflow. Each business service is responsible for its own workflow. It may call on other business services, which may in turn have their own workflow.
Each application needs a wrapper. The wrapper allows us to talk to the application using common business objects in the medium of communication. Question-Answer Scenarios are usually implemented as remote procedure calls to the business process wrapper. Programmers may be tempted to implement these as SQL calls (JDBC calls in Java). SQL calls from the client is a bad idea because it exposes the implementation and increases coupling between the client and the service. The application that “owns” a database will have a wrapper. The business process wrapper calls the application wrapper when it needs a service or some information from the application. The application wrapper may implement SQL calls if that is the only way to handle a query. In general, if the application has an API for data access, you should not by-pass it.
Some things about the work breakdown structure (WBS) for the implementation of a message are reasonably obvious … and other things are not quite so obvious. Much enterprise EAI and SOA occurs in an environment dominated by pre-Java “legacy” applications. The consequence: A lot of work goes into reverse-engineering the legacy application to discover how to detect an event or how to properly process an in-coming event or request. When a new request is received by a legacy application, there is almost always a pre-existing capability. The trick is to find a way to properly invoke the existing capability. When an event notice is needed, the trick is to find a way to reliably detect the event and then gather the appropriate data to at least partially populate the business objects referenced by the event. The teams that implement sender-side logic and receiver-side logic can proceed independently because our procedure defines the message in a way that avoids coupling between the sender and receiver. This greatly simplifies design coordination between teams, but release schedules still require appropriate synchronization. When work is delegated, you almost always have to deal with business-process exceptions for any non-trivial process. Consequently, you need to implement a workflow for each request-response scenario. Incidentally, we discovered a simple trick – most of the time you don’t need BEPL, a fancy workflow engine, or even a process diagram. We define the workflow via a “next step” table: next step = fc( prior step, completion status ) Let the publish-subscribe framework (For example, JMS) handle the dispatch of the next step.
The environment in which I work contains a large number of “legacy” applications and a large number of purchased applications. In both cases, implementing new interfaces tends to require a lot of “design discovery” work. Basically, this means reverse-engineering the application. Where is the best talent for doing this kind of reverse-engineering? If you cannot draft the original author, the next best talent is the maintenance team. Project teams are a poor choice for the design of general purpose components or tools. The pressure and limited view of the project tend to constrain the designer’s thinking. It is a better idea to staff a small team and make them responsible for developing tools. Create a tooling team if you can. If the project is delegating wrapper work to the legacy application’s maintenance teams, and tooling to the tooling team, what is the core team doing? It turns out they have plenty of challenging work to do. They need to discover the process exceptions and implement a workflow for each message. If the system is to have long-term durability and reliability, a suite of automated end-to-end test cases needs to be implemented. That is a lot of work. The most important concept that I learned from the study of concurrent engineering practices is that you need to pay particular attention to interfaces. They need to be clearly specified and changes must be tightly controlled. There are numerous examples of what can happen when one fails to do this. (For example: The spacecraft misses the planet Mars.) This is one more reason to focus attention on the messages that are exchanged between business processes and to assure they are clearly stated.
When the telephone system adds another feature to their services, do they ask millions of customers to re-test the service? No way! Similarly, a few moments of thought leads to the conclusion that you cannot expect the constituents of a component or service to re-test their application when one of the clients causes a new feature to be added. To be safe and reliable, you must maintain automated test suites, and they must be comprehensive. With anything less than that, you are putting your enterprise at risk. I recommend that your developers learn how to develop “mocks” and study “need-driven development,” which is a variation on test-driven development. In addition, you need to think carefully about: How will you test and handle business-process exceptions? - Do your tests generate all of the process exceptions? - What is your default, fall-through, trouble-shooting strategy? When are heuristics involved in data translations? - By definition, a heuristic is situational - and may not be stable. - You need a clear strategy for packaging and testing data translation components.
If you don’t already operate some kind of configuration control system (not just a source repository), it is just a matter of time before you need one. Some work-a-rounds … short-term fixes: A simple strategy is to require each “service owner” to maintain a list of clients. Other strategies include tracking check-outs, registering users, and examining Java jar files (assemblies in C#). If you have a centralized source repository, you can do a lot with it. But you still need to know whom to contact when you need to send a notice. There are products you can buy that provide for this. But beware the technology fix. The important question is how the configuration data fits into your IT business processes. Unless you build it into your process, it will not be used, not be complete, and not be accurate.
The secret of being agile in Enterprise Integration is understanding that integration is really about improving communication between business processes. However, a great deal of confusion, strong coupling, and non-essential work will result if the communication is treated at the level of files, records, and/or fields (tables, rows, and/or columns). The trick to achieving agility in EAI and SOA is to apply object-oriented analysis and object-oriented design concepts at the level of business processes and services. The secret sauce is to focus on the messages exchanged between the business processes . The message is the feature, the user story, the scenario for release planning. It serves a similar function for test planning. A mistake I see frequently is to confuse the interface with the pipe. Focus on the message and implement it so that the transport layer does not matter. Do not code to the pipe; code to the message so that the choice of pipe does not matter. The same message can be delivered in multiple ways – JMS, REST, even file-transfer can be used for asynchronous request-response and event notice scenarios. Similarly, RMI, WSDL, REST, and JDBC an be used for question-answer scenarios. In any case, the message is the same regardless of the choice of transport. The other secret to agility is delegation . The project team can and should delegate work to specialists – legacy application maintenance teams for new senders and receivers, and a tooling team for new infrastructure tooling. This delegation of responsibility to other designers and developers requires management strategies borrowed from concurrent engineering – because it is concurrent engineering. The essence of agility is reduced time to solution. Simplification reduces time – focus on the message. Concurrent engineering starts with the interfaces. Once again, the secret is to focus attention on the messages exchanged between business processes.
Author: Richard A. Green - GreenRA@DTEenergy.com - 313-235-8672 Title: Principal Analyst - Role: Enterprise Architect Enterprise Architecture – Information Technology Services – DTE Energy DTE Energy 2000 Second Avenue MD 1032 GO Detroit, Michigan 48226
Agile SOA - Agile EAI
Agile SOA Agile EAIHow do we achieve agilityin Enterprise Integration?
What is Enterprise Integration? Enterprise Integration is the main activity of an in-house software development organization. Building software products is not our main activity. Why? DTE is in a mature industry and a relatively stable environment. As a result we can buy almost all of the software we need cheaper than we can build it. But we still need to create interfaces for almost everything we buy. Enterprise Integration means building application-to-application (A2A) and business-to-business (B2B) interfaces.
ContentsContext What is Enterprise Integration?Analysis The message is the feature-set.Planning Release planning is a multi-player game.Managing Concurrent engineering requires a different approach to project management.Testing Re-usable components and services require automated test suites.Deploying Deployment requires configuration control.
The message is the feature. The message is the feature in EAI and SOA. Analysis: Enterprise Patterns. How to identify the messages quickly and simply. Messages are speech acts. Identify messages in complete sentences. Nouns identify business objects (data entities).
The message is the feature … because: Enterprise Integration implements communications between business processes. The simple way to plan the integration features is to think about the messages that are exchanged between business processes. A major application is the IT proxy or interface for some business process. A2A or B2B -- It is the same thing. What if your enterprise decides to out-source a business process?
Analysis: What to look for …In-coming Messages Out-going Messages Request Response Work Order Request Work Order Complete Question Answer Work Order Status Progress / Exception Entity State (data) Business Object (data) Subscribed Event Published Event External Event Event I Observe or Validate of Interest to Me of Interest to Others Data Subscribed Data Published
Messages are speech acts.Express each message as a complete sentence. Request: Please install a new service S at location L. Response: Service S installed at location L. L Question: What is the status of work order W? Notice: Inventory asset A moved from location L1 to location L2.
Noun ==> Business Object Write each message as a complete sentence. Use nouns in a consistent way. Avoid application-specific jargon. Request: Please install a new service S at location L. L Response: Service S installed at location L. L Question: What is the status of work order W? W Notice: Inventory asset A moved from location L1 to location L2. L2
Message Patterns ==> Design Patterns Request-Response (work order … publish-subscribe) Client: Please replace meter M at location L with a meter of type T. Service: Meter M1 with reading R1 replaced with meter M2 reading R2. Question-Answer (query … remote procedure call) Client: Order status S = What is the status of work order W? Service: Work order W has status S. Event Notice (event … publish-subscribe) Publisher: Work order W will be delayed due to problem P. Subscriber: (updates local work management)
Example: Metering ServiceIn-coming Messages Out-going Messages Request: (We accept the request.) Request: (We delegate work to …) Please install a meter of type T Please install a meter of type T for service S at location L. for service S at location L. Query: (We answer the question.) Query: (We ask the delegate.) Metric M = How much energy Status S = What is status has been used at meter M? of work order W? Response: (When the work is Publish: (We publish the event.) done, the delegate responds with work Meter M is monitoring service S order results.) Work order W completed at location L. normally with results R Notice: (We subscribe to notice.) Meter M needs a new battery before date D.
Business Collaboration - a Use Case ScenarioFrom To MessageClient Metering Please replace meter M at location L with a meter of type T. Service (This is work order WO-A received from the Client.)Metering Field Please replace meter M at location L with a meter of type T.Service Service (This is work order WO-B delegating work to Field Service.)Field Metering Meter M1 reading R1 replaced with meter M2 reading R2.Service Service (Work order WO-B completion report received from Field Service.)Metering Client Meter M1 reading R1 replaced with meter M2 reading R2.Service (This is work order WO-A completion report sent to Client.)Metering Subscribers Meter M1 reading R1 replaced with meter M2 reading R2.Service (This event notice is published for all interested subscribers.)Subscriber (self) Each subscriber updates its world-model (stored in a database) to reflect the business event. There may be additional side-effects.
Release planning is a multi-player game.Planning and Budgeting Business Process -- owner / governor Application Portfolio -- owner / governor Component Library -- architect / plannerDesign and Construction Software Architect -- designer / planner / coordinator Project Manager -- planner / coordinator Software Developer -- builder / tester / operator / maintainer
How do you plan a release?A release is a coherent set of features for a specific audience (UI feature set) for a specific process (EAI feature set)Ask which features are necessary and sufficient. Is data replication necessary and sufficient? Is passive notification of exceptions sufficient? Is workflow needed?
Interface Packaging ==> Wrappers Create services by building wrappers on business services, within legacy applications, and the message delivery system. wrappersBusiness Process A Business Process B Message Delivery Service Application publish subscribe Application 1 4 Application Question Application 2 5 Application Answer Application 3 6
Business Service Wrapper Business Process == Business Service Request Application Response A Business Application Process B WorkflowPublished Event ApplicationSubscribed Event C
Legacy Application Wrappers Business Process == Business Service Request Application Response A Application Question-Answer Database BPublished Event ApplicationSubscribed Event C
Work Breakdown StructureSender Receiver Detect the event / need. Receive the request. Assemble the business objects Translate the business objects into the message. into local form. Send the message. Call the equivalent method.What about workflow? Workflow Responsibilities Workflow is essential Delegate work as needed. for handling Respond to process exceptions. business-process exceptions. Each process manages Publish events. its own workflow Feed back results. and no other workflow. Collect statistics.
Concurrent EngineeringWho should do which components? Each application’s maintenance team has the local knowledge to … Write logic to detect events. Write logic to translate between local representation and standard representation. Write application wrappers (API). Each business-process team has the knowledge to … Define and implement local workflows. Define and implement end-to-end tests. Design business-process monitoring (dashboard displays and reports). A tooling team has the skill and focus to … Create commonly-used components and services. Messages Create and maintain a repertoire of common business objects. and business objects The project coordinators have the skills and tools to … provide Identify project-to-project dependencies. clear and succinct points of contact Coordinate release plans. for design discussions and release coordination
SOA requires automated test suites.Why do you need automated test suites? If you have a service with 10-20 clients, and you are about to deploy a new release … You can ask all of the clients to re-test. -OR- Some services You can exhaustively test the service have 100’s to assure backward compatibility. of clients For a service of any complexity, automated tests are the only robust solution. By definition, integration tests cross system boundaries. Business process tests tend to require set-up, tear-down, etc.
Configuration Control Challenges Where is each component used? To whom do we send release notices? Where is this service used? When the service is down, who is affected? Which version is used where? When a method is deprecated, who is at risk?
Summary Business processes communicate via messages. That means the message is the feature (the user story) for agile EAI and agile SOA. Secret Sauce: Identify each message with a complete sentence. Secret Organization: Concurrent engineering reduces time to solution.
To achieve EAI / SOA agility: Focus on the business-processmessage exchanges.