Integrating Systems with NIEM using SOA


Published on

Presentation for the NIEM National Training Event on 9/30/2009. If I had could have renamed it then the title would be Service-Oriented Event-Driven.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Welcome, thank you for joining us in Track 4B for NIEM and SOA. I’m Chris Deweese and today I’d like to talk to you about the path to SOA and where NIEM comes into play. We’re going to cover some of the theory around SOA and how you can connect the dots using NIEM.That said, I’ll give you a fair warning and this is one I picked up from a fellow Microsoft MVP, Lee Brandt who says “These opinions are mine and they are subject to change. It could be next week, tomorrow, or even today” Because nothing I know is truly static and things change over time. This presentation has changed from when I submitted it several months back. Fortunately Anthony was ok with that<smile>
  • As I mentioned, my name is Chris Deweese and I’m currently a .NET architect/developer/project lead working in the justice sector. I spend most of my days working on integration & info sharing projects at all levels. This past July I was awarded with a Microsoft Most Valuable Professional which is a truly humbling experience because I in no way feel like I am on the same level as many of the MVPs out there, but someone seemed to think so, so I fooled the right person <LOL>Leave me feedback on Twitter or as I like to say “The Twitter” @cdeweese and don’t’ forget to use our hash tag #niemnteAnd finally I’m an avid Xbox player and you probably won’t hear from me much this fall when all the games come out before Christmas, and yes, my wife is fully supportive of the hobby. Enough of that, let’s start our journey to SOA harmony with NIEM.
  • *act like phone is ringing and answer*Oh, sorry, phone rang and I had to answer it. <LOL>Yesterday when Anthony Hoang and I were reviewing this presentation we heard tires screech and we ran to the window. The change in the normal traffic flow sounds signaled to us that we had to go look.As software engineers and business people we talk about process. But what are we really talking about? Process is really a series of events that happen which indicate a change in state, that we react to. My phone rings, it wasn’t ringing, but now it is, so I react and answer it. When we model processes in our systems traditionally it has been very linear and we follow a path. If this, then that. Do this, then go to that. Which is really more a limitation of the languages we crafted to define and execute commands within computer systems.But what if…
  • What if we modeled our systems not linearly, but using events, significant changes in state which we record? What if we look at our systems through the lens of what events occur in them?
  • What if we broadcast those events so that other systems know about those events? So that other systems could consume those events and react to them.What if my system could broadcast out some event and data about that event and other systems could pick that up, react to the change and make something happen within each independent system.
  • What would we see? Think about it, what could our systems do that we don’t realize because we’ve never put on that lens? Every process becomes a series of events – CourtInitiateWarrant, CourtInitiateWarrantReceived, LawEnforcementAcceptedWarrant, WarrantExecuted
  • What questions would we have, what concerns would be there?Is this possible? Can it be true? Are you serious? Are you trying to sell me something? <LOL>
  • How could we share the information about those events, the details, in a way that other systems can consume, react to, and do something useful? How could we make the information open and understandable by different systems on different technology stacks?Where’s he going with this…what conference am I at? <LOL>
  • How could we do this without disrupting our systems so that they remain loosely coupled? I don’t want my system to hang if that other guys is trying to consume one of my events. My system has work to do! I don’t trust the other guys system.How can we make sure our systems can remain autonomous. I don’t need the other system in order for mine to function, but it sure would be nice if I could get data from it.These are key questions to ask.
  • The message bus. Our vehicle to allow systems to publish events and information about those events. A way for systems to remain autonomous, loosely coupled by only being dependent on the bus and the events broadcast by the bus, not dependent on the other systems directly. A way for our messages to be dynamically subscribed to and consumed. The bus is the end game for SOA that’s really just the beginning. Because once you reach this you will see possibilities in your systems that you could not see before.
  • The bus provides the mechanism for you to publish events from your system and for others to subscribe to those events and be notified when an event is broadcast.
  • The bus allows systems to continue operating and publishing events while simultaneously receiving events that are subscribed to. It does this without requiring that all systems are on-line. If someone’s system goes off-line it can catch up when it comes back on-line.
  • Xml schema allows us to define the payload of our events in a way that can be shared and consumed by other systems and humans a like.Wonder what we might define that schema in?
  • The bus provides a new way for our systems to monitor series of events over time. As events occur we can build up a saga, or long running story that can take seconds or hours or days and when each piece is complete our system can alter it’s state and broadcast an event. Consider the process of a trial.In Justice, how many of these do we have? Court trials, incarcerations, investigations – these things take months, years. Series of related events that started with an event, a crime, an accident. Something that sets in motion all our processes that have their own check points where state changes and there is significant data associated with that change.
  • The bus presents us with possibilities we hadn’t considered. We can think about our systems differently because we have a new way to share information.
  • That’s impossible. Not impossible, Mr. Anderson. Improbable.Anyone want to guess the movie reference? <SMILE>
  • Hopefully someone in here is thinking “Ok, what could this possibly look like. I need something other than pictures of actual buses.”
  • How about an inside pic? <LOL> No, ok <smile>In this very basic diagram we see 4 systems that are directly connected to the bus and that System A publishes to the bus, B subscribes, C publishes and subscribes, and D subscribes. At a high level all of these systems are connected to the bus directly and have been modeled in a way to better integrate with the bus.But what if we cannot directly connect a system to the bus? Like I have these old mainframe systems still around (what’s up with that??) and now I have to put them on the bus. Forget it. It’s like pulling teeth to get the data out as it is!
  • So in this scenario we have several things going on here. System A needs to publish to the bus but can’t so we use an adapter. The adapter is pulling data from System A and determining when a state change has occurred and publishing that to the bus. System B is using an adapter in the reverse. As the bus publishes events to subscribers the adapter for system B is taking those events and turning them into units of work to be sent to system B.System C is using an adapter to both publish and subscribe. And System D is still directly connected because it’s awesome.A key thing to note in this scenario is that the adapter will take very large units of work from the bus, say “SubjectArrested” and turn that into small units of work for the subscribing system. So in System B, when we receive a SubjectArrested that may translate into us saving a base record, then saving a person record, or updating a charge record. So the adapter exists to handle the translation from the events on the bus to the units of work in the system it is adapting those events to.In all reality you would have some adapter even in scenarios like system D here, but we’re at a little higher level so I’m using this to illustrate the point.But what if this is across department or even organizational boundaries?
  • In this one we see how web services come into play to allow us to connect systems across organizational boundaries. Organization 1 is using the web service to publish and subscribe to messages. Organizations 2 and 3 are using web services to expose their systems to their partner organizations but they do not have a bus internally.What we can create here by modeling our information exchange is the series of events each partner will broadcast and what information will be sent. Partners can then implement the web services for events they will accept and can internally perform whatever work is necessary. This is really an evolution of the adapters in the previous diagram.Let’s talk about the process of modeling such an exchange and what we can leverage.
  • How many people here have taken the JIEM certification? For those of you that haven’t definitely look into it. JIEM is a methodology and tooling that helps you model exchanges and is a great fit for looking at the various events that need to happen when reviewing processes. It has a great synergy with NIEM and is definitely worth your time as it will help you visualize and conceptualize these things.
  • The NIEM IEPD process provides us a way to model the data that the systems will exchange. Each event can be described in NIEM with the data elements it will contain. This allows implementers and business analysts to talk about these events and what data will be sent for each event.Each event can be analyzed and integrators can determine what units of work each event represents to their systems. Great care must be taken to be sure that events contain the data needed or that integrators can derive or locate that data in order to perform their units of work.
  • Our partners can determine how they react to those events and what events are published from their systems. Leveraging adapters we can break the events down into the units of work each system supports. This is how we can integrate existing systems with NIEM using SOA and an event driven architecture. This is where changing how we think about our systems and taking a new perspective helps us see what we couldn’t see before. This is the unstated promise of SOA and the possibilities that are yet to be discovered.
  • This is our last stop because now it’s time for discussion. We covered the benefits of moving to a service bus: services can remain autonomous, loosely coupled, and the bus can provide you with a framework for concerns such as security, transactions, durability, and recoverability.We covered The benefits of using IEPDs to separate the content of your messages from the actual channels you’ll use to communicate be they message queues, web services, or other types of channels. IEPDs allow you to define the content of messages and send that definition to other service authors who want to connect to the bus and receive your event messages.And Finally, to get started you can begin by modeling the exchange, using techniques and tooling such as JIEM, to understand the process and the events that compose it. Then you can build IEPDs around those events and their messages so that all partners in the exchange have a clear understanding of what data is sent and when.I told you all of this not because I have done it all right, but because I’ve gotten it wrong. Two years ago when encountered the bus I didn’t accept it, it sounded too much like a product, but now I see what the benefits could have been, because I should have used a bus and I missed the mark. Do you users care if you use a bus or not? No, but they will care when your information sharing project eats their record and you have to explain what happened.Let’s open it up for discussion and questions – what are your thoughts or experiences around SOA and ESB or what questions do you have?
  • Integrating Systems with NIEM using SOA

    1. 1. Integrating Existing Systems with NIEM using SOA<br />Photo by Señor Codo via Flickr<br />
    2. 2. .NET Architect / Developer<br />Specialties: SOA, Xml, Distributed Messaging<br />Primarily work on System Integration & Information Sharing projects (Local, State & Federal)<br />2009 Microsoft Solutions Architect MVP<br />Twitter @cdeweese<br /><br />Avid Xbox player (when kids are sleeping)<br />About Chris Deweese<br />Photo by gwenael.piaser via Flickr<br />
    3. 3. Our world is made up of <br />events, changes in state,that we react to.<br />Photo by E01 via Flickr<br />
    4. 4. What if we model our systems around the events that occur within each system?<br />Photo by julipan via Flickr<br />
    5. 5. What if we broadcast those events?<br />Photo by Pedro Moura Pinheiro via Flickr<br />
    6. 6. What possibilities would we see?<br />Photo by escapethematrix via Flickr<br />
    7. 7. What questions would we ask?<br />Photo by Feuillu via Flickr<br />
    8. 8. How can we share information about these events?<br />Photo by zzathras777 via Flickr<br />
    9. 9. How could we better connect our systems while keeping them loosely coupled?<br />Photo by cowfish via Flickr<br />
    10. 10. On the path to SOA, <br />all roads lead to the bus.<br />Photo by Geoff LMV via Flickr<br />
    11. 11. The bus allows interested parties to subscribeto events and be notified when one occurs.<br />Photo by quaisi via Flickr<br />
    12. 12. The bus allows systems to remain autonomous.<br />Photo by Thomas Hawk via Flickr<br />
    13. 13. Xml schema allows us to define the data elements of our events.<br />Photo by tricky ™ via Flickr<br />
    14. 14. The bus allows our systems to monitor<br />processes, series of events, over time.<br />Photo by DWinton via Flickr<br />
    15. 15. The bus allows us to model our systemsin new ways.<br />Photo by simonbooth via Flickr<br />
    16. 16. Publish. Subscribe. It’s that simple.<br />Photo by nagillum via Flickr<br />
    17. 17. What does the bus look like?<br />Photo by crafterm via Flickr<br />
    18. 18. Photo by sludgegulper via Flickr<br />
    19. 19. Photo by dugspr via Flickr<br />
    20. 20. Photo by monojussi via Flickr<br />
    21. 21. JIEM can help us model the exchange.<br />Photo by Matito via Flickr<br />
    22. 22. The NIEM IEPD process can help us model<br />the data.<br />Photo by jepoirrier via Flickr<br />
    23. 23. We can build the exchangebased on significant events that are published to partners.<br />Photo by mag3737 via Flickr<br />
    24. 24. Last stop. Let’s talk.<br />Photo by riebschlager via Flickr<br />