Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SOA & EDA follow up

101,387 views

Published on

This is a follow up to my SOA & EDA slide set http://www.slideshare.net/jeppec/soa-and-event-driven-architecture-soa-20 which caused some confusion/misinterpretations and a response from Jean-Jacque Dubray here http://www.slideshare.net/jdubray/soa-vs-eda

The discussion continued afterwards online and this slide set attempts to both sum of the different opinions and also respond to what I believe are misinterpretations (perhaps due to lack of a sound track). I've added links to the online discussion, articles, videos and blog posts that I hope can be of benefit to all interested in SOA & EDA.

Published in: Technology
  • Ali, please define "loose coupling" in a less than two paragraphs, then we'll talk. While you are at it, could explain how you achieve consistency in a distributed system.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Henrik, your argument : 'information dictates level of coupling' is absolutely wrong! Information or business processes do not dictate anything by themselves, our way of interpretation, modeling and engineering dose so instead. You guys do not understand a bit of what I'm talking about because SOA and ESB is your business, your only asset. I'm sorry but time comes and forces you to accept one simple rule: loose coupling is everything and much more superior to artificial , architect-made, 'need' for synchronization.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hi, the link to the Google doc is not working. Can you share the document again?
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Event driven architecture is a bad choice since you shutdown the feedback loop. The business rules decide if anything can be put on a event but it comes down to that the customer need to know what happens. You could 'store' the result of the event in some BPM but then you have coupled everything, and also a user doesn't know if the thing he just did actually happened. Just because you are adding events doesn't mean that you are decoupling anything. Yeah sure its easy to add listeners but thats a technical thing, the information is dictating the level of coupling. Events are made for decoupling technical things, not business related stuff. SOA 2.0 is even worse than 'SOA 1.0'.

    If you are using ESBs for anything else than protocol translation, abstractions of endpoints and service namespace abstraction, you are breaking encapsulation. Even service namespace abstraction is a close call of violating encapsulation. Why? Because the ESB has too much information about the service which it shouldn't have. This applies to orchestration as well.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Now to be very specific, since our positions are a lot closer. The core of the argument is: 'Is an ESB really the problem?' or is it the solution?

    The argument is that an ESB costs too much, it's too hard to use, ... whatever.

    The reality is that you have open source ESBs today: WSO2, JBoss, ... They are essentially free if you do not wish to pay for support, and paying for support is very reasonable. Cost is no longer an issue.

    Re-exposing an integration point from a system of record via WSO2 ESB for instance is trivial, you just create a 'proxy' and within minutes you have a 'managed' service.

    What the ESB gives you is the ability to hide change when necessary, keep the interface to the SoR stable until a consumer is ready to upgrade. If you believe in the definition of reuse I gave, you understand that you need a mechanism to hide change to old consumers. Both forward compatibility can be used and ESB proxy can be used.

    Now if you 'give up' completely on reuse, you apply what I call an 'immutable versioning strategy', whatever is currently consumed is duplicated and changed. That works well for 2-3 versions, after 25 versions of each service in production you are dead. I wrote a post on 'Understanding the cost of versioning: http://www.ebpml.org/blog2/index.php/2013/11/25/understanding-the-costs-of-versioning'
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

SOA & EDA follow up

  1. 1. SOA & EDA This is a response and also my recap of the discussions that followed Jean- Jacques Dubray’s response http://www.slideshare.net/jdubray/soa-vs-eda to my original SOA & EDA presentation http://www.slideshare.net/jeppec/soa-and-event-driven-architecture-soa-20 all of which originated from a “discussion” on InfoQ.com http://www.infoq.com/news/2014/08/microservices_ballmud Some of the discussion is still evolving in this document trying to make examples of the different approaches to SOAhttps://docs.google.com/a/tigerteam.dk/document/d/1Vt96dy3HUzSPD42k6l JFGNNequPIQPuRkFNU99CmBlg and a lot of interaction occurred on Twitter here the last entry as of this presentation https://twitter.com/clemensv/status/507188489163923456
  2. 2. Hopefully this response can correct what I believe are misinterpretations of my points of view and perhaps any clear up some presumptions about my lack of knowledge ;) I respect Jean Jacques deep knowledge of and contribution to SOA. Some of his ideas I fully agree with. The form of communication can be bit harsh at times. I hope this is because he feels deeply about the subject and not a sign of disrespect*? * Maybe I’m wrong? http://savas.me/2008/04/soa-cohesion-granularity-coupling-rudeness-eh/ http://www.ebpml.org/blog/76.htm
  3. 3. I’ve added my comments to Jean-Jacques comments to my original slide set to try and keep it simple. Where possible I’ve linked to online discussions, references, presentation, blog posts, videos of different people that give a more deeper or better explanation/information/argumentation than I can put into this slideset.
  4. 4. This slide set is not intended to do bashing or to muddy up the waters further. Hopefully it can clear up some confusion and hopefully people will benefit from it and use it to widen the discussion of and knowledge about different SOA approaches There are no consensus or conclusions as of now and I’m not sure there will be anytime soon…
  5. 5. A little while into the discussion it became clear to me that Jean- Jacque, on the basis of my SOA & EDA slide set, had deduced that I implied a versus between SOA and EDA. To be clear: I believe SOA and EDA are twins. I believe there is a need for both Orchestration as well as for an Event Driven approach to Service integration/coordination. For me it is not an either/or For me it’s that both are needed and they have different strengths and weaknesses. The requirements IMO determine which is most applicable.
  6. 6. To make the different SOA approaches more clear we created a document In https://docs.google.com/document/d/1Vt96dy3HUzSPD42k6lJFGNNequPIQ PuRkFNU99CmBlg Jean-Jacques wrote: The orchestration is not just a “nice to have” it is key to delivering a superior user experience. To which my response was: With this design and the requirements, we need immediate confirmation and therefore events are useless …. Just to be clear: I think Orchestration is useful and needed. A SOA without Orchestration doesn’t make sense in the same sense the SOA without events makes little sense. Events are useful across temporal boundaries where there’s no business expectation for immediate consistency or confirmation.
  7. 7. Which spawned the following Twitter discussion
  8. 8. So we agree that orchestration/coordination is important in SOA. The point of my SOA & EDA slide set was to introduce people to how business events can be used to trigger business processes
  9. 9. There are problems with SOA  Systems are more fragile  Development and Maintenance costs are higher  Your services are not being reused  You thought SOA would solve your integration problems, but they’ve gotten worse  No one wants to build or maintain services  System performance is worse Source: Jeppe Cramon
  10. 10. All these symptoms can be experienced if you do not understand what you are doing  Systems are more fragile This is because there are more moving parts and services are shared assets and when changes are needed it impacts/breaks more components  Development and Maintenance costs are higher To Quote Peter H. : “the second you introduce distributed, you need to leverage infrastructure that addresses network latency, fault tolerance, message serialization, unreliable networks, asynchronicity, versioning, varying loads within the application tiers etc […] Takes top level talent at the moment, not something all enterprises have access to.” Source: Jean-Jacques Dubray
  11. 11. Systems are more fragile • I think we agree on more moving parts and a distributed programming model are large contributors • What I’ve experienced is that the ad-hoc way many services/systems are coupled to each other result in more fragility than necessary – E.g. when I call a service that calls 3 services, which then again several other services. Suddenly the stability of the service I call is dependent upon the stability of the combined list of services. – Sometimes upfront (speculative) thinking in reuse can cause a landscape with too many small services (could perhaps be called the microservice/nanoservice trap) that provide very narrow functionality. To perform their job they need many other services to provide functionality. • Many call this ad-hoc integration for SOA – but it’s probably closer to EAI • Other times fragility is inflicted because temporal boundaries, across which there is no business expectation of immediate consistency or confirmation, are ignored – So in stead of decoupling what could have been decoupled we end still try to couple them because we only see one solution/style of integration.
  12. 12. Development costs are higher • We agree that integration contributes a lot of this • The main reason I hear from developers is that exposing their services over the company’s central ESB is very time consuming – I think we both agree that the ESB doesn’t need to be centralized, but most companies seem to end up in a broker/hub-spoke ESB model (probably because of licensing costs or organizational structure) • This of course varies with ESB product, organization, team skills, etc. • In the last project we used Oracle ESB (OSB) to expose an internal Services to the outside. This required security, proxying and message transformation, etc. – A model driven Java version of this code was built in 4 hours (including transformation, mediation and automated tests) – The OSB version took several days • IMO that’s not SOA’s fault, it’s not even the ESB patterns fault – it’s due to how some organization approach SOA and the tools they choose to use – which in turn give SOA a bad reputation
  13. 13. All these symptoms can be experienced if you do not understand what you are doing  Your services are not being reused In SOA reuse happens the other way around, when a new consumer wants to reuse a service, there is a very low probability that the service will be reused as is. It generally requires some small changes, when you change the service, what happens is that it is the old consumer which reuses the new version of the services, unless you adopt a “compatible versioning” strategy where new versions of services are compatible with older consumers. Source Jean-Jacques Dubray
  14. 14. Services are not being reused • I think it is very true that reuse happens the other way around – I’ve also seen Consumer driven interfaces/contracts yield good results. – Bill Poole has an interesting point on why Consumer Driven Contracts isn’t as necessary with his style of Autonomous services and Event driven Pub/sub http://bill-poole.blogspot.dk/2008/08/consumer-driven-contracts.html • IMO there are also other reasons for lack of reuse. Among the most common I’ve seen: – Often we are in reality just talking about a bunch of webservices with very little governance or even a service registry to help people figure out which services are already available – Not invented here – I will build it my self because … (bad excuse or good excuse) – It takes too long to get anything across the ESB service team, let’s just bypass them for now, build what we need and deal with it later – Some don’t see reuse as the end goal. Udi Dahan made a comment sometime ago stating “For me, I’ve given up on the whole reuse thing. I think it’s a crock. Has been since OO.” – Ref:http://www.udidahan.com/2008/11/01/soa-eda-and-cep- a-winning-combo/
  15. 15. All these symptoms can be experienced if you do not understand what you are doing  “You thought SOA would solve your integration problems, but they’ve gotten worse”  Well, following what you say, would certainly create a massive mess with events flying every where and everyone write code to correlate their content  world class encapsulation, cohesion and lose coupling  “No one wants to build or maintain services”  That’s why you need a dedicated service team  It also fit well the architectural paradigm of intent / consistency with SoR exposing Integration Points  “System performance is worse”  Not quite sure about that. I did some measurements on Synapse for instance and the added latency is less than 50 ms (measured around 10 ms, but just to be safe, I say 50) Source: Jean-Jacques Dubray
  16. 16. Create a massive mess with events flying every where • To detail my perspective further: I favor a composite service approach combined with business events. This doesn’t mean that events are used for everything or that orchestration isn’t used. • There’s video taped version of an earlier (but similar) talk http://www.tigerteam.dk/talks/IDDD-What-SOA-do-you-have-talk/ (from 16:15) where you can hear me say “I’m not saying it’s the perfect way to do it” and later say “it’s not really the scalable model” • The entire Matador example was intended to plant the idea that Business Events can play a role in temporally decoupling services, trigger business processes and that they CAN (in a few situations) be used to duplicate data between Services/SoR’s (depending on if you view services can be autonomous and therefore can own their business data) • I’ve also covered the challenge with using Events in a blog post where I wrote “Using Data duplication over Events is a well known, technology neutral, pattern for slowly separating monoliths into autonomous services, but it’s not the final solution for event based integration.”
  17. 17. “System performance is worse” • In general the ESB only adds very little latency – no argument there • Most of the performance problems I’ve seen is in solutions where you have services calling services calling services/systems (layered SOA of sorts) and in the mean time you have a client/user waiting (several seconds) for a response. And if one of these services were down, then “nothing” worked. – Nothing says layered SOA cannot perform, but the results I’ve seen point towards many latency and stability problems and a general “it’s not my problem that this services isn’t working, my service responded in time, etc.” • In the name of reuse I’ve seen one service call, from a portal widget, result in 12 additional services/system being called, where they with a different service boundary design could have sufficed with 3-4 services. – Udi has blog post about how he believes Composite UI’s can help you design different service boundaries http://www.udidahan.com/2012/06/23/ui-composition-techniques-for-correct-service-boundaries/ and http://www.udidahan.com/2012/07/09/ui-composition-vs-server-side-orchestration/ – On page 12 of your interaction document you argue why you believe this is not a good approach https://docs.google.com/document/d/1Vt96dy3HUzSPD42k6lJFGNNequPIQPuRkFNU99CmBlg • In you have multiple systems/SoR’s with overlapping responsibilities, such as 3 different Life Insurance policy systems, and you want to give the customer an overview of all Life insurance policies in one call, this should definitely happen as Orchestration that insulates any client from knowing anything about how many underlying systems there are and what their contracts might look like.
  18. 18. If you have arrived at the realization that SOA is a pain and costs too much without enabling business agility or reducing your costs, you are not the only one. You will have to change your approach to achieve the benefits of SOA If only no one would listen to hipsters, pundits and other self proclaimed experts Jean-Jacques Dubray’s comment in dark blue
  19. 19. Change your approach to SOA • If people are happy with the SOA they have, then I don’t think there should be any reason to change • If their SOA is a big mess (and thereby most likely broken) – and they’re unhappy - then searching for other approaches/guidance is IMO a good idea – Then they can hopefully decide who they trust • I hope we can agree that there are many styles of SOA (it seems each person has his) – We’ve discussed it online and it seems that reaching a consensus is hard • That fact that you find me a hipster or self proclaimed expert is probably valid to you and I’m not sure I can do more than this to change your mind (if at all possible)
  20. 20. Have our mindset changed over the last 40 years? Clearly yours has not Jean-Jacques Dubray’s comment in dark blue
  21. 21. My mind hasn’t changed? • I think having had the soundtrack could have helped with some of the misinterpretations • My mind has changed – Back in the days I viewed Layered SOA as the only option for SOA. Listening to people like Udi Dahan, Clemens Vaster, Bill Poole and Greg Young brought a different perspective on how Services could be build, how boundaries could be identified – how autonomy of service, command, documents and events play an important role together with existing SOA techniques such as Orchestration and mediation – Udi has written about Layered SOA challenges: http://msdn.microsoft.com/en-us/library/bb245672.aspx – And Bill Poole has also covered it: http://bill-poole.blogspot.dk/2008/05/layered-service-models-are-bad.html • I think being pragmatic is important – finding the right cost benefit solution depending on existing systems/services/skills/maturity. • This presentation might not sound pragmatic and it was not intended to be – it was intended to be “provocative” and make people stop and think to see another way to view SOA than the traditional Request/Response style integration and coupling services to services to services… • The critique of SOA problems in this talk is based on what most companies in reality have: – A bunch of webservices that communicate with other services using Request/Response over HTTP (i.e. RPC) and an ESB put in the middle as a broker. Business/IT alignment is for most of these companies completely alien and most of them end up creating a bigger and bigger mess every day, experience worse performance and try to mend it using caches (without having proper invalidation mechanism put in place) and other remedies (more hardware) – Actually Udi puts it quite clear (http://www.udidahan.com/2010/11/15/the-known-unknowns-of-soa/) • Most projects which bandy about the SOA acronym aren’t actually made up of services – they’re made up of XML over HTTP functions calling other XML over HTTP functions, eventually calling XML over HTTP databases. You can layer as much XML and HTTP as you want on top of it, but at the end of the day, most projects are just functions calling functions calling databases – in other words, procedural programming in the large, and no amount of SOAP will wash away the stink. • The slides on “have our mind changed” is meant to highlight that back in the days we IMO had something similar to autonomous services (data and logic present together where it was needed) and document based integration (a somewhat forgotten integration form) that Greg has been talking about lately https://skillsmatter.com/skillscasts/4081-greg-young- 4081
  22. 22. To be clear – none of the examples represent in my opinion SOA The 4 tenets of SOA 1. Services are autonomous Services ensure consistency – Encapsulation & Cohesion at a bigger scale. – A service is autonomous if it doesn’t rely on other services to complete its job 2. Services have explicit boundaries Service Interfaces are Intentional – Services should not be too familiar with each other. There’s a clear boundary to where the responsibilities of each services starts and ends – Boundaries are made unclear when a service rely on other services to complete its job 3. Services share contract and schema, not class or type – Encapsulation improved. We need not worry about how things are implemented (languages or platform dependent information) 4. Service interaction is controlled by a policy Ok – Controls under which rules/form of technical communication between services take place (e.g. using encryption incl. algorithm and keys) Whatever Jean-Jacques Dubray’s comment in dark blue
  23. 23. Service are autonomous • I agree that Services ensure Consistency • Service Autonomy is where I think we probably differ most – I think it’s a good idea to create autonomous services where the service owns its own data • On InfoQ you wrote “There is no "Autonomy" in SOA, services are not Autonomous. It's a fact, you can't fight it, you must embrace it. • Here’s how I’ve understood your approach: SoR’s are autonomous and Services are not autonomous. Service offer services to consumers through intentional interface (interface/impl. separation). Services plays the role of coordinators, which ensure that SoR’s eventually become consistent – Perhaps you’re spot on. I have used a similar approach in different situations (e.g. where I have multiple SoR’s – like multiple CRM’s, Life Insurance systems or SoR’s that should be replaceable). • What lead me to autonomous Services were the compelling cases and descriptions people such as Udi Dahan, Clemens Vaster, Bill Poole, etc. presented and the cases they’ve mentioned. It doesn’t just work on paper, but also in real life. • I especially like Udi’s approach with splitting Services into Business Components and splitting these into Autonomous Components (some call them Business Service and Services)http://www.udidahan.com/2008/04/23/visual-cobol-enterprise-processes-and- soa/
  24. 24. Services are autonomous Bill Poole gives a very good description of why he think services should be autonomous (http://bill-poole.blogspot.dk/2008/03/services-are-autonomous-but-why.html): “The services are autonomous tenet was intended by Don Box to mean that services are deployed and versioned independently from each other. I and some others such as Udi Dahan extend this constraint to also stipulate that services should be able to function autonomously at run time. That is, they should be able to as much as possible function without influence from other services. That is not to say that services live in a vaccuum. Services must of course be able to interact with other services. The nature of these interactions however should be such that no service should rely on the immediate availability of another service in order to handle a message. That is, we want to eliminate temporal coupling between services by limiting ourselves to asynchronous communication between services. The more autonomous our services are, the more loosely coupled they are. And loose coupling is what gives us business agility, which is the key business driver for SOA. Without this kind of autonomy between services, one service failing could cause a domino effect causing many other services across the enterprise to fail. In my opinion this degree of coupling is unacceptable.”
  25. 25. Services are autonomous • Clemens Vasters (Microsoft) has also been talking about Services being Autonomous (picture borrowed from one of his presentations):
  26. 26. Services are autonomous From our joined conversation on Twitter, Clemens explained that Autonomous Services is what allows the Azure teams to scale development or operations * Link provided by Clemens: http://channel9.msdn.com/Blogs/Subscribe/Wheres-the-ESB
  27. 27. And the discussion continued later https://twitter.com/clemensv/status/506868256234536960
  28. 28. Are Services autonomous? As a response to our discussion you (Jean- Jacques) posted a blog post which explains and exemplifies your view on services vs. SoR’s http://www.chorusjs.com/2014/09/03/everythi ng-i-needed-to-know-about-soa-i-learned-it-at-the- gas-station/
  29. 29. Arnon on Service Autonomy Arnon believes that services can be autonomous. If it’s the exact same kind of autonomy as Clemens, Udi and Bill I’m not sure
  30. 30. Are services autonomous or not? I think each person that presented their view on whether Services are autonomous or not, made compelling cases. The discussion didn’t get as wide as I had hoped, so there’s feedback from Arnon and Udi Dahan that didn’t get on the table.
  31. 31. A Service is • A technical authority for a specific business capability – very similar to Bounded Contexts • The owner of all the data and business rules that support this business capability – like Bounded Contexts When you have 500 apps, can you please show me where the bounded contexts are? There is the “theory” and the practice. Each app is a bounded context? A Service is equivalent to a Bounded Context Jean-Jacques Dubray’s comment in dark blue
  32. 32. A Service is like a bounded context Adrian Cockcroft (previously of Netflix) also describes Services (or Microservices) as being equivalent to Bounded Contexts* NetFlix has MANY autonomous services/bounded contexts which seems to work for them. The same seems to be the case for Azure (and as far I can tell also for Amazon). *See http://www.infoq.com/interviews/adrian-cockcroft-microservices- devops see section 3 and 4.
  33. 33. A Service is DDD’s Bounded Context “concept” is very flexible, so I’d like to present another definition of Service which Udi Dahan uses (see http://www.udidahan.com/2010/11/15/the-known-unknowns-of-soa/) : “A service is the technical authority for a specific business capability. Any piece of data or rule must be owned by only one service.” What this means is that even when services are publishing and subscribing to each other’s events, we always know what the authoritative source of truth is for every piece of data and rule. Also, when looking at services from the lense of business capabilities, what we see is that many user interfaces present information belonging to different capabilities – a product’s price alongside whether or not it’s in stock. In order for us to comply with the above definition of services, this leads us to an understanding that such user interfaces are actually a mashup – with each service having the fragment of the UI dealing with its particular data. Ultimately, process boundaries like web apps, back-end, batch-processing are very poor indicators of service boundaries. We’d expect to see multiple business capabilities manifested in each of those processes.
  34. 34. A Service is Bill Poole has a very nice description of why he thing it is important to align services with business capabilities: “The advantage of business capabilities is their remarkable level of stability. If we take a typical insurance organisation, it will likely have sales, marketing, policy administration, claims management, risk assessment, billing, payments, customer service, human resource management, rate management, document management, channel management, commissions management, compliance, IT support and human task management capabilities. In fact, any insurance organisation will very likely have many of these capabilities.” See http://bill-poole.blogspot.dk/2008/07/business-capabilities. html
  35. 35. Yes, we all know that autonomy and lose coupling are synonymous to consistency The more autonomous services are, the more loosely coupled they are. Loose coupling is what drives business agility – which is why we wanted SOA in the first place The CAR theorem postulates that you can’t have Consistency, Agility and a high degree of Relationality at the same time. You have to pick two. Jean-Jacques Dubray’s comment in dark blue
  36. 36. Loose coupling Requires good encapsulation and high cohesion We of course all know that encapsulation and high cohesion drive consistency Jean-Jacques Dubray’s comment in dark blue
  37. 37. CAR theorem Google didn’t find anything on CAR theorem But from your description, I don’t disagree with it.
  38. 38. WebServices and in general synchronous integration has nothing to do with real SOA Some call this pattern for SOA 1.0 to distinguish them selves from the old approach and the mess it causes Because “real” SOA is a protocol? Jean-Jacques Dubray’s comment in dark blue
  39. 39. Because “real” SOA is a protocol? • SOA is protocol and messaging pattern agnostic • The reason why I mentioned WebServices and Synchronous communication (Request/Response over HTTP) specifically is, as mentioned before, because that’s what 95% of all Services are (my experience – yours may vary). • What I’ve seen in many organizations is that many developers are NOT familiar with the fallacies of distributed computing, eventual consistency, orchestration, compensations, idempotence, etc. • Their contracts are mostly ad hoc – there’s very little alignment to anything but team structure • For many of the orchestrations I’ve seen have been built in these environments, with a lack of understanding for how much can go wrong with remote calls, it’s sometimes amazing that they don’t end up with more inconsistent systems/services (or maybe they do and they just don’t spot it).
  40. 40. Layered Architectures typically leaves all orchestration to a central manager (ESB) where business processes are coordinated through spaghetti code (BPEL) I am actually the (pre) author of that figure coming from BPMN. There is no ESB and no BPEL in it, this is actually a collaboration diagram, which I can be credited for pushing into BPMN BPEL is actually the only technology available today to achieve consistency at a reasonable cost. You simply have no freaking clue as to what you are talking about Jeppe: Yes it’s BPMN and not BPEL – my apologies Can we get on? Jean-Jacques Dubray’s comment in dark blue
  41. 41. You simply have no freaking clue as to what you are talking about • Because I show a BPMN diagram and say BPEL? – Honest mistake – I hope we can see past it • It was not the content of the diagram per se that was the point (BPMN or BPEL) – I needed a complex looking diagram to serve a point • Just to be clear - I have no problem with BPEL and neither with BPMN • The reason why I say BPEL spaghetti is because that’s what I typically see in companies where they try to mend badly designed services using BPEL (this is not BPEL’s fault – if anything it’s a statement of what can be accomplished using BPEL). • A centralized ESB’s is one of the problem I see many organizations battle with. In theory it should work (with clustering and all), but I’ve so many times heard “the ESB is slow again”, “none of our services work”, “who forgot to increase the DB quota for the ESB log tables”, etc. • The point of my talk was to raise awareness to alternatives to Layered SOA and Request/Response style integration. – Just like my presentation(s) on CQRS try to raise awareness that having a single (relational) model, which is what most choose per default, to fulfill the needs of an entire application is not the ONLY solution – and sometimes it’s directly the worst solution (and other times CQRS is like shooting sparrows using a canon)
  42. 42. These BPEL processes typically break all Service encapsulation as they are data and feature envious BPEL is actually the only technology available today to achieve consistency at a reasonable cost. You simply have no freaking clue as to what you are talking about This hurts our coupling an autonomy even further These past two slides is the reason why I am so mad at Jeppe. It shows that no only he has no understanding whatsoever as to what he is talking about, and he has no restraint in what he is claiming Jean-Jacques Dubray’s comment in dark blue
  43. 43. You simply have no freaking clue as to what you are talking about • As I’ve mentioned I believe Autonomous Services combined with events and a composite UI is a good approach to SOA – I know you disagree and that’s okay. – The reason for my comment “breaks all encapsulation” is because autonomous services, as explained by e.g. Udi Dahan, Clemens Vasters or Bill Poole, favor strong encapsulation and therefore dislikes Commands issued between services as it creates too much expectancy on behavior and often creates a strong temporal coupling. They prefer using events for interacting between services, when possible. • I’ve also seen the introduction of business events can help battle some of BPEL mess people create – to quote your book* “Events enable even looser coupling mechanisms because independent business processes may be triggered by an event. Otherwise we would have to specify a single business process encompassing several subprocess definitions, making it hard to change and maintain. However the use of events makes it just a “tiny bit” more difficult to understand the overall behavior of the system”. • I think your split in SoR’s and Services, focus on Business Entity Lifecycle (which I agree with) is interesting – maybe it’s better – maybe it’s not – that was what I was trying to find out on InfoQ before the discussion side railed • I’m not saying don’t use orchestration or BPEL – In reaction to commands/events you could run an Orchestration using BPEL (in slide 95 I used a Process Manager – which achieves roughly the same) * Link http://www.infoq.com/minibooks/composite-software-construction
  44. 44. You simply have no freaking clue as to what you are talking about • I’m also not saying “do EDA for everything” – be picky about what events you expose (in line with Pat Hellands “Data on the outside vs. data on the inside”*) – EDA is useless if you need authoritative answers or a quick response. • To me SOA and EDA are twins/two sides of the same coin or to quote your book “I’d like to make the point again that in reality, there is no difference between Service Oriented Architecture and Event Driven Architecture, they are part of the same programming model.” – I guess we agree on that they’re not two architectures separate from each other • What I was building up to in the original presentation was to present a different way of viewing/building Services that are Autonomous, which combined with event publishing and UI composition can result in a lower degree of coupling than many of the the layered approach I’ve seen – To quote Udi Dahan: “In short, the use of UI composition allows services to collect their own data, making it so anything outside that service doesn’t depend on those data structures which makes both development and versioning much easier. Technical failure conditions can be mitigated at infrastructure levels in most cases and other business logic concerns can be addressed asynchronously with respect to the data collection.” - http://www.udidahan.com/2012/07/09/ui-composition- vs-server-side-orchestration/ * http://msdn.microsoft.com/en-us/library/ms954587.aspx
  45. 45. What we have with classic layered SOA is a HARD COUPLED architecture You have no understanding as to what coupling means Jean-Jacques Dubray’s comment in dark blue
  46. 46. You have no understanding as to what coupling means • I didn’t quantify which types of coupling I meant – but you make a very general assumption • Coupling is a lot of thing – one of the important ones is dependencies, measured using – Afferent coupling (Ca) – who depends on you – Efferent coupling (Ce) – who do you depend on • For services these types of coupling are also interesting – Temporal coupling – Platform coupling – Spatial coupling • There’s also differences in behavioral coupling between e.g. Command, Event and Document based communication between services – http://bill-poole.blogspot.dk/2008/04/avoid-command-messages.html and http://bill-poole.blogspot.com/2008/04/when-event- messages-are-not-ideal.html – Ian S Robinson has covered Behavioral coupling vs. Temporal coupling here http://iansrobinson.com/2009/04/27/temporal-and- behavioural-coupling/ – And you also provided interesting insight here http://www.ebpml.org/blog/190.htm – I like Commands because they capture the intention of state transition (which can result in an Event being published). Excessive use of services calling each other using Command messages is where I think we can get into trouble. Use it when needed and avoid it if the requirements doesn’t mandate it. – I’ve seen composite UI’s, where each service provides it’s own UI parts, work quite well. These UI parts are prime candidates for sending Commands to their own Service implementation (keeping encapsulation and not exposing Commands outside the service boundary unless necessary). This is the approach that Udi Dahan is promoting (see e.g. http://vimeo.com/44235655). • With multiple channels to support this becomes an increased burden – so it’s not a panacea • What I’ve seen with many layered service solutions is that they often end up have a high degree of temporal and behavioral coupling (a service depending on many services that depend on my services and most integrated using Request/Response). – Sometimes this is because that’s how reality is and then one must deal with that the best possible (intentional interfaces or consumer driven interfaces can be good options to decouple service consumers) – IMO, when designing new services we should strive to minimize the degrees of coupling by having autonomous services, use events (when/where they make sense) and apply the composite application/ui pattern (when it’s possible) – Other times adding layers (or indirection) solve problems better, but they also bring their own issues – sometimes the tradeoffs favor adding a layer, other times they don’t IMO – A layered approach can have a reasonable coupling, but those I’ve seen/worked with quickly ended up knowing a little too much and doing too much. Probably mostly due to the lack of events for temporal decoupling
  47. 47. Service reuse multiplies our direct and especially indirect dependencies which creates high coupling Could please create the same picture with code duplication? Specially the stateful code that is needed for achieving consistency My new Service that wants to reuse other services Service that is going to be reused Reusable Service Reusable Service Reusable Service Reusable Service Reusable Service DB service Another Service that is going to be reused Jean-Jacques Dubray’s comment in dark blue
  48. 48. Same with duplication My point was that service reuse, like that depicted on the previous slide (services calling services calling services), should be scrutinized. If that’s reality (due to existing systems, fixed boundaries) then that’s different from a situation where you’re defining new service boundaries. Are the boundaries the best or could we introduce a better business domain/solution domain alignment, e.g. by aligning to business capabilities, with perhaps fewer service dependencies as a result? Bill Poole has some interesting perspective on reusehttp://bill-poole. blogspot.dk/2008/04/soa-and-reuse.html Udi has also covered some points to consider http://www.udidahan.com/2009/06/07/the-fallacy-of-reuse/
  49. 49. Business Events help us achieve autonomy, loose coupling and encapsulation  Encapsulation - because we don’t need to supply our services internal data (because no on else needs to know them) – internal data will only be accessible through the Services UI (which takes part in a Composite UI)  Events only need to explain WHAT happened and what it relates to and very few business data (which are typically only needed for replication scenarios) And so, how do you do exception handling and roll back with business events? You emit more message events? Shit happens you know. Where is the context? Jean-Jacques Dubray’s comment in dark blue
  50. 50. Exception handling / rollback • It really depends on the type of business process and the required business consistency requirements – I think your question was based on the assumption that I didn’t have any coordination (inside my service) to coordinate things – I do  – On slide 95 I use the Process Manager* pattern from the EIP book, which reacts to CustomerBilled (or CustomerBillingFailed). It will most like also use timeouts, to escalate problem if CustomerBilled/CustomerBillingFailed isn’t received within a certain period of time (e.g. NServiceBus’s Saga.RequestTimeOut) – The alternative to events is handling it with an orchestration • Udi has nice description on an event driven process where not all items are available in the Inventory here http://www.udidahan.com/2008/11/01/soa-eda-and-cep-a-winning-combo/ • Another blog post where Udi describes his Event driven vs orchestration approach here http://www.udidahan.com/2012/07/09/ui-composition-vs-server-side-orchestration/ – He ends with given this explanation for why he uses it: “I’ve spent the better part of 5 years travelling around the world talking to hundreds of people in quite a few business domains, and every single time I’ve found it possible to apply these principles.” • I know you and Udi have also “discussed” some of this online herehttp://www.ebpml.org/blog2/index.php/2010/08/31/soa-bpm-rest-udi-s-epiphany and here http://www.udidahan.com/2010/08/31/race-conditions-dont-exist/ • What I like about this using events for this, is that it mimics how business already runs (reactively). Even if we had a requirement that items had to be in stock before we accepted an order, we could still run into “race conditions” where Sam who was pulling the items from the Stock broke the last Vase that we just promised was in Stock. Should we rollback the order or should we handle it reactively and perhaps ship the items already in stock together, wait for the Vase to come back in stock and then ship everything together. • For me, Events are very useful for capturing/reacting to this type of exception/compensation handling and perhaps trigger other processes (like ordering items, etc.) nicely in combination with a Saga/ProcessManager/etc. Could the same be accomplished using command based interaction and Orchestration – sure – there are many ways to skin a cat (I’ve heard – haven’t actually tried to skin one ;)
  51. 51. What if an exception happens • Here’s a an old blog by Udi that covers some perspectives on this http://www.drdobbs.com/web-development/ dont-eda-between-existing-systems/228700784: When we use SOA/EDA (the same thing as far as I'm concerned), the top-level building block used is the Service. A service may make use of a number of existing systems to perform its work. The business-level events that we publish (and subscribe to) are done by the service, not the existing systems. If there's any orchestration/workflow that needs to be done as a result of a service receiving an event, it is done entirely internal to that service. Inter-service orchestrations don't really exist, as in there is no orchestration coordinator that is not in a service. And the orchestration coordinators within a service don't touch other services' back-end systems - if anything, they publish other business level events. Be aware: when just starting out on an SOA, you'll find that multiple Services make use of the same backend systems. This may be necessary, but not a desirable state to stay in for too long since it embodies the most insidious and invisible kind of inter-service coupling there is. “So what if no one picks the message up? Is that an error?” The answer is mu. If a service publishes a business-event (message) and no other services currently care, that's fine. It's not an error. Actually, you'd probably have some kind of infrastructure "queue" where messages that haven't been received more than X time get sent to, so that the event isn't "lost". On the other hand, within a service - if an existing system sends out a message that needs to arrive at another system, and that message doesn't arrive or isn't picked up "in time", that is an error. This is one of the advantages SOA brings to the table in terms of EDA (again, the same as far as I'm concerned). You get simple messaging semantics between services, while within the "sphere of control" of a service you need, and more importantly can do more complex messaging and orchestration. Bottom line: you need higher abstractions than your existing systems to employ EDA effectively.
  52. 52. Business Events Messages and Business Processes By publishing Events messages from our Services we can communicate with each other and also drive Business Processes Events drive business processes? You must be a candidate for the Turing Award. Jean-Jacques Dubray’s comment in dark blue
  53. 53. Events drive business processes? You must be a candidate for the Turing Award. • Why thanks! Though you were probably sarcastic ;) • It could be my Danish background which let me to use the word drive – perhaps the word trigger is more correct in English? • I cannot take credit for this idea – the inspiration among many came from here http://www.udidahan.com/2012/07/09/ui-composition-vs-server-side-orchestration/ orhttp://www.udidahan.com/wp-content/uploads/Event-Driven-Architecture-SOA-through-the-looking- glass.pdf or http://www.udidahan.com/2008/11/01/soa-eda-and-cep-a-winning-combo/ or http://arnon.me/2010/09/soa-contracts-events-ownership/ or http://www.udidahan.com/2011/04/08/integration-how-and-where/ or http://bill-poole. blogspot.dk/2008/04/avoid-command-messages.html or http://bill-poole. blogspot.dk/2008/05/layered-service-models-are-bad_20.html • Greg Young has similar material in his online course http://subscriptions.viddler.com/GregYoung • And I believe you’re saying something along those lines in your book “Events enable even looser coupling mechanisms because independent business processes may be triggered by an event. Otherwise we would have to specify a single business process encompassing several subprocess definitions, making it hard to change and maintain.” • Arnon Rotem-Gal-Oz also seems to agree
  54. 54. Events trigger processes IMO Bill Pooles description here makes a good point: http://bill-poole.blogspot.dk/2008/07/business-capabilities.html “Business capabilities are hierarchical in nature. We can decompose a business capability into smaller capabilities that support the broader capability. We only decompose a capability into smaller capabilities where there is sufficient distinctiveness between those sub-capabilities to warrant separate and distinct process maps to describe those sub-capabilities. Of course there are business processes that span multiple business capabilities. However, those processes are realised as event driven process chains. Each business process that executes within a business capability has the ability to raise business relevant events. Other business processes that execute within other capabilities subscribe to these events in order to trigger the execution of these processes. As such, end-to-end business processes are realised implicitly by virtue of the event publications and subscriptions between processes defined within each business capability. Note that this notion of business event publication and subscription is not a technology concern. It is simply a stable means of describing how business processes execute within an enterprise.”
  55. 55. Business Events example Sales Service Order Accepted Invoicing Service Retail System Order Accepted Customer Billed Message Channel The problem with SOA discussions is that when you take simplistic examples everything works. You need to “scale” these examples, real world stuff, not just PowerPoint Architectures We use the Order Accepted event message published from the Sales Service to drive the Invoicing/Billing of the customer. The billing part of the process also use Business Events, in this case Customer Billed Please provide end-to-end sequence diagrams including the implementation of the services, which you will discover is best implemented by … an orchestration that waits on the proper events Event, to indicate that its part of the process is completed. Because we use asynchronous messaging we can still accept orders in the sales service even though the invoicing services is down. The Order Accepted event message will remain in the Message Channel until the Invoicing Service is ready to process it. Jean-Jacques Dubray’s comment in dark blue
  56. 56. My example (with coordination)* * Taken from the working document https://docs.google.com/document/d/1Vt96dy3HUzSPD42k6lJFGNNequPIQPuRkFNU99CmBlg
  57. 57. Domain Events can also be used for data replication This is typically used for read-mostly master data that is needed in many business-domains. This way we avoid having to call a synchronous services to query the latest value(s) The response to any business event must be coordinated, Pub/Sub does not work in the business world Jean-Jacques Dubray’s comment in dark blue
  58. 58. Business Events must be coordinated • I agree, they need to be coordinated. • Udi Dahan covers some of this in his “Avoid a failed SOA”.http://www.infoq.com/presentations/S OA-Business-Autonomous-Components
  59. 59. Understanding SOA is actually pretty simple  Services provides a normalized interface to the systems of record  They ensure consistency of both the Queries and Command  Services serve data from the Systems of Truth  They make sure all SoRs are properly updated (or rolled-back)  Services call Integration Points on the SoR (not services)  Service interfaces are intentional, they express the intent of the consumer, how this intent is realized is under the control of the service  Technically there are two software paradigms that are essential to SOA  Mediation (e.g. Apache Synapse)  Orchestration (e.g. Apache ODE)  None of which are part of Jeppe’s presentation,  Yes, events are also part of the picture but only to reflect changes in the SoR when these changes are not “intentional”  The response to events is coordinated … by an orchestration Source: Jean-Jacques Dubray
  60. 60. Your definition of SOA • First of all: I have no problem with your definition of SOA – I think it can work quite well • Second: I think that Udi Dahans/Clemens Vasters/Bill Poole’s (and others) have a SOA definition, involving autonomous services, that is also compelling and also has a proven track record (which has also been my experience). • IMO too few people know EDA in relationship to SOA – my presentation had as a goal to present business events + EDA and some of the possibilities (and in the voice track add more distinctions and pitfalls) • I’m not sure that a consensus can be reached – but I applaud every attempt to widen the understanding of the different approaches. – As Clemens wrote: • I hope we can continue a good discussion forward 

×