TechDays 2010 Portugal - Event Driven Architectures - 16x9

2,500 views
2,334 views

Published on

Microsoft TechDays 2010 Portugal Session

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,500
On SlideShare
0
From Embeds
0
Number of Embeds
79
Actions
Shares
0
Downloads
41
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide
  • Inversion of communication. In contrast to the direct communication frequently used in a composite SOA, or other architectures, communications is done asynchronously through publishing events
  • Notable event happens, Initiating downstream action(s). Simple event processing is commonly used to drive the real-time flow of work—taking lag time and cost out of a business.
  • Ordinary events are screened for notability and streamed to information subscribersCommonly used to drive the real-time flow of information in and around the enterprise, which enables in-time decision making
  • Complex event processing evaluates a confluence of events and then takes action. The events (notable or ordinary) may cross event types and occur over a long period of time. The event correlation may be causal, temporal, or spatial. CEP is commonly used to detect and respond to business anomalies, threats, and opportunities.
  • Event Metadata. A good event-driven architecture has a strong metadata architecture. Event metadata includes event specifications and event processing rules. Event specifications must be made available to event generators, event format transformers, event processing engines, and subscribers. While no standards currently exist for event definition and processing notation, it is only a matter of time.Event Processing. The cores of event processing are the engine and the event occurrence data. Simple event engines are often homegrown. Complex event engines should be acquired from a CEP engine provider. Event occurrence data is typically persisted and retained for audit and trend analysis.Event Tooling. Event development tools are required to define event specifications and processing rules, and to manage subscriptions. Event management tools provide administration and monitoring of the event processing infrastructure, monitoring of event flows, and visibility into event generation and processing statistics.Enterprise Integration. An enterprise integration backbone plays a large role in event-driven architecture. Some of the required integration services are: event preprocessing (filters, routes, and transformations), event channel transport, service invocation, business process invocation, publication and subscription, and enterprise information access.Sources and Targets. These are the resources of the enterprise—applications, services, business processes, data stores, people, and automated agents—that generate events and/or perform an event-driven action.
  • The third layer, the event processing engine, is the logical location where the event is identified and the appropriate reaction is selected to be executed. Multiple reactions can be chosen by the engine for the same event. For example, the processing engine can decide to forward the event to other event processors. On the other hand it could choose to invoke a downstream activity like invoking a part of a business process, such as reserving some seats in the VIP area of a party, or send an email to the original requestor to tell him or her that everything went horribly wrong and he or she will get their money back. The primary tool for processing events on the Azure platform is the worker role; it continuously polls a queue for new events and responds to them by executing custom code. Every queuing mechanism discussed in the previous paragraph supports the queue-peek-lock mechanism to ensure that when a message is read from the queue, it does not really get removed from the queue. Until the processor has finished his work and explicitly removes it from the queue, the message will only go invisible for some time. If the processor fails to remove the message within an allotted time frame it will reappear as this indicates that the processor has failed to handle the message. This mechanism provides some sort of retry mechanism to make the processing a bit more resilient to the challenges that the cloud presents us. The .Net Service Bus provides another feature that is vital for the implementation of an event process layer, routers. A Router is a publish/subscribe message distribution primitive that allows to push events to subscribers. Routers can be configured through policies, just like queues, to distribute the events they receive to all or one of the subscribers, meaning they can be used to forward messages from one or multiple publishers to one or multiple subscribers. The real beauty however is the fact that queues and routers can subscribe and read from each other, allowing the composition of these primitives in a customized event processing engine, as shown in the example below. They are especially powerful when combined together with worker roles that take care of any complex routing, filtering logic or correlation of events.
  • EVENT GENERATORS. Every event is generated from a source. The source might be an application, data store, service, business process, transmitter, sensor, or collaboration tool (IM, email). An ordinary event may be evaluated for notability by an event preprocessor (router, filter), resulting in the generation of a new notable event. Because of the variety of event generators, not all events will be generated in the required format for event processing. In those cases, the events need to be transformed to the required (enterprise standard)format prior to being deposited in the event channel.EVENT CHANNEL. The event channel, typically a messaging backbone, transports standard formatted events between event generators, event processing engines, and downstream subscribers.EVENT PROCESSING. In the event processing layer, upon receipt, events are evaluated against event processing rules, and actions are initiated. The event processing rules and actions are defined in accordance to the needs of the interested parties, not of the event generators. The actions include invoking a service, initiating a business process, publishing the event out to a subscription hub, directly notifying humans or systems, generating a new event, and/or capturing the eventfor historical purposes. Events are processed by engines. A simple engine processes each event occurrence independently. A complex engine processes new event occurrences in context of prior and future events.DOWNSTREAM EVENT-DRIVEN ACTIVITY. A single event, or event correlation, may initiate numerous downstream activities. The invocation of the activity might be a push by the event processing engine (service invocation, business process initiation,notification) or a pull by subscribers of event publications. Subscribers might be humans, applications, active business processes, data warehouses, performance dashboards, and/or automated agents. Events should be published in the standard event format. Transformation to subscriber-specific formats is typically done by an enterprise integration backbone.
  • The circles at the top of the picture denote decoupling points (events), the linking pins between loosely coupled systems. At these decoupling points components can be connected and disconnected without any change of the connected systems (peers). All data exchange between the domains takes place only at this point and not at the lower levels.Within a reuse domain (see figure 1) a finer grained EDA may be implemented. The more fine-grained EDA, the more flexible the IT-systems are, but also to smaller the reuse domains will be.
  • BPEL, however, has a procedural nature and runtime implementations need a controlling BPEL-engine. This isnot a problem in case of SOA, but to achieve the aimed loose coupling of EDA it is. Good support for EDA wouldrather be a declarative model than a procedural model.
  • TechDays 2010 Portugal - Event Driven Architectures - 16x9

    1. 1. Event-Driven Architecture Como, Quando e Porquê?<br />ARC316<br />NunoGodinho<br />Partner & CTO @ ITech4All<br />nuno.godinho@itech4all.com<br />@NunoGodinho<br />
    2. 2. Session abstract<br />Session title<br />
    3. 3. Speaker Bio and Photo<br />Speaker Name<br />
    4. 4. Nuno Filipe Godinho<br />Partner & CTO @ ITech4all<br />Mail: Nuno.Godinho@itech4all.com<br />Nuno.Godinho@sapo.pt<br />Blogs: http://pontonetpt.com/blogs/nunogodinho<br />http://xamlpt.com/blogs/nunogodinho<br />http://weblogs.asp.net/nunogodinho<br />http://msmvps.org/blogs/nunogodinho<br />Twitter: @NunoGodinho<br />About Me<br />
    5. 5. Introduction<br />Event Driven Architecture<br />SOA vs EDA<br />How to Implement<br />Resources<br />Q&A<br />Agenda<br />
    6. 6. Introduction<br />
    7. 7. SOA is a synchronous RPC (mostly over Web services)<br />EDA is SOA<br />The best of EDA and SOA is combined in SOA 2.0<br />Introduction<br />Common Thoughts about SOA and EDA<br />
    8. 8. Moving towards on-Demand Business. Why?<br />Organizations tend to change their structure frequently<br />Parts of the business process are outsourced to External Partners<br />Departments and business units are seen now as service providers<br />Focus not only internally on the organization, but they are seeking for external markets to offer their services<br />Introduction<br />Trend Change<br />
    9. 9. Moving towards on-Demand Business. Why?<br />Everything is moving toward on-demand business where service providers react to impulses from the environment.<br />Loose coupling between application components<br />Introduction<br />Trend Change<br />
    10. 10. Linking application to simplify and automate a process, while avoiding changes to existing applications<br />Introduction<br />Sample Painpoint : EAI – Enterprise Application Integration<br />
    11. 11. File Tranfer<br />Import and Export Files<br />Common Issues:<br />Various Formats <br />Needs Translation<br />Manually Loaded<br />Introduction<br />How to Solve this painpoint? <br />
    12. 12. Shared Databases<br />Making integration on the Database Level<br />Several Applications share the same Database<br />Common Issues:<br />Data is tightly coupled to multiple applications<br />Incremental or partial updates are impossible<br />Introduction<br />How to Solve this painpoint? <br />
    13. 13. Web Services<br />Providing Services to automate the integration<br />Common Issues:<br />Requires services to be available at invocation<br />Results in multiple call stacks<br />Resource consuming<br />Yet another Remote Procedure Call (RPC, COM, Corba, DCOM, ...)<br />Introduction<br />How to Solve this painpoint? <br />
    14. 14. Messaging<br />Using Messaging mechanisms<br />Basics:<br />Defined data format<br />Asynchronous Operations<br />Minimized Coupling<br />Fault Tolerance<br />Data Freshness<br />Introduction<br />How to Solve this painpoint? <br />
    15. 15. Messaging<br />Defined data format<br />Contract between Producer and Consumer<br />Internal formats remain private<br />Integration is maintained at the edge of the application, to prevent internal changes to impact the interface<br />Introduction<br />How to Solve this painpoint? <br />
    16. 16. Messaging<br />Asynchronous Operations<br />Processes run on their own context<br />Source process continues<br />No need for the service to be always available<br />Introduction<br />How to Solve this painpoint? <br />
    17. 17. Messaging<br />Minimized Coupling<br />Additive Model reducing the changes needed to existing systems<br />Reduces dependencies<br />Increases service reuse<br />Enable Isolated Service Testing<br />Introduction<br />How to Solve this painpoint? <br />
    18. 18. Messaging<br />Fault Tolerance<br />Durable Messaging<br />Improved recoverability<br />Compensating Actions to handle partial failures<br />Introduction<br />How to Solve this painpoint? <br />
    19. 19. Messaging<br />Data Freshness<br />Favors small data transfer<br />Minimized data exchange latency<br />Introduction<br />How to Solve this painpoint? <br />
    20. 20. Events<br />Represent a change in state<br />Self-Contained<br />Pure and complete representation of an Event<br />No references to other data sources<br />Reduces dependencies, Loosens coupling <br />Uniquely indentified<br />Enabled idempotent handling events<br />Improves tracebility of related processing<br />Allows correlation with related events<br />Introduction<br />How to Solve this painpoint? <br />
    21. 21. Events<br />Time relevant, not time sensitive<br />Sourced using messaging<br />Observable <br />Published events can be observed by multiple subscribers<br />Event stream Processing<br />Introduction<br />How to Solve this painpoint? <br />
    22. 22. Event Types<br />Execution<br />Lifecycle<br />Management<br />Business<br />Introduction<br />How to Solve this painpoint? <br />
    23. 23. Event Driven Architecture<br />
    24. 24. Architecture pattern that orchestrates behavior around:<br />Production<br />Detection <br />Consumption of events as well as the responses they invoke.<br />A method for building enterprise systems in which events flow between decoupled components and services<br />A maintainable, sustainable and extensible model for building complex, distributed applications<br />Event Driven Architecture<br />What is EDA?<br />
    25. 25. Suited for Asynchronous, unpredictable environments<br />Extremely Loosely Coupled <br />Inversion of communication <br />Event Driven Architecture<br />What is EDA?<br />
    26. 26. Companies must manage and react to a large number events every day in real time<br />Real time trade settlement systems<br />Flight reservation system<br />Streaming stock data<br />Real time vehicle location for transportation companies<br />Stock Exchange Market systems<br />System designers normally must support both events and Services<br />Systems must be “Business Oriented”<br />Event Driven Architecture<br />Why do we need EDA?<br />
    27. 27. Concerns events that are directly related to specific, measurable changes of condition<br />Event Driven Architecture<br />Simple Event Processing<br />
    28. 28. Both ordinary and notable events happen<br />Event Driven Architecture<br />Stream Event Processing<br />
    29. 29. Allows patterns of simple and ordinary events to be considered to infer that a complex event has occurred<br />Event Driven Architecture<br />Complex Event Processing<br />
    30. 30. SOA-2: Event-Driven Service Oriented Architecture<br />Event Driven Architecture<br />What is Complex Event Processing?<br />Event Stream Processing<br />Complex Event Processing<br />time<br />1<br />2<br />3<br />4<br />5<br />6<br />7<br />8<br />9<br />Event pattern of significance where prompt detection and action can have materially impact<br />
    31. 31. SOA-2: Event-Driven Service Oriented Architecture<br />Event Driven Architecture<br />What is Complex Event Processing?<br />WHEN<br /> MSFT price moves outside 2% of MSFT-15-minute-VWAP<br />FOLLOWED-BY(<br />S&P moving by 0.5%<br />AND(<br />HPQ’s price moves up by 5%<br />OR<br />MSFT’s price moves down by 2%<br /> )<br />)<br />! <br />ALL WITHIN<br /> any 2 minute time period<br />S&P500<br />! <br />! <br />MSFT 15-MIN-VWAP<br />! <br />NYSE<br />NASDAQ<br />THEN<br />BUY MSFT<br /> SELL HPQ<br />time<br /><ul><li> Multiple data streams
    32. 32. Temporal sequencing
    33. 33. Complex event sequences</li></ul> Real-time Data Streams<br /><ul><li> Real-time constraints
    34. 34. Automated actions</li></li></ul><li>Event Driven Architecture<br />Implementation Components<br />
    35. 35. Event Driven Architecture<br />Event Processing Engine<br />
    36. 36. Event Generator<br />Event Channel<br />Event Processing Engine<br />Downstream Activity<br />Event Driven Architecture<br />Event Flow Layers<br />
    37. 37. SOA-2: Event-Driven Service Oriented Architecture<br />Event-driven Process<br />Automating the basic process<br />OTHER SERVICES<br />OTHER SERVICES<br />OTHER SERVICES<br />Investment manager wants order management system integrated with trading desk<br />ECN<br />ORDER MANAGEMENT SYSTEM<br />BROKER<br />TRADINGDESK<br />EXCHANGE<br />TRADER<br />
    38. 38. SOA-2: Event-Driven Service Oriented Architecture<br />Event-driven Processes<br />Operational improvement<br />OTHER SERVICES<br />OTHER SERVICES<br />COMPLIANCEENGINE<br />OTHER SERVICES<br />TRADINGDESK<br />President wants compliance engine to monitor trading activities to eliminate cash liability during market swings<br />INBOUNDTRANSFORMATION<br />OUTBOUNDTRANSFORMATION<br />?<br />ECN<br />ORDER MANAGEMENT SYSTEM<br />BROKER<br />EXCHANGE<br />TRADER<br />FUND MANAGER<br />
    39. 39. SOA-2: Event-Driven Service Oriented Architecture<br />Event-driven Processes<br />Keeping up with regulations<br />OTHER SERVICES<br />COMPLIANCE ENGINE<br />LOGGING SERVICE<br />OTHER SERVICES<br />OTHER SERVICES<br />TRADINGDESK<br />DB<br />Board wants trades logged for Sarbanes-Oxley and integrated with company-wide risk management<br />INBOUNDTRANSFORMATION<br />OUTBOUNDTRANSFORMATION<br />TOCORPORATERISK MGT<br />?<br />ECN<br />ORDER MANAGEMENT SYSTEM<br />BROKER<br />EXCHANGE<br />TRADER<br />FUND MANAGER<br />
    40. 40. SOA vs EDA<br />
    41. 41. SOA vs EDA<br />SOA Core Services<br />
    42. 42. SOA provides an organizing and delivery paradigm that derives greater value by reusing existing software solutions rather than duplicating capabilities<br />SOA establishes services as the mechanism by which needs and capabilities are brought together<br />SOA standardizes the necessary interfaces and behavior to support interaction<br />SOA vs EDA<br />What is SOA?<br />Service<br />S<br />Capabilities performed by one for another to achieve a desired outcome<br />Oriented<br />O<br />When capabilities are self-contained and independent to enable a collection of services to be linked together to solve a business problem<br />Architecture<br />A<br />The fundamental organization of a system embodied in its capabilities, their interactions, and the environment<br />
    43. 43. SOA vs EDA<br />Service Oriented Architecture<br />Applications are composed in design-time<br />Linear flow between services<br />Predictable behavior<br />Request/Response is common, and often overused<br />Event Driven Architecture<br />Applications are composed at run-time<br />Asynchronous components<br />Reactive behavior<br />
    44. 44. Support strong cohesion in the business processes, situations where all process steps are under one control<br />Commonly applied to:<br />Verticalinteraction between the hierarchical layers of functional decomposition<br />Functional request-and-reply processes such as man-machine dialogues; the user waits for an answer<br />Processes with a transactional nature which require commit and rollback facilities<br />Data enrichment in a message to be published to bring the message to its full content in a formal format<br />SOA vs EDA<br />When to use SOA?<br />
    45. 45. Support independency between business process steps<br />Commonly applied to:<br />Horizontal communication between tiers in a process chain<br />Workflow type of processes<br />Processes that cross recognizable functional organization borders, external (B2B) as well as internal<br />SOA vs EDA<br />When to use EDA?<br />
    46. 46. SOA vs EDA<br />Visualization<br />
    47. 47. How to Implement<br />
    48. 48. Using Web service technologies today, and additional SOAP-aware message queuing infrastructure.<br />Current ESB infrastructures provide a way of message queuing combined with Web service technologies.<br />SOA and EDA implementations must be regarded in the context of Business Process Management (BPM)<br />How to Implement<br />
    49. 49. Modern BPM-tools are based on BPEL (Business Process Execution Language)<br />Current BPEL implementation focuses strongly on the command-and-control model, the orchestration of services, and so on SOA<br />Beside orchestration BPEL - to a certain extend - also supports workflow, a kind of choreography, which goes in the direction of EDA<br />BPEL has a procedural nature<br />EDA would rather be a declarative model.<br />How to Implement<br />
    50. 50. Preparatory steps:<br />Model business requirements into functions at the granularity level of the desired autonomy. <br />Outline the application landscape to identify all affected systems. <br />Map the application landscape to the business function model. <br />Identify applications that cross functional borders as potential "agility bottlenecks" (assign a special high priority to those applications that are required to cross external organization borders). <br />Basis for the decoupled service boundaries <br />How to Implement<br />Scenarios where SOA and EDA can easily co-exist<br />
    51. 51. Resources<br />
    52. 52. Enterprise Integration Patterns (GregorHohpe and Bobby Woolf )<br />Patterns for Enterprise Architecture (Martin Fowler)<br />SOA Patterns (ArnonRotem-Gal-Oz, Eric Bruno, UdiDahan)<br />Resources<br />Books<br />
    53. 53. Event Processing: Designing IT Systems for Agile Companies (K. Chandy, W. Schulte)<br />Event-Driven Architecture: How SOA Enables the Real-Time Enterprise (Hugh Taylor, Angela Yochem, Les Phillips, Frank Martinez)<br />SOA Design Patterns (The Prentice Hall Service-Oriented Computing Series from Thomas Erl) <br />Resources<br />Books<br />
    54. 54. http://soapatterns.net/<br />http://www.enterpriseintegrationpatterns.com/<br />Resources<br />Sites<br />

    ×