Event Driven Architectures


Published on

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • As defined by Gartner SOA is classified into multiple styles RPC-Style , WOA-style , Event Driven SOA
  • Application generated events are generally Coarse grained. Database generated events are generally grained
  • kind of ID and version, you can often detect duplicates yourself and so not require to pay the performance cost of XA
  • Event Driven Architectures

    1. 1. Event Driven Architectures Avinash Ramineni
    2. 2. Agenda • • • • • • • What is Event Driven Architecture How is EDA related to SOA Event Capture/Generation Event Processing Event Delivery Guarantees Complex Event Processing Events, Internet of Things and Big Data
    3. 3. Event • Significant change in the state of an Object – – – – – Bank Transaction Students grade gets posted New Order gets created Customer Profile gets updated Change in Heart Beat • Characteristics of an Event – Events are broadcast – Communications are asynchronous – Events are fine grained
    4. 4. Event Driven Architecture (EDA) • Handling of the Events as they occur and the transfer of events among the systems in a loosely coupled way • Enables Situation Awareness and enables sense-and-respond behavior
    5. 5. Event Driven Architecture (EDA) • Characteristics of EDA – – – – – – – Producer/Consumers are not aware of each other Little or no statefullness Distributed Loosely Coupled Platform and Language Independent Reliable Fast and Highly Scalable
    6. 6. EDA extension of SOA Component software done right RPC-style SOA Computing with event objects SOA Web-oriented architecture (WOA) Web Architecture REST, URIs and HTTP Simple Web sites Event-driven SOA Event Processing Non-SOA EDA and CEP
    7. 7. EDA extension to SOA • EDA similar to SOA facilitates Agility by using Modular Design and Separation of Concerns • EDA fits well into organizations that have some SOA infrastructure in place • EDA adds another dimension / Possibilities to SOA – Events are generally stateless and do not have much content • While processing the events that do not have much data in them, the consuming applications call the services exposed to get the actual data • Publish – Subscribe Model – Similar to Asynchronous Messaging Architectures
    8. 8. How to generate Events • Application Generated – Application generates an event when a business transaction completes – Events generated by parsing the access logs/ server logs – Polling web pages – capturing the events from the services • Database Generated – Database Trigger generated Events • On insert/update/delete – During database replication
    9. 9. Fallacies of Distributed Systems • The network is reliable • Latency is zero • Bandwidth is infinite • The network is secure • Topology does not change • Transport cost is zero • Network is homogeneous
    10. 10. Usecase: 1 • Send an email when a customers profile gets updated Customer Profile – Assumptions: • There is an Application A that provides a updateProfileService • There is an Application B that provides a sendEmailService • All the Events are delivered with Once and Only Once SLA and the Events are processed Profile Updated Event Event A Channel Update DB B DB
    11. 11. Considerations while generating Events • The overhead involved in capturing events must be as low as possible to avoid impairing the performance of a business transaction. • Business Transaction and the Event generation needs to be in the same transaction – 1,2 needs to be part of a same transaction • Both 1 and 2 should succeed or both should fail 2. Profile Updated Event A Event Channel 1. Update DB B DB
    12. 12. Usecase: 1 – Web service • Successful Update of Profile -- Call a service to publish a Profile update Event • Can a database transaction and Web service call be made atomic? try{ try{ 1. Update Profile on DB 2. Call web service to publish profile updated event } catch(Exception e) { rollback(); } commit; 1. Update Profile on DB } catch(Exception e) { rollback(); } commit(); 2. Call web service to publish profile updated event • Can Compensating Transaction work ?
    13. 13. Usecase: 1 – JMS • Successful Update of Profile -- Put a message on to Message Broker A 2. Profile Updated Event (JMS message) Event Channel 1. Update DB DB B • Possible to make 1 and 2 into a single transaction but it requires – Distributed Transaction.
    14. 14. Distributed/XA Transactions (1) • Follows 2 Phase Commit protocol • Very Chatty protocol and does a lot of logging to be able to recover from any failure scenario. • Too much overhead to 99.9% of the cases to handle less than 0.1% of cases • Increases the chances for deadlocks on the database. • Lowers the overall performance of the system. • Bane of scalability. Grinds the entire system to halt by adding overhead to each and every transaction.
    15. 15. Distributed/XA Transactions (2) • • • • Availability of the Systems goes down. XA Configuration is complicated Difficult to test. Many People tend to believe that using JTA implementation of transaction manager will take care of a XA transactions • Many a time people end up using JTA Manager even while dealing with single resource.
    16. 16. Usecase: 1 – Local Database • Successful Update of Profile -- Put a message on to Message Broker A 1. Update DB Event Channel 2. Store Event 3. Batch process to send events to Event Channel DB B • 1 and 2 are in same transaction. 3 can be retried multiple times till they succeed.
    17. 17. Usecase: 1 – Local JMS • Successful Update of Profile -- Put a message on to Message Broker queue 2 A 3. Publish to Event Channel(JMS message) Event Channel Send Event to Local Queue 1. Update DB DB B • Possible to make 1 and 2 into a single transaction without a Distributed Transaction as long as Queue is backed by a same database
    18. 18. Usecase: 1 – Other Ways • Successful Update of Profile -- Put a message on to Message Broker / Call a web service to publish an Event • Have a Reconciliation Process in place to verify that there is an event generated for every Business Transaction – Have a unique Id (may be stored in DB) along with update and use that Id as Correlation Id with the Events Archive – If it events don’t tally up, recreate those events • Use the DB triggers to generate events – Write to a different local table – Call a web service to send the event to event channel (have seen people do that)
    19. 19. Event Delivery • Event Delivery Guarantees – Reliability • At least Once – Duplicate events can be delivered • At most Once – Some events can be lost • Once and only Once – Each is delivered Exactly once – Order of Delivery • In Order Guaranteed • In Order not Guaranteed
    20. 20. Considerations while Processing Events • Processing an Event and associated Business Transaction needs to be in the same transaction • 3,2 needs to be part of a same transaction • Both 3 and 2 should succeed or both should fail 1. Profile Updated Event Event Channel 2. Send Email B 3. Acknowledge Event Process success
    21. 21. Usecase: 1 – Consumer (1) • On Profile Update Event -- Call a service to send Email and on success acknowledge process successful onMessage try { sendEmail(); jms.commit() } catch (Exception e) { jms.rollback() } • What if jms.commit() fails ??
    22. 22. Usecase: 1 – Consumer (2) • If Jms.commit() fails , message gets delivered again onMessage try { if I have not processed this message successfully before { do some stuff in the database / Idempotency Logic /JMSRelivered flag jdbc.commit; } jms.commit() } catch (Exception e) { jms.rollback() }
    23. 23. Work Around for At-Least-Once • Idempotent Operation – An idempotent operations means that the result of a successful performed request is independent of the number of times it is executed. • Design consumer to be an Idempotent Consumer – By Understanding the Business rules it can be achieved relatively easy • Good Practice to make all the write operations Idempotent !! • Idempotency logic can be built on Event Ids, Business Ids – orderId, customerId, etc based on the Business scenario
    24. 24. What If In-order Delivery is required • In-order delivery of events – Limit the number of event publisher/consumer instance for these specific events to 1 • Brings up questions like if one of the event is having an issue – all the events of that event type can come to a stand still – AVOID the need for In-order Delivery of Events • By limiting the content in the Event , we can avoid a lot of cases where In-order delivery is required • Using Message Selectors to select and aggregate the events can solve a few of the cases
    25. 25. Usecase: 2 • Student Information System (SIS) maintain a list of students in a course roster. The students can be added or dropped from the course roster. SIS sends an event to Learning Management System (LMS) when the roster gets updated. • Lets us assume that SIS is sending the course roster as part of the event. • Processing of this Event requires the Roster Change events to be delivered in order to maintain the data from getting corrupted.
    26. 26. Usecase: 2 – Requires In-Order 1. Roster Change Event SIS Event Channel 2. Deliver Roster Change Event LMS DB LMS DB
    27. 27. Usecase: 2 - Solution • Remove the roster data from the event and just pass a roster Id • On Delivery of the Event , the LMS system would use the rosterId from the event, makes a service call to the source system (SIS) to get the latest Roster data and builds the classes in LMS • Even if the events are delivered out of order , the roster data on LMS will be always be the latest. • By making the processor on LMS idempotent, we can even avoid the database update
    28. 28. Usecase: 2 1. Roster Change Event SIS Event Channel 2. Deliver Roster Change Event 3. Get the latest roster for the Id LMS DB LMS DB 4. Update the LMS database
    29. 29. Best Practices • Make Event Publishers responsible for making sure that events are generated for every Business transaction and are published to the event channel • Make Event Consumers responsible for making sure that they react to the Stimuli from the event • Dashboards in place to make sure failed Events are detected and acted upon • Collects Stats to see if the events need to be made finer or coarser • Maintain Event Catalogs and self service capabilities to subscribe to events
    30. 30. Complex Event Processing (CEP) • CEP tap the events happening with in an organization and provide a situation awareness and enable better and faster decisions – Identify complicated events that are inferred from event pattern detection, event pattern interpretors and so on – Ex: Monitoring – capture events from ticket system, network load, performance and send out notification bring up new servers into the pool Traditional BI finds "needles in haystacks." Event-driven CEP finds "needles" in continuously arriving streams of "hay."
    31. 31. Events ,Big Data and Internet of Things • With Big Data hype and with the “capture-it-all“ approach and IoT , EDA and Event Driven programming will get a huge boost in the coming years. • Events and EDA due to their characteristics will be very well suited for Ubiquitous Computing • We might start seeing an advent of Complex Event Applications by being used by various gadgets – The concepts of EDA and Events will be a perfect Match for gadgets like Basis ,FIT
    32. 32. Questions ? Industry Experience avinash@clairvoyantsoft.com Twitter:@avinashramineni