Your SlideShare is downloading. ×
Event Driven Architecture
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Event Driven Architecture

10,146
views

Published on

A whirlwind tour of Event Driven Architecture, extensibility, Domain Driven Design, Command and Query Responsibility Segregation (CQRS) and Complex Event Processing

A whirlwind tour of Event Driven Architecture, extensibility, Domain Driven Design, Command and Query Responsibility Segregation (CQRS) and Complex Event Processing

Published in: Technology, Business

0 Comments
19 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,146
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
309
Comments
0
Likes
19
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Driven Architecture Lourens Naudé – SAPO Codebits 2010
  • 2. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 About • Developer @ http://www.wildfireapp.com • Ruby / C • Well versed full stack • Active interest in Event Driven technologies and related business processes
  • 3. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Agenda • Business and systems growth • Events • (Lack of) control • Event Driven Architecture • Domain Driven Design primer • Command and Query Responsibility Segregation • Complex Event Processing
  • 4. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Maintenance cost is multiple times more than development cost
  • 5. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 http://en.wikipedia.org/wiki/Open/closed_principle “Systems should ideally be open for extension, but closed for modification”
  • 6. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Driving forces for change • Fulfillment of business requirements • Business expansion: rollout of new processes to drive sales, market penetration etc • Mergers and acquisitions • Legislation
  • 7. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Modification risks • Introduces regression testing and a QA burden – never compromise a working system for a functional requirement • Profiling overhead for (soft) realtime systems • Excessive “refactoring” of existing stable interfaces • May fracture a business domain
  • 8. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 EDA imposes a system design that's geared for extension, interoperability and unanticipated use.
  • 9. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Interoperability VS Integration • Two or more systems performing a task / process together • Integration implies a formal link / bridge between two systems - high coupling • Building with interoperability in mind is the easiest way for long-term integration • Should be accounted for in system lifecycle as well: delays, cost, communication etc.
  • 10. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Governance (lay down the law) • Technology standards, SLAs, development processes • Cannot achieve interoperation without governance • Poor governance: inappropriate policies, people (habits) and business structures • Policies should be enforced at runtime in an auditable manner • Otherwise like criminal law without police and courts
  • 11. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Business layers
  • 12. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Business Layers • Physical world: sensors and users as actors • Transactional: physical world interactions with the business / domain layer • Intelligence: reporting tier and insights, most notably sales and marketing performance • This is where EDA really shines: visibility into business processes • Transparent operation for easier compliance and lower support costs
  • 13. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Events
  • 14. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Definition • Anything that happened (or didn't happen) • A change in state • A condition that triggers a notification • An event is always named in the past tense and is immutable CustomerAddressChanged InventoryRelocated OrderShipped
  • 15. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Three levels of detail • Basic fact that an event occurred • Event definition required in order to recognize an event • Event detail / context • Always attempt to capture events at the point of greatest information value (context) • This reduces downstream lookup and correlation overhead
  • 16. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Types • Notification: informational • Alert: notification that should cause a response • Event Object: record of an event - just data, no behavior • Business Event: meaningful for conducting trade activities • Complex: abstraction of one or more other events that represents summary level insights
  • 17. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Notification Types • Observational • Reports an event but does not change anything LowStock CartCleared • Transactional • Reports an event and causes change • Reliable transport • Originates from within the org. OrderShipped CardProcessed
  • 18. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 (Lack of) Control
  • 19. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Driven Programming • Control flow is driven by external events • NO central controller (main) - contrary to what we're taught starting off as devs • No well defined flow of data
  • 20. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 EDA Components • Event producers • Event consumers / listeners • Event processors • Event reactions • Messaging infrastructure
  • 21. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Lack of control in EDA • An EDA is driven by extensibility and not controllability • Transaction state is managed by events, not some central controller • Loosely coupled, asynchronous and stateless • Tell downstream components what happened, but not what to do. • Business events enable a system to operate under control, but without central control
  • 22. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Reduced coupling • There's always some coupling • We strive to reduce the degree of coupling • Always coupled to data
  • 23. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Stateless Order process • Modifying processes managed by a central controller requires changes to a component as well as the controller • Move state / data from the central controller to events
  • 24. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Driven Architecture
  • 25. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Use cases • Government: security, border control, taxes etc. • Health: stream processing + research • Compliance: guard against violations in cross country laws • Track and trace: Fedex, UPS etc. • Service Level Agreements
  • 26. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 When not to use an EDA • If the potential for reuse is very little • Problem being solved is lower down the application stack • Network bandwith and capacity is limited – serialization overheads and tends to be chatty • Security boundaries: immutability of events that carry sensitive data
  • 27. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 EDA Workloads • Simple event processing: narrow design • Stream processing: high message volume, but only relevant events is filtered • Complex Event Processing (CEP): able to correlate multiple streams
  • 28. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Interaction Types • Time-driven: actions initiated at specified times, eg. batch workflows etc. • Request-driven: initiated by client and completes when the provider replies, eg. client server • Event-driven: initiated by events and participants is open ended
  • 29. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Hybrid sytems Most event driven systems also include time and request driven components.
  • 30. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Components • Event producers • Event consumers / listeners • Event processors • Event reactions • Messaging infrastructure
  • 31. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Nervous System Analogy • Hand is the event producer • Spinal cord is the ESB • Brain is the event listener • Brain is the event processor • Hand is the event reaction – we pull away
  • 32. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Producers • Generates event data OR transforms it into a format that the EDA can understand • Data isn't always generated to feed an EDA, especially if the producer is a legacy component
  • 33. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Consumers / Listeners • Knows how to differentiate an event from data it receives • Can only “hear” events it's supposed to • Should be able to detect and interpret an event
  • 34. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Processors • Able to asses event importance and derives the next action to take • No value in not handling a perceived event • "Do nothing" is a valid reaction as well
  • 35. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Reactions • Automated responses • Notifications to participants or people • Human reactions / interventions for offline processes
  • 36. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Messaging Infrastructure • Communication link between EDA participants • Doesn't have to be a single piece of infrastructure - distinct backbones should be able to share information • Embrace standards and be as universal as possible • Promote decoupling between participants - pub/sub
  • 37. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Data in Event Driven Systems
  • 38. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Data • Business event data held in an event object • Context of what happened ProductCategorized('dsaed32wed', 'Coffee') OrderCancelled('ko90323', 'No funds')
  • 39. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 State Data • Core system data that changes as a result of operational activity Order('ko90323', 'Tom Jones', 13.45 'open') Product('dsaed32wed', 'Delta', 'Coffee', 5.8)
  • 40. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Reference Data • Stable / somewhat permanent - could change, but a lot less frequently than state data • Reference data would be replaced whereas state data would change incrementally. SupportedCurrency('USD') ProductCategory('Coffee') Address('Some Street', 'CA', 'USA')
  • 41. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Lifecycle of Data • New state data = state data + event data applied • State data is the accumulation of event data over time • Event data as deltas Product('dsaed32wed', 'Delta', 'Coffee', 5.8) ProductCategorized('dsaed32wed', 'Tea') Product('dsaed32wed', 'Delta', 'Tea', 5.8)
  • 42. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Domain Driven Design primer
  • 43. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Domain Driven Design • Suitable for behavior driven systems • Modeling is often data and not behavior driven • A Domain Model is a representation of relationships between Entities and Values • Try to have a business analyst review processes before introducing new architecture
  • 44. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Common Language • Participants: Developers, Experts and Stakeholders • Nouns: things with / without identity • Verbs: actions, hints at state changes • Domain models evolve through language • We often write different to how we speak and think • Most failures of systems is communication failures of people
  • 45. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Entities • Has an identity within the system • State can change Product Customer Order
  • 46. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Value Objects (Values) • No identity • Immutable • Represented by it's structure - structural equivalence Money OrderStatus Timezone
  • 47. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Domain Services • Actions, operations and activities • Business driven intent • Stateless • Should handle behavior that doesn't map well to any particular entity ShippingQuotesService PaymentProcessingService
  • 48. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Aggregates • A group of Entities that belong together • An instance of a process • A single unit of work – consistency / transactional boundary • eg. Order and Order items • Order item is useless without an Order • Remove Order item: reconstitute the Order, find item via Order, remove item • Usually identified by cascading delete requirements
  • 49. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Rental administration example
  • 50. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Command and Query Responsibility Segregation
  • 51. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Either perform an action, or return a result Not both.
  • 52. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Overview • CAP theorem: only can have 2 of 3 • Consistency steps aside for Availability and Partitionability • Somewhat stale data is often acceptable • Business requirements define by how much • Split systems into distinct command and query components (read and write)
  • 53. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Conceptual view
  • 54. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Benefits • Split teams: high level + good communicators on the write side, junior devs on the read side • Ability to scale command and reporting sides independently • Auditable system (dependent on history) • Selectively disable functionality during upgrades on the write side whilst the reporting side is still available • Lends itself well to behavior driven extension • Easy to test
  • 55. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 CQRS Components • Commands: drives change in the system • Internal events: captures intent (event store) • External events: republished to let other components know • Queries: examines state changes
  • 56. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Eventual Consistency • All data is stale, unless the whole system uses pessimistic locking • Most web apps request data *before* they render screens • Add 250ms to 500ms on average for rendering and additional server side blocking operations • Users don't react immediately to UI changes • Strictly consistent on the write side, relaxed (eventual) consistency on the read side
  • 57. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Rethinking UI / adaptors • Users don't think in data, but in tasks and processes • Cannot get benefits of CQRS without considering the UI first • Focus on capturing user intent explicitly • User experience should be an integral part of any business process
  • 58. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Ambigious UI UpdateComment / CommentUpdated
  • 59. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Task Driven UI ApproveComment / CommentApproved
  • 60. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Commands • A well defined task with parameters • Represents a process, not an entity • Distinct and transactional unit of work: wholly accepted or rejected • The only way to change data in a system • Just the right amount of granularity and intent to not cause concurrency conflicts of updates on large entities • Rejectable with exception or fault event • Idempotent – should never be reprocessed
  • 61. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Rethinking Validations • There's a fundamental difference between a validation and a business rule • Validation: well structured and formatted etc. • Business rule: uniqueness constraints etc. • Validate in client and in command handler - not all clients are well behaved • Valid commands can fail, most often through optimistic concurrency conflicts • Common for offline processes to reject requests
  • 62. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Queries and Reports • Read only view of data - examines state in the system • Never goes through the domain model • Table per UI view (assuming relational storage) • Single call to the reporting store returns all (or most) data for a screen • Minimal / no transformation - reporting tier is a cache
  • 63. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Aggregates and Events ---------- begin consistency boundary ---------- => Commands::CategorizeProduct("blends", "1517fd7efbffacdf928056656bbb61cf") rebuild Aggregates::Product(1517fd7efbffacdf928056656bbb61cf) from Events::ProductCreated({:category=>"coffee", :product_id=>"1517fd7efbffacdf928056656bbb61cf", :sku=>"XYZ", :name=>"Delta"}) Aggregates::Product @ version 1 <= Events::ProductCategorized({:category=>:blends, :product_id=>"1517fd7efbffacdf928056656bbb61cf"}) Aggregates::Product @ new version 2 ---------- end consistency boundary ----------
  • 64. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Code example class Aggregates::Product ...... def rename(name) apply Event(:ProductRenamed, uuid, name) end private def apply_product_renamed(event) @name = event.name end ...... end
  • 65. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Internal Events • Domain / Event store is append only: no deletes, ad hoc reads. • Compensating events to delete • Event store is a history of intent • Replay: events allow for reproducing error conditions by rebuilding an aggregate via snapshot up to just before a problem occurred • Republishes any handled events to other components
  • 66. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 External Events • Events handled by our domain that's been republished to the rest of our system • Communicates any state changes to the reporting tier • Integration / interoperation hook for business growth
  • 67. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 ORM Issues • Suitable for structural models only • Structural models tend to loose information • Mapping objects to structed data is difficult to optimize • CRUD – what changes on UPDATE ?
  • 68. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Testing • Command -> Domain -> [Event, Event …] • Able to assert behaviors in two ways • What happened ? • What didn't happen ? • Same pattern for testing the reporting side • Event -> View -> assert state from getters
  • 69. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Assertion example def test_relocate_inventory Domain << Command(:CreateInventory, uuid, 123, 'Delta', 'A', 'X',10) Domain << Command(:RelocateInventory, uuid, 'B', 'Y') assert_published :InventoryCreated, :InventoryRelocated end
  • 70. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Complex Event Processing
  • 71. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Overview • Connecting the dots • Summary level information gathered from base events • Patterns identified by business experts, but enforced at runtime by CEP software • Value is proportional to accuracy and relevance • Accurate data can however still be irrelevent • Allows for Management by Exception
  • 72. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Consciousness / awareness • Without consciousness can't react to situations without human intervention • SOA's the nervous system • EDA brings awareness • The core of Military systems – situational awareness on the battlefield derived from input streams
  • 73. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Stream • Linear (ordered by time) sequence of events • Produced by a single component / system • Little aggregation required • High throughput • Minimal insights into what happened • Example: market data feed
  • 74. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Cloud • Collection of Event Streams within a system • Aggregates streams into event sets • Much lower throughput • Very good situational awareness • Example: trading strategies that's dependent on multiple market feeds, realtime news sources etc.
  • 75. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Event Cloud layout
  • 76. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 CEP in Retrospective • Adds value to Event Driven systems • “Complex” is misleading – it simplifies input and streams • Matches real-time events from an Event Cloud with patterns from historical data and detect or predict situations • Event -> correlate -> assess -> decide -> action
  • 77. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 Summary • Restist modification • Tasks and intent – behavior matters • Communicate • Reduce coupling and avoid central control
  • 78. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929 @methodmissing http://www.github.com/methodmissing CQRS Example https://github.com/methodmissing/aftermath Thanks ! Questions ?