Event Driven Architecture

Event Driven Architecture Lourens Naudé – SAPO Codebits 2010 Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

About • Developer @ http://www.wildfireapp.com • Ruby / C • Well versed full stack • Active interest in Event Driven technologies and related business processes Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Agenda • Business and systems growth • Events • (Lack of) control • Event Driven Architecture • Domain Driven Design primer • Command and Query Responsibility Segregation • Complex Event Processing Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Maintenance cost is multiple times more than development cost Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

http://en.wikipedia.org/wiki/Open/closed_principle “Systems should ideally be open for extension, but closed for modification” Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Driving forces for change • Fulfillment of business requirements • Business expansion: rollout of new processes to drive sales, market penetration etc • Mergers and acquisitions • Legislation Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Modification risks • Introduces regression testing and a QA burden – never compromise a working system for a functional requirement • Profiling overhead for (soft) realtime systems • Excessive “refactoring” of existing stable interfaces • May fracture a business domain Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

EDA imposes a system design that's geared for extension, interoperability and unanticipated use. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Interoperability VS Integration • Two or more systems performing a task / process together • Integration implies a formal link / bridge between two systems - high coupling • Building with interoperability in mind is the easiest way for long-term integration • Should be accounted for in system lifecycle as well: delays, cost, communication etc. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Governance (lay down the law) • Technology standards, SLAs, development processes • Cannot achieve interoperation without governance • Poor governance: inappropriate policies, people (habits) and business structures • Policies should be enforced at runtime in an auditable manner • Otherwise like criminal law without police and courts Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Business layers Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Business Layers • Physical world: sensors and users as actors • Transactional: physical world interactions with the business / domain layer • Intelligence: reporting tier and insights, most notably sales and marketing performance • This is where EDA really shines: visibility into business processes • Transparent operation for easier compliance and lower support costs Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Events Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Definition • Anything that happened (or didn't happen) • A change in state • A condition that triggers a notification • An event is always named in the past tense and is immutable CustomerAddressChanged InventoryRelocated OrderShipped Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Three levels of detail • Basic fact that an event occurred • Event definition required in order to recognize an event • Event detail / context • Always attempt to capture events at the point of greatest information value (context) • This reduces downstream lookup and correlation overhead Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Types • Notification: informational • Alert: notification that should cause a response • Event Object: record of an event - just data, no behavior • Business Event: meaningful for conducting trade activities • Complex: abstraction of one or more other events that represents summary level insights Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Notification Types • Observational • Transactional • Reports an event • Reports an event but and does not change causes change anything • Reliable transport • Originates from within the org. LowStock OrderShipped CartCleared CardProcessed Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

(Lack of) Control Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Driven Programming • Control flow is driven by external events • NO central controller (main) - contrary to what we're taught starting off as devs • No well defined flow of data Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

EDA Components • Event producers • Event consumers / listeners • Event processors • Event reactions • Messaging infrastructure Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Lack of control in EDA • An EDA is driven by extensibility and not controllability • Transaction state is managed by events, not some central controller • Loosely coupled, asynchronous and stateless • Tell downstream components what happened, but not what to do. • Business events enable a system to operate under control, but without central control Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Reduced coupling • There's always some coupling • We strive to reduce the degree of coupling • Always coupled to data Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Stateless Order process • Modifying processes managed by a central controller requires changes to a component as well as the controller • Move state / data from the central controller to events Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Driven Architecture Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Use cases • Government: security, border control, taxes etc. • Health: stream processing + research • Compliance: guard against violations in cross country laws • Track and trace: Fedex, UPS etc. • Service Level Agreements Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

When not to use an EDA • If the potential for reuse is very little • Problem being solved is lower down the application stack • Network bandwith and capacity is limited – serialization overheads and tends to be chatty • Security boundaries: immutability of events that carry sensitive data Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

EDA Workloads • Simple event processing: narrow design • Stream processing: high message volume, but only relevant events is filtered • Complex Event Processing (CEP): able to correlate multiple streams Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Interaction Types • Time-driven: actions initiated at specified times, eg. batch workflows etc. • Request-driven: initiated by client and completes when the provider replies, eg. client server • Event-driven: initiated by events and participants is open ended Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Hybrid sytems Most event driven systems also include time and request driven components. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Components • Event producers • Event consumers / listeners • Event processors • Event reactions • Messaging infrastructure Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Nervous System Analogy • Hand is the event producer • Spinal cord is the ESB • Brain is the event listener • Brain is the event processor • Hand is the event reaction – we pull away Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Producers • Generates event data OR transforms it into a format that the EDA can understand • Data isn't always generated to feed an EDA, especially if the producer is a legacy component Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Consumers / Listeners • Knows how to differentiate an event from data it receives • Can only “hear” events it's supposed to • Should be able to detect and interpret an event Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Processors • Able to asses event importance and derives the next action to take • No value in not handling a perceived event • "Do nothing" is a valid reaction as well Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Reactions • Automated responses • Notifications to participants or people • Human reactions / interventions for offline processes Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Messaging Infrastructure • Communication link between EDA participants • Doesn't have to be a single piece of infrastructure - distinct backbones should be able to share information • Embrace standards and be as universal as possible • Promote decoupling between participants - pub/sub Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Data in Event Driven Systems Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Data • Business event data held in an event object • Context of what happened ProductCategorized('dsaed32wed', 'Coffee') OrderCancelled('ko90323', 'No funds') Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

State Data • Core system data that changes as a result of operational activity Order('ko90323', 'Tom Jones', 13.45 'open') Product('dsaed32wed', 'Delta', 'Coffee', 5.8) Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Reference Data • Stable / somewhat permanent - could change, but a lot less frequently than state data • Reference data would be replaced whereas state data would change incrementally. SupportedCurrency('USD') ProductCategory('Coffee') Address('Some Street', 'CA', 'USA') Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Lifecycle of Data • New state data = state data + event data applied • State data is the accumulation of event data over time • Event data as deltas Product('dsaed32wed', 'Delta', 'Coffee', 5.8) ProductCategorized('dsaed32wed', 'Tea') Product('dsaed32wed', 'Delta', 'Tea', 5.8) Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Domain Driven Design primer Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Domain Driven Design • Suitable for behavior driven systems • Modeling is often data and not behavior driven • A Domain Model is a representation of relationships between Entities and Values • Try to have a business analyst review processes before introducing new architecture Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Common Language • Participants: Developers, Experts and Stakeholders • Nouns: things with / without identity • Verbs: actions, hints at state changes • Domain models evolve through language • We often write different to how we speak and think • Most failures of systems is communication failures of people Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Entities • Has an identity within the system • State can change Product Customer Order Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Value Objects (Values) • No identity • Immutable • Represented by it's structure - structural equivalence Money OrderStatus Timezone Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Domain Services • Actions, operations and activities • Business driven intent • Stateless • Should handle behavior that doesn't map well to any particular entity ShippingQuotesService PaymentProcessingService Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Aggregates • A group of Entities that belong together • An instance of a process • A single unit of work – consistency / transactional boundary • eg. Order and Order items • Order item is useless without an Order • Remove Order item: reconstitute the Order, find item via Order, remove item • Usually identified by cascading delete requirements Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Rental administration example Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Command and Query Responsibility Segregation Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Either perform an action, or return a result Not both. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Overview • CAP theorem: only can have 2 of 3 • Consistency steps aside for Availability and Partitionability • Somewhat stale data is often acceptable • Business requirements define by how much • Split systems into distinct command and query components (read and write) Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Conceptual view Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Benefits • Split teams: high level + good communicators on the write side, junior devs on the read side • Ability to scale command and reporting sides independently • Auditable system (dependent on history) • Selectively disable functionality during upgrades on the write side whilst the reporting side is still available • Lends itself well to behavior driven extension • Easy to test Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

CQRS Components • Commands: drives change in the system • Internal events: captures intent (event store) • External events: republished to let other components know • Queries: examines state changes Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Eventual Consistency • All data is stale, unless the whole system uses pessimistic locking • Most web apps request data *before* they render screens • Add 250ms to 500ms on average for rendering and additional server side blocking operations • Users don't react immediately to UI changes • Strictly consistent on the write side, relaxed (eventual) consistency on the read side Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Rethinking UI / adaptors • Users don't think in data, but in tasks and processes • Cannot get benefits of CQRS without considering the UI first • Focus on capturing user intent explicitly • User experience should be an integral part of any business process Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Ambigious UI UpdateComment / CommentUpdated Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Task Driven UI ApproveComment / CommentApproved Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Commands • A well defined task with parameters • Represents a process, not an entity • Distinct and transactional unit of work: wholly accepted or rejected • The only way to change data in a system • Just the right amount of granularity and intent to not cause concurrency conflicts of updates on large entities • Rejectable with exception or fault event • Idempotent – should never be reprocessed Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Rethinking Validations • There's a fundamental difference between a validation and a business rule • Validation: well structured and formatted etc. • Business rule: uniqueness constraints etc. • Validate in client and in command handler - not all clients are well behaved • Valid commands can fail, most often through optimistic concurrency conflicts • Common for offline processes to reject requests Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Queries and Reports • Read only view of data - examines state in the system • Never goes through the domain model • Table per UI view (assuming relational storage) • Single call to the reporting store returns all (or most) data for a screen • Minimal / no transformation - reporting tier is a cache Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Aggregates and Events ---------- begin consistency boundary ---------- => Commands::CategorizeProduct("blends", "1517fd7efbffacdf928056656bbb61cf") rebuild Aggregates::Product(1517fd7efbffacdf928056656bbb61cf) from Events::ProductCreated({:category=>"coffee", :product_id=>"1517fd7efbffacdf928056656bbb61cf", :sku=>"XYZ", :name=>"Delta"}) Aggregates::Product @ version 1 <= Events::ProductCategorized({:category=>:blends, :product_id=>"1517fd7efbffacdf928056656bbb61cf"}) Aggregates::Product @ new version 2 ---------- end consistency boundary ---------- Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Code example class Aggregates::Product ...... def rename(name) apply Event(:ProductRenamed, uuid, name) end private def apply_product_renamed(event) @name = event.name end ...... end Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Internal Events • Domain / Event store is append only: no deletes, ad hoc reads. • Compensating events to delete • Event store is a history of intent • Replay: events allow for reproducing error conditions by rebuilding an aggregate via snapshot up to just before a problem occurred • Republishes any handled events to other components Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

External Events • Events handled by our domain that's been republished to the rest of our system • Communicates any state changes to the reporting tier • Integration / interoperation hook for business growth Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

ORM Issues • Suitable for structural models only • Structural models tend to loose information • Mapping objects to structed data is difficult to optimize • CRUD – what changes on UPDATE ? Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Testing • Command -> Domain -> [Event, Event …] • Able to assert behaviors in two ways • What happened ? • What didn't happen ? • Same pattern for testing the reporting side • Event -> View -> assert state from getters Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Assertion example def test_relocate_inventory Domain << Command(:CreateInventory, uuid, 123, 'Delta', 'A', 'X',10) Domain << Command(:RelocateInventory, uuid, 'B', 'Y') assert_published :InventoryCreated, :InventoryRelocated end Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Complex Event Processing Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Overview • Connecting the dots • Summary level information gathered from base events • Patterns identified by business experts, but enforced at runtime by CEP software • Value is proportional to accuracy and relevance • Accurate data can however still be irrelevent • Allows for Management by Exception Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Consciousness / awareness • Without consciousness can't react to situations without human intervention • SOA's the nervous system • EDA brings awareness • The core of Military systems – situational awareness on the battlefield derived from input streams Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Stream • Linear (ordered by time) sequence of events • Produced by a single component / system • Little aggregation required • High throughput • Minimal insights into what happened • Example: market data feed Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Cloud • Collection of Event Streams within a system • Aggregates streams into event sets • Much lower throughput • Very good situational awareness • Example: trading strategies that's dependent on multiple market feeds, realtime news sources etc. Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Event Cloud layout Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

CEP in Retrospective • Adds value to Event Driven systems • “Complex” is misleading – it simplifies input and streams • Matches real-time events from an Event Cloud with patterns from historical data and detect or predict situations • Event -> correlate -> assess -> decide -> action Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Summary • Restist modification • Tasks and intent – behavior matters • Communicate • Reduce coupling and avoid central control Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

Questions ? @methodmissing http://www.github.com/methodmissing CQRS Example https://github.com/methodmissing/aftermath Thanks ! Wildfire Interactive, Inc. | 167 Hamilton Ave, Palo Alto CA 94301 Suite 200 | (888) 274-0929

http://blog.methodmissing.com	783
http://www.xydo.com	1
http://www.hanrss.com	1
http://feeds.feedburner.com	1

Event Driven Architecture

by Lourens Naudé on Nov 12, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

4 Embeds 786

Statistics

Event Driven Architecture — Presentation Transcript