Eda on the azure services platform
Upcoming SlideShare
Loading in...5
×
 

Eda on the azure services platform

on

  • 2,162 views

The slides for my talk on how to implement an event driven architecture on top of the azure platform

The slides for my talk on how to implement an event driven architecture on top of the azure platform

Statistics

Views

Total Views
2,162
Views on SlideShare
2,145
Embed Views
17

Actions

Likes
0
Downloads
30
Comments
0

1 Embed 17

http://www.slideshare.net 17

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • Services do not communicate directlyInstead event generators constantly report what is happening Resulting in a stream of eventsEvent consumers may subscribe to these events And might react to them
  • Event generatorEvent sources come in many forms: Devices, humans, software, nature, …Event generators capture events from the sources, and translate them into a format understood by the event channelEvents are pushed onto the event channel by the generatorEvent ChannelThe event channel is a communication medium,transports the events throughout the systemPreferably it is durable, supports local transactionsAzure offerings that can fulfill this role: Queue storage, .NetServicebus’ messagebuffers, WCF’s RelaybindingsEvent Processing EngineContinuous and incremental processing of event streams with near-zero latencyIdentification of eventsStanding queries (aggregation, grouping, …)Event correlationEvent transformationWindow inspectionEvent forwardingDownstream activityResult of event processingPush : Service Invocation, Notification (Email, sms), Business Process Invocation (Workflow)Pull : Message handler from queue, RSS, Event Stores
  • Separation betweenreads (queries) and writes (commands)QueryUI queries directly from data storeData is potentially staleIs read-only from a UI perspectiveCan cache the data in memoryWill send messages to the command processing sideCommand processingAsynchronous by defaultTypically on other data or event storedoesn’t make sense for Azure table storageSynchronization Real time by means of eventsOr background processes
  • Used as a libraryIn a worker roleNot as easy to deploy as it looks …Standing QueriesImplemented with LinqOn top of the CepStream constructCustom AdaptersInput AdapterOutput AdapterImplemented on top of the custom NServiceBus transport for queue storage

Eda on the azure services platform Eda on the azure services platform Presentation Transcript

  • Windows Azure User Group
    Event Driven Architecture on the Azure Services Platform27/05/2010
  • Yours truly
    Yves Goeleven
    Solution Architect @ Capgemini
    MS Community Lead @ Capgemini
    Board member @ Azug
    Contact me @
    Cloudshaper.wordpress.com
    Mscop.be.capgemini.com
    Twitter.com/YvesGoeleven
    Facebook
    Linkedin
    Yves@goeleven.com
  • Introduction
    What are we going to discuss today?
  • Agenda
    Introduction
    Design challenges
    Event driven architecture
    Event Flow Layers
    Event Processing Styles
    Extreme Loose coupling
    Relationship to CQRS
    Answering the challenges
    Technology Mapping
    Demo & Implementation Details
    Questions
  • Windows Azure
  • .Net Service Bus
  • SQL Azure
    Business Analytics
    Reporting
    Data Sync
    Database
  • The beauty of cloud computing
    Infinite compute and storage capacity
    On demand
    Extreme flexibility
    Without upfront investments
    Reduces overall costs
    Better for the environment
  • Design challenges
    Aren’t there any downsides to cloud computing then?
  • Impact of scale
    Large scale systems suffer from
    Data latency
    No distributed transactions
    Global systems
    No guaranteed instance availability
  • Latency
    Azure Fabric
    Data overlay
    Data replication over peers
    Multiple failure & upgrade domains
    Machines in different racks
    Or even different datacenters
    Depending on traffic and data size
    This might lead to latency
  • No distributed transactions
    The very nature of 2 phase commit
    Makes each machine confirm twice in the commit process
    Adding more machines
    Makes the commit process exponentially slower
    This drastically limits scalability
    So distributed transactions are not supported in the cloud!
  • Instance availability
    Azure is a global platform
    Can be used by anyone from anywhere, 24 / 7
    No windows for maintenance and upgrades
    The automated update system will take your system down
    whenever it feels like it!
    Failure & upgrade domains counteract this
    Need at least 2 instances of each role
  • Traditional composite SOA
    Synchronous request reply integration
    It assumes little to no latency
    It assumes data from queries to be accurate
    It assumes distributed transactions
    To undo the work of services in case of failure
    It assumes service availability
    As consumer is waiting for service completion
  • Event Driven Architecture
    If synchronous composite SOA is unsuitable for the cloud, what is?
  • What’s an event anyway
    Notable, interesting, thing that happens in the business
    State change, opportunity, threshold, deviation
    From a technical perspective
    Message: header & payload
    Events are completely self descriptive
    All relevant state is encapsulated
    But whom decides what ‘interesting’ means?
  • Inversion of communication
    Event generators
    EventStream
    Time
    Eventconsumers
  • Eventflowlayers
    Event generator
    Event Channel
    Event Processing Engine
    Downstream activity
    Event Processing Actions
    Publish
    Event Processing Engine
    Notify
    Standing
    Query
    Invoke service
    Start business process
    Capture
  • Event Processing Styles
    What does ‘interesting event’ mean ?
  • SimpleEvent Processing
    Event generator
    Event Channel
    Event Processing Engine
    Downstream activity
    Event Processing Actions
    Publish
    Event Processing Engine
    Notify
    Simple Event
    Invoke service
    Start business process
    Capture
  • StreamEvent Processing
    Event generator
    Event Channel
    Event Processing Engine
    Downstream activity
    Event Processing Actions
    Publish
    Event Processing Engine
    Notify
    Simple Event
    Invoke service
    Start business process
    Capture
  • Complex Event Processing
    Event generator
    Event Channel
    Event Processing Engine
    Downstream activity
    Event Processing Actions
    Publish
    Event Processing Engine
    Notify
    Simple Event
    Invoke service
    Complex Event Series
    Start business process
    Capture
  • Extreme Loose Coupling
    Event driven architecture delivers…
  • Extreme loose coupling
    Decoupling in multiple dimensions
    Implementation
    Event generator and consumer’s implementation are not bound
    They even don’t know about the other’s existance
    All information encapsulated in event
    Distribution
    Event generator and consumer can be physically separated
    As long as they can access the event channel
  • Extreme loose coupling
    Decoupling (Continued)
    Time
    Events encapsulate all information
    They can be stored for later processing
    Evolution
    Event generators and consumers can evolve independently
    Usually they are ‘added’ instead of ‘changed’, contributing to ease of management
  • Brewers CAP Theorem
    A distributed system can satisfy any two of these guarantees at the same time, but not all three
    Consistency
    all nodes see the same data at the same time
    Availability
    node failures do not prevent survivors from continuing to operate
    Partition Tolerance
    The system continues to operate even if it becomes partitioned due to loss of connectivity
  • Eventual consistency
    EDA sacrifices consistency
    But will eventually become consistent
    Compensating events
    Errors and exceptions are events as well
    And should end up in the event stream
    Compensating transactions
    Event Consumers should subscribe to compensating events
    To make things right again
  • Eventual consistency
    Local transactions, or similar mechanics, are still supported
    So even if a role goes down while processing a compensating event
    The local transaction will ensure that the event gets processed later
    Event consumers must be completely autonomous
    They should not rely on other services
    The long forgotten SOA principle ‘Service Autonomy’, is mandatory in an EDA
  • Service Autonomy
    How to achieve it?
    Make every event consumer (aggregate root)
    a stand alone state machine
    Blog Comment (Submitted, Accepted, Rejected)
    Encapsulate state transitions with
    Tentative operations
    Submit / Accept / Reject
    Internally it uses
    the currently known state
    risk mitigation logic
  • Service Autonomy
    Give it a memory (Partner state machine)
    The relevant last known states of it’s remote partners
    Required to properly implement risk mitigation logic
    This allows for
    Stale, locally stored, state
    out of order events
  • Apology based computing
    Eventual Consistency presents some challenges to the UI
    Users need to be informed that the system
    Is working on it, instead of waiting for the result
    The system will get back to them later
    In case of failure or success
    Task oriented UI’s
    support this better than data oriented UI’s
  • Relationship to CQRS
    Haven’t I seensome of thisbefore?
  • Similar to CQRS*
    Events
    * Command QueryResponsibilitySegregation
    Browser
    Workerrole
    Web role
    Submit New
    Blog Post
    Command
    Request/Reply
    Publish
    Query
    Request/Reply
    View Recent
    Blog Posts
  • It happened, live with it
    Events
    Blog Post
    Submitted
    Workerrole
    Browser
    Web role
    Event
    Oneway
    EventStream
    Query
    Request/Reply
    View Recent
    Blog Posts
  • Open for extension
    ExtensionPoints
    Blog Post
    Submitted
    Workerrole
    Browser
    Web role
    Event
    Oneway
    EventStream
    Query
    Request/Reply
    View Recent
    Blog Posts
  • Answering The Challenges
    How does all of thissolve the design challenges in the cloud?
  • Answering the challenges
    Latency
    Event consumers listen to what’s happening
    They react to events when they receive them
    Time between event occurrence and processing
    Is probably irrelevant for the event consumer
    As all information is encapsulated inside the event
    Take into account that events may arrive out of order
    Risk mitigation strategies required
    Based on local state and partner state machine
  • Answering the challenges
    Availability
    If the event channel technology is durable
    Like azure storage queues
    Event consumers can go down without impact
    They will receive the events once they come back up again
  • Answering the challenges
    Availability
    The absence of events in the stream
    Can be detected by a stream event processing
    This can result into a new event
    Triggering a self healing process
    Or human intervention
  • Answering the challenges
    No distributed transactions
    Compensating events
    Compensation logic , local transactions
    Humans can subscribe to them as well
    To fix the issues manually
  • Technology Mapping
    What technology could I use to implement this…
  • Event channel
    Table storage queues
    Durable
    might lead to dead queues though
    Queue management required
    Supports queue-peek-lock
    Similar to local transactions
    .NetServicebus’ messagebuffers
    Temporarily durable
    will disappear if unused
    Supports queue-peek-lock
    Similar to local transactions
    WCF’s Relaybindings
    Can reach outside of the cloud
    Even across NAT &Firewalls
    No transaction support
  • Event processing engine
    Microsoft StreamInsight
    Stream event processing
    Complex event processing
    .NetServicebus used to* have routers
    Ideal for forwarding events
    Supported any / all forwarding
    *Rumored to come back soon
  • Event processing engine
    Open source to the rescue
    NServicebus
    Library to implement service busses
    Routing scenarios
    Publisher (All)
    Distributor (Any)
    Simple event processing
    Message handlers
    Complex event processing
    Saga pattern
  • Demo
    Canyou show me anexample?
  • Architecture
    Web
    Web
    Web
    Web
    Web
    Queue Storage
    StreamInsight
    NServicebus
    AzureRoles
    CEP
    Publisher
    Publisher
    (All)
    Table
    Storage
    Distributor
    Distributor
    (Any)
    Worker
    Worker
    Worker
    Worker
    Worker
  • Implementation Details
    How the helldidyou do that!
  • NHibernate
    Interaction with Azure is through REST API’s
    Manually crafting requests
    Manually parsing responses
    Is cumbersome
    Co-created an NHibernate driver
    To do the heavy lifting of interacting with table storage
    Provides persistence ignorance
  • NhibernateDriver
    Walkingthrough the code…
  • NServiceBus
    Performs messaging on Queue Storage
    Client API ( IBus )
    Server API ( IHandleMessages<T> )
    Pub/Sub with durable subscriptions
    Distribution based on worker availability
    Uses Nhibernate and azure storage driver internally
    For all it’s storage needs
  • NServiceBus
    Custom extensions
    Azure Queue Transport
    Subscription storage
    Worker availability management
    Saga support (not yet finished)
  • Nservice Bus extensions
    Walkingthrough the code…
  • Modules
    In order to cut costs
    Modules loaded ad runtime
    From blob storage
    For all components
    Allows for dynamic views, message handlers and CEP queries
    Plan to switch to MEF
    StructureMap as IOC
    Wires it all together
  • CMS Module WORKER
    Walkingthrough the code…
  • StreamInsight
    Event
    Event
    Event
    Event
    Event
    Event
    Event
    Event
    Event
    Event consumers
    Event generators
    CEP Application at Runtime
    CEP Engine
    Devices, Sensors
    Pagers & Monitoring devices
    Standing Queries
    Output Adapters
    Input Adapters
    Web & Worker Roles
    Web & Worker roles
    Event stores & Databases
    Event Stores & Databases
    Monitoring Systems
    Static reference data
    55
  • STREAMINSIGHT EXTENSIONS
    Walkingthrough the code…
  • Web Role
    ASP.Net MVC
    Heavy use of JQuery
    Asynchronous semantics
    WYSYWIG style
    Extremely apology based
    Nhibernate’s second level cache on by default
    Effectively queries memory most of the time
    Synchronization through events of course
    Switching to IronRuby as view engine
    Views stored in blob storage
    Allows for runtime customization
  • CMS Module web
    Walkingthrough the code…
  • RUNTIME
    Walkingthrough the code…
  • Let’sRecap
    We’refinallythere!
  • ANY QUESTIONS?
    Wouldthisguystill have a sociallife?
  • Thank you for coming!
    Spread the word! www.azug.be