Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova, Microsoft ) Kafka Summit SF 2019

2,145 views

Published on

Apache Kafka is changing the way we build scalable and highly available software systems. Providing a simplified path to eventual consistency and event sourcing Kafka gives us the platform to make these patterns a reality for a much broader segment of applications and customers than was possible in the past. Cloud Events is an interoperable specification for eventing that is part of the CNCF. This session will combine open source and open standards to show you how you can build highly reliable application that scale linearly, provide interoperability and are easily extensible leveraging both push and pull semantics. Concrete real world examples will be shown of how Kafka makes event sourcing more approachable and how streams and events complement each other including the difference between business events and technical events.

Published in: Technology
  • Login to see the comments

Building Event Driven Architectures with Kafka and Cloud Events (Dan Rosanova, Microsoft ) Kafka Summit SF 2019

  1. 1. Building Event Driven Architectures with Kafka and Cloud Events @DanRosanova Group Principal Program Manager Azure Messaging Microsoft
  2. 2. Who am I and why am I here? • Product owner for Azure’s messaging services • Career long Messenger • Background in trading and financial systems
  3. 3. Examining the parts of this talk EVENT DRIVEN ARCHITECTURE APACHE KAFKA CNCF CLOUD EVENTS
  4. 4. What is Event Driven Architecture (EDA) LOOSELY COUPLED ASYNCHRONOUS (NORMALLY) DOES NOT PROVIDE A MEANINGFUL DIRECT RESPONSE An architectural style (paradigm) built around reactive extensibility with the following core principles
  5. 5. What do you mean by “Event-Driven”? • Event Notification (what we will talk about) • Event-carried state transfer (partially what we’ll talk about) • Event Sourcing (not covered today) https://martinfowler.com/articles/201701-event-driven.html photo: Christopher Ferguson Martin Fowler
  6. 6. Why use EDA? Designed for distributed systems Simplifies horizontal scale Builds in resiliency
  7. 7. What is an event A SIGNIFICANT CHANGE IN STATE E.G. A LIGHT TURNING FROM OFF TO ON OR A TRADE FROM OPEN TO FILLED FROM HERE FORWARD WHAT WE REALLY MEAN IS AN EVENT NOTIFICATION
  8. 8. Attributes of events SELF CONTAINED AND INDEPENDENT CONTAINS PROVENANCE NOT TIED TO THE LOGIC THAT IT WILL ULTIMATELY TRIGGER NORMALLY CONSISTS OF A HEADER AND A BODY MAY OR MAY NOT CONTAIN THE ACTUAL STATE CHANGE EVENTS ARE NOT COMMANDS
  9. 9. Provenance Who the source is “When” it happened What was the cause of the event
  10. 10. Deep thoughts If an event happens and there’s no one to receive it
  11. 11. Components of Event Driven Architecture Emitter ConsumerChannel • Generates and sends events • Unaware of the existence of consumers • Does not expect responses directly • Reacts to events • Directly processes • Forwards • May raise events of its own • Communication conduit • Exclusively responsible for the distribution of events • Does not imply transport or protocol
  12. 12. Event Processing Styles Simple event processing (SEP) •Individual single events •Consumer directly processes •No further action Events stream processing (ESP) •Unit of Work is stream •Unending sequence (codata) •Consumer will have some memory / state Complex event processing (CEP) •Matching patterns across events and sources •Not bound by time
  13. 13. Example stream processor simpleMA(float[] prices, int period) float result; for(int i=0;i<period;i++) result += prices[prices.length – i]; return result/period
  14. 14. Another example stream processor Delay ReverbFlange
  15. 15. Extended from SEP to CEP • Creating derivative events • Joining streams • Over differing time windows
  16. 16. Beyond the simple diagram – how EDAs really look Price Emitter Moving Average Processor RSI Processor MACD Processor Trade Signal CEP Trading Component
  17. 17. Consumer Benefits of EDA: Scale out design • Flowing changes through the EDA makes scaling out trivial • As scale grows just add more nodes • Easy! Emitter Consumer Consumer
  18. 18. Benefits of EDA: Regionally distributed systems • Add content-based routing to deliver data for where it’s needed • Zero change for emitter • Each region emits • Each region receives
  19. 19. Benefits of EDA: Side by side deployment • Start at the end and work backwards • Add a new version of the consumer FIRST • Consumers and Emitters exist side by side for some amount of time Emitter v1 Application v2 Application v1Application v3 Emitter v2 Emitter v3
  20. 20. Benefits of EDA: Side by side deployment • Real systems are not so simple • There may be hundreds of components • Downtime is not as widely accepted as it used to be Web Frontend Inventory Management Fraud Detection
  21. 21. Obvious drawbacks of EDA Eventual consistency The channel is now REALLY important End to end visibility Missed events Replay Event storms CQRS
  22. 22. Hidden drawback of EDA: Semantic coupling SUBSCRIPTIONS TYPES (SCHEMA) PATTERNS USED FOR MATCHING EVENTS Behavioral dependencies between components or services <or> How to build a distributed monolith
  23. 23. What is Apache Kafka®
  24. 24. If you’re asking me, on the last session of the last day about what Kafka is?
  25. 25. Where have you been all week?
  26. 26. Key characteristics of Apache Kafka A DISTRIBUTED STREAMING PLATFORM MANIFESTED AS PARTITIONED APPEND ONLY LOG WITH CLIENT-SIDE CURSOR AND FAULT TOLERANCE
  27. 27. What Apache Kafka is not
  28. 28. Cloud Events • A specification for describing event data in common formats to provide interoperability across services, platforms and systems • An open standard built in an open forum via the CNCF’s Serverless Working Group • With many contributors across our industry (from Alibaba to VMWare: can we a W,X,Y, or Z contributor?)
  29. 29. Why I believe strongly in open standards REDUCES BARRIERS TO ENTRY SPURS INNOVATION PROTECTS EVERYONE INVOLVED
  30. 30. https://ourworldindata.org/economic-growth
  31. 31. An example event { "id" : "A234-1234-1234", "source" : "https://github.com/cloudevents/spec/pull", "specversion" : "1.0-rc1", "type" : "com.github.pull.create", "subject" : "123", "time" : "2018-04-05T17:31:00Z", "datacontenttype" : "text/xml", "data" : "<much wow="xml"/>" } RequiredOptional
  32. 32. Bringing it all together
  33. 33. Kafka as a channel: Topics Emitter Consumer • Obvious choice • Solves replay • Each consumer gets its own offsets • Decouples time • Cloud Events already has a Kafka Transport Binding Consumer
  34. 34. Does this mean Kafka for every channel? No!
  35. 35. Kafka as a consumer: KStreams • Kstreams can materialize a view for the scale out model we saw earlier • Can filter and forward • Joins and aggregations make CEP possible KStream Topic 1 Topic 2 Topic 3
  36. 36. Kafka for end to end visibility and logging • Anything making changes can write its own events to a centralized event log • You can forward the actual channel data as well KStream Topic 1 Topic 2 Topic 3 Service 1 Service 2 Service 3 Emitter Topic Global Log Consumer Topic ConsumerService N
  37. 37. Kafka as a router: Connect, MirrorMaker • Write to local Kafka, replicate and route as needed • Outbox for local services in app, DC, or Geo App 1 App Health
  38. 38. Key take aways AVOID THE ONE TOOL / TOPIC TO RULE THEM ALL GO LOCAL – USE A LOCAL CLUSTER / COMPUTE USE PORTABLE AND OPEN FORMATS PLAN FOR VERSIONING AHEAD OF TIME – CHANGE HAPPENS
  39. 39. Q&A
  40. 40. Open Stand

×