Introduction to leveraging Azure Event Hubs from Apache Kafka
•Download as PPTX, PDF•
3 likes•1,302 views
Introduction to Event Driven Architectures and how Azure Event Hubs enable a hybrid streaming data pipeline architecture with Apache Kafka. Presented at the 2019 Global Integration Bootcamp, Auckland, New Zealand.
Recommended
Recommended
More Related Content
Recently uploaded
Recently uploaded (20)
Featured
Featured (20)
Introduction to leveraging Azure Event Hubs from Apache Kafka
- 1. Introduction to leveraging Azure Event Hubs from Apache Kafka Presented by Nikolai Blackie @Adaptiv
- 2. About Nikolai • Co-founder and Principal Architect @ Adaptiv Integration • Partners with Dell Boomi, Microsoft and MuleSoft • Hobbies: Cycling, Classic Cars, Rick & Morty!!! • http://www.adaptiv.co.nz/team-member/nikolai-blackie/ • https://twitter.com/nikolaiblackie/ • https://www.linkedin.com/in/nikolaiblackie/
- 3. Legacy Application Architecture Database REST Endpoint POST /GIBSReview/{Session} New Reviews Fraud Detection Job (daily schedule) Nikolai s Mums Reviews Valid Reviews Poll >95% <95% REST Endpoint GET /GIBSStats Statistic Calculation Job By Category/Overall GIBS Stats Poll
- 4. Event-Driven Architectures • “Events” refer to things that happen in the systems • They often do not carry much data with it • EDA promotes the detection, consumption of, and reaction to events, rather than actions Event Transport Event Producer HTTPS Topic Consumer Event Event Producer AMQP Topic Consumer Event Event Producer Kafka 1.0 Topic Consumer Event
- 5. Is Not Life Simply a Series of Events? Message driven vs. Event Driven Its level of abstraction - messages are a technical artifact but events relate to a business occurrence. A traditional ESB tends to use "nouns" as its messages, whereas event- driven architectures are more around "verbs", if that makes sense. It's not really well defined anywhere that I've seen; no industry consensus yet. Other than "messages are old, events are new, pay us a bunch of money" Philip Durant - [21/3/19 5:23 PM]
- 6. Four Patterns of EDA What do you mean by “Event-Driven”? Martin Fowler 07 February 2017
- 7. Event Notification System • Pros • Reverses dependencies • Decouples systems • Cons • No statement of behaviour • Difficult to debug Customer Management Address Change Event Address Change Topic Insurance Quoting Consumer Consumer Change Address Customer Management Calculate premium change Insurance Quoting Change Address
- 8. Event Carried State Transfer • Pros • Decoupling • Resilient to downstream systems fails • Reducing load • Cons • Duplicate copy of event data • Eventual Consistency Customer Management Address Change Event Address Change Topic Insurance Quoting Customers Customers X
- 9. Event Sourcing • App state & replayable event log • Git, general ledgers are event sourced • Can replay events to restore app state • Pros: audit, debug, history, calculate corrections, in memory • Cons: External systems, event schema, ids, versioning, complexity Application Write Events Event Store Application State Application Read Event Handlers Event Queue/ Topic
- 10. Command Query Responsibility Segregation
- 11. Typical Event Streaming Data Pipeline Data Producers Collection/Staging Ingress/Capture Processing Storage Presentation/Action
- 12. Azure Data Pipelines Ingest Event Hubs Stage Blob Storage Process Databricks Stream Analytics Store SQL DW Cosmos DB Present/ Act Analysis Services PowerBI Apps
- 13. Kafka and Event Hubs Event Hub Topic Event Producers Partition HTTPS AMQP Kafka Partition Partition Partition Consumer Group Receiver Receiver Consumer Group Receiver Receiver
- 14. Distinguishing between Service Bus, Event Hubs, and Event Grid Service Purpose Type When to use Event Grid Reactive programming Event distribution (discrete) React to status changes Notification Systems Event Hubs Big data pipeline Event streaming (series) Telemetry and distributed data streaming Service Bus High-value enterprise messaging Message Order processing and financial transactions
- 15. Demo • Kafka Producer • Event Hub Topic • Kafka Consumer
- 16. Streaming Event Driven Architecture Event Hub Topic GIBSReview POST Review Event Producer Review Topic Streaming Fraud Detection >95% <95% Fraud Topic Valid Review Topic Machine Learning Model Streaming Statistics App Category Stats Topic Overall Stats Topic GIBSStats GET Application State State Sink
- 17. Why use event hubs for Kafka https://channel9.msdn.com/Shows/Azure-Friday/Azure-Event-Hubs-for-Apache-Kafka
- 18. Takeaways • Avoid tightly coupled architectures • Leverage event driven architecture appropriately • Event Hubs for Kafka • Simplifies infrastructure concerns • Provides streaming ramp to the cloud • Best of both worlds, local control, cloud scale & simplicity
- 21. Questions?
Editor's Notes
- Nikolai What are events, what EDA is about EDA vs. SOA Lightweight events rather than service call contracts
- Event producers: Any entity that sends data to an event hub. Event publishers can publish events using HTTPS or AMQP 1.0 or Apache Kafka (1.0 and above) Partitions: Each consumer only reads a specific subset, or partition, of the message stream. Consumer groups: A view (state, position, or offset) of an entire event hub. Consumer groups enable multiple consuming applications to each have a separate view of the event stream, and to read the stream independently at their own pace and with their own offsets. Throughput units: Pre-purchased units of capacity that control the throughput capacity of Event Hubs. Event receivers: Any entity that reads event data from an event hub. All Event Hubs consumers connect via the AMQP 1.0 session, and events are delivered through the session as they become available. All Kafka consumers connect via the Kafka protocol 1.0 and later.