Downloaded 154 times
![30
Event Streaming Platform - Value per Use Case
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate
Risk (protect
money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360 Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log
aggregation, Splunk
replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)](https://image.slidesharecdn.com/kaiwaehnerapachekafkavstraditionalmiddleware-190522012020/75/Apache-Kafka-vs-Integration-Middleware-MQ-ETL-ESB-Friends-Enemies-or-Frenemies-Kai-Waehner-Confluent-Kafka-Summit-London-2019-29-2048.jpg)
Uploaded byconfluent
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or Frenemies? (Kai Waehner, Confluent) Kafka Summit London 2019
Apache Kafka can act as both an enemy and a friend to traditional middleware like message queues, ETL tools, and enterprise service buses. As an enemy, Kafka replaces many of the individual components and provides a single scalable platform for messaging, storage, and processing. However, Kafka can also integrate with traditional middleware as a friend through connectors and client APIs, allowing certain use cases to still leverage existing tools. In complex environments with both new and legacy systems, Kafka acts as a "frenemy" - replacing some functions but integrating with other existing technologies to provide a bridge to new architectures.
More Related Content
Similar to Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or Frenemies? (Kai Waehner, Confluent) Kafka Summit London 2019
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Apache Kafka vs. Integration Middleware (MQ, ETL, ESB) - Friends, Enemies or Frenemies? (Kai Waehner, Confluent) Kafka Summit London 2019
- 1. 1 Apache Kafka vs.Traditional Middleware (MQ, ETL, ESB) Friends, Enemies or Frenemies? Kai Waehner Technology Evangelist kontakt@kai-waehner.de LinkedIn @KaiWaehner www.confluent.io www.kai-waehner.de
- 2. 2 Apache Kafka vs.Traditional Middleware Agreement: Kafka is the de facto standard for … • messaging at scale! • decoupling of microservices! • reliable, lightweight stream processing! Controversial discussion: Use Apache Kafka as middleware!
- 3. 3 Agenda 1.Traditional Middleware 2.Event StreamingPlatform 3.Enemies 4.Friends 5.Frenemies
- 4. 4 Agenda 1.Traditional Middleware 2.Event StreamingPlatform 3.Enemies 4.Friends 5.Frenemies
- 5. 6 Source A Source B SinkX Sink Y Connector A REST / SOAP Connector X JMS ETL / ESB Product XYZ Dream
- 6. 7 ETL / ESB Passive Server SourceA Source B Sink X Sink Y Connector A REST / SOAP Connector X JMS ETL / ESB Active Server Messaging System A (Real Time) Messaging System B (Big Data) Database (Important Data) In-Memory Data Grid (Cache) API Gateway Stream Processing Engine Build your custom integration layer… At least 4-5 different products or open source frameworks Another integration tool (because one tool cannot integrate everything, buy more…) Reality
- 7. 8 Challenges with currentintegration architectures (MQ / ETL / ESB) Zoo of technologies • Integration platform (Extract-Transform-Load / Enterprise service Bus) • + additional “optional” components • Messaging System (Message Queue, Peer-to-Peer) • In-Memory Cache or Data Grid • Database • Streaming Engine • API Gateway • …
- 8. 9 Challenges with currentintegration architectures (MQ / ETL / ESB) Architectures with limited scalability and availability • No end-to-end scalability (only parts are built for high volume of messages and high throughput) • No native, built-in scalability (you cannot just add a broker to the running system) • Broker-per-scenario (e.g. hundreds of MQ clusters) • Active-passive clustering • Downtime for maintenance, upgrades, configuration changes • No backwards-compatibility
- 9. 10 Challenges with currentintegration architectures (MQ / ETL / ESB) Tight coupling • Platform-centric integration implementation (à no separation of concerns, vendor lock-in) • Synchronous communication via request-response (SOAP / REST / gRPC / …) • No handling of backpressure or unavailable consumers (push-based messaging queues)
- 10. 11 Business Digitalization Trendsare Driving the Need to Process Events at a whole new Scale, Speed and Efficiency The World has Changed Mobile Cloud Microservices Internet of Things Machine Learning
- 11. 12 #NoMQ #NoESB #NoETL canhandle this well! Tight coupling, limited scale, zoo of components You need to think event-based to realize these use cases! You need to leverage a single, scalable and loosely coupled platform Event Streaming Platform! ?
- 12. 13 Agenda 1.Traditional Middleware 2.Event StreamingPlatform 3.Enemies 4.Friends 5.Frenemies
- 13. 14 Source A Source B SinkX Sink Y Connector A Java Connector X REST Event Streaming Platform Hmm… Yet another one of those magic boxes in the middle, huh?
- 14. Events What is anevent?
- 15.
- 16. 17 Events A Sale AnInvoice A Trade A Customer Experience
- 17. 18 Where are they? Eventshaven’t had a proper home in infrastructure or in code. They are implicit. Here!
- 18. 19 Implementing an Event-drivenArchitecture Requires a Paradigm Shift Relational DB From data represented in static tables and accessed with RPC... ...to data represented as streams of events Apps Microservices SaaS apps Custom apps Data warehouse
- 19. 20 Haven’t we seenall this before?
- 20. 21 What’s different thistime around? (Published in 2009) (Published in 2004)
- 21. 22 A Streaming Platformis the Underpinning of an Event-driven Architecture Ubiquitous connectivity Globally scalable platform for all event producers and consumers Immediate data access Data accessible to all consumers in real time Single system of record Persistent storage to enable reprocessing of past events Continuous queries Stream processing capabilities for in-line data transformation Microservices DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experience s Streams of real time events Stream processing apps
- 22. 23 Agenda • Traditional Middleware •Event Streaming Platform • Enemies • Friends • Frenemies
- 23. 24 Source A Source B SinkX Sink Y Connector A SOAP Connector X REST Integration Layer Middleware: Message Queue… ETL… Enterprise Service Bus… Event Streaming Platform: Apache Kafka. Spoilt for Choice!
- 24. 25 Business Digitalization Trendsare Driving the Need to Process Events at a whole new Scale, Speed and Efficiency Why do you need an Event Streaming Platform? Remember: Mobile Cloud Microservices Internet of Things Machine Learning The World has Changed!
- 25. 26 26 Fast (Low Latency) EventStreaming Paradigm
- 26.
- 27. 28 ● Global-scale ● Real-time ●Persistent Storage ● Stream Processing Apache Kafka: The De-facto Standard for Real-Time Event Streaming Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka
- 28. 29 Apache Kafka atScale at Tech Giants > 4.5 trillion messages / day > 6 Petabytes / day “You name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data
- 29. 30 Event Streaming Platform- Value per Use Case Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
- 30. 31 Why Apache Kafkainstead of traditional middleware? Event Streaming Platform The core is event-based Supports real time stream processing But also Fire-and-Forget, Publish / Subscribe, Request-Response / RPC, Batch, … Single infrastructure Messaging, storage, processing Scalability and throughput can be increased dynamically Reliability and zero downtime High availability Exactly-once semantics Rolling upgrades and dynamic configuration changes Backwards compatibility Decoupling of clients Agile microservices Dumb pipes, smart endpoints Handling backpressure No vendor lock-in
- 31. 32 Why Apache Kafkainstead of traditional middleware? ”Eat your own dog food!”
- 32. 33 Enemies! ESB MQ Storage Streaming Engine Messaging: KafkaCore Storage: Kafka Core Caching: Kafka Core Real-Time, Batch: Kafka Clients Integration: Kafka Connect Stream Processing: Kafka Streams / KSQL Request-Response: REST Proxy ”Eat your own dog food” vs. Enemy 1 Enemy 2 Enemy 3 Enemy 4 …. More components, clusters, technologies means more conflicts, incompatibility, operations burden! Enemy 5
- 33. 34 ETL / ESB Kafka BrokerX Source A Source B Sink X Sink Y Kafka Connect Python Kafka Connect Java Kafka Broker 1 Schema Registry REST Proxy Kafka Streams / KSQL No need for another infrastructure, cluster, database… High availability and scale handled by Kafka Topics! Monitoring Tool “Yet another Kafka addon”
- 34. 35 Independent, scalable, reliablecomponents (server and client side!) read, write App (Kafka Streams) Kafka (data) More Apps (KSQL, Connect, Python, REST, “You-name-it”) BookingsTeam FraudTeam … MobileTeam …
- 35. 39 Don’t use your“ESB knowledge” with Kafka! https://www.thoughtworks.com/radar/techniques/ recreating-esb-antipatterns-with-kafka !
- 36. 40 Kafka Connect Kafka Cluster CRM Integration Domain-DrivenDesign for your Integration Layer Legacy Integration Custom Application ESB Connector Java / KSQL / Kafka Streams Schema Registry Event Streaming Platform CRM Domain Legacy Domain Payment Domain è Independent and loosely coupled, but scalable, highly available and reliable!
- 37. 41 Agenda • Traditional Middleware •Event Streaming Platform • Enemies • Friends • Frenemies
- 38. 42 Friends! • Kafka Connectconnectors (JMS, IBM MQ, RabbitMQ, etc.) • JMS Client (Kafka-native JMS Implementation) • ESB or ETL tools with their own connectors • Kafka’s Client APIs (like Java, .NET, Go, Python, JavaScript) • REST Proxy • Etc. Plenty of integration options between Kafka and traditional middleware Traditional Middleware Apache Kafka
- 39. 43 Why friends? Kafka isNOT THE ALLROUNDER for every single problem! Maybe you don’t need a scalable, reliable, distributed system, but “just”: • Synchronous Web Service (SOAP / WS* or REST) integration between two endpoints • Integration with legacy components (Cobol, Edifact, …) • Point-to-point messaging with active / passive HA for (“a few”) messages • Very specific messaging solution for “less than millisecond performance” • Managed file transfer • API Management • “Real” BATCH Processing (Hadoop, Flink, Informatica, …) X
- 40. 44 Legacy integration àThis is where the ESB / ETL tool shines! • Visual coding for complex graphical mappings • Integration components for complex legacy standard software and protocols • SAP BAPI / iDoc, Cobol, Edifact, SOAP / WS*, etc.
- 41. 45 Agenda • Traditional Middleware •Event Streaming Platform • Enemies • Friends • Frenemies
- 42. 46 Big Bang failsfor critical 24/7 deployments No!
- 43. 47 Some technologies neverdie… Cough… Can we ever get away from all legacy applications? Probably not! Cobol
- 44. 48 RPC / Request-Response…You can use the ESB for it. BUT: REST SOAP RPC JMS Java … it can quickly become very complex … and doesn’t scale!
- 45. 49 Embrace data thatlives and flows between services
- 46. 50 An Event StreamingPlatform gives services independence Orders Customers Payments Stock Freedom to tap into and manage shared data REST JMS ESB REST CRM Mainframe SOAP … Kafka Kafka Kafka Kafka
- 47. 51 If you reallyneed Request-Response with Kafka à A Correlation ID is your friend!
- 48. 52 Before IQ L WithIQ J If you need to combine RPC with Streaming: Interactive Queries (IQ) (Kafka Streams)
- 49. 53 Confluent’s Streaming MaturityModel - where are you? Value Maturity (Investment & time) 2 Enterprise Streaming Pilot / Early Production Pub + Sub Store Process 5 Central Nervous System 1 Developer Interest Pre-Streaming 4 Global Streaming 3 SLA Ready, Integrated Streaming Projects Platform
- 50. 54 Frenemies? How to integrate theold and new world?
- 51. 55 Mainframe offloading toApache Kafka Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Transaction Data Vendor Description Starbucks Coffee Walmart Blu-Ray Transaction Description Integration via - Kafka Connect (JMS, MQ) - REST Proxy - 3rd party CDC tool - Etc. Website Client profiles Mainframe MIPS = $$ Ability to migrate back if not working well
- 52. 56 MQ Integration (andlater Replacement) Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Transaction Data Messaging Solution (IBM MQ, RabbitMQ, etc.) Application 1) Legacy MQ Communication with App 2) Kafka for decoupling between MQ and App 3) Direct communication via Kafka (no MQ anymore)
- 53. 57 New, Innovative Projectsand Applications Date Amount 1/27/2017 $4.56 1/22/2017 $32.14 Transaction Data Messaging Solution (IBM MQ, RabbitMQ, etc.) Application Kafka Microservices Agile, lightweight (but scalable robust) Kafka microservice Big Data project (Elastic, Spark, AWS Sagemaker, …) 1) Direct Legacy MQ Communication with App 2) Kafka for decoupling between MQ and App 3) Direct communication via Kafka (no MQ anymore) 4) New projects and applications (independent or related to the existing migration projects) External Solution
- 54. 58 Use the righttool for the job (and combine them where it makes sense!)
- 55.