David Ware, profile picture
Uploaded byDavid Ware
PDF, PPTX2,240 views

IBM MQ and Kafka, what is the difference?

The document compares IBM MQ and Apache Kafka, highlighting their distinct messaging capabilities and use cases. IBM MQ focuses on assured asynchronous message delivery and transactional message exchange, while Kafka emphasizes event streaming and scalability for microservices architectures. Choosing the appropriate tool depends on specific enterprise messaging requirements, such as critical data exchange, event-driven interaction, and system consumption models.

Software◦
Related topics:
Apache Kafka• Event-Driven•

Embed presentation

Download as PDF, PPTX
TechCon 2020 IBM MQ and Kafka, what is the difference? David Ware IBM MQ Chief Architect
Messaging is essential for building fully connected, efficient and scalable solutions. More now than ever before Messaging covers a broad spectrum of requirements
Messaging Scalability Security Management High Availability There are many capabilities essential to every enterprise messaging solution IBM MQ and Kafka have both repeatedly been proven to tick these boxes…
© 2020 IBM Corporation So just pick one and move on?
© 2020 IBM Corporation So just pick one and move on? Perhaps, if you think everything looks like a nail
© 2020 IBM Corporation So just pick one and move on? Perhaps, if you think everything looks like a nail Otherwise, let’s focus on what you’re trying to achieve…
Critical data exchange: work that needs to be done Critical applications demand assured asynchronous interactions Essential capabilities: End-to-end once and once only Deliveryü Fine grain messaging Messages typically represent commands, queries and operations The message is a way to pass control from the originator of the message to the consumer
Event Driven: building scalable microservices Microservices increases the need for communication. API-based interactions can build fragile and unscalable tight bonds between components Topics and Subscriptions Publishing and subscribing to events relaxes the coupling of microservices Events are messages that communicate that something has occurred Essential capabilities:
Event Streaming: the expanding need for messaging Event Streaming brings data together from disparate sources, enabling even more responsive and engaging experiences for a wider set of users Scalable Subscription Stream History Efficient cloud and analytics applications utilize local decoupled buffers of event data Essential capabilities:
The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
© 2020 IBM Corporation To choose the right tool you’re probably going to need to go to the next level…
© 2020 IBM Corporation Message consumption
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 … • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
© 2019 IBM Corporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER 0 1 23 45 67
© 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots
© 2019 IBM Corporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots Kafka topic partitions…
© 2020 IBM Corporation Queues and topics
© 2019 IBM Corporation Comparing queues and topics QUEUE 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER PRODUCER PRODUCER 0 1 23 45 6 7 TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER CONSUMER 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Queues Topics • Each messages goes to a single consumer from the queue • Each message goes to every subscriber to the topic
© 2019 IBM Corporation CONSUMERCONSUMER MQ and Kafka topics TOPIC PARTITION 0 1 2 3 4 5 6 7 CONSUMER GROUP CONSUMER GROUP PRODUCER PRODUCER CONSUMER CONSUMER TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER 1 2 3 4 5 6 70 1 2 3 4 5 6 70 IBM MQ Kafka
© 2019 IBM Corporation Queues and topics IBM MQ Kafka Queues Primary MQ concept Queues are administratively or dynamically created Queues can be partitioned for scale All messages are stored on queues, even publications n/a Topics Core MQ capability Topics are dynamically created as needed by applications Each subscribing application has access to a dedicated queue of to-be-consumed messages matching their subscription. Primary Kafka concept Rolling history of messages maintained by Kafka. Topics can be partitioned for scale… Consumer’s offset within a shared stream of messages is maintained by Kafka.
© 2020 IBM Corporation Scaling
Horizontal scaling – workload balance the messages PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER IBM MQ Kafka PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER
Kafka Clusters • A Kafka system is typically made up of multiple active nodes • Topics can be partitioned to enable workload balancing across those nodes, improving scale and availability of the topics • Kafka workload balances messages across partitions, either randomly or based on application provided context to enable per-partition ordering • Each topic partition is active in one broker at a time • The messages held in a topic partition can be replicated to other brokers to ensure data integrity across failures and automatic recovery
© 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 Kafka partitions and consumer groups 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
© 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
© 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
© 2019 IBM Corporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups - p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers X
IBM MQ Clusters and workload balancing PRODUCE MESSAGES TO QUEUE A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER • A single running instance of MQ is a queue manager. • These can act independently or collectively to build horizontally scaled solutions. • An MQ Cluster is a dynamically provisioned set of queue managers with the ability to partition queues across them. • MQ will workload balance inbound messages from producing applications across each partition of the queue. • With MQ’s Uniform Cluster capability, MQ will workload balance multiple consuming applications across each queue manager, ensuring messages are consumed efficiently. • MQ does not provide simple message ordering in these topologies • MQ’s HA technologies provide data integrity across failures and automatic recovery
© 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances
© 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20, … 01,11, … X
© 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20 01,11 03,13,23
© 2019 IBM Corporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20,40 01,11 03,13,23,43 CONSUMER CONSUMER CONSUMER CONSUMER 30,50,70 32,52 31 21
© 2019 IBM Corporation Scaling IBM MQ Kafka Consuming application is the bottleneck Scale out the consuming application Messaging system is the bottleneck Scale out the messaging system Increase the number of consuming applications from the existing queue/subscription. Increase the number of topic partitions to support a greater number of consuming applications in a consumer group. Increase the number of queue managers deployed and workload balance messages and consumers across them. Increase the number of Kafka broker nodes deployed and redistribute the topic partitions across them. †Good practice is to capacity plan for highest load and pre-define multiple topic partitions to support more consumers than you currently require, enabling more dynamic consumer scaling https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ † † † Unlike MQ, where new queue managers do not require copies of the existing messages, Kafka requires event histories to be copied to a new broker before it can be effective, slowing down dynamic scaling https://www.infoq.com/presentations/cloud-native-kafka-netflix/ ††
© 2020 IBM Corporation Conversational messaging
© 2019 IBM Corporation ConversationConversation ConversationConversation Fine grain communication? • MQ is designed for tens of thousands of lightweight queues and hundreds of thousands of dynamic topic strings per queue manager • This enables each conversation (e.g. reply flow) to be a dedicated queue or topic • Each consumer only sees its intended message • Dynamic create/deletion of queues and topic strings IBM MQ Equivalent Kafka model … • Kafka is designed for less, larger, configured topic partitions per broker • Recommendations are typically for 2-4k topic partitions per broker* • Consumers process every message in the topic from a point forward • This model is considered a Kafka anti-pattern. Recommendation is to rearchitect application to a stream processing pattern if possible** • https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions ** https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka PRODUCERPRODUCER PRODUCERPRODUCER CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … TOPIC 2 30 1 … …
© 2020 IBM Corporation Message delivery
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Any problem in the messaging system or the application may result in messages being lost Messages are not lost in the event of a failure, but any message may be duplicated Messages are not lost in the event of a failure and each message can be processed individually to prevent possible duplication An application can coordinate the processing of a message with a specific update to an external resource, such as a database
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications Ideal for event streaming use case Ideal for critical data exchange use case
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Higher levels of message reliability typically reduce the need for applications to consider error scenarios. However, higher message reliability typically reduces overall message performance and can limit overall availability The trade off Event streaming criticaldata exchange Reliability Performance
© 2019 IBM Corporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Kafka IBM MQ https://medium.com/@andrew_schofield/does-apache-kafka-do-acid-transactions-647b207f3d0e Sweet spots Default behavior. Some messages may be lost following an application or Kafka failure, but others persisted Default behavior. All messages of this type lost following an MQ failure by design Kafka well designed for this, producers can be configured to retry failed sends and consumers typically periodically update their offset Possible to design an application to behave like this through message batching and protocol optimizations, but not a key MQ scenario Possible within a closed Kafka system, but much harder with external systems. Technically possible to lock Kafka down to confirm every message for individual acknowledgment but loses benefits A natural fit for the destructive consume approach of MQ coupled with its core persistence model Supports Kafka-only transactions for exactly-once stream processing applications, but no external resource coordination A core MQ capability is as a resource manager (or coordinator) for global transactions Event streaming criticaldata exchange
© 2020 IBM Corporation Message availability
© 2019 IBM Corporation Per-message availability IBM MQ Kafka Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes. PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9
© 2019 IBM Corporation Per-message availability IBM MQ Kafka PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 active/active partitioning of queues and topics, each partition is active/passive Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes.
© 2020 IBM Corporation Summary
© 2020 IBM Corporation If you want to build the best solution, you’re probably going to need more than a hammer
The right tool for the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
Thank you David Ware IBM MQ Chief Architect — www.linkedin.com/in/dware1 © Copyright IBM Corporation 2020. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and ibm.com are trademarks of IBM Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available at Copyright and trademark information. 57TechCon 2020 © 2020 IBM Corporation
TechCon 2020 / © 2020 IBM Corporation 58

More Related Content

PDF
IBM MQ cloud architecture blueprint
byMatt Roberts
 
PPTX
Building an Active-Active IBM MQ System
bymatthew1001
 
PDF
IBM MQ High Availability 2019
byDavid Ware
 
PPT
IBM WebSphere MQ Introduction
byejlp12
 
PDF
IBM MQ - High Availability and Disaster Recovery
byMarkTaylorIBM
 
PPT
IBM Websphere MQ Basic
byPRASAD BHATKAR
 
PDF
Websphere MQ (MQSeries) fundamentals
byBiju Nair
 
PDF
IBM MQ Basics
bysubbareddyalamur
 
IBM MQ cloud architecture blueprint
byMatt Roberts
 
Building an Active-Active IBM MQ System
bymatthew1001
 
IBM MQ High Availability 2019
byDavid Ware
 
IBM WebSphere MQ Introduction
byejlp12
 
IBM MQ - High Availability and Disaster Recovery
byMarkTaylorIBM
 
IBM Websphere MQ Basic
byPRASAD BHATKAR
 
Websphere MQ (MQSeries) fundamentals
byBiju Nair
 
IBM MQ Basics
bysubbareddyalamur
 

What's hot

PDF
IBM MQ - What's new in 9.2
byDavid Ware
 
PPTX
Connecting mq&kafka
byMatt Leming
 
PDF
IBM MQ V9 Overview
byMarkTaylorIBM
 
PDF
IBM MQ Whats new - up to 9.3.4.pdf
byRobert Parker
 
PPTX
IBM MQ Overview (IBM Message Queue)
byJuarez Junior
 
PPTX
Introduction to Apache Kafka
byJeff Holoman
 
PDF
Kafka Overview
byiamtodor
 
PDF
Fault tolerant and scalable ibm mq
byDavid Ware
 
PPT
IBM MQ Online Tutorials
byBigClasses.com
 
PDF
IBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
byDavid Ware
 
PDF
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
PPTX
Apache kafka
byViswanath J
 
PDF
Fundamentals of Apache Kafka
byChhavi Parasher
 
PPTX
Overview - ESBs and IBM Integration Bus
byJuarez Junior
 
PPTX
MQ Infrastructure of Today and Tomorrow
byProlifics
 
PPT
WebSphere MQ introduction
byMikhailVladimirov
 
PDF
Apache Kafka - Martin Podval
byMartin Podval
 
PPT
Amqp Basic
byRahul Agrawal
 
PDF
Introduction To Flink
byKnoldus Inc.
 
PDF
Kong API Gateway.pdf
byAvinashUpadhyaya3
 
IBM MQ - What's new in 9.2
byDavid Ware
 
Connecting mq&kafka
byMatt Leming
 
IBM MQ V9 Overview
byMarkTaylorIBM
 
IBM MQ Whats new - up to 9.3.4.pdf
byRobert Parker
 
IBM MQ Overview (IBM Message Queue)
byJuarez Junior
 
Introduction to Apache Kafka
byJeff Holoman
 
Kafka Overview
byiamtodor
 
Fault tolerant and scalable ibm mq
byDavid Ware
 
IBM MQ Online Tutorials
byBigClasses.com
 
IBM MQ: Managing Workloads, Scaling and Availability with MQ Clusters
byDavid Ware
 
Apache Kafka Architecture & Fundamentals Explained
byconfluent
 
Apache kafka
byViswanath J
 
Fundamentals of Apache Kafka
byChhavi Parasher
 
Overview - ESBs and IBM Integration Bus
byJuarez Junior
 
MQ Infrastructure of Today and Tomorrow
byProlifics
 
WebSphere MQ introduction
byMikhailVladimirov
 
Apache Kafka - Martin Podval
byMartin Podval
 
Amqp Basic
byRahul Agrawal
 
Introduction To Flink
byKnoldus Inc.
 
Kong API Gateway.pdf
byAvinashUpadhyaya3
 

Similar to IBM MQ and Kafka, what is the difference?

PPTX
Kafka 101
byAparna Pillai
 
PPTX
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
byChristopher Curtin
 
PPTX
Kafka and ibm event streams basics
byBrian S. Paskin
 
PDF
09-Kafka Overview - All you need to know
byUttamMittal2
 
PDF
Confluent Messaging Modernization Forum
byconfluent
 
PDF
ADV Slides: Trends in Streaming Analytics and Message-oriented Middleware
byDATAVERSITY
 
PDF
Evolving from Messaging to Event Streaming
byconfluent
 
PDF
Streaming Analytics unit 2 notes for engineers
byManjuAppukuttan2
 
PPTX
Design Patterns for working with Fast Data
byMapR Technologies
 
PDF
Messaging With Apache ActiveMQ
byBruce Snyder
 
PDF
Event driven-arch
byMohammed Shoaib
 
PDF
[@NaukriEngineering] Messaging Queues
byNaukri.com
 
PPTX
Design Patterns for working with Fast Data in Kafka
byIan Downard
 
PPTX
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
byLucas Jellema
 
PDF
RabbitMQ vs Apache Kafka - Part 1
byErlang Solutions
 
PPTX
Virtual Meetup Sweden - Reacting to an event driven world
byGrace Jansen
 
PDF
Introduction to Apache Kafka
byvishnu rao
 
PDF
A Crash Course in Designing Messaging Apis With Jack Vanlightly | Current 2022
byHostedbyConfluent
 
PPTX
Kafka101
byAparna Pillai
 
PDF
NDC London 2017 - The Data Dichotomy- Rethinking Data and Services with Streams
byBen Stopford
 
Kafka 101
byAparna Pillai
 
Kafka 0.8.0 Presentation to Atlanta Java User's Group March 2013
byChristopher Curtin
 
Kafka and ibm event streams basics
byBrian S. Paskin
 
09-Kafka Overview - All you need to know
byUttamMittal2
 
Confluent Messaging Modernization Forum
byconfluent
 
ADV Slides: Trends in Streaming Analytics and Message-oriented Middleware
byDATAVERSITY
 
Evolving from Messaging to Event Streaming
byconfluent
 
Streaming Analytics unit 2 notes for engineers
byManjuAppukuttan2
 
Design Patterns for working with Fast Data
byMapR Technologies
 
Messaging With Apache ActiveMQ
byBruce Snyder
 
Event driven-arch
byMohammed Shoaib
 
[@NaukriEngineering] Messaging Queues
byNaukri.com
 
Design Patterns for working with Fast Data in Kafka
byIan Downard
 
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
byLucas Jellema
 
RabbitMQ vs Apache Kafka - Part 1
byErlang Solutions
 
Virtual Meetup Sweden - Reacting to an event driven world
byGrace Jansen
 
Introduction to Apache Kafka
byvishnu rao
 
A Crash Course in Designing Messaging Apis With Jack Vanlightly | Current 2022
byHostedbyConfluent
 
Kafka101
byAparna Pillai
 
NDC London 2017 - The Data Dichotomy- Rethinking Data and Services with Streams
byBen Stopford
 

More from David Ware

PDF
Whats new in MQ V9.1
byDavid Ware
 
PDF
IBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
byDavid Ware
 
PDF
IBM MQ What's new - Sept 2022
byDavid Ware
 
PDF
IBM MQ Update, including 9.1.2 CD
byDavid Ware
 
PDF
Designing IBM MQ deployments for the cloud generation
byDavid Ware
 
PPT
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
byDavid Ware
 
PDF
IBM MQ: Using Publish/Subscribe in an MQ Network
byDavid Ware
 
PPT
IBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
byDavid Ware
 
PDF
InterConnect 2016: IBM MQ self-service and as-a-service
byDavid Ware
 
PDF
Whats new in IBM MQ; V9 LTS, V9.0.1 CD and V9.0.2 CD
byDavid Ware
 
PDF
What's new in IBM MQ, March 2018
byDavid Ware
 
PDF
InterConnect 2016: What's new in IBM MQ
byDavid Ware
 
Whats new in MQ V9.1
byDavid Ware
 
IBM MQ: An Introduction to Using and Developing with MQ Publish/Subscribe
byDavid Ware
 
IBM MQ What's new - Sept 2022
byDavid Ware
 
IBM MQ Update, including 9.1.2 CD
byDavid Ware
 
Designing IBM MQ deployments for the cloud generation
byDavid Ware
 
IBM WebSphere MQ: Managing Workloads, Scaling and Availability with MQ Clusters
byDavid Ware
 
IBM MQ: Using Publish/Subscribe in an MQ Network
byDavid Ware
 
IBM WebSphere MQ: Using Publish/Subscribe in an MQ Network
byDavid Ware
 
InterConnect 2016: IBM MQ self-service and as-a-service
byDavid Ware
 
Whats new in IBM MQ; V9 LTS, V9.0.1 CD and V9.0.2 CD
byDavid Ware
 
What's new in IBM MQ, March 2018
byDavid Ware
 
InterConnect 2016: What's new in IBM MQ
byDavid Ware
 

Recently uploaded

PDF
nsfconvertersoftwaretoconvertNSFtoPST.pdf
byNSF Converter Software
 
PDF
Constraints First - Why Our On-Prem Ticketing System Starts With Limits, Not ...
bySpirit Face
 
PDF
How Modern Custom Software is Revolutionizing Mortgage Lending Processes -Ma...
byMatellio
 
PPTX
Why Your Business Needs Snowflake Consulting_ From Data Silos to AI-Ready Cloud
byChetu
 
PDF
Operating System (OS) :UNIT-I Introduction to Operating System BCA SEP SEM-II...
byKuvempu University
 
PDF
Manual vs Automated Accessibility Testing – What to Choose in 2025.pdf
bykalichargn70th171
 
PDF
Cloud-Based Underwriting Software for Insurance
byInsurance Tech Services
 
PPTX
Managed Splunk Partner vs In-House: Cost, Risk & Value Comparison
byssuser6ab5a5
 
PDF
How NetSuite Cloud ERP Helps Businesses Overcome Legacy System Downtime.
byTechWize
 
PDF
Blueprint to build quality before the code exists - StackConnect Milan 2025
byLudovicoBesana1
 
PDF
Red Hat Summit 2025 - Triton GPU Kernel programming.pdf
bySanjeev Rampal
 
PDF
Transforming Compliance Through Policy & Procedure Management
byInsurance Tech Services
 
PDF
Cybersecurity Alert- What Organisations Must Watch Out For This Christmas Fes...
byCybernetic Global Intelligence
 
PDF
What Is A Woman (WIAW) Token – Smart Contract Security Audit Report by EtherA...
byEtherAuthority
 
PDF
SecureChain AI (SCAI) Token – Smart Contract Security Audit Report by EtherAu...
byEtherAuthority
 
PDF
Here’s the case study that shows how companies lose ₹2–6 Cr annually due to m...
bysiddhantdazzlo
 
PDF
Data structure using C :UNIT-I Introduction to Data structures and Stacks BCA...
byKuvempu University
 
PPTX
Lecture 3 - Scheduling - Operating System
bynurulalomador
 
PPTX
AI Clinic Management Software for Otolaryngology Clinics Bringing Precision, ...
byEasy Clinic
 
PDF
Database Management Systems(DBMS):UNIT-II Relational Data Model BCA SEP SEM ...
byKuvempu University
 
nsfconvertersoftwaretoconvertNSFtoPST.pdf
byNSF Converter Software
 
Constraints First - Why Our On-Prem Ticketing System Starts With Limits, Not ...
bySpirit Face
 
How Modern Custom Software is Revolutionizing Mortgage Lending Processes -Ma...
byMatellio
 
Why Your Business Needs Snowflake Consulting_ From Data Silos to AI-Ready Cloud
byChetu
 
Operating System (OS) :UNIT-I Introduction to Operating System BCA SEP SEM-II...
byKuvempu University
 
Manual vs Automated Accessibility Testing – What to Choose in 2025.pdf
bykalichargn70th171
 
Cloud-Based Underwriting Software for Insurance
byInsurance Tech Services
 
Managed Splunk Partner vs In-House: Cost, Risk & Value Comparison
byssuser6ab5a5
 
How NetSuite Cloud ERP Helps Businesses Overcome Legacy System Downtime.
byTechWize
 
Blueprint to build quality before the code exists - StackConnect Milan 2025
byLudovicoBesana1
 
Red Hat Summit 2025 - Triton GPU Kernel programming.pdf
bySanjeev Rampal
 
Transforming Compliance Through Policy & Procedure Management
byInsurance Tech Services
 
Cybersecurity Alert- What Organisations Must Watch Out For This Christmas Fes...
byCybernetic Global Intelligence
 
What Is A Woman (WIAW) Token – Smart Contract Security Audit Report by EtherA...
byEtherAuthority
 
SecureChain AI (SCAI) Token – Smart Contract Security Audit Report by EtherAu...
byEtherAuthority
 
Here’s the case study that shows how companies lose ₹2–6 Cr annually due to m...
bysiddhantdazzlo
 
Data structure using C :UNIT-I Introduction to Data structures and Stacks BCA...
byKuvempu University
 
Lecture 3 - Scheduling - Operating System
bynurulalomador
 
AI Clinic Management Software for Otolaryngology Clinics Bringing Precision, ...
byEasy Clinic
 
Database Management Systems(DBMS):UNIT-II Relational Data Model BCA SEP SEM ...
byKuvempu University
 

IBM MQ and Kafka, what is the difference?

  • 1.
    TechCon 2020 IBM MQand Kafka, what is the difference? David Ware IBM MQ Chief Architect
  • 2.
    Messaging is essentialfor building fully connected, efficient and scalable solutions. More now than ever before Messaging covers a broad spectrum of requirements
  • 3.
    Messaging Scalability Security Management High Availability There aremany capabilities essential to every enterprise messaging solution IBM MQ and Kafka have both repeatedly been proven to tick these boxes…
  • 4.
    © 2020 IBMCorporation So just pick one and move on?
  • 5.
    © 2020 IBMCorporation So just pick one and move on? Perhaps, if you think everything looks like a nail
  • 6.
    © 2020 IBMCorporation So just pick one and move on? Perhaps, if you think everything looks like a nail Otherwise, let’s focus on what you’re trying to achieve…
  • 7.
    Critical data exchange:work that needs to be done Critical applications demand assured asynchronous interactions Essential capabilities: End-to-end once and once only Deliveryü Fine grain messaging Messages typically represent commands, queries and operations The message is a way to pass control from the originator of the message to the consumer
  • 8.
    Event Driven: buildingscalable microservices Microservices increases the need for communication. API-based interactions can build fragile and unscalable tight bonds between components Topics and Subscriptions Publishing and subscribing to events relaxes the coupling of microservices Events are messages that communicate that something has occurred Essential capabilities:
  • 9.
    Event Streaming: theexpanding need for messaging Event Streaming brings data together from disparate sources, enabling even more responsive and engaging experiences for a wider set of users Scalable Subscription Stream History Efficient cloud and analytics applications utilize local decoupled buffers of event data Essential capabilities:
  • 10.
    The right toolfor the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
  • 11.
    © 2020 IBMCorporation To choose the right tool you’re probably going to need to go to the next level…
  • 12.
    © 2020 IBMCorporation Message consumption
  • 13.
    © 2019 IBMCorporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption
  • 14.
    © 2019 IBMCorporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
  • 15.
    © 2019 IBMCorporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7
  • 16.
    © 2019 IBMCorporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
  • 17.
    © 2019 IBMCorporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
  • 18.
    © 2019 IBMCorporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER 0 0 1 2 3 4 5 6 7 CONSUMER • Consumers never delete a message • Consumers can move forwards and backwards • Historic data remains available for new and old consumers • Messages are removed by the system independently from the applications, whether they’ve been consumed or not • The system continues to accept new data as old messages are removed Destructive consumption Non-destructive consumption 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 … • Messages processed once and then deleted • Minimal data stored • Messages stay until they are consumed • When space is filled, new messages are blocked
  • 19.
    © 2019 IBMCorporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
  • 20.
    © 2019 IBMCorporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER
  • 21.
    © 2019 IBMCorporation Consumption models 0 1 2 3 4 5 6 7 CONSUMER • In-order message delivery • Fixed consumer scale per sequence 0 1 2 3 4 5 6 7 • Out of order message processing across consumers • Easy dynamic consumer scale Exclusive consumption Shared consumption 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER 0 1 23 45 67
  • 22.
    © 2019 IBMCorporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots
  • 23.
    © 2019 IBMCorporation Consumption models IBM MQ Kafka Destructive consumption Messages only remain in the system until an application has consumed them or they expire individually n/a Non-destructive consumption MQ niche: Messages can be browsed first, but ultimately, they still need to be destructively consumed (or expired) Messages remain available, independent of the consuming applications. Duration and volume is controlled per topic Exclusive consumption Exclusive access can be controlled administratively or per application Consumers always have exclusive access to a sequence of messages Shared consumption Unlimited number of consumers for a sequence of messages n/a Solution sweet spots Kafka topic partitions…
  • 24.
    © 2020 IBMCorporation Queues and topics
  • 25.
    © 2019 IBMCorporation Comparing queues and topics QUEUE 0 1 2 3 4 5 6 7 CONSUMER CONSUMER CONSUMER PRODUCER PRODUCER 0 1 23 45 6 7 TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER CONSUMER 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 0 1 2 3 4 5 6 7 Queues Topics • Each messages goes to a single consumer from the queue • Each message goes to every subscriber to the topic
  • 26.
    © 2019 IBMCorporation CONSUMERCONSUMER MQ and Kafka topics TOPIC PARTITION 0 1 2 3 4 5 6 7 CONSUMER GROUP CONSUMER GROUP PRODUCER PRODUCER CONSUMER CONSUMER TOPIC 0 1 2 3 4 5 6 7 SUBSCRIPTION SUBSCRIPTION PRODUCER PRODUCER CONSUMER CONSUMER 1 2 3 4 5 6 70 1 2 3 4 5 6 70 IBM MQ Kafka
  • 27.
    © 2019 IBMCorporation Queues and topics IBM MQ Kafka Queues Primary MQ concept Queues are administratively or dynamically created Queues can be partitioned for scale All messages are stored on queues, even publications n/a Topics Core MQ capability Topics are dynamically created as needed by applications Each subscribing application has access to a dedicated queue of to-be-consumed messages matching their subscription. Primary Kafka concept Rolling history of messages maintained by Kafka. Topics can be partitioned for scale… Consumer’s offset within a shared stream of messages is maintained by Kafka.
  • 28.
    © 2020 IBMCorporation Scaling
  • 29.
    Horizontal scaling –workload balance the messages PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER IBM MQ Kafka PRODUCE MESSAGES T O Q U E U E A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER
  • 30.
    Kafka Clusters • AKafka system is typically made up of multiple active nodes • Topics can be partitioned to enable workload balancing across those nodes, improving scale and availability of the topics • Kafka workload balances messages across partitions, either randomly or based on application provided context to enable per-partition ordering • Each topic partition is active in one broker at a time • The messages held in a topic partition can be replicated to other brokers to ensure data integrity across failures and automatic recovery
  • 31.
    © 2019 IBMCorporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 Kafka partitions and consumer groups 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
  • 32.
    © 2019 IBMCorporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
  • 33.
    © 2019 IBMCorporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers
  • 34.
    © 2019 IBMCorporation CONSUMER GROUP A TOPIC PARTITION 0 PARTITION 1 PARTITION 2 CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER GROUP B CONSUMER CONSUMER Kafka partitions and consumer groups - p0, offset 70 p1, offset 31 p2, offset 52 p0, offset 70 p1, offset 31 p2, offset 52 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular partition in order • Changing the number of partitions may result in out-of-order messages • There can be more partitions than consumers in a consumer group, but no benefit to more consumers X
  • 35.
    IBM MQ Clustersand workload balancing PRODUCE MESSAGES TO QUEUE A CONSUMER CONSUMER CONSUMER CONSUMER CONSUMER • A single running instance of MQ is a queue manager. • These can act independently or collectively to build horizontally scaled solutions. • An MQ Cluster is a dynamically provisioned set of queue managers with the ability to partition queues across them. • MQ will workload balance inbound messages from producing applications across each partition of the queue. • With MQ’s Uniform Cluster capability, MQ will workload balance multiple consuming applications across each queue manager, ensuring messages are consumed efficiently. • MQ does not provide simple message ordering in these topologies • MQ’s HA technologies provide data integrity across failures and automatic recovery
  • 36.
    © 2019 IBMCorporation MQ Clustering and application rebalancing APPLICATION QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances
  • 37.
    © 2019 IBMCorporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20, … 01,11, … X
  • 38.
    © 2019 IBMCorporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20 01,11 03,13,23
  • 39.
    © 2019 IBMCorporation MQ Clustering and application rebalancing APPLICATION A QUEUE/SUBSCRIPTION INSTANCE 0 INSTANCE 1 INSTANCE 2 CONSUMER CONSUMER CONSUMER 01 11 21 31 02 12 22 32 42 52 00 10 20 30 40 50 60 70 PRODUCER • Each consumer sees messages from a particular queue instance at any one time • Limited message ordering guarantees, particularly for producers • There should be at least as many consumers as instances, but can be increased independently to instances 00,10,20,40 01,11 03,13,23,43 CONSUMER CONSUMER CONSUMER CONSUMER 30,50,70 32,52 31 21
  • 40.
    © 2019 IBMCorporation Scaling IBM MQ Kafka Consuming application is the bottleneck Scale out the consuming application Messaging system is the bottleneck Scale out the messaging system Increase the number of consuming applications from the existing queue/subscription. Increase the number of topic partitions to support a greater number of consuming applications in a consumer group. Increase the number of queue managers deployed and workload balance messages and consumers across them. Increase the number of Kafka broker nodes deployed and redistribute the topic partitions across them. †Good practice is to capacity plan for highest load and pre-define multiple topic partitions to support more consumers than you currently require, enabling more dynamic consumer scaling https://www.confluent.io/blog/how-choose-number-topics-partitions-kafka-cluster/ † † † Unlike MQ, where new queue managers do not require copies of the existing messages, Kafka requires event histories to be copied to a new broker before it can be effective, slowing down dynamic scaling https://www.infoq.com/presentations/cloud-native-kafka-netflix/ ††
  • 41.
    © 2020 IBMCorporation Conversational messaging
  • 42.
    © 2019 IBMCorporation ConversationConversation ConversationConversation Fine grain communication? • MQ is designed for tens of thousands of lightweight queues and hundreds of thousands of dynamic topic strings per queue manager • This enables each conversation (e.g. reply flow) to be a dedicated queue or topic • Each consumer only sees its intended message • Dynamic create/deletion of queues and topic strings IBM MQ Equivalent Kafka model … • Kafka is designed for less, larger, configured topic partitions per broker • Recommendations are typically for 2-4k topic partitions per broker* • Consumers process every message in the topic from a point forward • This model is considered a Kafka anti-pattern. Recommendation is to rearchitect application to a stream processing pattern if possible** • https://cwiki.apache.org/confluence/display/KAFKA/KIP-578%3A+Add+configuration+to+limit+number+of+partitions ** https://www.quora.com/How-many-topics-can-be-created-in-Apache-Kafka PRODUCERPRODUCER PRODUCERPRODUCER CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 0 PRODUCER QUEUE/TOPIC Conversation CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … CONSUMER 2 30 1 … TOPIC 2 30 1 … …
  • 43.
    © 2020 IBMCorporation Message delivery
  • 44.
    © 2019 IBMCorporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Any problem in the messaging system or the application may result in messages being lost Messages are not lost in the event of a failure, but any message may be duplicated Messages are not lost in the event of a failure and each message can be processed individually to prevent possible duplication An application can coordinate the processing of a message with a specific update to an external resource, such as a database
  • 45.
    © 2019 IBMCorporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications
  • 46.
    © 2019 IBMCorporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Suitable for applications that regularly resend messages or can handle losses E.g. “I have 17 widgets this hour” Applications need to be designed to ignore or handle duplicate messages. E.g. “I have increased my widget stock to 17” Applications reliably knows that each message will be processed only once, almost independently of application logic. E.g. “Order me a widget” Applications can update multiple resources without fear of them ending up out of step. E.g. “Transfer $50 from my bank account to pay for the widget” Applications Ideal for event streaming use case Ideal for critical data exchange use case
  • 47.
    © 2019 IBMCorporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Higher levels of message reliability typically reduce the need for applications to consider error scenarios. However, higher message reliability typically reduces overall message performance and can limit overall availability The trade off Event streaming criticaldata exchange Reliability Performance
  • 48.
    © 2019 IBMCorporation Message delivery TRANSACTIONALLY COORDINATED EXACTLY ONCE AT LEAST ONCE AT MOST ONCE 1 2 3 4 8 9 1 2 3 4 5 6 7 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 Kafka IBM MQ https://medium.com/@andrew_schofield/does-apache-kafka-do-acid-transactions-647b207f3d0e Sweet spots Default behavior. Some messages may be lost following an application or Kafka failure, but others persisted Default behavior. All messages of this type lost following an MQ failure by design Kafka well designed for this, producers can be configured to retry failed sends and consumers typically periodically update their offset Possible to design an application to behave like this through message batching and protocol optimizations, but not a key MQ scenario Possible within a closed Kafka system, but much harder with external systems. Technically possible to lock Kafka down to confirm every message for individual acknowledgment but loses benefits A natural fit for the destructive consume approach of MQ coupled with its core persistence model Supports Kafka-only transactions for exactly-once stream processing applications, but no external resource coordination A core MQ capability is as a resource manager (or coordinator) for global transactions Event streaming criticaldata exchange
  • 49.
    © 2020 IBMCorporation Message availability
  • 50.
    © 2019 IBMCorporation Per-message availability IBM MQ Kafka Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes. PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9
  • 51.
    © 2019 IBMCorporation Per-message availability IBM MQ Kafka PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 PassiveActive 1 2 3 4 5 6 7 8 9 1 2 3 4 5 6 7 8 9 active/active partitioning of queues and topics, each partition is active/passive Application view System view Messages are tied to a queue manager being available. Physical location of queue manager obfuscated from the application. Messages are tied to a topic partition being available. Physical location of topic partition within the Kafka cluster obfuscated from the application. Each queue manager can be configured to be replicated and automatically fail over across systems. Topic partitions can be configured to be replicated and automatically fail over across multiple Kafka broker nodes.
  • 52.
    © 2020 IBMCorporation Summary
  • 53.
    © 2020 IBMCorporation If you want to build the best solution, you’re probably going to need more than a hammer
  • 54.
    The right toolfor the job Fine grain messaging End-to-end once and once only Deliveryü Topics and Subscriptions Critical data exchange Event driven Event streaming Apache Kafka Focused on streaming of events IBM MQ Focused on message exchange and transactions connectivity Scalable Subscription Stream History
  • 55.
    Thank you David Ware IBMMQ Chief Architect — www.linkedin.com/in/dware1 © Copyright IBM Corporation 2020. All rights reserved. The information contained in these materials is provided for informational purposes only, and is provided AS IS without warranty of any kind, express or implied. Any statement of direction represents IBM’s current intent, is subject to change or withdrawal, and represent only goals and objectives. IBM, the IBM logo, and ibm.com are trademarks of IBM Corp., registered in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of IBM trademarks is available at Copyright and trademark information. 57TechCon 2020 © 2020 IBM Corporation
  • 56.
    TechCon 2020 /© 2020 IBM Corporation 58