Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Red Hat JBoss AMQ and Apache Kafka :
Which to Use ?
Christian Posta (@christianposta)
Principal Architect
Paolo Patierno (...
Christian Posta
● Author “Microservices for Java Developers”
● Committer Apache Camel, ActiveMQ, ActiveMQ PMC,
Fabric8.io
...
Paolo Patierno
● Messaging and IoT team
● Lead/Committer on Eclipse Hono
● Committer on Eclipse Vert.x and Paho
● Hacking ...
Looking closely at JBoss AMQ
and Apache Kafka
Seriously? We are still talking about messaging?
How hard can it be!
● Send a message
● Receive a message
● Get my messages in order
● Get them once and only once
Why don’t we just pick the “fastest”
one and be done?
Safety guarantees Scalability
Recency
Reliability
Availability
Performance
JBoss AMQ
JBoss AMQ
● Standards based (JMS 1.x/2.x)
● Multi protocol (AMQP 1.0, MQTT, STOMP, OpenWire)
● Based on open community, Ap...
Producing messages
Durability
Dispatching messages
Consuming messages
Recap:
● For persistent messages, block waiting for acknowledgement
● Broker persists messages before continuing
● Broker ...
Apache Kafka
Apache Kafka
● Developed originally at LinkedIn
● Built to support high throughput, high ingest of “log data”
● Kafka is (...
Producing messages
Durability
Producing messages
Producing messages
Dispatching messages
Dispatching messages
Consuming messages
Recap:
● Send data as fast as possible
● Make efficient usage of transmission via batching
● Brokers write messages but do...
Two different brokers with different tradeoffs for
different use cases. They co-exist in your
architecture just fine.
Use JBoss AMQ for:
● You care about individual messages
● You want clients to use standard APIs (e.g., JMS) or wire protoc...
Use Kafka for:
● You care about messages in volume
● You care about raw throughput, high performance
● You need sliding-wi...
Shhh…. Sneak peak...
Find it from O’Reilly week of May 22nd!
… but they can live together …
Bridging AMQP to Kafka
Why ?
● AMQP 1.0 …
○ … is supported by AMQ 7 Broker
○ … is THE protocol for AMQ 7 Interconnect
○ … ...
Bridging AMQP to Kafka
What does “bridging” mean ?
● Mapping AMQP semantics on Kafka
● From/to an AMQP “address” to/from a...
Kafka
Cluster
Bridging AMQP to Kafka
Source endpoints
Sink endpoints
Producers
Consumers
Senders
Receivers
● Attach a link on the Kafka [topic] address, specifying [group.id] for consumer only
● QoS : settlement implies …
○ … wai...
… and message structure ?
● AMQP “full featured” messages vs Kafka raw bytes
● Conversion
○ Default : AMQP body to/from Ka...
Clients : AMQP to Kafka
● A JMS/AMQP client doesn’t need changes
○ Interacting with an A-MQ queue/topic is the same as Kaf...
Demo : the deployment
AMQ 7 Interconnect
AMQ 7 Broker
Apache Kafka
AMQP - Kafka
Kafka Web UI App
JMS/AMQP clients
Messagin...
Demo : use cases
● Apache Kafka
○ Getting messages published by AMQP/JMS clients
○ Re-read the stream
■ from beginning
■ F...
● One Kafka topic / one partition
● JMS/AMQP sender
● Native Kafka and JMS/AMQP receivers (belonging to different consumer...
● Asking for a filter/selector when receiver connects
● It’s up to the broker delivering only messages which meet the filt...
Demo : request/reply
AMQ 7 Interconnect
AMQ 7 Broker
JMS client
JMS server
AMQ 7 Interconnect
AMQP client AMQP server
● “S...
● Message as a Service - EnMasse : https://github.com/EnMasseProject/enmasse
● AMQP - Kafka bridge : https://github.com/En...
@christianposta
THANK YOU
@ppatierno
THANK YOU
plus.google.com/+RedHat
linkedin.com/company/red-hat
youtube.com/user/RedHatVideos
facebook.com/redhatinc
twitte...
Red Hat JBoss AMQ and Apache Kafka : which to use ?
Red Hat JBoss AMQ and Apache Kafka : which to use ?
Red Hat JBoss AMQ and Apache Kafka : which to use ?
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Hands on with CoAP and Californium
Next
Upcoming SlideShare
Hands on with CoAP and Californium
Next
Download to read offline and view in fullscreen.

30

Share

Red Hat JBoss AMQ and Apache Kafka : which to use ?

Download to read offline

From the Red Hat Summit 2017
Apache Kafka has recently become an interesting option for messaging. Kafka boasts some impressive features and performance numbers, so should you use it? In this session, we'll take a closer look at Kafka, the problems it was intended to solve and put it up against traditional brokers like Red Hat JBoss A-MQ / Apache ActiveMQ. We'll walk through the tradeoffs of each, and help you reason through the right messaging brokers for your use cases. Finally, we'll see how message brokers can be used in a complementary manner, bridging between the AMQP protocol provided by JBoss A-MQ and Apache Kafka.

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Red Hat JBoss AMQ and Apache Kafka : which to use ?

  1. 1. Red Hat JBoss AMQ and Apache Kafka : Which to Use ? Christian Posta (@christianposta) Principal Architect Paolo Patierno (@ppatierno) Senior Software Engineer
  2. 2. Christian Posta ● Author “Microservices for Java Developers” ● Committer Apache Camel, ActiveMQ, ActiveMQ PMC, Fabric8.io ● Contributor to Apache Kafka in spare time ● Worked with large Microservices, web-scale companies ● Blogger, speaker about microservices, DevOps, cloud, distributed systems, messaging Principal Architect at Red Hat christian@redhat.com @christianposta
  3. 3. Paolo Patierno ● Messaging and IoT team ● Lead/Committer on Eclipse Hono ● Committer on Eclipse Vert.x and Paho ● Hacking low constrained devices in spare time ● Technologies and protocols “globetrotter” ● Blogger and speaker about distributed systems, messaging, IoT and embedded “world” Senior Software Engineer at Red Hat ppatierno@redhat.com @ppatierno
  4. 4. Looking closely at JBoss AMQ and Apache Kafka
  5. 5. Seriously? We are still talking about messaging?
  6. 6. How hard can it be! ● Send a message ● Receive a message ● Get my messages in order ● Get them once and only once
  7. 7. Why don’t we just pick the “fastest” one and be done?
  8. 8. Safety guarantees Scalability Recency Reliability Availability Performance
  9. 9. JBoss AMQ
  10. 10. JBoss AMQ ● Standards based (JMS 1.x/2.x) ● Multi protocol (AMQP 1.0, MQTT, STOMP, OpenWire) ● Based on open community, Apache ActiveMQ ● Highly available, fault tolerant, performant ● Trusted and used in many highly critical use cases around the world ● Uses high-level “queueing” constructs such as “queue”, “topic” “durable topic” etc
  11. 11. Producing messages
  12. 12. Durability
  13. 13. Dispatching messages
  14. 14. Consuming messages
  15. 15. Recap: ● For persistent messages, block waiting for acknowledgement ● Broker persists messages before continuing ● Broker responsible for dispatching to consumers ● Broker can handle redelivery on failover ● Consumers acknowledge each message ● Broker has transactional capabilities including XA transaction
  16. 16. Apache Kafka
  17. 17. Apache Kafka ● Developed originally at LinkedIn ● Built to support high throughput, high ingest of “log data” ● Kafka is (at its core) a distributed “log” ● Built for data pipeline between services and analytic clusters ● Highly parallelizable ● Persistent/replicated ● Key/Value structure (not “message” in the traditional sense”) ● Uses concept of publish-subscribe “topics”
  18. 18. Producing messages
  19. 19. Durability
  20. 20. Producing messages
  21. 21. Producing messages
  22. 22. Dispatching messages
  23. 23. Dispatching messages
  24. 24. Consuming messages
  25. 25. Recap: ● Send data as fast as possible ● Make efficient usage of transmission via batching ● Brokers write messages but don’t sync ● Rely on batching/write back to actual disk when the OS flushes ● Not responsible for dispatching, tracking acks etc ● Consumers track position in the log ● Consumers coordinate to balance consumption of messages
  26. 26. Two different brokers with different tradeoffs for different use cases. They co-exist in your architecture just fine.
  27. 27. Use JBoss AMQ for: ● You care about individual messages ● You want clients to use standard APIs (e.g., JMS) or wire protocols (e.g., AMQP) ● You need transactional sends and receives ● You’re doing request-reply messaging ● Using selectors ● Heterogeneous client/protocol messaging (ie, AMQP, MQTT, STOMP, etc) ● You send metadata/headers/properties with your messages ● You need Time-To-Live semantics (TTL) ● Need Dead Letter Queue semantics (DLQ) ● You don’t want to implement broker functionality in your clients (ie, partitioning, dispatching, coordination)
  28. 28. Use Kafka for: ● You care about messages in volume ● You care about raw throughput, high performance ● You need sliding-window replay abilities ● Large numbers of subscribers for published events ● You need to finely control the parallelism/scalability of consumers ● You want to leverage application-level replication vs HA storage ● You need total order guarantees at the partition level
  29. 29. Shhh…. Sneak peak... Find it from O’Reilly week of May 22nd!
  30. 30. … but they can live together …
  31. 31. Bridging AMQP to Kafka Why ? ● AMQP 1.0 … ○ … is supported by AMQ 7 Broker ○ … is THE protocol for AMQ 7 Interconnect ○ … is supported by all AMQ 7 Clients ● Maybe you have an application … ○ … with JMS/AMQP senders ○ … but now you need a different way to consume ○ … or a different feature (i.e. “re-read” stream) ● Avoiding changes on the senders side
  32. 32. Bridging AMQP to Kafka What does “bridging” mean ? ● Mapping AMQP semantics on Kafka ● From/to an AMQP “address” to/from a Kafka “topic” ● Addressing topic “partitions” through AMQP ● Handling QoS and flow control ● Dealing with messages : ○ AMQP full featured packets ○ Kafka simple raw bytes
  33. 33. Kafka Cluster Bridging AMQP to Kafka Source endpoints Sink endpoints Producers Consumers Senders Receivers
  34. 34. ● Attach a link on the Kafka [topic] address, specifying [group.id] for consumer only ● QoS : settlement implies … ○ … waiting for Kafka “acks” from leaders/followers ○ … and Kafka “offsets” commit ● Flow : credits implies … ○ … sender can send on Kafka “acks” ○ … paused/resumed Kafka consumer transfer () / disposition() / flow (credits) attach ([topic]/group.id/[group.id]) What about semantics … AMQP - Kafkaattach ([topic]) transfer () / disposition() / flow (credits) [topic] P0 P1 P2 sende r receiver
  35. 35. … and message structure ? ● AMQP “full featured” messages vs Kafka raw bytes ● Conversion ○ Default : AMQP body to/from Kafka bytes (losing metadata) ○ JSON : mapping AMQP body, properties, annotations … ○ Raw : raw AMQP message bytes become Kafka message (only for AMQP “enabled” clients on both sides)
  36. 36. Clients : AMQP to Kafka ● A JMS/AMQP client doesn’t need changes ○ Interacting with an A-MQ queue/topic is the same as Kafka topic ○ Just change the address ● A simplified API ○ Kafka protocol needs to open multiple connections to partition “leaders” ○ One connection, links multiplexing for interacting with Kafka topics (partitions) ○ The bridge does the “magic”
  37. 37. Demo : the deployment AMQ 7 Interconnect AMQ 7 Broker Apache Kafka AMQP - Kafka Kafka Web UI App JMS/AMQP clients Messaging-as-a-Service (EnMasse)
  38. 38. Demo : use cases ● Apache Kafka ○ Getting messages published by AMQP/JMS clients ○ Re-read the stream ■ from beginning ■ From a specific offset ● AMQ ○ Filtering on messages ■ Using a “selector” on a message property ○ Request/reply : ■ Using “store and forward” ■ Using “direct messaging”
  39. 39. ● One Kafka topic / one partition ● JMS/AMQP sender ● Native Kafka and JMS/AMQP receivers (belonging to different consumer groups) Demo : AMQP - Kafka AMQ 7 Interconnect Apache Kafka AMQP - Kafka Kafka Web UI App JMS/AMQP clients
  40. 40. ● Asking for a filter/selector when receiver connects ● It’s up to the broker delivering only messages which meet the filter ● No filtering at application level Demo : filtering AMQ 7 Interconnect AMQ 7 BrokerFilter : “count % 2 = 0” JMS/AMQP receiver JMS/AMQP sender count : <val>
  41. 41. Demo : request/reply AMQ 7 Interconnect AMQ 7 Broker JMS client JMS server AMQ 7 Interconnect AMQP client AMQP server ● “Store and forward” is used ● Queue for request and temporary queue for response hosted on broker ● “Direct messaging” is used ● Requests and replies are just routed through the router ● Address for request, a dynamic one for response
  42. 42. ● Message as a Service - EnMasse : https://github.com/EnMasseProject/enmasse ● AMQP - Kafka bridge : https://github.com/EnMasseProject/amqp-kafka-bridge ● Kafka deployment : https://github.com/EnMasseProject/barnabas ● Demo code : https://github.com/ppatierno/amqp-kafka-demo Demo : material
  43. 43. @christianposta THANK YOU @ppatierno
  44. 44. THANK YOU plus.google.com/+RedHat linkedin.com/company/red-hat youtube.com/user/RedHatVideos facebook.com/redhatinc twitter.com/RedHatNews
  • MykhailoDubovytskyi

    Aug. 15, 2019
  • MykhailoDubovytskyi

    Aug. 15, 2019
  • SalvatoreSannino1

    Oct. 25, 2018
  • RajivRanjan250

    Oct. 12, 2018
  • atul018

    Sep. 3, 2018
  • chandrapaladugula

    Aug. 8, 2018
  • AlexandreBrandoLusto

    Jul. 25, 2018
  • Gudimetla

    Jun. 6, 2018
  • ArkadiuszWielga

    Apr. 7, 2018
  • ChiaChunShih

    Mar. 12, 2018
  • JulHuang

    Jan. 29, 2018
  • JianYang9

    Nov. 28, 2017
  • gabrielemuscas82

    Oct. 19, 2017
  • JosebaIakiFuertesdel

    Sep. 29, 2017
  • VenkataRavuri

    Jun. 28, 2017
  • lavillar

    Jun. 15, 2017
  • rnaszcyn

    May. 23, 2017
  • ceposta

    May. 22, 2017
  • PaulBrown86

    May. 19, 2017
  • viliamkasala

    May. 12, 2017

From the Red Hat Summit 2017 Apache Kafka has recently become an interesting option for messaging. Kafka boasts some impressive features and performance numbers, so should you use it? In this session, we'll take a closer look at Kafka, the problems it was intended to solve and put it up against traditional brokers like Red Hat JBoss A-MQ / Apache ActiveMQ. We'll walk through the tradeoffs of each, and help you reason through the right messaging brokers for your use cases. Finally, we'll see how message brokers can be used in a complementary manner, bridging between the AMQP protocol provided by JBoss A-MQ and Apache Kafka.

Views

Total views

15,574

On Slideshare

0

From embeds

0

Number of embeds

542

Actions

Downloads

317

Shares

0

Comments

0

Likes

30

×