Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Red Hat JBoss AMQ and Apache Kafka : which to use ?


Published on

From the Red Hat Summit 2017
Apache Kafka has recently become an interesting option for messaging. Kafka boasts some impressive features and performance numbers, so should you use it? In this session, we'll take a closer look at Kafka, the problems it was intended to solve and put it up against traditional brokers like Red Hat JBoss A-MQ / Apache ActiveMQ. We'll walk through the tradeoffs of each, and help you reason through the right messaging brokers for your use cases. Finally, we'll see how message brokers can be used in a complementary manner, bridging between the AMQP protocol provided by JBoss A-MQ and Apache Kafka.

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website!
    Are you sure you want to  Yes  No
    Your message goes here

Red Hat JBoss AMQ and Apache Kafka : which to use ?

  1. 1. Red Hat JBoss AMQ and Apache Kafka : Which to Use ? Christian Posta (@christianposta) Principal Architect Paolo Patierno (@ppatierno) Senior Software Engineer
  2. 2. Christian Posta ● Author “Microservices for Java Developers” ● Committer Apache Camel, ActiveMQ, ActiveMQ PMC, ● Contributor to Apache Kafka in spare time ● Worked with large Microservices, web-scale companies ● Blogger, speaker about microservices, DevOps, cloud, distributed systems, messaging Principal Architect at Red Hat @christianposta
  3. 3. Paolo Patierno ● Messaging and IoT team ● Lead/Committer on Eclipse Hono ● Committer on Eclipse Vert.x and Paho ● Hacking low constrained devices in spare time ● Technologies and protocols “globetrotter” ● Blogger and speaker about distributed systems, messaging, IoT and embedded “world” Senior Software Engineer at Red Hat @ppatierno
  4. 4. Looking closely at JBoss AMQ and Apache Kafka
  5. 5. Seriously? We are still talking about messaging?
  6. 6. How hard can it be! ● Send a message ● Receive a message ● Get my messages in order ● Get them once and only once
  7. 7. Why don’t we just pick the “fastest” one and be done?
  8. 8. Safety guarantees Scalability Recency Reliability Availability Performance
  9. 9. JBoss AMQ
  10. 10. JBoss AMQ ● Standards based (JMS 1.x/2.x) ● Multi protocol (AMQP 1.0, MQTT, STOMP, OpenWire) ● Based on open community, Apache ActiveMQ ● Highly available, fault tolerant, performant ● Trusted and used in many highly critical use cases around the world ● Uses high-level “queueing” constructs such as “queue”, “topic” “durable topic” etc
  11. 11. Producing messages
  12. 12. Durability
  13. 13. Dispatching messages
  14. 14. Consuming messages
  15. 15. Recap: ● For persistent messages, block waiting for acknowledgement ● Broker persists messages before continuing ● Broker responsible for dispatching to consumers ● Broker can handle redelivery on failover ● Consumers acknowledge each message ● Broker has transactional capabilities including XA transaction
  16. 16. Apache Kafka
  17. 17. Apache Kafka ● Developed originally at LinkedIn ● Built to support high throughput, high ingest of “log data” ● Kafka is (at its core) a distributed “log” ● Built for data pipeline between services and analytic clusters ● Highly parallelizable ● Persistent/replicated ● Key/Value structure (not “message” in the traditional sense”) ● Uses concept of publish-subscribe “topics”
  18. 18. Producing messages
  19. 19. Durability
  20. 20. Producing messages
  21. 21. Producing messages
  22. 22. Dispatching messages
  23. 23. Dispatching messages
  24. 24. Consuming messages
  25. 25. Recap: ● Send data as fast as possible ● Make efficient usage of transmission via batching ● Brokers write messages but don’t sync ● Rely on batching/write back to actual disk when the OS flushes ● Not responsible for dispatching, tracking acks etc ● Consumers track position in the log ● Consumers coordinate to balance consumption of messages
  26. 26. Two different brokers with different tradeoffs for different use cases. They co-exist in your architecture just fine.
  27. 27. Use JBoss AMQ for: ● You care about individual messages ● You want clients to use standard APIs (e.g., JMS) or wire protocols (e.g., AMQP) ● You need transactional sends and receives ● You’re doing request-reply messaging ● Using selectors ● Heterogeneous client/protocol messaging (ie, AMQP, MQTT, STOMP, etc) ● You send metadata/headers/properties with your messages ● You need Time-To-Live semantics (TTL) ● Need Dead Letter Queue semantics (DLQ) ● You don’t want to implement broker functionality in your clients (ie, partitioning, dispatching, coordination)
  28. 28. Use Kafka for: ● You care about messages in volume ● You care about raw throughput, high performance ● You need sliding-window replay abilities ● Large numbers of subscribers for published events ● You need to finely control the parallelism/scalability of consumers ● You want to leverage application-level replication vs HA storage ● You need total order guarantees at the partition level
  29. 29. Shhh…. Sneak peak... Find it from O’Reilly week of May 22nd!
  30. 30. … but they can live together …
  31. 31. Bridging AMQP to Kafka Why ? ● AMQP 1.0 … ○ … is supported by AMQ 7 Broker ○ … is THE protocol for AMQ 7 Interconnect ○ … is supported by all AMQ 7 Clients ● Maybe you have an application … ○ … with JMS/AMQP senders ○ … but now you need a different way to consume ○ … or a different feature (i.e. “re-read” stream) ● Avoiding changes on the senders side
  32. 32. Bridging AMQP to Kafka What does “bridging” mean ? ● Mapping AMQP semantics on Kafka ● From/to an AMQP “address” to/from a Kafka “topic” ● Addressing topic “partitions” through AMQP ● Handling QoS and flow control ● Dealing with messages : ○ AMQP full featured packets ○ Kafka simple raw bytes
  33. 33. Kafka Cluster Bridging AMQP to Kafka Source endpoints Sink endpoints Producers Consumers Senders Receivers
  34. 34. ● Attach a link on the Kafka [topic] address, specifying [] for consumer only ● QoS : settlement implies … ○ … waiting for Kafka “acks” from leaders/followers ○ … and Kafka “offsets” commit ● Flow : credits implies … ○ … sender can send on Kafka “acks” ○ … paused/resumed Kafka consumer transfer () / disposition() / flow (credits) attach ([topic]/[]) What about semantics … AMQP - Kafkaattach ([topic]) transfer () / disposition() / flow (credits) [topic] P0 P1 P2 sende r receiver
  35. 35. … and message structure ? ● AMQP “full featured” messages vs Kafka raw bytes ● Conversion ○ Default : AMQP body to/from Kafka bytes (losing metadata) ○ JSON : mapping AMQP body, properties, annotations … ○ Raw : raw AMQP message bytes become Kafka message (only for AMQP “enabled” clients on both sides)
  36. 36. Clients : AMQP to Kafka ● A JMS/AMQP client doesn’t need changes ○ Interacting with an A-MQ queue/topic is the same as Kafka topic ○ Just change the address ● A simplified API ○ Kafka protocol needs to open multiple connections to partition “leaders” ○ One connection, links multiplexing for interacting with Kafka topics (partitions) ○ The bridge does the “magic”
  37. 37. Demo : the deployment AMQ 7 Interconnect AMQ 7 Broker Apache Kafka AMQP - Kafka Kafka Web UI App JMS/AMQP clients Messaging-as-a-Service (EnMasse)
  38. 38. Demo : use cases ● Apache Kafka ○ Getting messages published by AMQP/JMS clients ○ Re-read the stream ■ from beginning ■ From a specific offset ● AMQ ○ Filtering on messages ■ Using a “selector” on a message property ○ Request/reply : ■ Using “store and forward” ■ Using “direct messaging”
  39. 39. ● One Kafka topic / one partition ● JMS/AMQP sender ● Native Kafka and JMS/AMQP receivers (belonging to different consumer groups) Demo : AMQP - Kafka AMQ 7 Interconnect Apache Kafka AMQP - Kafka Kafka Web UI App JMS/AMQP clients
  40. 40. ● Asking for a filter/selector when receiver connects ● It’s up to the broker delivering only messages which meet the filter ● No filtering at application level Demo : filtering AMQ 7 Interconnect AMQ 7 BrokerFilter : “count % 2 = 0” JMS/AMQP receiver JMS/AMQP sender count : <val>
  41. 41. Demo : request/reply AMQ 7 Interconnect AMQ 7 Broker JMS client JMS server AMQ 7 Interconnect AMQP client AMQP server ● “Store and forward” is used ● Queue for request and temporary queue for response hosted on broker ● “Direct messaging” is used ● Requests and replies are just routed through the router ● Address for request, a dynamic one for response
  42. 42. ● Message as a Service - EnMasse : ● AMQP - Kafka bridge : ● Kafka deployment : ● Demo code : Demo : material
  43. 43. @christianposta THANK YOU @ppatierno
  44. 44. THANK YOU