Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Overview of
Message Queues
Bozhidar Bozhanov
Vanity slide
• Senior...whataver
• http://techblog.bozho.net
• http://twitter.com/bozhobg
Message queues
Източник: CloudAMQP
Why queues?
• Decoupling:
• service.performAction(params);
• sender.send(params);
• receiver.subscribe(actionType);
• Asyn...
Why queues (2)?
• Fault-tolerace
• Stopping the processing component doesn’t stop
the application
• Spikability
• (limited...
Use-cases
• Gathering statistics
• Visitor stats, used functionalities
• Sending emails
• Push
• Load balancing workers (e...
Types of queues
• Message Queue Brokers
• Usually used as a synonym of “message queue”
• Distributed / brokerless
• In-mem...
Broker
Source: RedHat
Brokerless
Source: ZeroMQ
In-memory
• Simple queues, without the need to
• distribute
• guarantee execution
• Brokerless in-memory
• whether message...
In the Java world
• JMS - JavaEE broker standard
• AMQP (RabbitMQ, ActiveMQ, Qpid)
• Kafka
• ZeroMQ
• JGroups
• Hazelcast
...
Complications
• Exactly-once delivery
• Consumer acknowledgments, publisher confirms
• Deliver order
• Persistence of data...
There are only two hard problems in distributed
systems:
2. Exactly-once delivery
1. Guaranteed order of messages
2. Exact...
Broker clusters
• No single point of failure
• How clients connect to the cluster?
• Hard-coded IPs
• DNS round-robin
• Lo...
ZeroMQ
• Allows both broker and brokerless
• Not a “system” but a “library”
• Sockets++
• Messaging patterns - “do-it-your...
It’s complicated…
“Simplicity is prerequisite for reliability”
-- E.Dijkstra
The more components to administer, the more
“...
Do we need a complex queues?
• Often - no.
• If you don’t need order guarantees, integration of
multiple systems and langu...
Synchronous calls
• If:
• You don’t have multiple appliations to integrate
• You don’t need background processing
• Compil...
Asynchronous
• Spring @Async / ExecutorService
• Risk of losing a message if an app node dies
• Option 1: you don’t care
•...
Database queues
• You already have a distributed component
– the database
• (regardless of whether it’s done by using shar...
Distributed batch processing
• How to have multiple worker nodes distribute
their work?
• Do we need multiple? Failover wo...
Queues and scalability?
• Often the two are connected
• But queues don’t always give scalability
• Scaling a broker isn’t ...
Scalability
• Horizontal scaling: just the application nodes
• Taking extra load (spikes):
• Message queues vs auto-scalin...
Options, options...
• Before choosing what to use:
• List all the mandatory features you need
• Try to simplify your use-c...
Conclusions
• No silver bullet
• It’s important to know all the options
• Use the simplest possible solution, but not
simp...
Thank you!
Overview of Message Queues
Upcoming SlideShare
Loading in …5
×

Overview of Message Queues

5,328 views

Published on

What are the reasons to use a message queue and what are the options in terms of technology

Published in: Software
  • Be the first to comment

Overview of Message Queues

  1. 1. Overview of Message Queues Bozhidar Bozhanov
  2. 2. Vanity slide • Senior...whataver • http://techblog.bozho.net • http://twitter.com/bozhobg
  3. 3. Message queues Източник: CloudAMQP
  4. 4. Why queues? • Decoupling: • service.performAction(params); • sender.send(params); • receiver.subscribe(actionType); • Async processing • Background, redundant • Integration • Multiple systems, multiple programming languages
  5. 5. Why queues (2)? • Fault-tolerace • Stopping the processing component doesn’t stop the application • Spikability • (limited to specific types of requests) • Event-sourcing friendly
  6. 6. Use-cases • Gathering statistics • Visitor stats, used functionalities • Sending emails • Push • Load balancing workers (e.g. video encoding, text analysis) • System integration
  7. 7. Types of queues • Message Queue Brokers • Usually used as a synonym of “message queue” • Distributed / brokerless • In-memory • Database-based
  8. 8. Broker Source: RedHat
  9. 9. Brokerless Source: ZeroMQ
  10. 10. In-memory • Simple queues, without the need to • distribute • guarantee execution • Brokerless in-memory • whether messages are stored on disk or in memory is an implementation detail
  11. 11. In the Java world • JMS - JavaEE broker standard • AMQP (RabbitMQ, ActiveMQ, Qpid) • Kafka • ZeroMQ • JGroups • Hazelcast • Redis • Spring @Async
  12. 12. Complications • Exactly-once delivery • Consumer acknowledgments, publisher confirms • Deliver order • Persistence of data • Transactional queues
  13. 13. There are only two hard problems in distributed systems: 2. Exactly-once delivery 1. Guaranteed order of messages 2. Exactly-once delivery (author: unknown)
  14. 14. Broker clusters • No single point of failure • How clients connect to the cluster? • Hard-coded IPs • DNS round-robin • Load-balancer • Multi-datacenter performance; split brain
  15. 15. ZeroMQ • Allows both broker and brokerless • Not a “system” but a “library” • Sockets++ • Messaging patterns - “do-it-yourself” • => more work • => learning curve
  16. 16. It’s complicated… “Simplicity is prerequisite for reliability” -- E.Dijkstra The more components to administer, the more “breakable” everything is.
  17. 17. Do we need a complex queues? • Often - no. • If you don’t need order guarantees, integration of multiple systems and languages, transactions • Alternative: • Simple synchronous calls • Async calls within a single app node • Database queue + batch processing • Hazelcast, JGroups • akka
  18. 18. Synchronous calls • If: • You don’t have multiple appliations to integrate • You don’t need background processing • Compile-time decoupling doesn’t give you much • You just don’t need a queue
  19. 19. Asynchronous • Spring @Async / ExecutorService • Risk of losing a message if an app node dies • Option 1: you don’t care • Option 2: client retry • akka – akka cluster
  20. 20. Database queues • You already have a distributed component – the database • (regardless of whether it’s done by using sharding, consistent hashing, master-slave) • Application node – stores in a table • Batch processor – reads the table
  21. 21. Distributed batch processing • How to have multiple worker nodes distribute their work? • Do we need multiple? Failover workers • Spring batch • Distributed locks • Hazelcast, Redisson, ZooKeeper • lock.tryLock(uniqueId){ process(uniqueId); }
  22. 22. Queues and scalability? • Often the two are connected • But queues don’t always give scalability • Scaling a broker isn’t trivial
  23. 23. Scalability • Horizontal scaling: just the application nodes • Taking extra load (spikes): • Message queues vs auto-scaling • Distributing resource-heavy tasks • Message queues vs spot instances
  24. 24. Options, options... • Before choosing what to use: • List all the mandatory features you need • Try to simplify your use-case • Why do you need a queue? • Distributing load • Integrating various components • “Because it’s cool”?
  25. 25. Conclusions • No silver bullet • It’s important to know all the options • Use the simplest possible solution, but not simpler
  26. 26. Thank you!

×