Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Message queues


Published on

Comparing message queues

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Message queues

  1. 1. Message Queues (RabbitMQ, CloudAMQP, AmazonSNS, Stormmq, ActiveMQ, …) By Ahmad Karawash 7/4/2016 1
  2. 2. List of Some Message Queues (MQ) I. RabbitMQ II. CloudAMQP (RabbitMQ as a service) III. Amazon SNS IV. Stormmq (MQ as a service) V. Apache ActiveMQ VI. SwiftMQ VII. Sparrow VIII.Starling IX. Kestrel X. Kafka (used by linkedin) 7/4/2016 2
  3. 3. I- RabbitMQ • Robust messaging for applications • Easy to use • Runs on all major operating systems • Supports a huge number of developer platforms • Open source and commercially supported • Languages: Java | .NET | Ruby | Python | PHP | Delphi/Free Pascal RabbitMQ Client | more... • Cloud: Cloud Foundry | EC2 | Google Compute Engine | more... 7/4/2016 3
  4. 4. I- RabbitMQ patterns 7/4/2016 4
  5. 5. II- CloudAMQP (RabbitMQ as a service) • CloudAMQP are managed RabbitMQ servers in the cloud • Hosted message queues that lets you pass messages between processes and other systems. • Messages are published to a queue by a producer, the consumers can then get the messages off the queue when the consumer wants to handle the messages. In-between, it can route, buffer, and persist the messages according to rules you give it. • Messages can be sent cross languages, platforms and OS, this way of handling messages decouple your processes and creates a highly scalable system. 7/4/2016 5
  6. 6. II- CloudAMQP instances 7/4/2016 6
  7. 7. II- CloudAMQP Console 7/4/2016 7
  8. 8. II- RabbitMQ Management Interface 7/4/2016 8
  9. 9. III- Amazon Simple Notification Service (Amazon SNS) • Send push messages, Deliver messages instantly to applications or users and eliminate polling in your apps. • Scale as your needs grow, Start fast with three simpleAPIs, and deliver an unlimited number of messages. • Use direct-addressing to send messages to individual devices or broadcast to multiple destinations at-once. • Use your choice of platforms, Directly access SNS using iOS, Android, Java, Python, PHP, Node.js, or .NET SDKs from AWS. • Deliver worldwide and across multiple protocols, Send notifications to Apple iOS, Android and other mobile devices, and destinations like Amazon SQS queues, Lambda functions, email addresses, SMS, and HTTP endpoints. • Easily connect with other AWS services, Use SNS as a message bus to send messages, alarms, and notifications from your AWS services such asAmazon RDS,CloudWatch, and S3 to other AWS services such as SQS and Lambda. • Get message delivery analytics, Get delivery status information viaAmazon CloudWatch on success rates, failure rates, and dwell times for mobile push messages as well as deliveries to SMS, SQS, HTTP, and Lambda destinations. • Usage-based pricing,You pay only for what you use with SNS. $0.50 per million publishes. 7/4/2016 9
  10. 10. III- How SNS works There are two types of clients: publishers and subscribers. • Publishers communicate asynchronously with subscribers by producing and sending a message to a topic, which is a logical access point and communication channel. • Subscribers (i.e., web servers, email addresses, Amazon SQS queues, AWS Lambda functions) consume or receive the message or notification over one of the supported protocols (i.e.,Amazon SQS, HTTP/S, email, SMS, Lambda) when they are subscribed to the topic. 7/4/2016 10
  11. 11. IV- Stormmq (MQ as a service) • Free. As In Beer: Send and receive messages absolutely free for up to 5 connected clients. No sign up charges. No subscriptions. Unless you want more. • Unlimited Monthly Messages: Send. Store. Receive. Any size. Any number: 1 or 1,000,000,000. Just pay for the storage you need. Consume messages quickly?Then storage is free. • Internet Scale: broker scales well beyond the C10K limit of traditional MQ technology, with no sync locks, and up to 850,000 messages / sec for ultimate users. Designed for today’s 48 core hydras. • PostgreSQL Admin Interface • Store & Forward • Never lose a message. Acknowledgments, partial re-delivery, the works. • Follows Publish & Subscribe • Secure by Default 7/4/2016 11
  12. 12. V- Apache ActiveMQ (Apollo) • Multi-protocol broker and supports STOMP, AMQP, MQTT, Openwire, SSL, andWebSockets. • Queue Browsers • Durable Subscriptions onTopics • Mirrored Queues • Reliable Messaging • Message Expiration, Message Swapping, Message Selectors, Message Groups • JAAS Authentication • ACL based Authorization • SSL/TLS Support and Certificate based Authentication • REST Management API 7/4/2016 12
  13. 13. VI- SwiftMQ • HighAvailability JMS Router with ACTIVE/STANDBY deployment and active replication. • Transparent JMS and JNDI failover, no extra reconnect code required. • Automatic and transparent resolve of all open transactions in transit, includingXA transactions, as well as any non-transacted acknowledge mode. • Full inbound and outbound duplicate message detection. Messages are never delivered twice. • No persistent messages lost during failover. Message sequence is guaranteed. • Choice of 3 different store types: Replicated File Store, Shared File Store, Shared JDBC Store. • Transparent reconnect from SwiftMQ Explorer, CLI, CLIAdminAPI. • Easy conversion to HighAvailability by HAWizard.Turns any existing configuration into HighAvailability. • HighAvailability test suite included.Test your JMS applications under a continuous failover scenario. • Additional Swiftlets included at no extra cost: JMS XA/ASF Swiftlet, Routing/UnlimtedSwiftlet, Standard File Store Swiftlet, JDBC Store Swiftlet. 7/4/2016 13
  14. 14. VII- Sparrow • fast lightweight queue written in Ruby that speaks memcache.That means you can use Sparrow with any memcached client library (Ruby or otherwise). • keeps messages in memory, but persists them to disk, using Sqlite, when the queue is shutdown. • Basic tests shows that Sparrow processes messages at a rate of around 1660 per second.The load Sparrow can cope with increases exponentially as you add to the cluster. • Sparrow's speed greatly increases when using the included client. • comes with built in support for daemonization and clustering. • If the Sparrow servers aren't available, queuing can fallback to Amazon SQS 7/4/2016 14
  15. 15. VIII- Starling • Starling is a powerful but simple messaging server that enables reliable distributed queuing with an absolutely minimal overhead. • It speaks the MemCache protocol for maximum cross-platform compatibility. • Any language that speaks MemCache can take advantage of Starling's queue facilities. • Only tested with Memcache client 1.7.x and with SystemTimer ( • Starling is “slow” as far as messaging systems are concerned. In practice, it's fast enough. 7/4/2016 15
  16. 16. IX- Kestrel • Fast, It runs on the JVM so it can take advantage of the hard work people have put into java performance. • Small, currently about 2500 lines of scala, because it relies on Netty (a rough equivalent of Danger's ziggurat or Ruby's EventMachine) -- and because Scala is extremely expressive. • Durable, Queues are stored in memory for speed, but logged into a journal on disk so that servers can be shutdown or moved without losing any data. • Reliable, A client can ask to "tentatively" fetch an item from a queue, and if that client disconnects from kestrel before confirming ownership of the item, the item is handed to another client. In this way, crashing clients don't cause lost messages. • It is not strongly ordered, While each queue is strongly ordered on each machine, a cluster will appear "loosely ordered" because clients pick a machine at random for each operation.The end result should be "mostly fair". • It is notTransactional, This is not a database. Item ownership is transferred with acknowledgement, but kestrel does not support grouping multiple operations into an atomic unit. 7/4/2016 16
  17. 17. X- Kafka • It is a distributed publish-subscribe messaging system. • Current usage of Kafka at LinkedIn includes: • Collecting log data (e.g., frontend activity events and service metrics) and loading them into Hadoop/DWH for offline reporting and ad hoc analysis. • Feeding log data directly to online applications, such as security, firehose (feeding Signal and LinkedIn Today), wotc, and realtime service dashboard. • Deploying derived data (e.g., customized emails and connection strength scores for Cloud) from Hadoop to production. • Automatic load balancing • Asynchronous send • Semantic partitioning 7/4/2016 17
  18. 18. ?@: 7/4/2016 18