Message queueing


Published on


Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • \n
  • \n
  • \n
  • \n
  • Network latency currently means processing slows down.\n
  • One "facility" is the zipping up of multiple events into a batch and sending the zip across the WAN in one go. This obviously complicates things further and introduces additional latency onto the message transmission.\n
  • \n
  • \n
  • \n
  • ... and I never actually came back to evaluate it\n
  • Is message queueing without a broker. Kind of like a glorified socket. Messages are routed in common MQ patterns right down at the network level. If you want store and forward, you're on your own for the persistence part.\n
  • \n
  • AMQP in a nutshell. I might add that was no help whatsoever in figuring this out.\n
  • There's a bunch of AMQP implementations out there, but in the interests of keeping my own sanity I only looked at the one most popular free implementation. Since they all end up implementing the same thing anyway.\n
  • \n
  • \n
  • Queues are where your messages end up in the broker. They sit there until a client (a.k.a. consumer) connects to the queue and siphons them off. Queues may be configured so messages are discarded if there isn’t a consumer ready to accept them. Multiple consumers may connect to a queue - the messages will be passed to each consumer in turn.\n
  • Exchanges are routers with routing tables that sit in front of queues. They're declared by consumers, just like queues, except that there's a default "just pass the message to the queue" exchange for the simple case. Every message has what’s known as a “routing key”, which is simply a string. The exchange has a list of bindings (routes) that say, for example, messages with routing key “X” go to queue “spam”.\n
  • Messages come in to the broker, are routed by exchanges and stored in queues until slurped off by consumers. Within a broker you may have multiple logical systems called virtual hosts. I'm not sure why. There's a default one and it's probably all I'll ever need.\nQueues and exchanges are created programmatically by your producers or consumers - not via a configuration file or command line program - your MQ configuration is in-line with your app code.\n
  • An interesting aside for performance - exchanges all run in their own processes so adding more exchanges is a way to spread load and increase throughput.\n
  • “routing rules” (or bindings) link an exchange to a queue based on a routing key. It is possible for two binding rules to use the same routing key. For example, maybe messages with the routing key “audit” need to go both to the “log-forever” queue and the “alert-the-big-dude” queue. To accomplish this, just create two binding rules (each one linking the exchange to one of the queues) that both trigger on routing key “audit”. In this case, the exchange duplicates the message and sends it to both queues.\n
  • There are multiple types of exchanges. They all do routing, but they accept different styles of binding “rules”. \n
  • A "direct" exchange only matches if the routing key is “dogs” or not. The default exchange is a Direct exchange.\n
  • For example, a “topic” exchange tries to match a message’s routing key against a wildcard pattern like “dogs.*”.\n
  • A "fanout" exchange ignores the routing key and distributes the messages to all queues.\n
  • There's two levels of persistence in play in RabbitMQ: the structure of the broker and the messages in the broker's queues.\n\n
  • You may mark your queues and exchanges as “durable” so the queue or exchange will be re-created automatically on reboot. It does not mean the messages in the queues will survive the reboot. They won’t.\n
  • When you publish your message to an exchange, you may set a flag called “Delivery Mode” to the value 2, which means “persistent”. “Delivery Mode” usually (depending on your AMQP library) defaults to a value of 1, which means “non-persistent”.\n
  • So the steps for persistent messaging are...\nIf you bind a durable queue to a durable exchange, RabbitMQ will automatically preserve the binding. Similarly, if you delete any exchange/queue (durable or not) any bindings that depend on it get deleted automatically.\n
  • RabbitMQ will not allow you to bind a non-durable exchange to a durable queue, or vice-versa. Both the exchange and the queue must be durable for the binding operation to succeed.\nYou cannot change the creation flags on a queue or exchange after you’ve created it. For example, if you create a queue as “non-durable”, and want to change it to “durable”, the only way to do this is to destroy the queue and re-create it. It’s a good reason to double check your declarations\n
  • This is my typical setup - I have hosts separated by network (WAN) out of my control. I need all the events generated on the remote hosts to be processed in a timely, reliable manner by the processing server.\n
  • One solution is to write a forwarding consumer on the remote host - pretty simple code but not very elegant.\n
  • An alternative setup we could use with the processing server consumer pulling from all the remote host queues. This makes it more configuration work when adding new remote hosts though.\n
  • Then I discovered the Shovel plugin. I've ignored exchanges up until now, but using Shovel allows an exchange on one host to pull messages from a queue and fire them at another queue. It basically runs an erlang client in the remote broker to forward the messages. In theory. In practise...\n
  • Look hard - it says "Fun" in there. This is NOT FUN. This was bloody hard to figure out. The main problem is that once you decide you're going to use Shovel, you're now programming Erlang. See previous statement about fun.\n
  • Not only am I now learning Erlang but the errors you get back for malformed configuration files (Erlang programs) are really unhelpful. This error basically says "there's an error somewhere in the 37 lines of your program."\n
  • This is my minimum shovel configuration, eventually discovered after much trial-and-error. I shall not share with you the entire volume of bizarre and obscure errors and warnings I waded through to get to this point, nor the number of dead-end alleyways the poor documentation led me down. Unfortunately when you start up the server with this configuration the log file will FILL with warnings about the queue not existing until a client connects and creates the queue.\n
  • This would probably be simpler if I was familiar with Erlang, but I found the documentation to be basically impenetrable. Also, for some reason I couldn't declare the queue without declaring an exchange. Which isn't needed if I don't declare the queue. There was an awful lot of stumbling around in the dark to get this working, but in the end it does.\n
  • Default login is guest/guest...\n
  • It tries to stay fairly independent of the underlying network support library. It uses amqplib underneath by default, which is what most of the libraries do.\n
  • This code connects to our RabbitMQ server using a blocking connection (send and wait for successful delivery at server before continuing); it declares a durable "helllo" queue and publishes a simple message to the queue. The queue and undelivered messages will be persistent across restarts of RabbitMQ. It's a slightly bizarrely wordy API. The exchange argument is required, but we use the "no-op" or "default" exchange here. More on them later.\n
  • This code connects to our RabbitMQ server using a blocking connection (listen and wait for messages); it also declares a "helllo" queue (just in case no publisher is connected) and consumes messages from the queue. You can see that a bunch of the code is the same - connections, channels and queues.\n
  • Here we see the publisher running on my laptop (local) and the consumer running on the server where the RabbitMQ server is also running. You can see that we can publish into the queue without anything consuming the messages, and we can consume published events immediately. We can also listen as a consumer with no publisher publishing. It's all quite disconnected.\n
  • By making one small change to our consumer, we can consume remote queues.\n
  • Here we set up a second consumer that will consume the remote queue on the server from the local host. Messages published to the queue will be passed to each consumer in turn round-robin style. When a consumer stops consuming the queue will transparently feed all messages to the remaining consumer. And of course this extends to having multiple publishers as well.\n
  • Can fix the hard-coded socket timeout issue, and a try/except might be able to handle the packet loss.\n
  • If you set up a queue with the wrong parameters you'll need to delete it. For some unknown reason the rabbitmq control program doesn't provide the ability to delete queues, so you need to do it from a client. This code does that.\n
  • Kombu is a messaging framework for Python. It replaces Carrot.\nThe aim of Kombu is to make messaging in Python as easy as possible by providing an idiomatic high-level interface for the AMQ protocol, and also provide proven and tested solutions to common messaging problems. Its "transports" include AMQP variations and non-AMQP "virtual transports" such as Redis, MongoDB, CouchDB and Beanstalk or "database transports" such as SQLAlchemy and Django ORM.\n
  • This took me some time to write as the documentation for Kombu is quite limited. Attempting to set things up using channels lead nowhere. I couldn't find clear documentation on how to create a channel. In the end I found the SimpleQueue which worked after some effort, but I'm still not clear on the details. Then I discovered that the default queue parameters were different (see next slide.)\n
  • When I was testing early on I wasn't using durable queues. kombu's SimpleQueue sets durable to True by default which caused the above bizarro error (which basically says I'm trying to use a queue with different parameters to those it was created with.) This error is not specific to Kombu, but it was unexpected and inexplicable when I first encountered it until I guessed at the durable parameter setting.\n
  • Took me a while to figure out how to disable durable.\n\nBut again, there's no channel - it's really just API noise for simple code like this.\n
  • The puka module implements a client for AMQP 0-9-1 protocol. It’s tuned to work with RabbitMQ broker, but should work fine with other message brokers that support this protocol. It tries to be a nicer API than pika, which is honestly quite appalling.\n
  • Everything in puka works off the Client which is quite different but pretty convenient. The API is basically the same as pika at the business end. The big difference is in the promises and the ability to wait on the promised action being completed successfully. Puka wants to be asynchronous - the wait() calls effectively force it to be synchronous for simple code like this.\nI needed to patch puka version 0.0.5 as mentioned in issue #15 on their tracker to fix a connection-time issue. Unfortunately even with that fix this was still fragile and inexplicably stopped working at one point. Puka also does away with channels like kombu's simple case.\n
  • STOMP is the Simple (or Streaming) Text Orientated Messaging Protocol\n
  • "STOMP is a very simple and easy to implement protocol, coming from the HTTP school of design; the server side may be hard to implement well, but it is very easy to write a client to get yourself connected. For example you can use Telnet to login to any STOMP broker and interact with it!"\n
  • The STOMP website kinda mirrors the AMQP website: there's a specification there but little else. Not so simple. Fortunately there's a bunch of Python STOMP client libraries.\n
  • Enabling STOMP was easy enough in the RabbitMQ configuration file - the docs were much clearer and the configuration much simpler than Shovel.\n
  • So here's one Python library, stompclient, publishing to my "hello" queue. \n
  • So... not a whole lot simpler...\n
  • I made a small change to the stompclient library to allow this usage, which I think is pretty darned simple - I believe it'll be in the next release. I'm not completely sold on STOMP yet though. I'm not sure it improves things over AMQP enough to switch from just using Kombu.\n
  • RPC engine. It is focused on real-time operation, but supports scheduling as well. The execution units, called tasks, are executed concurrently on a single or more worker servers. It's built over kombu and a variety of non-AMQP backends. The RPC nature and backend generality limit the abilities of MQ somewhat, though is nice if you're just after worker management (which I'm not.)\n
  • I found this worked sometimes, but not other times. RabbitMQ did NOT like me putting this in the middle of the Shovel setup, but the Python libraries I used were happy to use it as a proxy.\n
  • Message queueing

    1. 1. Message Queueing from an MQ noobs perspective
    2. 2. Preamble• I work for a global telco• We push telco events and messages around• All should be processed quickly and reliably• Some generate a response
    3. 3. Scenario• We send messages around in small text files, sometimes with RFC 2822 headers• Use the UNIX sendf/recfiled mechanism
    4. 4. Problem• Its entirely intolerant to faults• ... and network latency
    5. 5. Our Solution• Add in our own "message queueing"• ... and some facilities to decouple delays
    6. 6. Shortcomings• Messages cannot be sent to multiple hosts• Target hosts are not easily reconfigurable• Its developed in-house and has issues
    7. 7. Scenario• multiple producers• a consumer and its backup• Systems separated by 100-400ms WAN • with occasional 100% packet loss
    8. 8. The Contenders
    9. 9. pymq• very hazy on the details• depends on MySQL? and Django?!?• ignored while looking at kombu / rabbitmq
    10. 10. ZeroMQ aka 0MQ"ZeroMQ is a message orientated IPC Library." - commenter on stackoverflow
    11. 11. AMQP• Is a standard for message queueing• Producers submit messages to brokers• Brokers consist of exchanges and queues• Exchanges route the messages• Queues store messages• Consumers pull messages out
    12. 12. RabbitMQ
    13. 13. RabbitMQProducer Broker Consumer
    14. 14. RabbitMQProducer Broker Exchange Queue Consumer
    15. 15. RabbitMQQueues
    16. 16. RabbitMQExchanges
    17. 17. RabbitMQ
    18. 18. RabbitMQExchanges
    19. 19. RabbitMQ Routing
    20. 20. RabbitMQExchange Types
    21. 21. RabbitMQDirect Exchanges
    22. 22. RabbitMQTopic Exchanges
    23. 23. RabbitMQFanout Exchanges
    24. 24. RabbitMQPersistence
    25. 25. RabbitMQBroker Persistence
    26. 26. RabbitMQMessage Persistence
    27. 27. RabbitMQ1. Mark the exchange “durable”.2. Mark the queue “durable”.3. Set the message’s “delivery mode” to a value of 2
    28. 28. RabbitMQPersistence Warnings
    29. 29. My Desired Setup Remote Host Processing Server Publisher Queue ebs Interw Queue Consumer
    30. 30. My Desired Setup Remote Host Processing Server Publisher Queue ebs Interw Queue Consumer Consumer
    31. 31. My Desired Setup Remote Host Processing Server Publisher Consumer ebs Interw Queue
    32. 32. ShovelRemote Host Processing Server Publisher Queue bs Queue rwe Consumer Inte Shovel
    33. 33. =ERROR REPORT==== 7-Aug-2012::16:50:32 ===** Generic server <0.366.0> terminating** Last message in was {$gen_cast,init}** When Server state == {state,undefined,undefined,undefined,undefined, hello_shovel, {shovel, {endpoint, [{amqp_params_direct,<<"guest">>,<<"/">>, rabbit@localhost,none,[]}], []}, {endpoint, [{amqp_params_network,<<"guest">>,<<"guest">>, <<>>,"hardrock",undefined,0,0,0,infinity,none, [#Fun<amqp_auth_mechanisms.plain.3>, #Fun<amqp_auth_mechanisms.amqplain.3>], [],[]}], []}, 0,on_confirm,#Fun<rabbit_shovel_sup.16.91344538>, #Fun<rabbit_shovel_sup.16.91344538>,<<"hello">>,5}, undefined,undefined,undefined,undefined,undefined}** Reason for termination ==** {{badmatch,{error,access_refused}}, [{rabbit_shovel_worker,make_conn_and_chan,1,[]}, {rabbit_shovel_worker,handle_cast,2,[]}, {gen_server2,handle_msg,2,[]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
    34. 34. error in config file "/usr/local/etc/rabbitmq/rabbitmq.config" (37): syntax errorbefore: .
    35. 35. Shovel[ {rabbitmq_shovel, [ {shovels, [ {hello_shovel, [ {sources, [ {brokers, ["amqp://"]} ]} , {destinations, [ {brokers, ["amqp://server"]} ]} , {queue, <<"hello">>} ]} ]} ]}].
    36. 36. [ Shovel {rabbitmq_shovel, [ {shovels, [ {hello_shovel, [ {sources, [ {brokers, ["amqp://"]} , {declarations, [ {exchange.declare, [ {exchange, <<"foo">>} , {type, <<"direct">>} , durable ]} , {queue.declare, [ {queue, <<"hello">>} ]} , {queue.bind, [ {exchange, <<"foo">>} , {queue, <<"hello">>} ]} ]} ]} , {destinations, [ {brokers, ["amqp://hardrock"]} ]} , {queue, <<"hello">>} ]} ]} ]}].
    37. 37. Installation• 1. install RabbitMQ • (install homebrew; brew install rabbitmq; rabbitmq-server [-detached]) • (yum install rabbitmq-server.noarch) • rabbitmqctl or http://server:55672/• 2. pip install kombu, or pika, or puka, or ...
    38. 38. pika"a pure-Python implementation of the AMQP 0-9-1 protocol"
    39. 39. Publisherimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel =, durable=True)channel.basic_publish(exchange=, routing_key=hello, body=Hello World!, properties=pika.BasicProperties(delivery_mode=2))print " [x] Sent Hello World!"connection.close() Publisher Queue
    40. 40. Consumerimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(localhost))channel =, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,)channel.basic_consume(callback, queue=hello)channel.start_consuming() Queue Consumer
    41. 41. Remote Consumer Local Publisher
    42. 42. Remote queuesimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel =, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,)channel.basic_consume(callback, queue=hello)channel.start_consuming() Host 1 Host 2 Publisher Queue Consumer Consumer
    43. 43. Remote Consumer Local Consumer Local Publisher
    44. 44. Issues• pikas BlockingConnection (my use case is simple enough) hard-codes the socket timeout and fails to cope with latency >1s• Fails to cope at all with packet loss
    45. 45. Deleting a Queueimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel =
    46. 46. Kombu"a messaging framework for Python"
    47. 47. Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn: with conn.SimpleQueue(hello) as queue: queue.put(Hello World!) print " [x] Sent Hello World!"
    48. 48. Bizarro errorsamqplib.client_0_8.exceptions.AMQPChannelException: (406,u"PRECONDITION_FAILED - parameters for queue hello in vhost /not equivalent", (50, 10), Channel.queue_declare)
    49. 49. Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost//) as conn: with conn.SimpleQueue(hello, queue_opts=dict(durable=False)) as queue: queue.put(Hello World!) print " [x] Sent Hello World!"
    50. 50. Puka"An opinionated RabbitMQ client"
    51. 51. Publisherimport pukaclient = puka.Client(amqp://localhost//)client.wait(client.connect())client.wait(client.queue_declare(queue=hello, durable=True))promise = client.basic_publish(exchange=, routing_key=hello, body=Hello World!)client.wait(promise)print " [x] Sent Hello World!"client.wait(client.close())
    52. 52. Other Plugins• Management (web interface, quite nice)• Federation (Shovel, but with MOAR)• STOMP protocol support
    53. 53. STOMPSimple (or Streaming) Text Orientated Messaging Protocol
    54. 54. STOMP "simple"
    55. 55. STOMP{rabbitmq_stomp, [ {tcp_listeners, [ {"", 61613} , {"::1", 61613} ]} , {default_user, [ {login, "guest"} , {passcode, "guest"} ]}]}
    56. 56. stompclientfrom stompclient import PublishClientclient = PublishClient(localhost, 61613)client.connect()client.send(/queue/hello, Hello, world!)client.disconnect()
    57. 57. stompclientfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn: with conn.SimpleQueue(hello) as queue: queue.put(Hello World!) print " [x] Sent Hello World!" from stompclient import PublishClient client = PublishClient(localhost, 61613) client.connect() client.send(/queue/hello, Hello, world!) client.disconnect()
    58. 58. stompclient from stompclient import PublishClient client = PublishClient(localhost, 61613) client.connect() client.send(/queue/hello, Hello, world!) client.disconnect()from stompclient import PublishClientwith PublishClient(localhost, 61613) as client: client.send(/queue/hello, Hello, world!)
    59. 59. Celery"a synchronous or asynchronous task queue/job queue based on distributed message passing"
    60. 60. TcpCatcher• TCP, SOCKS, HTTP(S) proxy & monitor• Can introduce latency and transmission errors• Understands HTTP and images• Can debug/interfere/log SSL traffic• Free: