Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Message Queueing  from an MQ noobs perspective
Preamble• I work for a global telco• We push telco events and messages around• All should be processed quickly and reliabl...
Scenario• We send messages around in small text  files, sometimes with RFC 2822 headers• Use the UNIX sendf/recfiled mechanism
Problem• Its entirely intolerant to faults• ... and network latency
Our Solution• Add in our own "message queueing"• ... and some facilities to decouple delays
Shortcomings• Messages cannot be sent to multiple hosts• Target hosts are not easily reconfigurable• Its developed in-house...
Scenario• multiple producers• a consumer and its backup• Systems separated by 100-400ms WAN • with occasional 100% packet ...
The Contenders
pymq• very hazy on the details• depends on MySQL? and Django?!?• ignored while looking at kombu / rabbitmq
ZeroMQ aka 0MQ"ZeroMQ is a message orientated IPC Library."              - commenter on stackoverflow
AMQP• Is a standard for message queueing• Producers submit messages to brokers• Brokers consist of exchanges and queues• E...
RabbitMQ
RabbitMQProducer              Broker                       Consumer
RabbitMQProducer                      Broker           Exchange                               Queue                       ...
RabbitMQQueues
RabbitMQExchanges
RabbitMQ
RabbitMQExchanges
RabbitMQ Routing
RabbitMQExchange Types
RabbitMQDirect Exchanges
RabbitMQTopic Exchanges
RabbitMQFanout Exchanges
RabbitMQPersistence
RabbitMQBroker Persistence
RabbitMQMessage Persistence
RabbitMQ1. Mark the exchange “durable”.2. Mark the queue “durable”.3. Set the message’s “delivery mode” to a value of 2
RabbitMQPersistence Warnings
RabbitMQAcknowledging
My Desired Setup   Remote Host                      Processing Server         Publisher                          Queue    ...
My Desired Setup   Remote Host                     Processing Server        Publisher                          Queue      ...
My Desired Setup   Remote Host                   Processing Server        Publisher                      Consumer         ...
ShovelRemote Host                      Processing Server     Publisher                           Queue                    ...
=ERROR REPORT==== 7-Aug-2012::16:50:32 ===** Generic server <0.366.0> terminating** Last message in was {$gen_cast,init}**...
error in config file "/usr/local/etc/rabbitmq/rabbitmq.config" (37): syntax errorbefore: .
Shovel[     {rabbitmq_shovel, [        {shovels, [           {hello_shovel, [               {sources, [                  {...
[                       Shovel     {rabbitmq_shovel, [        {shovels, [           {hello_shovel, [               {source...
Installation• 1. install RabbitMQ • (install homebrew; brew install rabbitmq;     rabbitmq-server [-detached])  • (yum ins...
pika"a pure-Python implementation of the       AMQP 0-9-1 protocol"
Publisherimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()c...
Consumerimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(localhost))channel = connection.channel(...
Remote Consumer   Local Publisher
Remote queuesimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channe...
Remote Consumer   Local Consumer   Local Publisher
Acknowledgementimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.chan...
Consumer   Publisher
Issues• pikas BlockingConnection (my use case is  simple enough) hard-codes the socket  timeout and fails to cope with lat...
Deleting a Queueimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.cha...
Kombu"a messaging framework for Python"
Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn:    with conn.SimpleQueue(hello...
Bizarro errorsamqplib.client_0_8.exceptions.AMQPChannelException: (406,u"PRECONDITION_FAILED - parameters for queue hello ...
Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost//) as conn:    with conn.SimpleQueue(hel...
Puka"An opinionated RabbitMQ client"
Publisherimport pukaclient = puka.Client(amqp://localhost//)client.wait(client.connect())client.wait(client.queue_declare(...
Other Plugins• Management (web interface, quite nice)• Federation (Shovel, but with MOAR)• STOMP protocol support
STOMPSimple (or Streaming) Text Orientated         Messaging Protocol
STOMP "simple"
STOMP{rabbitmq_stomp, [    {tcp_listeners, [        {"127.0.0.1", 61613}      , {"::1",    61613}    ]}  , {default_user, ...
stompclientfrom stompclient import PublishClientclient = PublishClient(localhost, 61613)client.connect()client.send(/queue...
stompclientfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn:    with conn.SimpleQueue(hel...
stompclient  from stompclient import PublishClient  client = PublishClient(localhost, 61613)  client.connect()  client.sen...
Celery"a synchronous or asynchronous task queue/job queue based on distributed message passing"
TcpCatcher• TCP, SOCKS, HTTP(S) proxy & monitor• Can introduce latency and transmission  errors• Understands HTTP and imag...
Message Queueing - by an MQ noob
Message Queueing - by an MQ noob
Upcoming SlideShare
Loading in …5
×

Message Queueing - by an MQ noob

3,700 views

Published on

This presentation was given at PyCon AU 2012 but not recorded. It was written as I learned about modern message queueing methods (in particular RabbitMQ.)

Published in: Technology
  • The 'bizarro error' occurs because of the equivalence checks in RabbitMQ. The amqp protocol specifies that exchanges and queues cannot be redeclared using different arguments. So you probably declared the 'hello' queue once, then changed some parameter (like durable, auto_delete etc), making the equivalence checks complain.

    I'm not a fan of the eq. checks, and I've never found them useful. They make it very hard to make changes in an application without breaking backwards compatibility. When the auto_delete flag on an exchange was discouraged and removed from Celery it meant that the new version could no longer talk to older versions because they disagreed on an unimportant flag. It could see if the exchange was already declared, but that will also cause the channel to crash forcing you to create a new channel, and with some clients even a new connection, because someone decided that queue_declare(passive=False) should raise an exception.
    But now I'm ranting. It's not a game stopper, but it is annoying, and it's not Kombu's fault.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Message Queueing - by an MQ noob

  1. 1. Message Queueing from an MQ noobs perspective
  2. 2. Preamble• I work for a global telco• We push telco events and messages around• All should be processed quickly and reliably• Some generate a response
  3. 3. Scenario• We send messages around in small text files, sometimes with RFC 2822 headers• Use the UNIX sendf/recfiled mechanism
  4. 4. Problem• Its entirely intolerant to faults• ... and network latency
  5. 5. Our Solution• Add in our own "message queueing"• ... and some facilities to decouple delays
  6. 6. Shortcomings• Messages cannot be sent to multiple hosts• Target hosts are not easily reconfigurable• Its developed in-house and has issues
  7. 7. Scenario• multiple producers• a consumer and its backup• Systems separated by 100-400ms WAN • with occasional 100% packet loss
  8. 8. The Contenders
  9. 9. pymq• very hazy on the details• depends on MySQL? and Django?!?• ignored while looking at kombu / rabbitmq
  10. 10. ZeroMQ aka 0MQ"ZeroMQ is a message orientated IPC Library." - commenter on stackoverflow
  11. 11. AMQP• Is a standard for message queueing• Producers submit messages to brokers• Brokers consist of exchanges and queues• Exchanges route the messages• Queues store messages• Consumers pull messages out
  12. 12. RabbitMQ
  13. 13. RabbitMQProducer Broker Consumer
  14. 14. RabbitMQProducer Broker Exchange Queue Consumer
  15. 15. RabbitMQQueues
  16. 16. RabbitMQExchanges
  17. 17. RabbitMQ
  18. 18. RabbitMQExchanges
  19. 19. RabbitMQ Routing
  20. 20. RabbitMQExchange Types
  21. 21. RabbitMQDirect Exchanges
  22. 22. RabbitMQTopic Exchanges
  23. 23. RabbitMQFanout Exchanges
  24. 24. RabbitMQPersistence
  25. 25. RabbitMQBroker Persistence
  26. 26. RabbitMQMessage Persistence
  27. 27. RabbitMQ1. Mark the exchange “durable”.2. Mark the queue “durable”.3. Set the message’s “delivery mode” to a value of 2
  28. 28. RabbitMQPersistence Warnings
  29. 29. RabbitMQAcknowledging
  30. 30. My Desired Setup Remote Host Processing Server Publisher Queue ebs Interw Queue Consumer
  31. 31. My Desired Setup Remote Host Processing Server Publisher Queue ebs Interw Queue Consumer Consumer
  32. 32. My Desired Setup Remote Host Processing Server Publisher Consumer ebs Interw Queue
  33. 33. ShovelRemote Host Processing Server Publisher Queue bs Queue rwe Consumer Inte Shovel
  34. 34. =ERROR REPORT==== 7-Aug-2012::16:50:32 ===** Generic server <0.366.0> terminating** Last message in was {$gen_cast,init}** When Server state == {state,undefined,undefined,undefined,undefined, hello_shovel, {shovel, {endpoint, [{amqp_params_direct,<<"guest">>,<<"/">>, rabbit@localhost,none,[]}], []}, {endpoint, [{amqp_params_network,<<"guest">>,<<"guest">>, <<>>,"hardrock",undefined,0,0,0,infinity,none, [#Fun<amqp_auth_mechanisms.plain.3>, #Fun<amqp_auth_mechanisms.amqplain.3>], [],[]}], []}, 0,on_confirm,#Fun<rabbit_shovel_sup.16.91344538>, #Fun<rabbit_shovel_sup.16.91344538>,<<"hello">>,5}, undefined,undefined,undefined,undefined,undefined}** Reason for termination ==** {{badmatch,{error,access_refused}}, [{rabbit_shovel_worker,make_conn_and_chan,1,[]}, {rabbit_shovel_worker,handle_cast,2,[]}, {gen_server2,handle_msg,2,[]}, {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,227}]}]}
  35. 35. error in config file "/usr/local/etc/rabbitmq/rabbitmq.config" (37): syntax errorbefore: .
  36. 36. Shovel[ {rabbitmq_shovel, [ {shovels, [ {hello_shovel, [ {sources, [ {brokers, ["amqp://"]} ]} , {destinations, [ {brokers, ["amqp://server"]} ]} , {queue, <<"hello">>} ]} ]} ]}].
  37. 37. [ Shovel {rabbitmq_shovel, [ {shovels, [ {hello_shovel, [ {sources, [ {brokers, ["amqp://"]} , {declarations, [ {exchange.declare, [ {exchange, <<"foo">>} , {type, <<"direct">>} , durable ]} , {queue.declare, [ {queue, <<"hello">>} ]} , {queue.bind, [ {exchange, <<"foo">>} , {queue, <<"hello">>} ]} ]} ]} , {destinations, [ {brokers, ["amqp://hardrock"]} ]} , {queue, <<"hello">>} ]} ]} ]}].
  38. 38. Installation• 1. install RabbitMQ • (install homebrew; brew install rabbitmq; rabbitmq-server [-detached]) • (yum install rabbitmq-server.noarch) • rabbitmqctl or http://server:55672/• 2. pip install kombu, or pika, or puka, or ...
  39. 39. pika"a pure-Python implementation of the AMQP 0-9-1 protocol"
  40. 40. Publisherimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)channel.basic_publish(exchange=, routing_key=hello, body=Hello World!, properties=pika.BasicProperties(delivery_mode=2))print " [x] Sent Hello World!"connection.close() Publisher Queue
  41. 41. Consumerimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(localhost))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,)channel.basic_consume(callback, queue=hello, no_ack=True)channel.start_consuming() Queue Consumer
  42. 42. Remote Consumer Local Publisher
  43. 43. Remote queuesimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,)channel.basic_consume(callback, queue=hello, no_ack=True)channel.start_consuming() Host 1 Host 2 Publisher Queue Consumer Consumer
  44. 44. Remote Consumer Local Consumer Local Publisher
  45. 45. Acknowledgementimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_declare(queue=hello, durable=True)print [*] Waiting for messages. To exit press CTRL+Cdef callback(ch, method, properties, body): print [x] Received %r % (body,) raw_input(Press enter to ACK) ch.basic_ack(delivery_tag=method.delivery_tag)channel.basic_consume(callback, queue=hello, no_ack=False)channel.start_consuming()
  46. 46. Consumer Publisher
  47. 47. Issues• pikas BlockingConnection (my use case is simple enough) hard-codes the socket timeout and fails to cope with latency >1s• Fails to cope at all with packet loss
  48. 48. Deleting a Queueimport pikaconnection = pika.BlockingConnection(pika.ConnectionParameters(server))channel = connection.channel()channel.queue_delete(queue=hello)channel.close()connection.close()
  49. 49. Kombu"a messaging framework for Python"
  50. 50. Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn: with conn.SimpleQueue(hello) as queue: queue.put(Hello World!) print " [x] Sent Hello World!"
  51. 51. Bizarro errorsamqplib.client_0_8.exceptions.AMQPChannelException: (406,u"PRECONDITION_FAILED - parameters for queue hello in vhost /not equivalent", (50, 10), Channel.queue_declare)
  52. 52. Publisherfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost//) as conn: with conn.SimpleQueue(hello, queue_opts=dict(durable=False)) as queue: queue.put(Hello World!) print " [x] Sent Hello World!"
  53. 53. Puka"An opinionated RabbitMQ client"
  54. 54. Publisherimport pukaclient = puka.Client(amqp://localhost//)client.wait(client.connect())client.wait(client.queue_declare(queue=hello, durable=True))promise = client.basic_publish(exchange=, routing_key=hello, body=Hello World!)client.wait(promise)print " [x] Sent Hello World!"client.wait(client.close())
  55. 55. Other Plugins• Management (web interface, quite nice)• Federation (Shovel, but with MOAR)• STOMP protocol support
  56. 56. STOMPSimple (or Streaming) Text Orientated Messaging Protocol
  57. 57. STOMP "simple"
  58. 58. STOMP{rabbitmq_stomp, [ {tcp_listeners, [ {"127.0.0.1", 61613} , {"::1", 61613} ]} , {default_user, [ {login, "guest"} , {passcode, "guest"} ]}]}
  59. 59. stompclientfrom stompclient import PublishClientclient = PublishClient(localhost, 61613)client.connect()client.send(/queue/hello, Hello, world!)client.disconnect()
  60. 60. stompclientfrom kombu import BrokerConnectionwith BrokerConnection(amqp://localhost) as conn: with conn.SimpleQueue(hello) as queue: queue.put(Hello World!) print " [x] Sent Hello World!" from stompclient import PublishClient client = PublishClient(localhost, 61613) client.connect() client.send(/queue/hello, Hello, world!) client.disconnect()
  61. 61. stompclient from stompclient import PublishClient client = PublishClient(localhost, 61613) client.connect() client.send(/queue/hello, Hello, world!) client.disconnect()from stompclient import PublishClientwith PublishClient(localhost, 61613) as client: client.send(/queue/hello, Hello, world!)
  62. 62. Celery"a synchronous or asynchronous task queue/job queue based on distributed message passing"
  63. 63. TcpCatcher• TCP, SOCKS, HTTP(S) proxy & monitor• Can introduce latency and transmission errors• Understands HTTP and images• Can debug/interfere/log SSL traffic• Free: www.tcpcatcher.fr

×