Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)

2,148 views

Published on

RabbitMQ is a message broker - an application that allows communication between applications by way of a message queuing system. In this talk, we’ll set up an RabbitMQ instance, take an intermediate-level look into the technical features it provides and also how you can apply RabbitMQ in your in applications to scale them efficiently.

Practical Message Queuing Using RabbitMQ (PHPem, 3rd July 2014)

  1. 1. Practical Message Queueing Using RabbitMQ James Titcumb PHPem 3rd July 2014
  2. 2. James Titcumb www.jamestitcumb.com www.protected.co.uk www.phphants.co.uk @asgrim Who is this guy?
  3. 3. Who are you? https://www.flickr.com/photos/akrabat/10168019755/
  4. 4. What is message queueing?
  5. 5. Separation of Concerns
  6. 6. Scaling with Rabbit RabbitMQApplication Background processing
  7. 7. Scaling with Rabbit RabbitMQApplication Background processing Background processing
  8. 8. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing
  9. 9. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing
  10. 10. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing Background processing
  11. 11. ● Fast logging solution Some real world uses...
  12. 12. ● Fast logging solution ● Background Processing Some real world uses...
  13. 13. ● Fast logging solution ● Background Processing ○ Sending emails Some real world uses...
  14. 14. ● Fast logging solution ● Background Processing ○ Sending emails ○ Sending SMS Some real world uses...
  15. 15. ● Fast logging solution ● Background Processing ○ Sending emails ○ Sending SMS ○ Analytics, reporting Some real world uses...
  16. 16. (on precise64, other OSs may vary) Installing RabbitMQ
  17. 17. ● add apt repo ○ deb http://www.rabbitmq.com/debian/ testing main ● add signing key ○ http://www.rabbitmq.com/rabbitmq-signing-key-public.asc ● apt-get update ● apt-get install rabbitmq-server ● rabbitmq-plugins enable rabbitmq_management ● sudo service rabbitmq-server restart Using Apt
  18. 18. http://localhost:15672/
  19. 19. Basic Message Queuing
  20. 20. Objective: Basic Queuing Producer Consumer test_queue 1 2 3 4 5
  21. 21. composer.json { "require": { "videlalvaro/php-amqplib": "2.*" } } then composer install
  22. 22. Please wait, connecting... use PhpAmqpLibConnectionAMQPConnection; $connection = new AMQPConnection( 'localhost', 5672, 'guest', 'guest', '/' ); $channel = $connection->channel();
  23. 23. basic/producer.php use PhpAmqpLibMessageAMQPMessage; $channel->queue_declare( 'test_queue', false, true, false, false); $message = new AMQPMessage('my test message'); $channel->basic_publish($message, '', 'test_queue');
  24. 24. basic/consumer.php $channel->basic_consume( 'test_queue', // Queue to consume '', // Consumer identifier false, true, // No-ack means messages are "auto acknowledged" false, // Exclusive - no other consumers can use the queue false, function(AMQPMessage $message) { echo $message->body . "n"; } ); while (count($channel->callbacks)) { $channel->wait(); }
  25. 25. What to expect...
  26. 26. Exchanges: Fanout
  27. 27. Objective: Fanout Exchange test_exchange amq.KfgPZ3PE amq.cK5Cp3FC Consumer Consumer Producer 1 1 2 2 3 3 4 4 5 5
  28. 28. fanout/producer.php use PhpAmqpLibMessageAMQPMessage; $channel->exchange_declare( 'test_exchange', 'fanout', false, false, false); $message = new AMQPMessage('my test message #' . $id); $channel->basic_publish($message, 'test_exchange');
  29. 29. fanout/consumer.php $q = $channel->queue_declare( '', // Lets RabbitMQ pick a name for queue false, false, false, true // Delete this queue ); $queue_name = $q[0]; $channel->exchange_declare( 'test_exchange', 'fanout', false, false, false); $channel->queue_bind($queue_name, 'test_exchange');
  30. 30. What to expect...
  31. 31. A word on Temporary Queues ● Only the “current” flow ● Specific use cases test_exchangeProducer Messages go nowhere
  32. 32. Exchanges: Direct
  33. 33. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer 3 Message Routing Keys 1 = orange 2 = banana 3 = apple 2 3 BK = orange, banana, apple Consumer1 2 3
  34. 34. direct/producer.php $channel->exchange_declare( 'test_direct', 'fanout', false, false, false); $messageContent = 'my test message, key=' . $routingKey; $message = new AMQPMessage($messageContent); $channel->basic_publish($message, 'test_direct', $routingKey);
  35. 35. direct/consumer.php $q = $channel->queue_declare('', false, false, false, true); $queue_name = $q[0]; $channel->exchange_declare( 'test_direct', 'direct', false, false, false); // Bind for each routing key we want (BINDING KEY) $channel->queue_bind($queue_name, 'test_direct', 'apple'); $channel->queue_bind($queue_name, 'test_direct', 'orange'); $channel->queue_bind($queue_name, 'test_direct', 'banana');
  36. 36. What to expect...
  37. 37. Exchanges: Topic
  38. 38. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer 1 2 Message Routing Keys 1 = red.vegetable 2 = green.vegetable 3 = green.fruit 4 = red.meat 5 = green.grass.long 1 2 3 4 5 BK = green.# Consumer2 3 5 BK = *.grass.* / *.*.long Consumer5
  39. 39. Real World Example
  40. 40. Fetch message Logging Sequence ApplicationBrowser Log Server HTTP request JSON via AMQP Error! HTTP response RabbitMQ
  41. 41. Flexibility! ● Temporary Queues ○ e.g. inspect “debug” messages
  42. 42. Flexibility! ● Temporary Queues ● Queues to log to DB
  43. 43. Flexibility! ● Temporary Queues ● Queues to log to DB ● Queues to email “alert/emergency”
  44. 44. Flexibility! ● Temporary Queues ● Queues to log to DB ● Queues to email “alert/emergency” ● Get creative with routing keys ○ RK = app.api.error … BK = #.api.error ○ RK = app.form.debug … BK = #.debug
  45. 45. Problem: SPOF
  46. 46. Solution 1: Clustering
  47. 47. Clustering RabbitMQ Node 1 RabbitMQ Node 3 RabbitMQ Node 2 RabbitMQ Node 4 RabbitMQ Node 5 RabbitMQ Node 6 Load Balance / Floating IP / Low TTL DNS etc.
  48. 48. Clustering ● Everything replicated, except queues
  49. 49. Clustering ● Everything replicated, except queues ● Types: ○ RAM ○ Disk
  50. 50. Clustering ● Everything replicated, except queues ● Types: ○ RAM ○ Disk ● Configuration: ○ CLI (rabbitmqctl) ○ Configuration files
  51. 51. Creating a cluster node1$ rabbitmqctl cluster_status Cluster status of node rabbit@node1 ... [{nodes,[{disc,[rabbit@node1]}]},{running_nodes,[rabbit@node1]}] ...done. node2$ rabbitmqctl cluster_status Cluster status of node rabbit@node2 ... [{nodes,[{disc,[rabbit@node2]}]},{running_nodes,[rabbit@node2]}] ...done. node3$ rabbitmqctl cluster_status Cluster status of node rabbit@node3 ... [{nodes,[{disc,[rabbit@node3]}]},{running_nodes,[rabbit@node3]}] ...done.
  52. 52. node2$ rabbitmqctl join_cluster --ram rabbit@node1 node3$ rabbitmqctl join_cluster rabbit@node2 node3$ rabbitmqctl cluster_status Cluster status of node rabbit@node3 ... [{nodes,[{disc,[rabbit@node3,rabbit@node1]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node1,rabbit@node3]}] ...done. Creating a cluster
  53. 53. node1$ rabbitmqctl stop_app node2$ rabbitmqctl forget_cluster_node rabbit@node1 node1$ rabbitmqctl reset node1$ rabbitmqctl start_app node2$ rabbitmqctl cluster_status Cluster status of node rabbit@node2 ... [{nodes,[{disc,[rabbit@node3]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node3]}] ...done. Managing Nodes ● Stopping/starting nodes ● Removing nodes:
  54. 54. Solution 2: HA
  55. 55. HA + Queue Mirroring RabbitMQ Node 1 RabbitMQ Node 2 Load Balance / Floating IP / Low TTL DNS etc.
  56. 56. https://github.com/asgrim/rmq-slides Have a go yourself!
  57. 57. Questions?
  58. 58. James Titcumb www.jamestitcumb.com www.protected.co.uk www.phphants.co.uk @asgrim Thanks for watching!

×