Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Adding 1.21 Gigawatts to Applications with RabbitMQ (PHP Oxford June Meetup 2016)

336 views

Published on

As your application grows, you soon realise you need to break up your application into smaller chunks that talk to each other. You could just use web services to interact, or you could take a more robust approach and use the message broker RabbitMQ.
In this talk, we will take a look at the techniques you can use to vastly enhance inter-application communication, learn about the core concepts of RabbitMQ, cover how you can scale different parts of your application separately, and modernise your development using a message-oriented architecture.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Adding 1.21 Gigawatts to Applications with RabbitMQ (PHP Oxford June Meetup 2016)

  1. 1. Adding 1.21 Gigawatts to Applications with RabbitMQ James Titcumb PHP Oxford June 2016
  2. 2. James Titcumb www.jamestitcumb.com www.roave.com www.phphants.co.uk www.phpsouthcoast.co.uk @asgrim Who is this guy?
  3. 3. What is a message? @asgrim
  4. 4. What is message queueing? @asgrim
  5. 5. Separation of Concerns @asgrim
  6. 6. Scaling with Rabbit RabbitMQApplication Background processing @asgrim
  7. 7. Scaling with Rabbit RabbitMQApplication Background processing Background processing @asgrim
  8. 8. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing @asgrim
  9. 9. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing @asgrim
  10. 10. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing Background processing @asgrim
  11. 11. Real world uses? @asgrim
  12. 12. SOA @asgrim
  13. 13. Why RabbitMQ? @asgrim
  14. 14. https://github.com/asgrim/rmq-slides Follow along here... @asgrim
  15. 15. (on trusty64 Vagrant, other OSs may vary) Installing RabbitMQ @asgrim
  16. 16. ● add apt repo ○ deb http://www.rabbitmq.com/debian/ testing main ● add signing key ○ http://www.rabbitmq.com/rabbitmq-signing-key-public.asc ● apt-get update ● apt-get install rabbitmq-server ● rabbitmq-plugins enable rabbitmq_management ● sudo service rabbitmq-server restart Using Apt @asgrim
  17. 17. http://localhost:15672/ @asgrim
  18. 18. http://localhost:15672/ @asgrim
  19. 19. http://localhost:15672/ @asgrim
  20. 20. http://localhost:15672/ @asgrim
  21. 21. http://localhost:15672/ @asgrim
  22. 22. Basic Message Queue @asgrim
  23. 23. Objective: Basic Queuing Producer Consumer test_queue 1 2 3 4 5 @asgrim
  24. 24. composer.json composer require videlalvaro/php-amqplib @asgrim
  25. 25. Please wait, connecting... 1 use PhpAmqpLibConnectionAMQPConnection; 2 3 $connection = new AMQPConnection( 4 'localhost', 5672, 'guest', 'guest', '/'); 5 6 $channel = $connection->channel(); @asgrim
  26. 26. basic/producer.php 1 use PhpAmqpLibMessageAMQPMessage; 2 3 $channel->queue_declare( 4 'test_queue', false, true, false, false); 5 6 $message = new AMQPMessage('my test message'); 7 $channel->basic_publish($message, '', 'test_queue'); @asgrim
  27. 27. basic/consumer.php 1 $channel->basic_consume( 2 'test_queue', '', false, true, false, false, 3 function(AMQPMessage $message) { 4 echo $message->body . "n"; 5 }); 6 7 while (count($channel->callbacks)) 8 $channel->wait(); @asgrim
  28. 28. What to expect... @asgrim
  29. 29. Exchanges: Fanout @asgrim
  30. 30. Objective: Fanout Exchange test_exchange amq.KfgPZ3PE amq.cK5Cp3FC Consumer Consumer Producer 1 1 2 2 3 3 4 4 5 5 @asgrim
  31. 31. 1 use PhpAmqpLibMessageAMQPMessage; 2 3 $channel->exchange_declare( 4 'test_exchange', 'fanout', false, false, false); 5 6 $message = new AMQPMessage('test msg #' . $id); 7 $channel->basic_publish($message, 'test_exchange'); fanout/producer.php @asgrim
  32. 32. fanout/consumer.php 1 $q = $channel->queue_declare( 2 '', false, false, false, true); 3 $queue_name = $q[0]; 4 5 $channel->exchange_declare( 6 'test_exchange', 'fanout', false, false, false); 7 $channel->queue_bind($queue_name, 'test_exchange'); @asgrim
  33. 33. What to expect... @asgrim
  34. 34. A word on Temporary Queues test_exchangeProducer Messages go nowhere @asgrim
  35. 35. Exchanges: Direct @asgrim
  36. 36. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer BK = orange, banana, apple Consumer @asgrim
  37. 37. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer MESSAGE ROUTING KEY = ORANGE BK = orange, banana, apple Consumer @asgrim
  38. 38. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer MESSAGE ROUTING KEY = BANANA BK = orange, banana, apple Consumer @asgrim
  39. 39. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer MESSAGE ROUTING KEY = APPLE BK = orange, banana, apple Consumer @asgrim
  40. 40. direct/producer.php 1 $channel->exchange_declare( 2 'test_direct', 'fanout', false, false, false); 3 4 $messageContent = 'test msg, key=' . $routingKey; 5 $message = new AMQPMessage($messageContent); 6 $channel->basic_publish( 7 $message, 'test_direct', $routingKey); @asgrim
  41. 41. direct/consumer.php 1 $q = $channel->queue_declare('', false, false, false, true); 2 $queue_name = $q[0]; 3 $channel->exchange_declare( 4 'test_direct', 'direct', false, false, false); 5 6 // Bind for each routing key we want (BINDING KEY) 7 $channel->queue_bind($queue_name, 'test_direct', 'apple'); 8 $channel->queue_bind($queue_name, 'test_direct', 'orange'); 9 $channel->queue_bind($queue_name, 'test_direct', 'banana'); @asgrim
  42. 42. What to expect... @asgrim
  43. 43. Exchanges: Topic @asgrim
  44. 44. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer BK = green.# Consumer BK = *.grass.* / *.*.long Consumer @asgrim
  45. 45. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer RED.VEGETABLE BK = green.# Consumer BK = *.grass.* / *.*.long Consumer @asgrim
  46. 46. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer GREEN.VEGETABLE BK = green.# Consumer BK = *.grass.* / *.*.long Consumer @asgrim
  47. 47. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer GREEN.GRASS.LONG BK = green.# Consumer BK = *.grass.* / *.*.long Consumer @asgrim
  48. 48. Real World Example @asgrim
  49. 49. Fetch message Logging Sequence ApplicationBrowser Log Server HTTP request JSON via AMQP Error! HTTP response RabbitMQ @asgrim
  50. 50. Flexibility! @asgrim
  51. 51. Parallel Processing @asgrim
  52. 52. test_exchange amq.KfgPZ3PE amq.cK5Cp3FC Consumer Producer 1 1 2 2 3 3 4 4 5 5 Consumer Consumer Consumer Consumer Consumer Consumer Consumer Parallel Processing @asgrim
  53. 53. Message Acknowledgement @asgrim
  54. 54. 1 $channel->basic_consume( 2 'test_queue', '', false, false, false, false, 3 function(AMQPMessage $message) { 4 echo $message->body . "n"; 5 throw new Exception('oh noes!!!'); 6 7 $channel = $message->delivery_info['channel']; 8 $tag = $message->delivery_info['delivery_tag']; 9 $channel->basic_ack($tag); 10 }); Message Acknowledgement @asgrim
  55. 55. RPC @asgrim
  56. 56. RPC example Producer Consumer test_queue 1 reply_to: amq.gen-Xa2 1’ @asgrim
  57. 57. TTL @asgrim
  58. 58. TTL - per queue message Producer Consumer test_queue 1 2 3 4 5 } 10s @asgrim
  59. 59. TTL - per message Producer Consumer test_queue 1 2 3 4 5 5s 3s 7s 1s 9s@asgrim
  60. 60. TTL - queue Producer test_queue } 5s (if no consumers) @asgrim
  61. 61. DLX @asgrim
  62. 62. DLX test_exchange amq.KfgPZ3PE Producer 1 dlx_exchange dlx_queue 1 @asgrim
  63. 63. Scheduling / Delayed messages @asgrim
  64. 64. Priority @asgrim
  65. 65. http://tryrabbitmq.com/ @asgrim
  66. 66. Infrastructure @asgrim
  67. 67. Problem: SPOF @asgrim
  68. 68. Solution 1: Clustering @asgrim
  69. 69. Clustering RabbitMQ Node 1 RabbitMQ Node 3 RabbitMQ Node 2 RabbitMQ Node 4 RabbitMQ Node 5 RabbitMQ Node 6 Load Balance / Floating IP / Low TTL DNS etc. @asgrim
  70. 70. Everything Replicates (except queues…) @asgrim
  71. 71. Disk / RAM @asgrim
  72. 72. Configuration... @asgrim
  73. 73. /etc/rabbitmq/rabbitmq.config [ {rabbit, [ {loopback_users, []}, {vm_memory_high_watermark, 0.8} ]} ]. @asgrim
  74. 74. /etc/rabbitmq/rabbitmq.config [{{{[{{[{{}}{][[[{[{{}[[}{[[{}[][}{}}}{}}{{,},]{ [[{rabbit, [{{}[[}{,,{}[][}{[][][{}{{{{}}}}[[}{{ {{}}{loopback_users, []},[][][]{}{}{}<}{[}[][][} [{{[{vm_memory_high_watermark, 0.8}]]{}{[[[]]{}] {{]}[{[{{}[[}{]]{}[][,{}[][}{[][][{}.[]}{]][][]} ]...{}[][,]{.}[][}{}[[[{}{][]}{}{}[}{}{}{]{}{}}[ @asgrim
  75. 75. node1$ rabbitmqctl cluster_status Cluster status of node rabbit@node1 ... [{nodes,[{disc,[rabbit@node1]}]},{running_nodes,[rabbit@node1]}] ...done. Creating a cluster @asgrim
  76. 76. node2$ rabbitmqctl join_cluster --ram rabbit@node1 node3$ rabbitmqctl join_cluster rabbit@node2 node3$ rabbitmqctl cluster_status Cluster status of node rabbit@node3 ... [{nodes,[{disc,[rabbit@node3,rabbit@node1]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node1,rabbit@node3]}] ...done. Creating a cluster @asgrim
  77. 77. Starting/Stopping Nodes @asgrim
  78. 78. node1$ rabbitmqctl stop_app node2$ rabbitmqctl forget_cluster_node rabbit@node1 node1$ rabbitmqctl reset node1$ rabbitmqctl start_app node2$ rabbitmqctl cluster_status Cluster status of node rabbit@node2 ... [{nodes,[{disc,[rabbit@node3]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node3]}] ...done. Removing Nodes @asgrim
  79. 79. Solution 2: HA @asgrim
  80. 80. HA + Queue Mirroring RabbitMQ Node 1 RabbitMQ Node 2 Load Balance / Floating IP / Low TTL DNS etc. @asgrim
  81. 81. https://github.com/asgrim/rmq-slides That link again... @asgrim
  82. 82. Any questions? :) James Titcumb @asgrim

×