Adding 1.21 Gigawatts to Applications with RabbitMQ (PHP Oxford June Meetup 2016)

1. Adding 1.21 Gigawatts to Applications with RabbitMQ James Titcumb PHP Oxford June 2016

2. James Titcumb www.jamestitcumb.com www.roave.com www.phphants.co.uk www.phpsouthcoast.co.uk @asgrim Who is this guy?

3. What is a message? @asgrim

4. What is message queueing? @asgrim

5. Separation of Concerns @asgrim

6. Scaling with Rabbit RabbitMQApplication Background processing @asgrim

7. Scaling with Rabbit RabbitMQApplication Background processing Background processing @asgrim

8. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing @asgrim

9. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing @asgrim

10. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing Background processing @asgrim

11. Real world uses? @asgrim

12. SOA @asgrim

13. Why RabbitMQ? @asgrim

14. https://github.com/asgrim/rmq-slides Follow along here... @asgrim

15. (on trusty64 Vagrant, other OSs may vary) Installing RabbitMQ @asgrim

16. ● add apt repo ○ deb http://www.rabbitmq.com/debian/ testing main ● add signing key ○ http://www.rabbitmq.com/rabbitmq-signing-key-public.asc ● apt-get update ● apt-get install rabbitmq-server ● rabbitmq-plugins enable rabbitmq_management ● sudo service rabbitmq-server restart Using Apt @asgrim

17. http://localhost:15672/ @asgrim

22. Basic Message Queue @asgrim

23. Objective: Basic Queuing Producer Consumer test_queue 1 2 3 4 5 @asgrim

24. composer.json composer require videlalvaro/php-amqplib @asgrim

25. Please wait, connecting... 1 use PhpAmqpLibConnectionAMQPConnection; 2 3 $connection = new AMQPConnection( 4 'localhost', 5672, 'guest', 'guest', '/'); 5 6 $channel = $connection->channel(); @asgrim

26. basic/producer.php 1 use PhpAmqpLibMessageAMQPMessage; 2 3 $channel->queue_declare( 4 'test_queue', false, true, false, false); 5 6 $message = new AMQPMessage('my test message'); 7 $channel->basic_publish($message, '', 'test_queue'); @asgrim

27. basic/consumer.php 1 $channel->basic_consume( 2 'test_queue', '', false, true, false, false, 3 function(AMQPMessage $message) { 4 echo $message->body . "n"; 5 }); 6 7 while (count($channel->callbacks)) 8 $channel->wait(); @asgrim

28. What to expect... @asgrim

29. Exchanges: Fanout @asgrim

30. Objective: Fanout Exchange test_exchange amq.KfgPZ3PE amq.cK5Cp3FC Consumer Consumer Producer 1 1 2 2 3 3 4 4 5 5 @asgrim

31. 1 use PhpAmqpLibMessageAMQPMessage; 2 3 $channel->exchange_declare( 4 'test_exchange', 'fanout', false, false, false); 5 6 $message = new AMQPMessage('test msg #' . $id); 7 $channel->basic_publish($message, 'test_exchange'); fanout/producer.php @asgrim

32. fanout/consumer.php 1 $q = $channel->queue_declare( 2 '', false, false, false, true); 3 $queue_name = $q[0]; 4 5 $channel->exchange_declare( 6 'test_exchange', 'fanout', false, false, false); 7 $channel->queue_bind($queue_name, 'test_exchange'); @asgrim

34. A word on Temporary Queues test_exchangeProducer Messages go nowhere @asgrim

35. Exchanges: Direct @asgrim

36. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer BK = orange, banana, apple Consumer @asgrim

37. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer MESSAGE ROUTING KEY = ORANGE BK = orange, banana, apple Consumer @asgrim

38. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer MESSAGE ROUTING KEY = BANANA BK = orange, banana, apple Consumer @asgrim

39. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer MESSAGE ROUTING KEY = APPLE BK = orange, banana, apple Consumer @asgrim

40. direct/producer.php 1 $channel->exchange_declare( 2 'test_direct', 'fanout', false, false, false); 3 4 $messageContent = 'test msg, key=' . $routingKey; 5 $message = new AMQPMessage($messageContent); 6 $channel->basic_publish( 7 $message, 'test_direct', $routingKey); @asgrim

41. direct/consumer.php 1 $q = $channel->queue_declare('', false, false, false, true); 2 $queue_name = $q[0]; 3 $channel->exchange_declare( 4 'test_direct', 'direct', false, false, false); 5 6 // Bind for each routing key we want (BINDING KEY) 7 $channel->queue_bind($queue_name, 'test_direct', 'apple'); 8 $channel->queue_bind($queue_name, 'test_direct', 'orange'); 9 $channel->queue_bind($queue_name, 'test_direct', 'banana'); @asgrim

43. Exchanges: Topic @asgrim

44. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer BK = green.# Consumer BK = *.grass.* / *.*.long Consumer @asgrim

45. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer RED.VEGETABLE BK = green.# Consumer BK = *.grass.* / *.*.long Consumer @asgrim

46. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer GREEN.VEGETABLE BK = green.# Consumer BK = *.grass.* / *.*.long Consumer @asgrim

47. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer GREEN.GRASS.LONG BK = green.# Consumer BK = *.grass.* / *.*.long Consumer @asgrim

48. Real World Example @asgrim

49. Fetch message Logging Sequence ApplicationBrowser Log Server HTTP request JSON via AMQP Error! HTTP response RabbitMQ @asgrim

50. Flexibility! @asgrim

51. Parallel Processing @asgrim

52. test_exchange amq.KfgPZ3PE amq.cK5Cp3FC Consumer Producer 1 1 2 2 3 3 4 4 5 5 Consumer Consumer Consumer Consumer Consumer Consumer Consumer Parallel Processing @asgrim

53. Message Acknowledgement @asgrim

54. 1 $channel->basic_consume( 2 'test_queue', '', false, false, false, false, 3 function(AMQPMessage $message) { 4 echo $message->body . "n"; 5 throw new Exception('oh noes!!!'); 6 7 $channel = $message->delivery_info['channel']; 8 $tag = $message->delivery_info['delivery_tag']; 9 $channel->basic_ack($tag); 10 }); Message Acknowledgement @asgrim

55. RPC @asgrim

56. RPC example Producer Consumer test_queue 1 reply_to: amq.gen-Xa2 1’ @asgrim

57. TTL @asgrim

58. TTL - per queue message Producer Consumer test_queue 1 2 3 4 5 } 10s @asgrim

59. TTL - per message Producer Consumer test_queue 1 2 3 4 5 5s 3s 7s 1s 9s@asgrim

60. TTL - queue Producer test_queue } 5s (if no consumers) @asgrim

61. DLX @asgrim

62. DLX test_exchange amq.KfgPZ3PE Producer 1 dlx_exchange dlx_queue 1 @asgrim

63. Scheduling / Delayed messages @asgrim

64. Priority @asgrim

65. http://tryrabbitmq.com/ @asgrim

66. Infrastructure @asgrim

67. Problem: SPOF @asgrim

68. Solution 1: Clustering @asgrim

69. Clustering RabbitMQ Node 1 RabbitMQ Node 3 RabbitMQ Node 2 RabbitMQ Node 4 RabbitMQ Node 5 RabbitMQ Node 6 Load Balance / Floating IP / Low TTL DNS etc. @asgrim

70. Everything Replicates (except queues…) @asgrim

71. Disk / RAM @asgrim

72. Configuration... @asgrim

73. /etc/rabbitmq/rabbitmq.config [ {rabbit, [ {loopback_users, []}, {vm_memory_high_watermark, 0.8} ]} ]. @asgrim

74. /etc/rabbitmq/rabbitmq.config [{{{[{{[{{}}{][[[{[{{}[[}{[[{}[][}{}}}{}}{{,},]{ [[{rabbit, [{{}[[}{,,{}[][}{[][][{}{{{{}}}}[[}{{ {{}}{loopback_users, []},[][][]{}{}{}<}{[}[][][} [{{[{vm_memory_high_watermark, 0.8}]]{}{[[[]]{}] {{]}[{[{{}[[}{]]{}[][,{}[][}{[][][{}.[]}{]][][]} ]...{}[][,]{.}[][}{}[[[{}{][]}{}{}[}{}{}{]{}{}}[ @asgrim

75. node1$ rabbitmqctl cluster_status Cluster status of node rabbit@node1 ... [{nodes,[{disc,[rabbit@node1]}]},{running_nodes,[rabbit@node1]}] ...done. Creating a cluster @asgrim

76. node2$ rabbitmqctl join_cluster --ram rabbit@node1 node3$ rabbitmqctl join_cluster rabbit@node2 node3$ rabbitmqctl cluster_status Cluster status of node rabbit@node3 ... [{nodes,[{disc,[rabbit@node3,rabbit@node1]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node1,rabbit@node3]}] ...done. Creating a cluster @asgrim

77. Starting/Stopping Nodes @asgrim

78. node1$ rabbitmqctl stop_app node2$ rabbitmqctl forget_cluster_node rabbit@node1 node1$ rabbitmqctl reset node1$ rabbitmqctl start_app node2$ rabbitmqctl cluster_status Cluster status of node rabbit@node2 ... [{nodes,[{disc,[rabbit@node3]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node3]}] ...done. Removing Nodes @asgrim

79. Solution 2: HA @asgrim

80. HA + Queue Mirroring RabbitMQ Node 1 RabbitMQ Node 2 Load Balance / Floating IP / Low TTL DNS etc. @asgrim

81. https://github.com/asgrim/rmq-slides That link again... @asgrim

82. Any questions? :) James Titcumb @asgrim