As your application grows, you soon realise you need to break up your application into smaller chunks that talk to each other. You could just use web services to interact, or you could take a more robust approach and use the message broker RabbitMQ. In this talk, we will take a look at the techniques you can use to vastly enhance inter-application communication, learn about the core concepts of RabbitMQ, cover how you can scale different parts of your application separately, and modernise your development using a message-oriented architecture.

1. Adding 1.21 Gigawatts to
Applications with RabbitMQ
James Titcumb
PHP Oxford June 2016

2. James Titcumb
www.jamestitcumb.com
www.roave.com
www.phphants.co.uk
www.phpsouthcoast.co.uk
@asgrim
Who is this guy?

3. What is a message?
@asgrim

4. What is message
queueing?
@asgrim

5. Separation of Concerns
@asgrim

6. Scaling with Rabbit
RabbitMQApplication
Background processing
@asgrim

7. Scaling with Rabbit
RabbitMQApplication
Background processing
Background processing
@asgrim

8. Scaling with Rabbit
RabbitMQApplication
Background processing
Background processing
Background processing
@asgrim

9. Scaling with Rabbit
RabbitMQApplication
Background processing
Background processing
Background processing
Background processing
@asgrim

10. Scaling with Rabbit
RabbitMQApplication
Background processing
Background processing
Background processing
Background processing
Background processing
@asgrim

11. Real world uses?
@asgrim

12. SOA
@asgrim

13. Why RabbitMQ?
@asgrim

14. https://github.com/asgrim/rmq-slides
Follow along here...
@asgrim

15. (on trusty64 Vagrant, other OSs may vary)
Installing RabbitMQ
@asgrim

16. ● add apt repo
○ deb http://www.rabbitmq.com/debian/ testing main
● add signing key
○ http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
● apt-get update
● apt-get install rabbitmq-server
● rabbitmq-plugins enable rabbitmq_management
● sudo service rabbitmq-server restart
Using Apt
@asgrim

17. http://localhost:15672/
@asgrim

18. http://localhost:15672/
@asgrim

19. http://localhost:15672/
@asgrim

20. http://localhost:15672/
@asgrim

21. http://localhost:15672/
@asgrim

22. Basic Message Queue
@asgrim

23. Objective: Basic Queuing
Producer Consumer
test_queue
1 2 3 4 5
@asgrim

24. composer.json
composer require videlalvaro/php-amqplib
@asgrim

25. Please wait, connecting...
1 use PhpAmqpLibConnectionAMQPConnection;
2
3 $connection = new AMQPConnection(
4 'localhost', 5672, 'guest', 'guest', '/');
5
6 $channel = $connection->channel();
@asgrim

26. basic/producer.php
1 use PhpAmqpLibMessageAMQPMessage;
2
3 $channel->queue_declare(
4 'test_queue', false, true, false, false);
5
6 $message = new AMQPMessage('my test message');
7 $channel->basic_publish($message, '', 'test_queue');
@asgrim

27. basic/consumer.php
1 $channel->basic_consume(
2 'test_queue', '', false, true, false, false,
3 function(AMQPMessage $message) {
4 echo $message->body . "n";
5 });
6
7 while (count($channel->callbacks))
8 $channel->wait();
@asgrim

28. What to expect...
@asgrim

29. Exchanges: Fanout
@asgrim

30. Objective: Fanout Exchange
test_exchange
amq.KfgPZ3PE
amq.cK5Cp3FC
Consumer
Consumer
Producer
1
1
2
2
3
3
4
4
5
5
@asgrim

31. 1 use PhpAmqpLibMessageAMQPMessage;
2
3 $channel->exchange_declare(
4 'test_exchange', 'fanout', false, false, false);
5
6 $message = new AMQPMessage('test msg #' . $id);
7 $channel->basic_publish($message, 'test_exchange');
fanout/producer.php
@asgrim

32. fanout/consumer.php
1 $q = $channel->queue_declare(
2 '', false, false, false, true);
3 $queue_name = $q[0];
4
5 $channel->exchange_declare(
6 'test_exchange', 'fanout', false, false, false);
7 $channel->queue_bind($queue_name, 'test_exchange');
@asgrim

33. What to expect...
@asgrim

34. A word on Temporary Queues
test_exchangeProducer
Messages
go nowhere
@asgrim

35. Exchanges: Direct
@asgrim

36. Objective: Direct Exchange
test_direct
BK = apple
BK = banana, apple
Consumer
Consumer
Producer
BK = orange, banana,
apple
Consumer
@asgrim

37. Objective: Direct Exchange
test_direct
BK = apple
BK = banana, apple
Consumer
Consumer
Producer
MESSAGE
ROUTING KEY
= ORANGE
BK = orange, banana,
apple
Consumer
@asgrim

38. Objective: Direct Exchange
test_direct
BK = apple
BK = banana, apple
Consumer
Consumer
Producer
MESSAGE
ROUTING KEY
= BANANA
BK = orange, banana,
apple
Consumer
@asgrim

39. Objective: Direct Exchange
test_direct
BK = apple
BK = banana, apple
Consumer
Consumer
Producer
MESSAGE
ROUTING KEY
= APPLE
BK = orange, banana,
apple
Consumer
@asgrim

40. direct/producer.php
1 $channel->exchange_declare(
2 'test_direct', 'fanout', false, false, false);
3
4 $messageContent = 'test msg, key=' . $routingKey;
5 $message = new AMQPMessage($messageContent);
6 $channel->basic_publish(
7 $message, 'test_direct', $routingKey);
@asgrim

41. direct/consumer.php
1 $q = $channel->queue_declare('', false, false, false, true);
2 $queue_name = $q[0];
3 $channel->exchange_declare(
4 'test_direct', 'direct', false, false, false);
5
6 // Bind for each routing key we want (BINDING KEY)
7 $channel->queue_bind($queue_name, 'test_direct', 'apple');
8 $channel->queue_bind($queue_name, 'test_direct', 'orange');
9 $channel->queue_bind($queue_name, 'test_direct', 'banana');
@asgrim

42. What to expect...
@asgrim

43. Exchanges: Topic
@asgrim

44. Objective:
Topic Exchange
test_topic
BK = *.vegetable
BK = #
Consumer
Consumer
Producer
BK = green.#
Consumer
BK = *.grass.* / *.*.long
Consumer
@asgrim

45. Objective:
Topic Exchange
test_topic
BK = *.vegetable
BK = #
Consumer
Consumer
Producer
RED.VEGETABLE
BK = green.#
Consumer
BK = *.grass.* / *.*.long
Consumer
@asgrim

46. Objective:
Topic Exchange
test_topic
BK = *.vegetable
BK = #
Consumer
Consumer
Producer
GREEN.VEGETABLE
BK = green.#
Consumer
BK = *.grass.* / *.*.long
Consumer
@asgrim

47. Objective:
Topic Exchange
test_topic
BK = *.vegetable
BK = #
Consumer
Consumer
Producer
GREEN.GRASS.LONG
BK = green.#
Consumer
BK = *.grass.* / *.*.long
Consumer
@asgrim

48. Real World Example
@asgrim

50. Fetch message
Logging Sequence
ApplicationBrowser Log Server
HTTP request
JSON via AMQP
Error!
HTTP response
RabbitMQ
@asgrim

51. Flexibility!
@asgrim

52. Parallel Processing
@asgrim

53. test_exchange
amq.KfgPZ3PE
amq.cK5Cp3FC
Consumer
Producer
1
1
2
2
3
3
4
4
5
5
Consumer
Consumer
Consumer
Consumer
Consumer
Consumer
Consumer
Parallel Processing
@asgrim

54. Message
Acknowledgement
@asgrim

55. 1 $channel->basic_consume(
2 'test_queue', '', false, false, false, false,
3 function(AMQPMessage $message) {
4 echo $message->body . "n";
5 throw new Exception('oh noes!!!');
6
7 $channel = $message->delivery_info['channel'];
8 $tag = $message->delivery_info['delivery_tag'];
9 $channel->basic_ack($tag);
10 });
Message Acknowledgement
@asgrim

56. RPC
@asgrim

57. RPC example
Producer Consumer
test_queue
1
reply_to: amq.gen-Xa2
1’
@asgrim

58. TTL
@asgrim

59. TTL - per queue message
Producer Consumer
test_queue
1 2 3 4 5
}
10s
@asgrim

60. TTL - per message
Producer Consumer
test_queue
1 2 3 4 5
5s 3s 7s 1s 9s@asgrim

61. TTL - queue
Producer
test_queue
}
5s (if no consumers)
@asgrim

62. DLX
@asgrim

63. DLX
test_exchange
amq.KfgPZ3PE
Producer 1
dlx_exchange
dlx_queue
1
@asgrim

64. Scheduling / Delayed messages
@asgrim

65. Priority
@asgrim

66. http://tryrabbitmq.com/
@asgrim

67. Infrastructure
@asgrim

68. Problem: SPOF
@asgrim

69. Solution 1: Clustering
@asgrim

70. Clustering
RabbitMQ
Node 1
RabbitMQ
Node 3
RabbitMQ
Node 2
RabbitMQ
Node 4
RabbitMQ
Node 5
RabbitMQ
Node 6
Load Balance / Floating IP / Low TTL DNS etc.
@asgrim

71. Everything Replicates
(except queues…)
@asgrim

72. Disk / RAM
@asgrim

73. Configuration...
@asgrim

74. /etc/rabbitmq/rabbitmq.config
[
{rabbit, [
{loopback_users, []},
{vm_memory_high_watermark, 0.8}
]}
].
@asgrim

75. /etc/rabbitmq/rabbitmq.config
[{{{[{{[{{}}{][[[{[{{}[[}{[[{}[][}{}}}{}}{{,},]{
[[{rabbit, [{{}[[}{,,{}[][}{[][][{}{{{{}}}}[[}{{
{{}}{loopback_users, []},[][][]{}{}{}<}{[}[][][}
[{{[{vm_memory_high_watermark, 0.8}]]{}{[[[]]{}]
{{]}[{[{{}[[}{]]{}[][,{}[][}{[][][{}.[]}{]][][]}
]...{}[][,]{.}[][}{}[[[{}{][]}{}{}[}{}{}{]{}{}}[
@asgrim

76. node1$ rabbitmqctl cluster_status
Cluster status of node rabbit@node1 ...
[{nodes,[{disc,[rabbit@node1]}]},{running_nodes,[rabbit@node1]}]
...done.
Creating a cluster
@asgrim

77. node2$ rabbitmqctl join_cluster --ram rabbit@node1
node3$ rabbitmqctl join_cluster rabbit@node2
node3$ rabbitmqctl cluster_status
Cluster status of node rabbit@node3 ...
[{nodes,[{disc,[rabbit@node3,rabbit@node1]},{ram,[rabbit@node2]}]},
{running_nodes,[rabbit@node2,rabbit@node1,rabbit@node3]}]
...done.
Creating a cluster
@asgrim

78. Starting/Stopping Nodes
@asgrim

79. node1$ rabbitmqctl stop_app
node2$ rabbitmqctl forget_cluster_node rabbit@node1
node1$ rabbitmqctl reset
node1$ rabbitmqctl start_app
node2$ rabbitmqctl cluster_status
Cluster status of node rabbit@node2 ...
[{nodes,[{disc,[rabbit@node3]},{ram,[rabbit@node2]}]},
{running_nodes,[rabbit@node2,rabbit@node3]}]
...done.
Removing Nodes
@asgrim

80. Solution 2: HA
@asgrim

81. HA + Queue Mirroring
RabbitMQ
Node 1
RabbitMQ
Node 2
Load Balance / Floating IP / Low TTL DNS etc.
@asgrim

82. https://github.com/asgrim/rmq-slides
That link again...
@asgrim

83. Any questions? :)
James Titcumb @asgrim