Scaling applications with RabbitMQ at SunshinePHP

4,646 views

Published on

Do you need to process thousands of images in the background for your web app?

Do you need to share data across multiple applications, probably written in different languages and sitting at different servers?

Your real time data feed is becoming slow because you are polling the database constantly for new data updates?

Do you need to scale information processing during peek times?

What about deploying new features with zero downtime? If any of these problems sound familiar then you probably need to use messaging in your application.

In this talk I will introduce RabbitMQ, a messaging and queue server that can help us tackle those problems. We will learn the benefits of a Queue Server and see how to integrate messaging into our applications. With this talk we hope that the term 'decoupling' gets a new, broader, meaning.

Published in: Technology

Scaling applications with RabbitMQ at SunshinePHP

  1. 1. Scaling Applications with RabbitMQ Alvaro Videla - RabbitMQ
  2. 2. Alvaro Videla • • • • • Developer Advocate at Pivotal / RabbitMQ! Co-Author of RabbitMQ in Action! Creator of the RabbitMQ Simulator! Blogs about RabbitMQ Internals: http://videlalvaro.github.io/internals.html! @old_sound — alvaro@rabbitmq.com — github.com/videlalvaro

  3. 3. About Me Co-authored! ! RabbitMQ in Action! http://bit.ly/rabbitmq
  4. 4. About this Talk • Exploratory Talk • A ‘what could be done’ talk instead of ‘this is how you do it’
  5. 5. What is RabbitMQ
  6. 6. RabbitMQ
  7. 7. RabbitMQ
  8. 8. RabbitMQ • Multi Protocol Messaging Server
  9. 9. RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)
  10. 10. RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot
  11. 11. RabbitMQ • Multi Protocol Messaging Server! • Open Source (MPL)! • Polyglot! • Written in Erlang/OTP
  12. 12. Multi Protocol http://bit.ly/rmq-protocols
  13. 13. Polyglot
  14. 14. Polyglot
  15. 15. Polyglot • Java
  16. 16. Polyglot • Java! • node.js
  17. 17. Polyglot • Java! • node.js! • Erlang
  18. 18. Polyglot • Java! • node.js! • Erlang! • PHP
  19. 19. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby
  20. 20. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net
  21. 21. Polyglot • Java! • node.js! • Erlang! • PHP! • Ruby! • .Net! • Haskell
  22. 22. Polyglot Even COBOL!!!11
  23. 23. Some users of RabbitMQ
  24. 24. Some users of RabbitMQ • Instagram
  25. 25. Some users of RabbitMQ • • Instagram! Indeed.com
  26. 26. Some users of RabbitMQ • • • Instagram! Indeed.com! Telefonica
  27. 27. Some users of RabbitMQ • • • • Instagram! Indeed.com! Telefonica! Mercado Libre
  28. 28. Some users of RabbitMQ • • • • • Instagram! Indeed.com! Telefonica! Mercado Libre! NHS
  29. 29. Some users of RabbitMQ • • • • • • Instagram! Indeed.com! Telefonica! Mercado Libre! NHS! Mozilla
  30. 30. http://www.rabbitmq.com/download.html Unix - Mac - Windows
  31. 31. Messaging with RabbitMQ A demo with the RabbitMQ Simulator https://github.com/RabbitMQSimulator/RabbitMQSimulator
  32. 32. http://tryrabbitmq.com
  33. 33. RabbitMQ Simulator
  34. 34. What you get out of the box
  35. 35. Some RabbitMQ Features
  36. 36. Some RabbitMQ Features • No message loss
  37. 37. Some RabbitMQ Features • No message loss • Persistent Messages
  38. 38. Some RabbitMQ Features • No message loss • Persistent Messages • Publisher Confirms
  39. 39. Some RabbitMQ Features • No message loss • Persistent Messages • Publisher Confirms • Message Acknowledgment
  40. 40. Some RabbitMQ Features • No message loss • Persistent Messages • Publisher Confirms • Message Acknowledgment • Mirrored Queues
  41. 41. Some RabbitMQ Features • No message loss • Persistent Messages • Publisher Confirms • Message Acknowledgment • Mirrored Queues • Message Ordering
  42. 42. Some RabbitMQ Features
  43. 43. Some RabbitMQ Features • Dead Letter Exchanges
  44. 44. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges
  45. 45. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs
  46. 46. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities
  47. 47. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities • Federation
  48. 48. Some RabbitMQ Features • Dead Letter Exchanges • Alternate Exchanges • Message and Queues TTLs • Consumer Priorities • Federation • Shovel
  49. 49. Preventing unbounded buffers with RabbitMQ http://bit.ly/1eCKYK8
  50. 50. RabbitMQ Management
  51. 51. Scenario Batch Processing
  52. 52. Requirements
  53. 53. Requirements • Generate XML
  54. 54. Requirements • Generate XML • Distribution Over a Cluster
  55. 55. Requirements • Generate XML • Distribution Over a Cluster • Elasticity - Add/Remove new workers
  56. 56. Requirements • Generate XML • Distribution Over a Cluster • Elasticity - Add/Remove new workers • No Code Changes
  57. 57. Design
  58. 58. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); ! $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'video-desc-ex'); ! $channel->close(); $conn->close();
  59. 59. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); ! $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'video-desc-ex'); ! $channel->close(); $conn->close();
  60. 60. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); ! $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'video-desc-ex'); ! $channel->close(); $conn->close();
  61. 61. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); ! $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'video-desc-ex'); ! $channel->close(); $conn->close();
  62. 62. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); ! $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'video-desc-ex'); ! $channel->close(); $conn->close();
  63. 63. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); ! $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'video-desc-ex'); ! $channel->close(); $conn->close();
  64. 64. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); ! $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  65. 65. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); ! $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  66. 66. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); ! $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  67. 67. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); ! $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  68. 68. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); ! $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  69. 69. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); ! $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  70. 70. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); ! $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); ! $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  71. 71. Scenario Upload Pictures
  72. 72. Requirements
  73. 73. Requirements • Upload Picture
  74. 74. Requirements • Upload Picture • Reward User
  75. 75. Requirements • Upload Picture • Reward User • Notify User Friends
  76. 76. Requirements • Upload Picture • Reward User • Notify User Friends • Resize Picture
  77. 77. Requirements • Upload Picture • Reward User • Notify User Friends • Resize Picture • No Code Changes
  78. 78. Design
  79. 79. Design
  80. 80. Design
  81. 81. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); ! $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'upload-pictures');
  82. 82. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); ! $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'upload-pictures');
  83. 83. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); ! $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'upload-pictures');
  84. 84. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); ! $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'upload-pictures');
  85. 85. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); ! $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); ! $channel->basic_publish($msg, 'upload-pictures');
  86. 86. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $channel->queue_declare('resize-picture', false, true, false, false); ! $channel->queue_bind('resize-picture', 'upload-pictures'); ! $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  87. 87. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $channel->queue_declare('resize-picture', false, true, false, false); ! $channel->queue_bind('resize-picture', 'upload-pictures'); ! $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  88. 88. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $channel->queue_declare('resize-picture', false, true, false, false); ! $channel->queue_bind('resize-picture', 'upload-pictures'); ! $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  89. 89. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $channel->queue_declare('resize-picture', false, true, false, false); ! $channel->queue_bind('resize-picture', 'upload-pictures'); ! $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  90. 90. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $channel->queue_declare('resize-picture', false, true, false, false); ! $channel->queue_bind('resize-picture', 'upload-pictures'); ! $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  91. 91. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); ! $channel->queue_declare('resize-picture', false, true, false, false); ! $channel->queue_bind('resize-picture', 'upload-pictures'); ! $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); ! while(count($channel->callbacks)) { $channel->wait(); }
  92. 92. Consumer Code $consumer = function($msg){ ! $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  93. 93. Consumer Code $consumer = function($msg){ ! $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  94. 94. Consumer Code $consumer = function($msg){ ! $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  95. 95. Consumer Code $consumer = function($msg){ ! $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  96. 96. Consumer Code $consumer = function($msg){ ! $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  97. 97. Scenario Distributed Logging
  98. 98. Requirements
  99. 99. Requirements • Several Web Servers
  100. 100. Requirements • Several Web Servers • Logic Separated by Module/Action
  101. 101. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels: info, warning and error
  102. 102. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels: info, warning and error • Add Remove log listeners at will
  103. 103. Design
  104. 104. Design
  105. 105. Design
  106. 106. Design
  107. 107. Design
  108. 108. Publisher Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $msg = new AMQPMessage('some log message', array('content_type' => 'text/plain')); ! $channel->basic_publish($msg, 'logs', 'server1.user.profile.info');
  109. 109. Publisher Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $msg = new AMQPMessage('some log message', array('content_type' => 'text/plain')); ! $channel->basic_publish($msg, 'logs', 'server1.user.profile.info');
  110. 110. Publisher Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $msg = new AMQPMessage('some log message', array('content_type' => 'text/plain')); ! $channel->basic_publish($msg, 'logs', 'server1.user.profile.info');
  111. 111. Publisher Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $msg = new AMQPMessage('some log message', array('content_type' => 'text/plain')); ! $channel->basic_publish($msg, 'logs', 'server1.user.profile.info');
  112. 112. Consumer Code Get messages sent by host: ! server1
  113. 113. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $channel->queue_declare('server1-logs', false, true, false, false); ! $channel->queue_bind('server1-logs', 'logs', 'server1.#');
  114. 114. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $channel->queue_declare('server1-logs', false, true, false, false); ! $channel->queue_bind('server1-logs', 'logs', 'server1.#');
  115. 115. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $channel->queue_declare('server1-logs', false, true, false, false); ! $channel->queue_bind('server1-logs', 'logs', 'server1.#');
  116. 116. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $channel->queue_declare('server1-logs', false, true, false, false); ! $channel->queue_bind('server1-logs', 'logs', 'server1.#');
  117. 117. Consumer Code Get all error messages
  118. 118. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $channel->queue_declare('error-logs', false, true, false, false); ! $channel->queue_bind('error-logs', 'logs', '#.error');
  119. 119. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $channel->queue_declare('error-logs', false, true, false, false); ! $channel->queue_bind('error-logs', 'logs', '#.error');
  120. 120. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $channel->queue_declare('error-logs', false, true, false, false); ! $channel->queue_bind('error-logs', 'logs', '#.error');
  121. 121. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); ! $channel->queue_declare('error-logs', false, true, false, false); ! $channel->queue_bind('error-logs', 'logs', '#.error');
  122. 122. Distributed Apps
  123. 123. Distributed Application App App App App
  124. 124. Distributed Application App App App App
  125. 125. Ad-hoc solution
  126. 126. A process that replicates data to the remote server
  127. 127. Possible issues • Remote server is offline • • • Prevent unbounded local buffers Prevent message loss Prevent unnecessary message replication • No need for those messages on remote server • Messages that became stale
  128. 128. Can we do better?
  129. 129. RabbitMQ Federation
  130. 130. RabbitMQ Federation • Supports replication across different administrative domains • Supports mix of Erlang and RabbitMQ versions • Supports Network Partitions • Specificity - not everything has to be federated
  131. 131. RabbitMQ Federation
  132. 132. RabbitMQ Federation
  133. 133. RabbitMQ Federation
  134. 134. RabbitMQ Federation • It’s a RabbitMQ Plugin
  135. 135. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators
  136. 136. RabbitMQ Federation • It’s a RabbitMQ Plugin • Internally uses Queues and Exchanges Decorators • Managed using Parameters and Policies
  137. 137. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation
  138. 138. Enabling the Plugin rabbitmq-plugins enable rabbitmq_federation rabbitmq-plugins enable rabbitmq_federation_management
  139. 139. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}'
  140. 140. Federating an Exchange rabbitmqctl set_parameter federation-upstream my-upstream ‘{“uri":"amqp://server-name","expires":3600000}' ! rabbitmqctl set_policy --apply-to exchanges federate-me "^amq." '{"federation-upstream-set":"all"}'
  141. 141. Federating an Exchange
  142. 142. With RabbitMQ we can
  143. 143. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP
  144. 144. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation
  145. 145. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using with different consumer strategies
  146. 146. With RabbitMQ we can • Ingest data using various protocols: AMQP, MQTT and STOMP • Distribute that data globally using Federation • Scale up using with different consumer strategies • Integrate across many platforms and programming languages
  147. 147. Questions?
  148. 148. Talk and Slides https://joind.in/talk/view/10525
  149. 149. Thanks Alvaro Videla - @old_sound Credits world map: wikipedia.org federation diagrams: rabbitmq.com

×