Scaling Web Apps With RabbitMQ - Erlang Factory Lite

7,687 views

Published on

Talk given at the Erlang Factory Lite - 2010 in Los Angeles.

Published in: Technology

Scaling Web Apps With RabbitMQ - Erlang Factory Lite

  1. 1. Scaling Web Apps With RabbitMQ ÁlvaroVidela | The NetCircle Erlang Factory Lite 2010
  2. 2. Who?
  3. 3. About Me • Development Manager at TheNetCircle.com • Writing “RabbitMQ in Action” for Manning • Blog: http://videlalvaro.github.com/ • Twitter: @old_sound
  4. 4. Why Do I need RabbitMQ?
  5. 5. The User
  6. 6. I don’t want to wait till your app resizes my image!
  7. 7. The Product Owner
  8. 8. Can we also notify the user friends when she uploads a new image?
  9. 9. Can we also notify the user friends when she uploads a new image? I forgot to mention we need it for tomorrow…
  10. 10. The Sysadmin
  11. 11. Dumb!You’re delivering full size images! The bandwidth bill has tripled!
  12. 12. Dumb!You’re delivering full size images! The bandwidth bill has tripled! We need this fixed for yesterday!
  13. 13. The Developer in the other team
  14. 14. I need to call your PHP stuff but from Python
  15. 15. I need to call your PHP stuff but from Python And also Java starting next week
  16. 16. You
  17. 17. FML!
  18. 18. Is there a solution?
  19. 19. RabbitMQ & AMQP
  20. 20. AMQP
  21. 21. AMQP • Advanced Message Queuing Protocol • Suits Interoperability • Completely Open Protocol • Binary Protocol • AMQP Model • AMQP Wire Format
  22. 22. AMQP Model • Exchanges • Message Queues • Bindings • Rules for binding them
  23. 23. AMQP Wire Protocol • Functional Layer • Transport Layer
  24. 24. Message Flow http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.html
  25. 25. Exchange Types • Fanout • Direct • Topic
  26. 26. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Fanout_Exchange.html
  27. 27. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Direct_Exchange.html
  28. 28. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Topic_Exchange.html
  29. 29. Usage Scenarios
  30. 30. Usage Scenarios • Batch Processing
  31. 31. Usage Scenarios • Batch Processing • Image Uploading
  32. 32. Usage Scenarios • Batch Processing • Image Uploading • Distributed Logging
  33. 33. Scenario Batch Processing
  34. 34. Requirements
  35. 35. Requirements • Generate XML
  36. 36. Requirements • Generate XML • Distribution Over a Cluster
  37. 37. Requirements • Generate XML • Distribution Over a Cluster • Elasticity - Add/Remove new workers
  38. 38. Requirements • Generate XML • Distribution Over a Cluster • Elasticity - Add/Remove new workers • No Code Changes
  39. 39. Design
  40. 40. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'video-desc-ex'); $channel->close(); $conn->close();
  41. 41. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'video-desc-ex'); $channel->close(); $conn->close();
  42. 42. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'video-desc-ex'); $channel->close(); $conn->close();
  43. 43. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'video-desc-ex'); $channel->close(); $conn->close();
  44. 44. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'video-desc-ex'); $channel->close(); $conn->close();
  45. 45. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'video-desc-ex'); $channel->close(); $conn->close();
  46. 46. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  47. 47. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  48. 48. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  49. 49. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  50. 50. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  51. 51. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  52. 52. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  53. 53. Scenario Upload Pictures
  54. 54. Requirements
  55. 55. Requirements • Upload Picture
  56. 56. Requirements • Upload Picture • Reward User
  57. 57. Requirements • Upload Picture • Reward User • Notify User Friends
  58. 58. Requirements • Upload Picture • Reward User • Notify User Friends • Resize Picture
  59. 59. Requirements • Upload Picture • Reward User • Notify User Friends • Resize Picture • No Code Changes
  60. 60. Design
  61. 61. Design
  62. 62. Design
  63. 63. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'upload-pictures');
  64. 64. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'upload-pictures');
  65. 65. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'upload-pictures');
  66. 66. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'upload-pictures');
  67. 67. Publisher Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'upload-pictures');
  68. 68. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $channel->queue_declare('resize-picture', false, true, false, false); $channel->queue_bind('resize-picture', 'upload-pictures'); $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  69. 69. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $channel->queue_declare('resize-picture', false, true, false, false); $channel->queue_bind('resize-picture', 'upload-pictures'); $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  70. 70. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $channel->queue_declare('resize-picture', false, true, false, false); $channel->queue_bind('resize-picture', 'upload-pictures'); $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  71. 71. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $channel->queue_declare('resize-picture', false, true, false, false); $channel->queue_bind('resize-picture', 'upload-pictures'); $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  72. 72. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $channel->queue_declare('resize-picture', false, true, false, false); $channel->queue_bind('resize-picture', 'upload-pictures'); $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  73. 73. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $channel->queue_declare('resize-picture', false, true, false, false); $channel->queue_bind('resize-picture', 'upload-pictures'); $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }
  74. 74. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  75. 75. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  76. 76. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  77. 77. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  78. 78. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); };
  79. 79. Scenario Distributed Logging
  80. 80. Requirements
  81. 81. Requirements • Several Web Servers
  82. 82. Requirements • Several Web Servers • Logic Separated by Module/Action
  83. 83. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels:
  84. 84. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels: • Info,Warning, Error
  85. 85. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels: • Info,Warning, Error • Add/Remove log listeners at will
  86. 86. Design
  87. 87. Design
  88. 88. Design
  89. 89. Design
  90. 90. Design
  91. 91. Publisher Code $channel->exchange_declare('logs', 'topic', false, true, false); $msg = new AMQPMessage('some log message', array('content_type' => 'text/plain')); $channel->basic_publish($msg, 'logs', 'server1.user.profile.info');
  92. 92. Publisher Code $channel->exchange_declare('logs', 'topic', false, true, false); $msg = new AMQPMessage('some log message', array('content_type' => 'text/plain')); $channel->basic_publish($msg, 'logs', server1.user.profile.info');
  93. 93. Publisher Code $channel->exchange_declare('logs', 'topic', false, true, false); $msg = new AMQPMessage('some log message', array('content_type' => 'text/plain')); $channel->basic_publish($msg, 'logs', server1.user.profile.info');
  94. 94. Publisher Code $channel->exchange_declare('logs', 'topic', false, true, false); $msg = new AMQPMessage('some log message', array('content_type' => 'text/plain')); $channel->basic_publish($msg, 'logs', server1.user.profile.info');
  95. 95. Consumer Code Get messages sent by host: server1
  96. 96. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('server1-logs', false, true, false, false); $channel->queue_bind('server1-logs', 'logs', 'server1.#');
  97. 97. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('server1-logs', false, true, false, false); $channel->queue_bind('server1-logs', 'logs', 'server1.#');
  98. 98. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('server1-logs', false, true, false, false); $channel->queue_bind('server1-logs', 'logs', 'server1.#');
  99. 99. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('server1-logs', false, true, false, false); $channel->queue_bind('server1-logs', 'logs', 'server1.#');
  100. 100. Consumer Code Get all error messages
  101. 101. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('error-logs', false, true, false, false); $channel->queue_bind('error-logs', 'logs', '#.error');
  102. 102. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('error-logs', false, true, false, false); $channel->queue_bind('error-logs', 'logs', '#.error');
  103. 103. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('error-logs', false, true, false, false); $channel->queue_bind('error-logs', 'logs', '#.error');
  104. 104. Consumer Code $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('error-logs', false, true, false, false); $channel->queue_bind('error-logs', 'logs', '#.error');
  105. 105. Why RabbitMQ?
  106. 106. RabbitMQ • Enterprise Messaging System • Open Source MPL • Written in Erlang/OTP • Commercial Support
  107. 107. Features • Reliable and High Scalable • Easy To install • Easy To Cluster • Runs on:Windows, Solaris, Linux, OSX • AMQP 0.8 - 0.9.1
  108. 108. Client Libraries • Java • .NET/C# • Erlang • Ruby, Python, PHP, Perl,AS3, Lisp, Scala, Clojure, Haskell
  109. 109. Docs/Support • http://www.rabbitmq.com/documentation.html • http://dev.rabbitmq.com/wiki/ • #rabbitmq at irc.freenode.net • http://www.rabbitmq.com/email-archive.html
  110. 110. One Setup for HA
  111. 111. Conclusion
  112. 112. Conclusion • Flexibility
  113. 113. Conclusion • Flexibility • Scalability
  114. 114. Conclusion • Flexibility • Scalability • Interoperability
  115. 115. Conclusion • Flexibility • Scalability • Interoperability • Reduce Ops
  116. 116. Questions?
  117. 117. Thanks! Álvaro Videla http://twitter.com/old_sound http://github.com/videlalvaro http://github.com/tnc http://www.slideshare.net/old_sound

×