Your SlideShare is downloading. ×
Scaling websites with RabbitMQ   A(rlvaro Videla)
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Scaling websites with RabbitMQ A(rlvaro Videla)

1,505
views

Published on

Published in: Technology

1 Comment
4 Likes
Statistics
Notes
No Downloads
Views
Total Views
1,505
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
50
Comments
1
Likes
4
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Scaling Web Apps With RabbitMQ Álvaro Videla | Liip AG HighLoad ++ 2011Thursday, September 29, 2011
  • 2. Who?Thursday, September 29, 2011
  • 3. About Me • Software Development at Liip AG • Writing “RabbitMQ in Action” for Manning • Blog: http://videlalvaro.github.com/ • Twitter: @old_soundThursday, September 29, 2011
  • 4. Why Do I need RabbitMQ?Thursday, September 29, 2011
  • 5. The UserThursday, September 29, 2011
  • 6. I don’t want to wait till your app resizes my image!Thursday, September 29, 2011
  • 7. The Product OwnerThursday, September 29, 2011
  • 8. Can we also notify the user friends when she uploads a new image?Thursday, September 29, 2011
  • 9. Can we also notify the user friends when she uploads a new image? I forgot to mention we need it for tomorrow…Thursday, September 29, 2011
  • 10. The SysadminThursday, September 29, 2011
  • 11. Dumb! You’re delivering full size images! The bandwidth bill has tripled!Thursday, September 29, 2011
  • 12. Dumb! You’re delivering full size images! The bandwidth bill has tripled! We need this fixed for yesterday!Thursday, September 29, 2011
  • 13. The Developer in the other teamThursday, September 29, 2011
  • 14. I need to call your PHP stuff but from PythonThursday, September 29, 2011
  • 15. I need to call your PHP stuff but from Python And also Java starting next weekThursday, September 29, 2011
  • 16. YouThursday, September 29, 2011
  • 17. FML!Thursday, September 29, 2011
  • 18. Is there a solution?Thursday, September 29, 2011
  • 19. RabbitMQ & AMQPThursday, September 29, 2011
  • 20. AMQPThursday, September 29, 2011
  • 21. AMQP • Advanced Message Queuing Protocol • Suits Interoperability • Completely Open Protocol • Binary Protocol • AMQP Model • AMQP Wire FormatThursday, September 29, 2011
  • 22. AMQP Model • Exchanges • Message Queues • Bindings • Rules for binding themThursday, September 29, 2011
  • 23. AMQP Wire Protocol • Functional Layer • Transport LayerThursday, September 29, 2011
  • 24. Message Flow http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.htmlThursday, September 29, 2011
  • 25. Exchange Types • Fanout • Direct • TopicThursday, September 29, 2011
  • 26. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Fanout_Exchange.htmlThursday, September 29, 2011
  • 27. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Direct_Exchange.htmlThursday, September 29, 2011
  • 28. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Topic_Exchange.htmlThursday, September 29, 2011
  • 29. Usage ScenariosThursday, September 29, 2011
  • 30. Usage Scenarios • Batch ProcessingThursday, September 29, 2011
  • 31. Usage Scenarios • Batch Processing • Image UploadingThursday, September 29, 2011
  • 32. Usage Scenarios • Batch Processing • Image Uploading • Distributed LoggingThursday, September 29, 2011
  • 33. Scenario Batch ProcessingThursday, September 29, 2011
  • 34. RequirementsThursday, September 29, 2011
  • 35. Requirements • Generate XMLThursday, September 29, 2011
  • 36. Requirements • Generate XML • Distribution Over a ClusterThursday, September 29, 2011
  • 37. Requirements • Generate XML • Distribution Over a Cluster • Elasticity - Add/Remove new workersThursday, September 29, 2011
  • 38. Requirements • Generate XML • Distribution Over a Cluster • Elasticity - Add/Remove new workers • No Code ChangesThursday, September 29, 2011
  • 39. DesignThursday, September 29, 2011
  • 40. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $msg = new AMQPMessage($video_info, array(content_type => text/plain, delivery_mode => 2)); $channel->basic_publish($msg, video-desc-ex); $channel->close(); $conn->close();Thursday, September 29, 2011
  • 41. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $msg = new AMQPMessage($video_info, array(content_type => text/plain, delivery_mode => 2)); $channel->basic_publish($msg, video-desc-ex); $channel->close(); $conn->close();Thursday, September 29, 2011
  • 42. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $msg = new AMQPMessage($video_info, array(content_type => text/plain, delivery_mode => 2)); $channel->basic_publish($msg, video-desc-ex); $channel->close(); $conn->close();Thursday, September 29, 2011
  • 43. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $msg = new AMQPMessage($video_info, array(content_type => text/plain, delivery_mode => 2)); $channel->basic_publish($msg, video-desc-ex); $channel->close(); $conn->close();Thursday, September 29, 2011
  • 44. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $msg = new AMQPMessage($video_info, array(content_type => text/plain, delivery_mode => 2)); $channel->basic_publish($msg, video-desc-ex); $channel->close(); $conn->close();Thursday, September 29, 2011
  • 45. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $msg = new AMQPMessage($video_info, array(content_type => text/plain, delivery_mode => 2)); $channel->basic_publish($msg, video-desc-ex); $channel->close(); $conn->close();Thursday, September 29, 2011
  • 46. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $channel->queue_declare(video-desc-queue, false, true, false, false); $channel->queue_bind(video-desc-queue, video-desc-ex); $channel->basic_consume(video-desc-queue, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 47. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $channel->queue_declare(video-desc-queue, false, true, false, false); $channel->queue_bind(video-desc-queue, video-desc-ex); $channel->basic_consume(video-desc-queue, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 48. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $channel->queue_declare(video-desc-queue, false, true, false, false); $channel->queue_bind(video-desc-queue, video-desc-ex); $channel->basic_consume(video-desc-queue, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 49. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $channel->queue_declare(video-desc-queue, false, true, false, false); $channel->queue_bind(video-desc-queue, video-desc-ex); $channel->basic_consume(video-desc-queue, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 50. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $channel->queue_declare(video-desc-queue, false, true, false, false); $channel->queue_bind(video-desc-queue, video-desc-ex); $channel->basic_consume(video-desc-queue, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 51. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $channel->queue_declare(video-desc-queue, false, true, false, false); $channel->queue_bind(video-desc-queue, video-desc-ex); $channel->basic_consume(video-desc-queue, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 52. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare(video-desc-ex, direct, false, true, false); $channel->queue_declare(video-desc-queue, false, true, false, false); $channel->queue_bind(video-desc-queue, video-desc-ex); $channel->basic_consume(video-desc-queue, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 53. Scenario Upload PicturesThursday, September 29, 2011
  • 54. RequirementsThursday, September 29, 2011
  • 55. Requirements • Upload PictureThursday, September 29, 2011
  • 56. Requirements • Upload Picture • Reward UserThursday, September 29, 2011
  • 57. Requirements • Upload Picture • Reward User • Notify User FriendsThursday, September 29, 2011
  • 58. Requirements • Upload Picture • Reward User • Notify User Friends • Resize PictureThursday, September 29, 2011
  • 59. Requirements • Upload Picture • Reward User • Notify User Friends • Resize Picture • No Code ChangesThursday, September 29, 2011
  • 60. DesignThursday, September 29, 2011
  • 61. DesignThursday, September 29, 2011
  • 62. DesignThursday, September 29, 2011
  • 63. Publisher Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $metadata = json_encode(array( image_id => $image_id, user_id => $user_id, ‘image_path => $image_path)); $msg = new AMQPMessage($metadata, array(content_type => application/json, delivery_mode => 2)); $channel->basic_publish($msg, upload-pictures);Thursday, September 29, 2011
  • 64. Publisher Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $metadata = json_encode(array( image_id => $image_id, user_id => $user_id, ‘image_path => $image_path)); $msg = new AMQPMessage($metadata, array(content_type => application/json, delivery_mode => 2)); $channel->basic_publish($msg, upload-pictures);Thursday, September 29, 2011
  • 65. Publisher Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $metadata = json_encode(array( image_id => $image_id, user_id => $user_id, ‘image_path => $image_path)); $msg = new AMQPMessage($metadata, array(content_type => application/json, delivery_mode => 2)); $channel->basic_publish($msg, upload-pictures);Thursday, September 29, 2011
  • 66. Publisher Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $metadata = json_encode(array( image_id => $image_id, user_id => $user_id, ‘image_path => $image_path)); $msg = new AMQPMessage($metadata, array(content_type => application/json, delivery_mode => 2)); $channel->basic_publish($msg, upload-pictures);Thursday, September 29, 2011
  • 67. Publisher Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $metadata = json_encode(array( image_id => $image_id, user_id => $user_id, ‘image_path => $image_path)); $msg = new AMQPMessage($metadata, array(content_type => application/json, delivery_mode => 2)); $channel->basic_publish($msg, upload-pictures);Thursday, September 29, 2011
  • 68. Consumer Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $channel->queue_declare(resize-picture, false, true, false, false); $channel->queue_bind(resize-picture, upload-pictures); $channel->basic_consume(resize-picture, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 69. Consumer Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $channel->queue_declare(resize-picture, false, true, false, false); $channel->queue_bind(resize-picture, upload-pictures); $channel->basic_consume(resize-picture, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 70. Consumer Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $channel->queue_declare(resize-picture, false, true, false, false); $channel->queue_bind(resize-picture, upload-pictures); $channel->basic_consume(resize-picture, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 71. Consumer Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $channel->queue_declare(resize-picture, false, true, false, false); $channel->queue_bind(resize-picture, upload-pictures); $channel->basic_consume(resize-picture, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 72. Consumer Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $channel->queue_declare(resize-picture, false, true, false, false); $channel->queue_bind(resize-picture, upload-pictures); $channel->basic_consume(resize-picture, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 73. Consumer Code $channel->exchange_declare(upload-pictures, fanout, false, true, false); $channel->queue_declare(resize-picture, false, true, false, false); $channel->queue_bind(resize-picture, upload-pictures); $channel->basic_consume(resize-picture, $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); }Thursday, September 29, 2011
  • 74. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta[image_id], $meta[image_path]); $msg->delivery_info[channel]-> basic_ack($msg->delivery_info[delivery_tag]); };Thursday, September 29, 2011
  • 75. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta[image_id], $meta[image_path]); $msg->delivery_info[channel]-> basic_ack($msg->delivery_info[delivery_tag]); };Thursday, September 29, 2011
  • 76. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta[image_id], $meta[image_path]); $msg->delivery_info[channel]-> basic_ack($msg->delivery_info[delivery_tag]); };Thursday, September 29, 2011
  • 77. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta[image_id], $meta[image_path]); $msg->delivery_info[channel]-> basic_ack($msg->delivery_info[delivery_tag]); };Thursday, September 29, 2011
  • 78. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta[image_id], $meta[image_path]); $msg->delivery_info[channel]-> basic_ack($msg->delivery_info[delivery_tag]); };Thursday, September 29, 2011
  • 79. Scenario Distributed LoggingThursday, September 29, 2011
  • 80. RequirementsThursday, September 29, 2011
  • 81. Requirements • Several Web ServersThursday, September 29, 2011
  • 82. Requirements • Several Web Servers • Logic Separated by Module/ActionThursday, September 29, 2011
  • 83. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels:Thursday, September 29, 2011
  • 84. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels: • Info, Warning, ErrorThursday, September 29, 2011
  • 85. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels: • Info, Warning, Error • Add/Remove log listeners at willThursday, September 29, 2011
  • 86. DesignThursday, September 29, 2011
  • 87. DesignThursday, September 29, 2011
  • 88. DesignThursday, September 29, 2011
  • 89. DesignThursday, September 29, 2011
  • 90. DesignThursday, September 29, 2011
  • 91. Publisher Code $channel->exchange_declare(logs, topic, false, true, false); $msg = new AMQPMessage(some log message, array(content_type => text/plain)); $channel->basic_publish($msg, logs, server1.user.profile.info);Thursday, September 29, 2011
  • 92. Publisher Code $channel->exchange_declare(logs, topic, false, true, false); $msg = new AMQPMessage(some log message, array(content_type => text/plain)); $channel->basic_publish($msg, logs, server1.user.profile.info);Thursday, September 29, 2011
  • 93. Publisher Code $channel->exchange_declare(logs, topic, false, true, false); $msg = new AMQPMessage(some log message, array(content_type => text/plain)); $channel->basic_publish($msg, logs, server1.user.profile.info);Thursday, September 29, 2011
  • 94. Publisher Code $channel->exchange_declare(logs, topic, false, true, false); $msg = new AMQPMessage(some log message, array(content_type => text/plain)); $channel->basic_publish($msg, logs, server1.user.profile.info);Thursday, September 29, 2011
  • 95. Consumer Code Get messages sent by host: server1Thursday, September 29, 2011
  • 96. Consumer Code $channel->exchange_declare(logs, topic, false, true, false); $channel->queue_declare(server1-logs, false, true, false, false); $channel->queue_bind(server1-logs, logs, server1.#);Thursday, September 29, 2011
  • 97. Consumer Code $channel->exchange_declare(logs, topic, false, true, false); $channel->queue_declare(server1-logs, false, true, false, false); $channel->queue_bind(server1-logs, logs, server1.#);Thursday, September 29, 2011
  • 98. Consumer Code $channel->exchange_declare(logs, topic, false, true, false); $channel->queue_declare(server1-logs, false, true, false, false); $channel->queue_bind(server1-logs, logs, server1.#);Thursday, September 29, 2011
  • 99. Consumer Code $channel->exchange_declare(logs, topic, false, true, false); $channel->queue_declare(server1-logs, false, true, false, false); $channel->queue_bind(server1-logs, logs, server1.#);Thursday, September 29, 2011
  • 100. Consumer Code Get all error messagesThursday, September 29, 2011
  • 101. Consumer Code $channel->exchange_declare(logs, topic, false, true, false); $channel->queue_declare(error-logs, false, true, false, false); $channel->queue_bind(error-logs, logs, #.error);Thursday, September 29, 2011
  • 102. Consumer Code $channel->exchange_declare(logs, topic, false, true, false); $channel->queue_declare(error-logs, false, true, false, false); $channel->queue_bind(error-logs, logs, #.error);Thursday, September 29, 2011
  • 103. Consumer Code $channel->exchange_declare(logs, topic, false, true, false); $channel->queue_declare(error-logs, false, true, false, false); $channel->queue_bind(error-logs, logs, #.error);Thursday, September 29, 2011
  • 104. Consumer Code $channel->exchange_declare(logs, topic, false, true, false); $channel->queue_declare(error-logs, false, true, false, false); $channel->queue_bind(error-logs, logs, #.error);Thursday, September 29, 2011
  • 105. Why RabbitMQ?Thursday, September 29, 2011
  • 106. RabbitMQ • Enterprise Messaging System • Open Source MPL • Written in Erlang/OTP • Commercial SupportThursday, September 29, 2011
  • 107. Features • Reliable and High Scalable • Easy To install • Easy To Cluster • Runs on: Windows, Solaris, Linux, OSX • AMQP 0.8 - 0.9.1Thursday, September 29, 2011
  • 108. Client Libraries • Java • .NET/C# • Erlang • Ruby, Python, PHP, Perl, AS3, Lisp, Scala, Clojure, HaskellThursday, September 29, 2011
  • 109. Docs/Support • http://www.rabbitmq.com/documentation.html • http://dev.rabbitmq.com/wiki/ • #rabbitmq at irc.freenode.net • http://www.rabbitmq.com/email-archive.htmlThursday, September 29, 2011
  • 110. One Setup for HAThursday, September 29, 2011
  • 111. ConclusionThursday, September 29, 2011
  • 112. Conclusion • FlexibilityThursday, September 29, 2011
  • 113. Conclusion • Flexibility • ScalabilityThursday, September 29, 2011
  • 114. Conclusion • Flexibility • Scalability • InteroperabilityThursday, September 29, 2011
  • 115. Conclusion • Flexibility • Scalability • Interoperability • Reduce OpsThursday, September 29, 2011
  • 116. Questions?Thursday, September 29, 2011
  • 117. Thanks!Thursday, September 29, 2011

×