Scaling webappswithrabbitmq
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Scaling webappswithrabbitmq

  • 24,320 views
Uploaded on

Talk given at the ECUG Conference in Beijing, October 17th. 2010

Talk given at the ECUG Conference in Beijing, October 17th. 2010

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
24,320
On Slideshare
24,314
From Embeds
6
Number of Embeds
3

Actions

Shares
Downloads
98
Comments
0
Likes
6

Embeds 6

http://www.linkedin.com 4
http://coderwall.com 1
https://www.linkedin.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Scaling Web Apps With RabbitMQ Alvaro Videla - ECUG Con 2010 Thursday, October 21, 2010
  • 2. About Me • Development Manager at TheNetCircle.com • Twitter: @old_sound • Blog: http://videlalvaro.github.com/ • Thursday, October 21, 2010
  • 3. Agenda • RabbitMQ • AMQP • Scaling Web Apps • High Availability Thursday, October 21, 2010
  • 4. RabbitMQ Thursday, October 21, 2010
  • 5. RabbitMQ • Enterprise Messaging System • Open Source MPL • Written in Erlang/OTP • Commercial Support Thursday, October 21, 2010
  • 6. Features • Reliable and High Scalable • Easy To install • Easy To Cluster • Runs on: Windows, Solaris, Linux, OSX • AMQP 0.8 - 0.9.1 Thursday, October 21, 2010
  • 7. Client Libraries • Java • .NET/C# • Erlang • Ruby, Python, PHP, Perl, AS3, Lisp, Scala, Clojure, Haskell Thursday, October 21, 2010
  • 8. Docs/Support • http://www.rabbitmq.com/documentation.html • http://dev.rabbitmq.com/wiki/ • #rabbitmq at irc.freenode.net • http://www.rabbitmq.com/email-archive.html Thursday, October 21, 2010
  • 9. AMQP Thursday, October 21, 2010
  • 10. AMQP • Advanced Message Queuing Protocol • Suits Interoperability • Completely Open Protocol • Binary Protocol • AMQP Model • AMQP Wire Format Thursday, October 21, 2010
  • 11. AMQP Model • Exchanges • Message Queues • Bindings • Rules for binding them Thursday, October 21, 2010
  • 12. AMQP Wire Protocol • Functional Layer • Transport Layer Thursday, October 21, 2010
  • 13. Message Flow http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/chap-Messaging_Tutorial-Initial_Concepts.html Thursday, October 21, 2010
  • 14. Exchange Types • Fanout • Direct • Topic Thursday, October 21, 2010
  • 15. Default Exchanges • amqp.fanout • amqp.direct • amqp.topic Thursday, October 21, 2010
  • 16. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Fanout_Exchange.html Thursday, October 21, 2010
  • 17. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Direct_Exchange.html Thursday, October 21, 2010
  • 18. http://www.redhat.com/docs/en-US/Red_Hat_Enterprise_MRG/1.0/html/Messaging_Tutorial/sect-Messaging_Tutorial-Initial_Concepts- Topic_Exchange.html Thursday, October 21, 2010
  • 19. Scaling Web Apps Thursday, October 21, 2010
  • 20. What means “to Scale” • Scale Up • Scale Down • Scale to New Product Requirements Thursday, October 21, 2010
  • 21. Scenario I Batch Processing Thursday, October 21, 2010
  • 22. Requirements Thursday, October 21, 2010
  • 23. Requirements • Generate XML Thursday, October 21, 2010
  • 24. Requirements • Generate XML • Distribution Over a Cluster Thursday, October 21, 2010
  • 25. Requirements • Generate XML • Distribution Over a Cluster • Elasticity - Add/Remove new workers Thursday, October 21, 2010
  • 26. Design Thursday, October 21, 2010
  • 27. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $msg = new AMQPMessage($video_info, array('content_type' => 'text/plain', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'video-desc-ex'); $channel->close(); $conn->close(); Thursday, October 21, 2010
  • 28. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('video-desc-ex', 'direct', false, true, false); $channel->queue_declare('video-desc-queue', false, true, false, false); $channel->queue_bind('video-desc-queue', 'video-desc-ex'); $channel->basic_consume('video-desc-queue', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); } Thursday, October 21, 2010
  • 29. Scenario II Upload Pictures Thursday, October 21, 2010
  • 30. Requirements Thursday, October 21, 2010
  • 31. Requirements • Upload Picture Thursday, October 21, 2010
  • 32. Requirements • Upload Picture • Reward User Thursday, October 21, 2010
  • 33. Requirements • Upload Picture • Reward User • Notify User Friends Thursday, October 21, 2010
  • 34. Requirements • Upload Picture • Reward User • Notify User Friends • Resize Picture Thursday, October 21, 2010
  • 35. Design Thursday, October 21, 2010
  • 36. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $metadata = json_encode(array( 'image_id' => $image_id, 'user_id' => $user_id, ‘image_path' => $image_path)); $msg = new AMQPMessage($metadata, array('content_type' => 'application/json', 'delivery_mode' => 2)); $channel->basic_publish($msg, 'upload-pictures'); $channel->close(); $conn->close(); Thursday, October 21, 2010
  • 37. Consumer Code $channel->exchange_declare('upload-pictures', 'fanout', false, true, false); $channel->queue_declare('resize-picture', false, true, false, false); $channel->queue_bind('resize-picture', 'upload-pictures'); $channel->basic_consume('resize-picture', $consumer_tag, false, false, false, false, $consumer); while(count($channel->callbacks)) { $channel->wait(); } Thursday, October 21, 2010
  • 38. Consumer Code $consumer = function($msg){ $meta = json_decode($msg->body, true); resize_picture($meta['image_id'], $meta['image_path']); $msg->delivery_info['channel']-> basic_ack($msg->delivery_info['delivery_tag']); if($msg->body == 'quit'){ $msg->delivery_info['channel']-> basic_cancel($msg->delivery_info['consumer_tag']); } }; Thursday, October 21, 2010
  • 39. Scenario III Distributed Logging Thursday, October 21, 2010
  • 40. Requirements Thursday, October 21, 2010
  • 41. Requirements • Several Web Servers Thursday, October 21, 2010
  • 42. Requirements • Several Web Servers • Logic Separated by Module/Action Thursday, October 21, 2010
  • 43. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels: Thursday, October 21, 2010
  • 44. Requirements • Several Web Servers • Logic Separated by Module/Action • Several Log Levels: • Info • Warning • Error Thursday, October 21, 2010
  • 45. Design Thursday, October 21, 2010
  • 46. Publisher Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('logs', 'topic', false, true, false); $msg = new AMQPMessage('some log message', array('content_type' => 'text/plain')); $channel->basic_publish($msg, 'logs', server1.user.profile.info'); $channel->close(); $conn->close(); Thursday, October 21, 2010
  • 47. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('server1-logs', false, true, false, false); $channel->queue_bind('server1-logs', 'logs', 'server1.#'); Thursday, October 21, 2010
  • 48. Consumer Code $conn = new AMQPConnection(HOST, PORT, USER, PASS, VHOST); $channel = $conn->channel(); $channel->exchange_declare('logs', 'topic', false, true, false); $channel->queue_declare('error-logs', false, true, false, false); $channel->queue_bind('error-logs', 'logs', '#.error'); Thursday, October 21, 2010
  • 49. One Setup for HA Thursday, October 21, 2010
  • 50. Questions? Thursday, October 21, 2010
  • 51. Thanks! Alvaro Videla http://twitter.com/old_sound http://github.com/videlalvaro http://github.com/tnc http://www.slideshare.net/old_sound Thursday, October 21, 2010