Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009

13,288 views
12,983 views

Published on

Presentation given at the "rails underground" conference about distributed and concurrent programming with ruby using RabbitMQ and EventMachine

Published in: Technology
2 Comments
36 Likes
Statistics
Notes
No Downloads
Views
Total views
13,288
On SlideShare
0
From Embeds
0
Number of Embeds
239
Actions
Shares
0
Downloads
348
Comments
2
Likes
36
Embeds 0
No embeds

No notes for slide

Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009

  1. 1. “Divide and conquer riding rabbits and trading gems” - a tale about distributed programming - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter Paolo Negri @hungryblank
  2. 2. Resources • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter
  3. 3. About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby
  4. 4. Summary:
  5. 5. Distributed Concurrent Programming
  6. 6. rabbitMQ http://www.flickr.com/photos/myxi/448253580
  7. 7. Control
  8. 8. The problem Given a dictionary of 1.000.000 search phrases, compare for each one the first page of results on google.com and bing.com
  9. 9. Don’t do it. It’s a violation of terms of service! (Sorry, I needed an example that required little or no explanation)
  10. 10. Comparison
  11. 11. The whole process
  12. 12. Great! but... • Fetching, parsing and analyzing 2.000.000 pages of results will take a long time • we want to collect the statistics in a limited time and this sequential approach is not quick enough
  13. 13. Distributed Computing Is a method of solving computational problem by dividing the problem into many tasks run simultaneously on many hardware or software systems (Wikipedia)
  14. 14. Map Reduce "Map" step the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)
  15. 15. Problems: • How many nodes? • How many workers? • Distribution mechanism to feed the workers?
  16. 16. What about queuing? • the master node takes the input, chops it up into smaller sub-problems, and publishes them in a queue • workers independently consume the content of the queue
  17. 17. Here comes • RabbitMQ is an implementation of AMQP, the emerging standard for high performance enterprise messaging • It’s opensource • Written in Erlang
  18. 18. Erlang? • general-purpose concurrent programming language designed by Ericsson • first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
  19. 19. Erlang - is coming back • Projects • CouchDB - RESTful document storage • WebMachine - REST toolkit • Nitrogen - web framework • Mochiweb - web framework • Ruby shops using it • 37 Signal - Campfire • Heroku
  20. 20. + Erlang It’s messages all the way down
  21. 21. Install it • sudo apt-get install rabbitmq • sudo gem install tmm1-amqp Note: rabbitMQ must be v1.6.0 and amqp gem v 0.6.4 to follow the code in the slides
  22. 22. Do it! - master node
  23. 23. Do it! - worker node
  24. 24. Get for free • Decoupling master/worker • Workers take care of feeding themselves • Flexible number of workers
  25. 25. Behind the scenes Worker1 Master msg A Exchange Queue Worker2 Worker3
  26. 26. Behind the scenes Worker1 Master Exchange Queue Worker2 msg A Worker3
  27. 27. Behind the scenes Worker1 Master Exchange Queue msg A Worker2 Worker3
  28. 28. Behind the scenes msg A Worker1 Master Exchange Queue Worker2 Worker3
  29. 29. RabbitMQ • Multiple exchanges (and multiple types of exchange) • Multiple queues • Queues are connected by bindings to exchanges • Exchanges route messages to queues
  30. 30. RabbitMQ • Exchanges and queues have names • BUT direct exchanges created implicitly are not public and don’t have name • Queues and messages are resident in RAM and can be persisted on disk (at a performance price)
  31. 31. What and where Master (ruby) RabbitMQ (Erlang) Worker TCP/IP (ruby) Worker (ruby) Exchange Queue Worker (ruby)
  32. 32. Problem #1 If a worker has a problem we might lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
  33. 33. Solution - ACK in worker
  34. 34. Acknoledgement • messages - 1 or more depending by prefetch settings - are passed to the client • the client processes and acknowledges the messages one by one • if the client connection closes => unacknowledged messages go back in the queue
  35. 35. http://www.flickr.com/photos/streetfly_jz/2770586821/ Problem #2 No convenient way to control the workers
  36. 36. System queue - worker
  37. 37. System queue - control • save it as system_command.rb • ruby system_command.rb halt
  38. 38. System queue Queue1 Worker1 Control script msg A Exchange Queue2 Worker2 Queue3 Worker3
  39. 39. System queue Queue1 Worker1 Control script Exchange Queue2 Worker2 msg A Queue3 Worker3
  40. 40. System queue msg A Queue1 Worker1 Control script Exchange msg A Queue2 Worker2 msg A Queue3 Worker3
  41. 41. EventMachine
  42. 42. EventMachine Is an implementation of Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming
  43. 43. without EM with EM Ruby process Ruby process Time code code network network operation Free operation use network use network operation operation Callback result result Free Free
  44. 44. EventMachine amqp gem is built on top of EventMachine => you’re in a context where you can leverage concurrent programming
  45. 45. EM - Deferrables “The Deferrable pattern allows you to specify any number of Ruby code blocks that will be executed at some future time when the status of the Deferrable object changes “
  46. 46. EM - Deferrables
  47. 47. EM - Deferrables
  48. 48. EM - Deferrables
  49. 49. EM - Deferrables
  50. 50. Deferrables without deferrables with deferrables GooglePage GooglePage BingPage BingPage Time
  51. 51. http://www.flickr.com/photos/philocrites/341850461/ Problem #3 How many of them? what are they doing?
  52. 52. Heartbeat - worker
  53. 53. Heartbeat monitor
  54. 54. Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor Worker3 msg B
  55. 55. Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor msg B Worker3
  56. 56. Heartbeat queue Worker1 Worker2 Exchange Queue msg B msg A Monitor Worker3
  57. 57. Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP
  58. 58. async vs sync Syncronous clients on github • famoseagle/carrot • celldee/bunny
  59. 59. Rabbits on github • danielsdeleo/moqueue Tests/stubbing • Check out ezmobius/nanite “self assembling fabric of ruby daemons” • Keep an eye on tonyg/rabbithub implementation of pubsubhubbub (PubSub over REST) • auser/alice web app to monitor RabbitMQ
  60. 60. More rabbits on github • tmm1/em-spec • eventmachine/eventmachine • tmm1/amqp • macournoyer/thin
  61. 61. Q&A ?
  62. 62. Thanks! (Thanks Mark!) Paolo Negri / hungryblank

×