“Divide and conquer
      riding rabbits and
        trading gems”
   - a tale about distributed programming -

   • http:...
Resources

• http://www.slideshare.net/hungryblank
• http://github.com/hungryblank/rabbit_starter
About me
Time
       GNU/Linux - Dbs
                          Perl
         “systems”
                                   ...
Summary:
Distributed
Concurrent
Programming
rabbitMQ




http://www.flickr.com/photos/myxi/448253580
Control
The problem
   Given a dictionary of
1.000.000 search phrases,
  compare for each one
the first page of results on
 google....
Don’t do it.
It’s a violation of terms
        of service!
(Sorry, I needed an example that
required little or no explanat...
Comparison
The whole process
Great! but...

• Fetching, parsing and analyzing
  2.000.000 pages of results will take a
  long time
• we want to collect...
Distributed Computing
Is a method of solving computational
problem by dividing the problem into
  many tasks run simultane...
Map Reduce "Map" step

  the master node takes the
input, chops it up into smaller
sub-problems, and distributes
   those ...
Problems:

• How many nodes?
• How many workers?
• Distribution mechanism to
 feed the workers?
What about queuing?

• the master node takes the input, chops
  it up into smaller sub-problems, and
  publishes them in a...
Here comes

• RabbitMQ is an implementation of AMQP,
  the emerging standard for high
  performance enterprise messaging
•...
Erlang?
• general-purpose concurrent programming
  language designed by Ericsson
• first version written by J. Armstrong in...
Erlang - is coming back
• Projects
 • CouchDB - RESTful document storage
 • WebMachine - REST toolkit
 • Nitrogen - web fr...
+ Erlang


It’s messages all the way down
Install it


    • sudo apt-get install rabbitmq
    • sudo gem install tmm1-amqp
Note: rabbitMQ must be v1.6.0 and amqp g...
Do it! - master node
Do it! - worker node
Get for free

• Decoupling master/worker
• Workers take care of feeding themselves
• Flexible number of workers
Behind the scenes

                            Worker1
Master

msg A    Exchange   Queue   Worker2


                     ...
Behind the scenes

                            Worker1
Master

         Exchange   Queue   Worker2
          msg A

      ...
Behind the scenes

                                  Worker1
Master

         Exchange   Queue msg A   Worker2


         ...
Behind the scenes

                            msg A   Worker1
Master

         Exchange   Queue           Worker2


     ...
RabbitMQ
• Multiple exchanges (and multiple types of
  exchange)
• Multiple queues
• Queues are connected by bindings to
 ...
RabbitMQ

• Exchanges and queues have names
• BUT direct exchanges created implicitly are
  not public and don’t have name...
What and where
                          Master
                          (ruby)
     RabbitMQ
      (Erlang)            W...
Problem #1




If a worker has a problem we might lose one
             or more messages
                    http://www.fli...
Solution - ACK in worker
Acknoledgement
• messages - 1 or more depending by
  prefetch settings - are passed to the client
• the client processes a...
http://www.flickr.com/photos/streetfly_jz/2770586821/



           Problem #2




No convenient way to control the workers
System queue - worker
System queue - control




• save it as system_command.rb
• ruby system_command.rb halt
System queue

                              Queue1   Worker1
Control script

   msg A           Exchange   Queue2   Worker...
System queue

                              Queue1   Worker1
Control script
                   Exchange   Queue2   Worker2...
System queue

                              msg A Queue1   Worker1
Control script
                   Exchange   msg A Queu...
EventMachine
EventMachine
Is an implementation of Reactor Pattern


• Non blocking IO and lightweight
  concurrency
• eliminate the com...
without EM                with EM
Ruby process          Ruby process
               Time
    code                  code


...
EventMachine

amqp gem is built on top of
EventMachine => you’re in a
  context where you can
    leverage concurrent
    ...
EM - Deferrables

“The Deferrable pattern allows you to specify
 any number of Ruby code blocks that will be
executed at s...
EM - Deferrables
EM - Deferrables
EM - Deferrables
EM - Deferrables
Deferrables
without deferrables      with deferrables


    GooglePage          GooglePage
                               ...
http://www.flickr.com/photos/philocrites/341850461/



          Problem #3




How many of them?
what are they doing?
Heartbeat - worker
Heartbeat monitor
Heartbeat queue

Worker1 msg A


Worker2           Exchange   Queue   Monitor


Worker3
          msg B
Heartbeat queue

Worker1

             msg A
Worker2     Exchange   Queue   Monitor
             msg B

Worker3
Heartbeat queue

Worker1


Worker2     Exchange   Queue msg B msg A   Monitor


Worker3
Clustering
         RabbitMQ - node A   RabbitMQ - node B

TCP/IP                                           TCP/IP



    ...
async vs sync
       Syncronous clients on github



• famoseagle/carrot
• celldee/bunny
Rabbits on github
• danielsdeleo/moqueue Tests/stubbing
• Check out ezmobius/nanite
  “self assembling fabric of ruby daem...
More rabbits on github

• tmm1/em-spec
• eventmachine/eventmachine
• tmm1/amqp
• macournoyer/thin
Q&A


 ?
Thanks!
(Thanks Mark!)

    Paolo Negri

/        hungryblank
Upcoming SlideShare
Loading in...5
×

Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009

12,037

Published on

Presentation given at the "rails underground" conference about distributed and concurrent programming with ruby using RabbitMQ and EventMachine

Published in: Technology
2 Comments
34 Likes
Statistics
Notes
No Downloads
Views
Total Views
12,037
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
343
Comments
2
Likes
34
Embeds 0
No embeds

No notes for slide

Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009

  1. 1. “Divide and conquer riding rabbits and trading gems” - a tale about distributed programming - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter Paolo Negri @hungryblank
  2. 2. Resources • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter
  3. 3. About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby
  4. 4. Summary:
  5. 5. Distributed Concurrent Programming
  6. 6. rabbitMQ http://www.flickr.com/photos/myxi/448253580
  7. 7. Control
  8. 8. The problem Given a dictionary of 1.000.000 search phrases, compare for each one the first page of results on google.com and bing.com
  9. 9. Don’t do it. It’s a violation of terms of service! (Sorry, I needed an example that required little or no explanation)
  10. 10. Comparison
  11. 11. The whole process
  12. 12. Great! but... • Fetching, parsing and analyzing 2.000.000 pages of results will take a long time • we want to collect the statistics in a limited time and this sequential approach is not quick enough
  13. 13. Distributed Computing Is a method of solving computational problem by dividing the problem into many tasks run simultaneously on many hardware or software systems (Wikipedia)
  14. 14. Map Reduce "Map" step the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)
  15. 15. Problems: • How many nodes? • How many workers? • Distribution mechanism to feed the workers?
  16. 16. What about queuing? • the master node takes the input, chops it up into smaller sub-problems, and publishes them in a queue • workers independently consume the content of the queue
  17. 17. Here comes • RabbitMQ is an implementation of AMQP, the emerging standard for high performance enterprise messaging • It’s opensource • Written in Erlang
  18. 18. Erlang? • general-purpose concurrent programming language designed by Ericsson • first version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability
  19. 19. Erlang - is coming back • Projects • CouchDB - RESTful document storage • WebMachine - REST toolkit • Nitrogen - web framework • Mochiweb - web framework • Ruby shops using it • 37 Signal - Campfire • Heroku
  20. 20. + Erlang It’s messages all the way down
  21. 21. Install it • sudo apt-get install rabbitmq • sudo gem install tmm1-amqp Note: rabbitMQ must be v1.6.0 and amqp gem v 0.6.4 to follow the code in the slides
  22. 22. Do it! - master node
  23. 23. Do it! - worker node
  24. 24. Get for free • Decoupling master/worker • Workers take care of feeding themselves • Flexible number of workers
  25. 25. Behind the scenes Worker1 Master msg A Exchange Queue Worker2 Worker3
  26. 26. Behind the scenes Worker1 Master Exchange Queue Worker2 msg A Worker3
  27. 27. Behind the scenes Worker1 Master Exchange Queue msg A Worker2 Worker3
  28. 28. Behind the scenes msg A Worker1 Master Exchange Queue Worker2 Worker3
  29. 29. RabbitMQ • Multiple exchanges (and multiple types of exchange) • Multiple queues • Queues are connected by bindings to exchanges • Exchanges route messages to queues
  30. 30. RabbitMQ • Exchanges and queues have names • BUT direct exchanges created implicitly are not public and don’t have name • Queues and messages are resident in RAM and can be persisted on disk (at a performance price)
  31. 31. What and where Master (ruby) RabbitMQ (Erlang) Worker TCP/IP (ruby) Worker (ruby) Exchange Queue Worker (ruby)
  32. 32. Problem #1 If a worker has a problem we might lose one or more messages http://www.flickr.com/photos/danzen/2288625136/
  33. 33. Solution - ACK in worker
  34. 34. Acknoledgement • messages - 1 or more depending by prefetch settings - are passed to the client • the client processes and acknowledges the messages one by one • if the client connection closes => unacknowledged messages go back in the queue
  35. 35. http://www.flickr.com/photos/streetfly_jz/2770586821/ Problem #2 No convenient way to control the workers
  36. 36. System queue - worker
  37. 37. System queue - control • save it as system_command.rb • ruby system_command.rb halt
  38. 38. System queue Queue1 Worker1 Control script msg A Exchange Queue2 Worker2 Queue3 Worker3
  39. 39. System queue Queue1 Worker1 Control script Exchange Queue2 Worker2 msg A Queue3 Worker3
  40. 40. System queue msg A Queue1 Worker1 Control script Exchange msg A Queue2 Worker2 msg A Queue3 Worker3
  41. 41. EventMachine
  42. 42. EventMachine Is an implementation of Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming
  43. 43. without EM with EM Ruby process Ruby process Time code code network network operation Free operation use network use network operation operation Callback result result Free Free
  44. 44. EventMachine amqp gem is built on top of EventMachine => you’re in a context where you can leverage concurrent programming
  45. 45. EM - Deferrables “The Deferrable pattern allows you to specify any number of Ruby code blocks that will be executed at some future time when the status of the Deferrable object changes “
  46. 46. EM - Deferrables
  47. 47. EM - Deferrables
  48. 48. EM - Deferrables
  49. 49. EM - Deferrables
  50. 50. Deferrables without deferrables with deferrables GooglePage GooglePage BingPage BingPage Time
  51. 51. http://www.flickr.com/photos/philocrites/341850461/ Problem #3 How many of them? what are they doing?
  52. 52. Heartbeat - worker
  53. 53. Heartbeat monitor
  54. 54. Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor Worker3 msg B
  55. 55. Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor msg B Worker3
  56. 56. Heartbeat queue Worker1 Worker2 Exchange Queue msg B msg A Monitor Worker3
  57. 57. Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP
  58. 58. async vs sync Syncronous clients on github • famoseagle/carrot • celldee/bunny
  59. 59. Rabbits on github • danielsdeleo/moqueue Tests/stubbing • Check out ezmobius/nanite “self assembling fabric of ruby daemons” • Keep an eye on tonyg/rabbithub implementation of pubsubhubbub (PubSub over REST) • auser/alice web app to monitor RabbitMQ
  60. 60. More rabbits on github • tmm1/em-spec • eventmachine/eventmachine • tmm1/amqp • macournoyer/thin
  61. 61. Q&A ?
  62. 62. Thanks! (Thanks Mark!) Paolo Negri / hungryblank
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×