Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009

“Divide and conquer riding rabbits and trading gems” - a tale about distributed programming - • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter Paolo Negri @hungryblank

Resources • http://www.slideshare.net/hungryblank • http://github.com/hungryblank/rabbit_starter

About me Time GNU/Linux - Dbs Perl “systems” PHP Python Ruby

Summary:

Distributed Concurrent Programming

rabbitMQ http://www.ﬂickr.com/photos/myxi/448253580

Control

The problem Given a dictionary of 1.000.000 search phrases, compare for each one the ﬁrst page of results on google.com and bing.com

Don’t do it. It’s a violation of terms of service! (Sorry, I needed an example that required little or no explanation)

Comparison

The whole process

Great! but... • Fetching, parsing and analyzing 2.000.000 pages of results will take a long time • we want to collect the statistics in a limited time and this sequential approach is not quick enough

Distributed Computing Is a method of solving computational problem by dividing the problem into many tasks run simultaneously on many hardware or software systems (Wikipedia)

Map Reduce "Map" step the master node takes the input, chops it up into smaller sub-problems, and distributes those to worker nodes. (Wikipedia)

Problems: • How many nodes? • How many workers? • Distribution mechanism to feed the workers?

What about queuing? • the master node takes the input, chops it up into smaller sub-problems, and publishes them in a queue • workers independently consume the content of the queue

Here comes • RabbitMQ is an implementation of AMQP, the emerging standard for high performance enterprise messaging • It’s opensource • Written in Erlang

Erlang? • general-purpose concurrent programming language designed by Ericsson • ﬁrst version written by J. Armstrong in 1986 • distributed • fault tolerant • soft real time • high availability

Erlang - is coming back • Projects • CouchDB - RESTful document storage • WebMachine - REST toolkit • Nitrogen - web framework • Mochiweb - web framework • Ruby shops using it • 37 Signal - Campﬁre • Heroku

+ Erlang It’s messages all the way down

Install it • sudo apt-get install rabbitmq • sudo gem install tmm1-amqp Note: rabbitMQ must be v1.6.0 and amqp gem v 0.6.4 to follow the code in the slides

Do it! - master node

Do it! - worker node

Get for free • Decoupling master/worker • Workers take care of feeding themselves • Flexible number of workers

Behind the scenes Worker1 Master msg A Exchange Queue Worker2 Worker3

Behind the scenes Worker1 Master Exchange Queue Worker2 msg A Worker3

Behind the scenes Worker1 Master Exchange Queue msg A Worker2 Worker3

Behind the scenes msg A Worker1 Master Exchange Queue Worker2 Worker3

RabbitMQ • Multiple exchanges (and multiple types of exchange) • Multiple queues • Queues are connected by bindings to exchanges • Exchanges route messages to queues

RabbitMQ • Exchanges and queues have names • BUT direct exchanges created implicitly are not public and don’t have name • Queues and messages are resident in RAM and can be persisted on disk (at a performance price)

What and where Master (ruby) RabbitMQ (Erlang) Worker TCP/IP (ruby) Worker (ruby) Exchange Queue Worker (ruby)

Problem #1 If a worker has a problem we might lose one or more messages http://www.ﬂickr.com/photos/danzen/2288625136/

Solution - ACK in worker

Acknoledgement • messages - 1 or more depending by prefetch settings - are passed to the client • the client processes and acknowledges the messages one by one • if the client connection closes => unacknowledged messages go back in the queue

http://www.ﬂickr.com/photos/streetﬂy_jz/2770586821/ Problem #2 No convenient way to control the workers

System queue - worker

System queue - control • save it as system_command.rb • ruby system_command.rb halt

System queue Queue1 Worker1 Control script msg A Exchange Queue2 Worker2 Queue3 Worker3

System queue Queue1 Worker1 Control script Exchange Queue2 Worker2 msg A Queue3 Worker3

System queue msg A Queue1 Worker1 Control script Exchange msg A Queue2 Worker2 msg A Queue3 Worker3

EventMachine

EventMachine Is an implementation of Reactor Pattern • Non blocking IO and lightweight concurrency • eliminate the complexities of high- performance threaded network programming

without EM with EM Ruby process Ruby process Time code code network network operation Free operation use network use network operation operation Callback result result Free Free

EventMachine amqp gem is built on top of EventMachine => you’re in a context where you can leverage concurrent programming

EM - Deferrables “The Deferrable pattern allows you to specify any number of Ruby code blocks that will be executed at some future time when the status of the Deferrable object changes “

EM - Deferrables

Deferrables without deferrables with deferrables GooglePage GooglePage BingPage BingPage Time

http://www.ﬂickr.com/photos/philocrites/341850461/ Problem #3 How many of them? what are they doing?

Heartbeat - worker

Heartbeat monitor

Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor Worker3 msg B

Heartbeat queue Worker1 msg A Worker2 Exchange Queue Monitor msg B Worker3

Heartbeat queue Worker1 Worker2 Exchange Queue msg B msg A Monitor Worker3

Clustering RabbitMQ - node A RabbitMQ - node B TCP/IP TCP/IP RabbitMQ - node C TCP/IP

async vs sync Syncronous clients on github • famoseagle/carrot • celldee/bunny

Rabbits on github • danielsdeleo/moqueue Tests/stubbing • Check out ezmobius/nanite “self assembling fabric of ruby daemons” • Keep an eye on tonyg/rabbithub implementation of pubsubhubbub (PubSub over REST) • auser/alice web app to monitor RabbitMQ

More rabbits on github • tmm1/em-spec • eventmachine/eventmachine • tmm1/amqp • macournoyer/thin

Thanks! (Thanks Mark!) Paolo Negri / hungryblank

http://skillsmatter.com	107
http://www.slideshare.net	66
http://www.lifeyun.com	17
http://thinkery.me	5
http://www.techgig.com	1
http://www.yatedo.com	1
http://courtneybolton.parasoup.de	1
http://www.apurva.com	1
http://clipboard.com	1
http://www.slashdocs.com	1

Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009

by Paolo Negri on Jul 25, 2009

Accessibility

Categories

Upload Details

Usage Rights

10 Embeds 201

Statistics

Distributed and concurrent programming with RabbitMQ and EventMachine Rails Underground 2009 Presentation Transcript