%w(map reduce).first
A Tale About Rabbits,
Latency and Slim
Crontabs
Paolo Negri
thanks to: www.autoscout24.de

Summary:

Map
http://www.matthiasdittrich.com/projekte/dliste/visualisations/index.html

rabbitMQ
http://www.flickr.com/photos/myxi/448253580

crontab diet
http://www.flickr.com/photos/tim_norris/2600843131/

Map Reduce
“Programming model for
processing and generating large
data sets”
(Google)

Map Reduce \"Map\" step
the master node takes the
input, chops it up into smaller
sub-problems, and distributes
those to worker nodes.
(Wikipedia)

The problem
Invoicing our clients

Is it as simple as...
clients.map do |client|
client.invoice
end

No!
Because the process is:
• distributed
• concurrent

Problems:
• How many nodes?
• How many workers?
• Distribution mechanism to
feed the workers?

What about queuing?
• the master node takes the input, chops
it up into smaller sub-problems, and
publishes them in a queue
• workers independently consume the
content of the queue

Here comes
• RabbitMQ is an implementation of AMQP,
the emerging standard for high
performance enterprise messaging
• It’s opensource
• Can be used to manage queues
• Written in Erlang

Erlang?
• Erlang is a general-purpose concurrent
programming language designed by Ericsson
• distributed
• fault tolerant
• soft real time
• high availability

Install it
• sudo apt-get install rabbitmq
• sudo gem install tmm1-amqp

Do it - master node

Use it - worker node

What and where
Master
(ruby)
RabbitMQ Worker
TCP/IP
(Erlang) (ruby)
Worker
(ruby)

Get for free
• Decoupling master/worker
• Workers take care of feeding themselves
• Flexible number of workers

Get for free
• RabbitMQ can be clustered
• Support of message acknowledgement
• Queues can be persisted on disk
(at a price)
• low latency

Queue
• Is an actual entity
• has a name
• can be inspected and managed

EventMachine

EventMachine
Is an implementation of Reactor Pattern
• Non blocking IO and lightweight
concurrency
• eliminate the complexities of high-
performance threaded network
programming

EventMachine

EventMachine
amqp gem is built on
EventMachine => you’re in a
context where you can
leverage concurrent
programming

EM - Deferrables

EM - Deferrables
“The Deferrable pattern allows you to specify
any number of Ruby code blocks that will be
executed at some future time when the status
of the Deferrable object changes “

EM - Deferrables

EM - Deferrables

Deferrables
without deferrables with deferrables
ClientStat ClientStat
Arrears
Arrears
Time

Achieved so far
• Easy distribution of tasks
• Architecture that supports arbitrary
number of workers (and masters)
• Concurrency within the single worker

More rabbits
Analogy with email system

Multicasting - producer

Multicasting - consumer

Multicasting
Cons1
Queue1
Publisher
Queue2 Cons2
Exchange
msg A
Cons3
Queue3

Multicasting
Cons1
msg A Queue1
Publisher
msg A Queue2 Cons2
Exchange
Cons3
msg A Queue3

Not only queues then
Use messages distribution to build the
nervous system of your app
• communication across hosts,
heterogeneous systems
• low latency
• clustering

Where to start?
crontab -l
5 * * * * bin/do_the_quick_thing.rb
0 2 * * * bin/do_the_scary_thing.rb

Cron
• Simple
• Reliable
• No maintenance
• Status is not explicit
• Locking?
• Shot and forget

Queue
• Distributed easily
• Reliable
• Can be inspected
• Add/decrease workers
• Makes you think!
• Adds more complexity

On github - Projects
• eventmachine/eventmachine
• tmm1/amqp
• macournoyer/thin
• famoseagle/carrot
• celldee/bunny
• ezmobius/nanite

Q&A
?

Thanks!
Paolo Negri
/ hungryblank