RabbitMQ and Nanite
Scalable Messaging
Mathias Meyer, Peritor GmbH

The Problem
• A scalable backend to offload asynchronous
tasks
• Self-configuring
• Self-healing
• Self-assembling
• But not a lot of people have it

The Problem
• I’m still learning about the nitty-gritty details

Protocols
• JMS
• MQ
• XMPP
• SMTP (yep, for real)
• STOMP
• RestMS
• Wtf?

AMQP!

AMQP
• Open standard for messaging middleware
• Vendor neutral
• Wire level protocol
• Mandates interoperatibility
• Abundant number of libraries that speak it

AMQP
• Supports publish/subscribe and point-to-point
messaging, among others

AMQP Components
• Broker
• Combination of Exchange and a Queue
• Exchange routes messages (stateless) based
on patterns
• Queue buffers messages (stateful)
• Bound by Bindings

AMQP Components
• Consumer
• Producer
• Messages (duh!)

RabbitMQ

RabbitMQ
• Erlang-based, independent AMQP
implementation
• Some 6k lines of code
• Apache Qpid (Java): 142k lines of code
• Also speaks STOMP, SMTP, XMPP and HTTP
• Message persistence uses a journal
• Wicked fast

RabbitMQ
• port install rabbitmq-server
• It just works™
• rabbitmqctl
• Can be clustered through Erlang’s distributed
nodes

Nanite
• A self-assembling fabric of Ruby daemons
• Uses mappers, agents and actors
• Mappers route message requests
• Agents handle messages
• Actors offer specific services bundled into
agents

Nanite
• Messages can be simple pushes
• Or requests waiting for a response

Nanite
Agent
Agent Mapper
RabbitMQ
Agent Mapper
Agent

Nanite Agents

Nanite Agents
class Reactor
include Nanite::Actor
expose :react
def react(payload)
"reacting to message with payload: #{payload}"
end
end

Nanite
Nanite.request('/reactor/react', 'good acting is reacting') do |r|
pr
end

Nanite
• Agents register themselves and advertise their
services, constantly pinging the mappers
• Mappers remove timed-out agents
• Work distributed based on agent load

Nanite
• Agents can interoperate with each other
during one single message processing
• Mapper state can (and should) be stored in
redis
• Mappers live inside your application or
standalone

Intermediate Messages
class Reactor
include Nanite::Actor
expose :react
def react(payload)
request("/worker/work", "kill all humans",
:intermediate_handler => lambda {|res| puts res})
# working, working...
yield "I'm almost done"
# cleanup the mess
"done reacting to message with payload: #{payload}"
end
end

Intermediate Messages
class Worker
include Nanite::Actor
expose :work
def work(payload)
Human.all.each do |human|
human.destroy
yield "Destroyed human #{human.name}"
end
"Killed all the humans"
end
end

Nanite Internals
• Based on EventMachine and Ruby amqp library
(also EventMachine-based)
• Everything runs in the EventMachine loop

Nanite Internals
• Uses a combination of the following queues to
work
• Registration queue
• Heartbeat queue
• Advertise queue
• Agent queues

Nanite Internals
• Messages can be encrypted and signed
between agents and mappers
• Sends marshaled Ruby (default),YAML or
JSON
• The code internals are actually mind-boggling

Nanite Boos
• Poor documentation
• A not too great test suite

So how do we use it?
That really is a topic for another talk, isn’t it?

So how do we use it?
That really is a topic for another talk, isn’t it?

Resources
• http://skillsmatter.com/podcast/erlang/alexis-richardson-
introduction-to-rabbitmq
• http://skillsmatter.com/podcast/erlang/rabbitmq-internal-
architecture-tony-garnock-jones
• http://brainspl.at/articles/2008/10/11/merbcamp-keynote-and-
introducing-nanite
• http://github.com/ezmobius/nanite
• http://www.rabbitmq.com
• http://www.amqp.org
• http://www.slideshare.net/somic/introduction-to-amqp-messaging-
with-rabbitmq