Vetebra
Nanite
George Palmer

Background processing
• Work should be moved to the background
to stop application server load
• This keeps website responsive
• Useful for: images, videos, web services,
slow database queries....anything that isn’t
quick!

Current background
offerings
• Fork into background of rails process
• eg spawn, run_later
• Record to database (or file) and
background daemon picks up from there
• eg background_job, delayed_job
• Fork onto some kind of queue
• eg backgroundRB, beanstalk, starling with workling

Introducing Nanite
• Developed by EngineYard
• Gives presence - we know what’s available
• Handles failure
• Instant scalability
• Event based architecture

Pre-reqs
• Erlang
• Not R12B5 (the version installed by
MacPorts)
• RabbitMQ
• Untar to erlang lib directory - most
likely /usr/local/lib/erlang/lib

AMQP
• Advanced Message Queuing Protocol
• Enterprise quality protocol developed by
several financial institutions
• Includes wire protocol to ensure language
neutral
• RabbitMQ implements AMQP 0.8

Installing AMQP for Ruby
$ sudo gem install eventmachine
$ git clone git://github.com/tmm1/amqp.git
$ cd amqp && rake gem && sudo gem install
amqp-<version>.gem

Nanite
NA A Nanite
Thin Actor
a M M
App
ni Q Q Nanite
Server
te P P Actor
Nanite Mappers m e s s a g e s Nanite Agents
NA A Nanite
Ruby a M M Actor
Script ni Q Q Nanite
te P P Actor

Nanite Mappers
• Control and track work
• Unlimited number can be run that get
updates from mapper exchange
• mapper exchange itself is just a heartbeat and
registration MQ
• Run either inside Rails/MERB app (on Thin)
or via command line

Nanite Agents
• Do the work
• A given nanite agent can have multiple
actors
• Scale by adding more agents
• Pings the mapper exchange every
@ping_time seconds to report health

Nanite Actors
class Manor < Nanite::Actor
expose :name
def name(vars)
# Do something interesting here
:result => “RubyManor”
end
end
Nanite::Dispatcher.register(Manor.new)

Agent directory
structure
+ myagent
+ actors
- manor.rb
+ files
- init.rb
- config.yml

Agent config.yml
---
:vhost: /nanite # Allow multiple agents with different queues [compulsory]
:user: nanite # Username for queue
:pass: testing # Password for queue
:identity: barney # Can be auto-generated but useful to send work to specific
# agents
:file_root: path # where to store any transfered files
:format: marshal # or :json
# Additional options include host and port. All options can be passed into
# nanite command so can avoid config file if want

Getting Started...
• Start RabbitMQ
• /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq-
server
• On first run nanite/bin/rabbitconf
• Sets up RabbitMQ with a vhost and users
for that vhost (more on this later)

Starting agents
• cd <agentdir> && <nanite basedir>/bin/
nanite
• I’ve been using:
cd <agentdir> && <nanite basedir>/bin/
nanite -t <identity> &
• Could be managed better through a
daemon/monitoring system though

Offloading work to
Nanite
• Use the following code:
Nanite.request(callable, params = ‘’, options =
{}) {|res| # use res to do something}
• callable is the actor and method - eg ‘/manor/
name’
• params are parameters for the callable method -
eg ‘2008’
• options includes timeout, target and the routing
algorithm - more on this later

Interfaces into the
Mappers
• Via console:
• nanite/nanite-mapper -i
• Via command line:
• See nanite/examples/cli.rb
• Via Rails/MERB app:
• See next slide....

Rails/MERB & Nanite
# Updates the user
def update
...
if (@user.save)
Nanite.request(‘/updates/twitter’, ‘georgio_1999’) {|res|
# This block won’t execute until the event fires
@user.status = res[:status]
@user.save
}
end
end
def ajax_call
# Must use database for state and not Nanite job
end

Allocation of work
• The pings are used by the mappers to find
the healthiest nanite agents
• If a nanite agent doesn’t ping inside a
window it is removed from the active list
(until it does ping again)
• perfect for busy or error hit nanites
• The default routing algorithm is based on
server load

Routing options
• In options argument of Nanite.request you
can choose:
• selector
• :least_loaded (default), :all, :random, :rr
• target - use this to target a specific agent
• timeout - override the default timeout
(60s)

Custom algorithms
• Nanites report state with their ping
• By default this is the server load
• Can override this by adding code to agent
init.rb
• Nanite.status_proc = lambda
{ MyApp.some_statistic_indicating_load }
• Must be comparable
• Can use with existing routing algorithms or
create own more complex ones

File transfers
• Nanite can handle file transmission
• Agents subscribe (for all actors) in init.rb
or Actors subscribe individually:
• Nanite.subscribe_to_files(domain)
• Mappers send via:
• Nanite.broadcast_file(filepath, opts)
• where opts can contain :domain
and :destination (destination filename)

The beauty of JSON
• Nanite is built on top of AMQP, so if the
queue items are serialised using JSON...
• Then Nanite isn’t needed at the agent
side
• Any AMQP implementing daemon can
read message and respond
• Useful for legacy code (or legacy people)

Understanding Security
• Security is implemented using RabbitMQ’s
vhosts and username/passwords
• The username/password is defined in
config.yml for each nanite
• Need to configure RabbitMQ using the
rabbitmqctl command
• Generally one vhost per application

Let’s play
• cd <git_resources>/nanite/examples/myagent
• ../../bin/nanite -t <identity> -h <host>
• Let’s use fullname for identity - eg georgepalmer
• Should see something like:
# subscribing to file broadcasts for foobar
# loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/gems.rb
# loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/mock.rb
# \"advertise_services\"
# [\"/gems/list\", \"/mock/list\"]

Resources
• RabbitMQ: http://www.rabbitmq.com
• Ruby AMQP with RabbitMQ tutorial:
http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html
• Nanite: http://github.com/ezmobius/nanite
• Doc isn’t great, code is very readable