Vetebra
Nanite
George Palmer
Background processing

• Work should be moved to the background
  to stop application server load
• This keeps website res...
Current background
       offerings
• Fork into background of rails process
  •   eg spawn, run_later

• Record to databas...
Introducing Nanite

• Developed by EngineYard
• Gives presence - we know what’s available
• Handles failure
• Instant scal...
Pre-reqs

• Erlang
 • Not R12B5 (the version installed by
    MacPorts)
• RabbitMQ
 • Untar to erlang lib directory - most...
AMQP
• Advanced Message Queuing Protocol
• Enterprise quality protocol developed by
  several financial institutions
• Incl...
Installing AMQP for Ruby


$ sudo gem install eventmachine
$ git clone git://github.com/tmm1/amqp.git
$ cd amqp && rake ge...
Nanite
         NA                         A   Nanite
   Thin                                 Actor
          a M         ...
Nanite Mappers
• Control and track work
• Unlimited number can be run that get
  updates from mapper exchange
 •   mapper ...
Nanite Agents
• Do the work
• A given nanite agent can have multiple
  actors
• Scale by adding more agents
• Pings the ma...
Nanite Actors

class Manor < Nanite::Actor
 expose :name

 def name(vars)
  # Do something interesting here
  :result => “...
Agent directory
        structure
+ myagent
 + actors
  - manor.rb
 + files
 - init.rb
 - config.yml
Agent config.yml
---
:vhost: /nanite   # Allow multiple agents with different queues [compulsory]
:user: nanite     # Usern...
Getting Started...

• Start RabbitMQ
 •   /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq-
     server

• On first run nan...
Starting agents
• cd <agentdir> && <nanite basedir>/bin/
  nanite
• I’ve been using:
  cd <agentdir> && <nanite basedir>/b...
Offloading work to
        Nanite
• Use the following code:
  Nanite.request(callable, params = ‘’,   options =
  {}) {|res...
Interfaces into the
         Mappers
• Via console:
 • nanite/nanite-mapper -i
• Via command line:
 • See nanite/examples/...
Rails/MERB & Nanite
# Updates the user
def update
 ...
 if (@user.save)
    Nanite.request(‘/updates/twitter’, ‘georgio_19...
Allocation of work
• The pings are used by the mappers to find
  the healthiest nanite agents
• If a nanite agent doesn’t p...
Routing options
• In options argument of        Nanite.request   you
  can choose:

 • selector
   •   :least_loaded (defa...
Custom algorithms
• Nanites report state with their ping
  •   By default this is the server load

• Can override this by ...
File transfers
• Nanite can handle file transmission
 • Agents subscribe (for all actors) in init.rb
    or Actors subscrib...
The beauty of JSON
• Nanite is built on top of AMQP, so if the
  queue items are serialised using JSON...
  • Then Nanite ...
Understanding Security
• Security is implemented using RabbitMQ’s
  vhosts and username/passwords
• The username/password ...
Let’s play
• cd <git_resources>/nanite/examples/myagent
• ../../bin/nanite -t <identity> -h <host>
    •      Let’s use fu...
Resources

• RabbitMQ: http://www.rabbitmq.com
• Ruby AMQP with RabbitMQ tutorial:
  http://hopper.squarespace.com/blog/20...
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Upcoming SlideShare
Loading in …5
×

Rubymanor - Nanite talk

4,851 views

Published on

Published in: Technology
0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,851
On SlideShare
0
From Embeds
0
Number of Embeds
99
Actions
Shares
0
Downloads
76
Comments
0
Likes
12
Embeds 0
No embeds

No notes for slide

Rubymanor - Nanite talk

  1. 1. Vetebra Nanite George Palmer
  2. 2. Background processing • Work should be moved to the background to stop application server load • This keeps website responsive • Useful for: images, videos, web services, slow database queries....anything that isn’t quick!
  3. 3. Current background offerings • Fork into background of rails process • eg spawn, run_later • Record to database (or file) and background daemon picks up from there • eg background_job, delayed_job • Fork onto some kind of queue • eg backgroundRB, beanstalk, starling with workling
  4. 4. Introducing Nanite • Developed by EngineYard • Gives presence - we know what’s available • Handles failure • Instant scalability • Event based architecture
  5. 5. Pre-reqs • Erlang • Not R12B5 (the version installed by MacPorts) • RabbitMQ • Untar to erlang lib directory - most likely /usr/local/lib/erlang/lib
  6. 6. AMQP • Advanced Message Queuing Protocol • Enterprise quality protocol developed by several financial institutions • Includes wire protocol to ensure language neutral • RabbitMQ implements AMQP 0.8
  7. 7. Installing AMQP for Ruby $ sudo gem install eventmachine $ git clone git://github.com/tmm1/amqp.git $ cd amqp && rake gem && sudo gem install amqp-<version>.gem
  8. 8. Nanite NA A Nanite Thin Actor a M M App ni Q Q Nanite Server te P P Actor Nanite Mappers m e s s a g e s Nanite Agents NA A Nanite Ruby a M M Actor Script ni Q Q Nanite te P P Actor
  9. 9. Nanite Mappers • Control and track work • Unlimited number can be run that get updates from mapper exchange • mapper exchange itself is just a heartbeat and registration MQ • Run either inside Rails/MERB app (on Thin) or via command line
  10. 10. Nanite Agents • Do the work • A given nanite agent can have multiple actors • Scale by adding more agents • Pings the mapper exchange every @ping_time seconds to report health
  11. 11. Nanite Actors class Manor < Nanite::Actor expose :name def name(vars) # Do something interesting here :result => “RubyManor” end end Nanite::Dispatcher.register(Manor.new)
  12. 12. Agent directory structure + myagent + actors - manor.rb + files - init.rb - config.yml
  13. 13. Agent config.yml --- :vhost: /nanite # Allow multiple agents with different queues [compulsory] :user: nanite # Username for queue :pass: testing # Password for queue :identity: barney # Can be auto-generated but useful to send work to specific # agents :file_root: path # where to store any transfered files :format: marshal # or :json # Additional options include host and port. All options can be passed into # nanite command so can avoid config file if want
  14. 14. Getting Started... • Start RabbitMQ • /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq- server • On first run nanite/bin/rabbitconf • Sets up RabbitMQ with a vhost and users for that vhost (more on this later)
  15. 15. Starting agents • cd <agentdir> && <nanite basedir>/bin/ nanite • I’ve been using: cd <agentdir> && <nanite basedir>/bin/ nanite -t <identity> & • Could be managed better through a daemon/monitoring system though
  16. 16. Offloading work to Nanite • Use the following code: Nanite.request(callable, params = ‘’, options = {}) {|res| # use res to do something} • callable is the actor and method - eg ‘/manor/ name’ • params are parameters for the callable method - eg ‘2008’ • options includes timeout, target and the routing algorithm - more on this later
  17. 17. Interfaces into the Mappers • Via console: • nanite/nanite-mapper -i • Via command line: • See nanite/examples/cli.rb • Via Rails/MERB app: • See next slide....
  18. 18. Rails/MERB & Nanite # Updates the user def update ... if (@user.save) Nanite.request(‘/updates/twitter’, ‘georgio_1999’) {|res| # This block won’t execute until the event fires @user.status = res[:status] @user.save } end end def ajax_call # Must use database for state and not Nanite job end
  19. 19. Allocation of work • The pings are used by the mappers to find the healthiest nanite agents • If a nanite agent doesn’t ping inside a window it is removed from the active list (until it does ping again) • perfect for busy or error hit nanites • The default routing algorithm is based on server load
  20. 20. Routing options • In options argument of Nanite.request you can choose: • selector • :least_loaded (default), :all, :random, :rr • target - use this to target a specific agent • timeout - override the default timeout (60s)
  21. 21. Custom algorithms • Nanites report state with their ping • By default this is the server load • Can override this by adding code to agent init.rb • Nanite.status_proc = lambda { MyApp.some_statistic_indicating_load } • Must be comparable • Can use with existing routing algorithms or create own more complex ones
  22. 22. File transfers • Nanite can handle file transmission • Agents subscribe (for all actors) in init.rb or Actors subscribe individually: • Nanite.subscribe_to_files(domain) • Mappers send via: • Nanite.broadcast_file(filepath, opts) • where opts can contain :domain and :destination (destination filename)
  23. 23. The beauty of JSON • Nanite is built on top of AMQP, so if the queue items are serialised using JSON... • Then Nanite isn’t needed at the agent side • Any AMQP implementing daemon can read message and respond • Useful for legacy code (or legacy people)
  24. 24. Understanding Security • Security is implemented using RabbitMQ’s vhosts and username/passwords • The username/password is defined in config.yml for each nanite • Need to configure RabbitMQ using the rabbitmqctl command • Generally one vhost per application
  25. 25. Let’s play • cd <git_resources>/nanite/examples/myagent • ../../bin/nanite -t <identity> -h <host> • Let’s use fullname for identity - eg georgepalmer • Should see something like: # subscribing to file broadcasts for foobar # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/gems.rb # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/mock.rb # quot;advertise_servicesquot; # [quot;/gems/listquot;, quot;/mock/listquot;]
  26. 26. Resources • RabbitMQ: http://www.rabbitmq.com • Ruby AMQP with RabbitMQ tutorial: http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html • Nanite: http://github.com/ezmobius/nanite • Doc isn’t great, code is very readable

×