Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Vetebra
Nanite
George Palmer
Background processing

• Work should be moved to the background
  to stop application server load
• This keeps website res...
Current background
       offerings
• Fork into background of rails process
  •   eg spawn, run_later

• Record to databas...
Introducing Nanite

• Developed by EngineYard
• Gives presence - we know what’s available
• Handles failure
• Instant scal...
Pre-reqs

• Erlang
 • Not R12B5 (the version installed by
    MacPorts)
• RabbitMQ
 • Untar to erlang lib directory - most...
AMQP
• Advanced Message Queuing Protocol
• Enterprise quality protocol developed by
  several financial institutions
• Incl...
Installing AMQP for Ruby


$ sudo gem install eventmachine
$ git clone git://github.com/tmm1/amqp.git
$ cd amqp && rake ge...
Nanite
         NA                         A   Nanite
   Thin                                 Actor
          a M         ...
Nanite Mappers
• Control and track work
• Unlimited number can be run that get
  updates from mapper exchange
 •   mapper ...
Nanite Agents
• Do the work
• A given nanite agent can have multiple
  actors
• Scale by adding more agents
• Pings the ma...
Nanite Actors

class Manor < Nanite::Actor
 expose :name

 def name(vars)
  # Do something interesting here
  :result => “...
Agent directory
        structure
+ myagent
 + actors
  - manor.rb
 + files
 - init.rb
 - config.yml
Agent config.yml
---
:vhost: /nanite   # Allow multiple agents with different queues [compulsory]
:user: nanite     # Usern...
Getting Started...

• Start RabbitMQ
 •   /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq-
     server

• On first run nan...
Starting agents
• cd <agentdir> && <nanite basedir>/bin/
  nanite
• I’ve been using:
  cd <agentdir> && <nanite basedir>/b...
Offloading work to
        Nanite
• Use the following code:
  Nanite.request(callable, params = ‘’,   options =
  {}) {|res...
Interfaces into the
         Mappers
• Via console:
 • nanite/nanite-mapper -i
• Via command line:
 • See nanite/examples/...
Rails/MERB & Nanite
# Updates the user
def update
 ...
 if (@user.save)
    Nanite.request(‘/updates/twitter’, ‘georgio_19...
Allocation of work
• The pings are used by the mappers to find
  the healthiest nanite agents
• If a nanite agent doesn’t p...
Routing options
• In options argument of        Nanite.request   you
  can choose:

 • selector
   •   :least_loaded (defa...
Custom algorithms
• Nanites report state with their ping
  •   By default this is the server load

• Can override this by ...
File transfers
• Nanite can handle file transmission
 • Agents subscribe (for all actors) in init.rb
    or Actors subscrib...
The beauty of JSON
• Nanite is built on top of AMQP, so if the
  queue items are serialised using JSON...
  • Then Nanite ...
Understanding Security
• Security is implemented using RabbitMQ’s
  vhosts and username/passwords
• The username/password ...
Let’s play
• cd <git_resources>/nanite/examples/myagent
• ../../bin/nanite -t <identity> -h <host>
    •      Let’s use fu...
Resources

• RabbitMQ: http://www.rabbitmq.com
• Ruby AMQP with RabbitMQ tutorial:
  http://hopper.squarespace.com/blog/20...
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Rubymanor - Nanite talk
Upcoming SlideShare
Loading in …5
×

Rubymanor - Nanite talk

4,943 views

Published on

Published in: Technology
  • Be the first to comment

Rubymanor - Nanite talk

  1. 1. Vetebra Nanite George Palmer
  2. 2. Background processing • Work should be moved to the background to stop application server load • This keeps website responsive • Useful for: images, videos, web services, slow database queries....anything that isn’t quick!
  3. 3. Current background offerings • Fork into background of rails process • eg spawn, run_later • Record to database (or file) and background daemon picks up from there • eg background_job, delayed_job • Fork onto some kind of queue • eg backgroundRB, beanstalk, starling with workling
  4. 4. Introducing Nanite • Developed by EngineYard • Gives presence - we know what’s available • Handles failure • Instant scalability • Event based architecture
  5. 5. Pre-reqs • Erlang • Not R12B5 (the version installed by MacPorts) • RabbitMQ • Untar to erlang lib directory - most likely /usr/local/lib/erlang/lib
  6. 6. AMQP • Advanced Message Queuing Protocol • Enterprise quality protocol developed by several financial institutions • Includes wire protocol to ensure language neutral • RabbitMQ implements AMQP 0.8
  7. 7. Installing AMQP for Ruby $ sudo gem install eventmachine $ git clone git://github.com/tmm1/amqp.git $ cd amqp && rake gem && sudo gem install amqp-<version>.gem
  8. 8. Nanite NA A Nanite Thin Actor a M M App ni Q Q Nanite Server te P P Actor Nanite Mappers m e s s a g e s Nanite Agents NA A Nanite Ruby a M M Actor Script ni Q Q Nanite te P P Actor
  9. 9. Nanite Mappers • Control and track work • Unlimited number can be run that get updates from mapper exchange • mapper exchange itself is just a heartbeat and registration MQ • Run either inside Rails/MERB app (on Thin) or via command line
  10. 10. Nanite Agents • Do the work • A given nanite agent can have multiple actors • Scale by adding more agents • Pings the mapper exchange every @ping_time seconds to report health
  11. 11. Nanite Actors class Manor < Nanite::Actor expose :name def name(vars) # Do something interesting here :result => “RubyManor” end end Nanite::Dispatcher.register(Manor.new)
  12. 12. Agent directory structure + myagent + actors - manor.rb + files - init.rb - config.yml
  13. 13. Agent config.yml --- :vhost: /nanite # Allow multiple agents with different queues [compulsory] :user: nanite # Username for queue :pass: testing # Password for queue :identity: barney # Can be auto-generated but useful to send work to specific # agents :file_root: path # where to store any transfered files :format: marshal # or :json # Additional options include host and port. All options can be passed into # nanite command so can avoid config file if want
  14. 14. Getting Started... • Start RabbitMQ • /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq- server • On first run nanite/bin/rabbitconf • Sets up RabbitMQ with a vhost and users for that vhost (more on this later)
  15. 15. Starting agents • cd <agentdir> && <nanite basedir>/bin/ nanite • I’ve been using: cd <agentdir> && <nanite basedir>/bin/ nanite -t <identity> & • Could be managed better through a daemon/monitoring system though
  16. 16. Offloading work to Nanite • Use the following code: Nanite.request(callable, params = ‘’, options = {}) {|res| # use res to do something} • callable is the actor and method - eg ‘/manor/ name’ • params are parameters for the callable method - eg ‘2008’ • options includes timeout, target and the routing algorithm - more on this later
  17. 17. Interfaces into the Mappers • Via console: • nanite/nanite-mapper -i • Via command line: • See nanite/examples/cli.rb • Via Rails/MERB app: • See next slide....
  18. 18. Rails/MERB & Nanite # Updates the user def update ... if (@user.save) Nanite.request(‘/updates/twitter’, ‘georgio_1999’) {|res| # This block won’t execute until the event fires @user.status = res[:status] @user.save } end end def ajax_call # Must use database for state and not Nanite job end
  19. 19. Allocation of work • The pings are used by the mappers to find the healthiest nanite agents • If a nanite agent doesn’t ping inside a window it is removed from the active list (until it does ping again) • perfect for busy or error hit nanites • The default routing algorithm is based on server load
  20. 20. Routing options • In options argument of Nanite.request you can choose: • selector • :least_loaded (default), :all, :random, :rr • target - use this to target a specific agent • timeout - override the default timeout (60s)
  21. 21. Custom algorithms • Nanites report state with their ping • By default this is the server load • Can override this by adding code to agent init.rb • Nanite.status_proc = lambda { MyApp.some_statistic_indicating_load } • Must be comparable • Can use with existing routing algorithms or create own more complex ones
  22. 22. File transfers • Nanite can handle file transmission • Agents subscribe (for all actors) in init.rb or Actors subscribe individually: • Nanite.subscribe_to_files(domain) • Mappers send via: • Nanite.broadcast_file(filepath, opts) • where opts can contain :domain and :destination (destination filename)
  23. 23. The beauty of JSON • Nanite is built on top of AMQP, so if the queue items are serialised using JSON... • Then Nanite isn’t needed at the agent side • Any AMQP implementing daemon can read message and respond • Useful for legacy code (or legacy people)
  24. 24. Understanding Security • Security is implemented using RabbitMQ’s vhosts and username/passwords • The username/password is defined in config.yml for each nanite • Need to configure RabbitMQ using the rabbitmqctl command • Generally one vhost per application
  25. 25. Let’s play • cd <git_resources>/nanite/examples/myagent • ../../bin/nanite -t <identity> -h <host> • Let’s use fullname for identity - eg georgepalmer • Should see something like: # subscribing to file broadcasts for foobar # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/gems.rb # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/mock.rb # quot;advertise_servicesquot; # [quot;/gems/listquot;, quot;/mock/listquot;]
  26. 26. Resources • RabbitMQ: http://www.rabbitmq.com • Ruby AMQP with RabbitMQ tutorial: http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html • Nanite: http://github.com/ezmobius/nanite • Doc isn’t great, code is very readable

×