Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Rubymanor - Nanite talk

4,946 views

Published on

Published in: Technology
  • Be the first to comment

Rubymanor - Nanite talk

  1. 1. Vetebra Nanite George Palmer
  2. 2. Background processing • Work should be moved to the background to stop application server load • This keeps website responsive • Useful for: images, videos, web services, slow database queries....anything that isn’t quick!
  3. 3. Current background offerings • Fork into background of rails process • eg spawn, run_later • Record to database (or file) and background daemon picks up from there • eg background_job, delayed_job • Fork onto some kind of queue • eg backgroundRB, beanstalk, starling with workling
  4. 4. Introducing Nanite • Developed by EngineYard • Gives presence - we know what’s available • Handles failure • Instant scalability • Event based architecture
  5. 5. Pre-reqs • Erlang • Not R12B5 (the version installed by MacPorts) • RabbitMQ • Untar to erlang lib directory - most likely /usr/local/lib/erlang/lib
  6. 6. AMQP • Advanced Message Queuing Protocol • Enterprise quality protocol developed by several financial institutions • Includes wire protocol to ensure language neutral • RabbitMQ implements AMQP 0.8
  7. 7. Installing AMQP for Ruby $ sudo gem install eventmachine $ git clone git://github.com/tmm1/amqp.git $ cd amqp && rake gem && sudo gem install amqp-<version>.gem
  8. 8. Nanite NA A Nanite Thin Actor a M M App ni Q Q Nanite Server te P P Actor Nanite Mappers m e s s a g e s Nanite Agents NA A Nanite Ruby a M M Actor Script ni Q Q Nanite te P P Actor
  9. 9. Nanite Mappers • Control and track work • Unlimited number can be run that get updates from mapper exchange • mapper exchange itself is just a heartbeat and registration MQ • Run either inside Rails/MERB app (on Thin) or via command line
  10. 10. Nanite Agents • Do the work • A given nanite agent can have multiple actors • Scale by adding more agents • Pings the mapper exchange every @ping_time seconds to report health
  11. 11. Nanite Actors class Manor < Nanite::Actor expose :name def name(vars) # Do something interesting here :result => “RubyManor” end end Nanite::Dispatcher.register(Manor.new)
  12. 12. Agent directory structure + myagent + actors - manor.rb + files - init.rb - config.yml
  13. 13. Agent config.yml --- :vhost: /nanite # Allow multiple agents with different queues [compulsory] :user: nanite # Username for queue :pass: testing # Password for queue :identity: barney # Can be auto-generated but useful to send work to specific # agents :file_root: path # where to store any transfered files :format: marshal # or :json # Additional options include host and port. All options can be passed into # nanite command so can avoid config file if want
  14. 14. Getting Started... • Start RabbitMQ • /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq- server • On first run nanite/bin/rabbitconf • Sets up RabbitMQ with a vhost and users for that vhost (more on this later)
  15. 15. Starting agents • cd <agentdir> && <nanite basedir>/bin/ nanite • I’ve been using: cd <agentdir> && <nanite basedir>/bin/ nanite -t <identity> & • Could be managed better through a daemon/monitoring system though
  16. 16. Offloading work to Nanite • Use the following code: Nanite.request(callable, params = ‘’, options = {}) {|res| # use res to do something} • callable is the actor and method - eg ‘/manor/ name’ • params are parameters for the callable method - eg ‘2008’ • options includes timeout, target and the routing algorithm - more on this later
  17. 17. Interfaces into the Mappers • Via console: • nanite/nanite-mapper -i • Via command line: • See nanite/examples/cli.rb • Via Rails/MERB app: • See next slide....
  18. 18. Rails/MERB & Nanite # Updates the user def update ... if (@user.save) Nanite.request(‘/updates/twitter’, ‘georgio_1999’) {|res| # This block won’t execute until the event fires @user.status = res[:status] @user.save } end end def ajax_call # Must use database for state and not Nanite job end
  19. 19. Allocation of work • The pings are used by the mappers to find the healthiest nanite agents • If a nanite agent doesn’t ping inside a window it is removed from the active list (until it does ping again) • perfect for busy or error hit nanites • The default routing algorithm is based on server load
  20. 20. Routing options • In options argument of Nanite.request you can choose: • selector • :least_loaded (default), :all, :random, :rr • target - use this to target a specific agent • timeout - override the default timeout (60s)
  21. 21. Custom algorithms • Nanites report state with their ping • By default this is the server load • Can override this by adding code to agent init.rb • Nanite.status_proc = lambda { MyApp.some_statistic_indicating_load } • Must be comparable • Can use with existing routing algorithms or create own more complex ones
  22. 22. File transfers • Nanite can handle file transmission • Agents subscribe (for all actors) in init.rb or Actors subscribe individually: • Nanite.subscribe_to_files(domain) • Mappers send via: • Nanite.broadcast_file(filepath, opts) • where opts can contain :domain and :destination (destination filename)
  23. 23. The beauty of JSON • Nanite is built on top of AMQP, so if the queue items are serialised using JSON... • Then Nanite isn’t needed at the agent side • Any AMQP implementing daemon can read message and respond • Useful for legacy code (or legacy people)
  24. 24. Understanding Security • Security is implemented using RabbitMQ’s vhosts and username/passwords • The username/password is defined in config.yml for each nanite • Need to configure RabbitMQ using the rabbitmqctl command • Generally one vhost per application
  25. 25. Let’s play • cd <git_resources>/nanite/examples/myagent • ../../bin/nanite -t <identity> -h <host> • Let’s use fullname for identity - eg georgepalmer • Should see something like: # subscribing to file broadcasts for foobar # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/gems.rb # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/mock.rb # quot;advertise_servicesquot; # [quot;/gems/listquot;, quot;/mock/listquot;]
  26. 26. Resources • RabbitMQ: http://www.rabbitmq.com • Ruby AMQP with RabbitMQ tutorial: http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html • Nanite: http://github.com/ezmobius/nanite • Doc isn’t great, code is very readable

×