Rubymanor - Nanite talk

Vetebra Nanite George Palmer

Background processing • Work should be moved to the background to stop application server load • This keeps website responsive • Useful for: images, videos, web services, slow database queries....anything that isn’t quick!

Current background offerings • Fork into background of rails process • eg spawn, run_later • Record to database (or ﬁle) and background daemon picks up from there • eg background_job, delayed_job • Fork onto some kind of queue • eg backgroundRB, beanstalk, starling with workling

Introducing Nanite • Developed by EngineYard • Gives presence - we know what’s available • Handles failure • Instant scalability • Event based architecture

Pre-reqs • Erlang • Not R12B5 (the version installed by MacPorts) • RabbitMQ • Untar to erlang lib directory - most likely /usr/local/lib/erlang/lib

AMQP • Advanced Message Queuing Protocol • Enterprise quality protocol developed by several ﬁnancial institutions • Includes wire protocol to ensure language neutral • RabbitMQ implements AMQP 0.8

Installing AMQP for Ruby $ sudo gem install eventmachine $ git clone git://github.com/tmm1/amqp.git $ cd amqp && rake gem && sudo gem install amqp-<version>.gem

Nanite NA A Nanite Thin Actor a M M App ni Q Q Nanite Server te P P Actor Nanite Mappers m e s s a g e s Nanite Agents NA A Nanite Ruby a M M Actor Script ni Q Q Nanite te P P Actor

Nanite Mappers • Control and track work • Unlimited number can be run that get updates from mapper exchange • mapper exchange itself is just a heartbeat and registration MQ • Run either inside Rails/MERB app (on Thin) or via command line

Nanite Agents • Do the work • A given nanite agent can have multiple actors • Scale by adding more agents • Pings the mapper exchange every @ping_time seconds to report health

Nanite Actors class Manor < Nanite::Actor expose :name def name(vars) # Do something interesting here :result => “RubyManor” end end Nanite::Dispatcher.register(Manor.new)

Agent directory structure + myagent + actors - manor.rb + ﬁles - init.rb - conﬁg.yml

Agent config.yml --- :vhost: /nanite # Allow multiple agents with different queues [compulsory] :user: nanite # Username for queue :pass: testing # Password for queue :identity: barney # Can be auto-generated but useful to send work to specific # agents :file_root: path # where to store any transfered files :format: marshal # or :json # Additional options include host and port. All options can be passed into # nanite command so can avoid config file if want

Getting Started... • Start RabbitMQ • /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq- server • On ﬁrst run nanite/bin/rabbitconf • Sets up RabbitMQ with a vhost and users for that vhost (more on this later)

Starting agents • cd <agentdir> && <nanite basedir>/bin/ nanite • I’ve been using: cd <agentdir> && <nanite basedir>/bin/ nanite -t <identity> & • Could be managed better through a daemon/monitoring system though

Ofﬂoading work to Nanite • Use the following code: Nanite.request(callable, params = ‘’, options = {}) {|res| # use res to do something} • callable is the actor and method - eg ‘/manor/ name’ • params are parameters for the callable method - eg ‘2008’ • options includes timeout, target and the routing algorithm - more on this later

Interfaces into the Mappers • Via console: • nanite/nanite-mapper -i • Via command line: • See nanite/examples/cli.rb • Via Rails/MERB app: • See next slide....

Rails/MERB & Nanite # Updates the user def update ... if (@user.save) Nanite.request(‘/updates/twitter’, ‘georgio_1999’) {|res| # This block won’t execute until the event ﬁres @user.status = res[:status] @user.save } end end def ajax_call # Must use database for state and not Nanite job end

Allocation of work • The pings are used by the mappers to ﬁnd the healthiest nanite agents • If a nanite agent doesn’t ping inside a window it is removed from the active list (until it does ping again) • perfect for busy or error hit nanites • The default routing algorithm is based on server load

Routing options • In options argument of Nanite.request you can choose: • selector • :least_loaded (default), :all, :random, :rr • target - use this to target a speciﬁc agent • timeout - override the default timeout (60s)

Custom algorithms • Nanites report state with their ping • By default this is the server load • Can override this by adding code to agent init.rb • Nanite.status_proc = lambda { MyApp.some_statistic_indicating_load } • Must be comparable • Can use with existing routing algorithms or create own more complex ones

File transfers • Nanite can handle ﬁle transmission • Agents subscribe (for all actors) in init.rb or Actors subscribe individually: • Nanite.subscribe_to_files(domain) • Mappers send via: • Nanite.broadcast_file(filepath, opts) • where opts can contain :domain and :destination (destination ﬁlename)

The beauty of JSON • Nanite is built on top of AMQP, so if the queue items are serialised using JSON... • Then Nanite isn’t needed at the agent side • Any AMQP implementing daemon can read message and respond • Useful for legacy code (or legacy people)

Understanding Security • Security is implemented using RabbitMQ’s vhosts and username/passwords • The username/password is defined in config.yml for each nanite • Need to configure RabbitMQ using the rabbitmqctl command • Generally one vhost per application

Let’s play • cd <git_resources>/nanite/examples/myagent • ../../bin/nanite -t <identity> -h <host> • Let’s use fullname for identity - eg georgepalmer • Should see something like: # subscribing to ﬁle broadcasts for foobar # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/gems.rb # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/mock.rb # quot;advertise_servicesquot; # [quot;/gems/listquot;, quot;/mock/listquot;]

Resources • RabbitMQ: http://www.rabbitmq.com • Ruby AMQP with RabbitMQ tutorial: http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html • Nanite: http://github.com/ezmobius/nanite • Doc isn’t great, code is very readable

http://syntaxritual.tumblr.com	32
http://www.rowtheboat.com	24
http://www.slideshare.net	13
http://rowtheboat.com	7
http://www.filestube.com	1
http://www.railsfire.com	1
http://safe.tumblr.com	1

Rubymanor - Nanite talk

by Georgio_1999 on Nov 22, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

7 Embeds 79

Statistics

Rubymanor - Nanite talk — Presentation Transcript