SlideShare a Scribd company logo
1 of 35
Download to read offline
Vetebra
Nanite
George Palmer
Background processing

• Work should be moved to the background
  to stop application server load
• This keeps website responsive
• Useful for: images, videos, web services,
  slow database queries....anything that isn’t
  quick!
Current background
       offerings
• Fork into background of rails process
  •   eg spawn, run_later

• Record to database (or file) and
  background daemon picks up from there
  •   eg background_job, delayed_job

• Fork onto some kind of queue
  •   eg backgroundRB, beanstalk, starling with workling
Introducing Nanite

• Developed by EngineYard
• Gives presence - we know what’s available
• Handles failure
• Instant scalability
• Event based architecture
Pre-reqs

• Erlang
 • Not R12B5 (the version installed by
    MacPorts)
• RabbitMQ
 • Untar to erlang lib directory - most
    likely /usr/local/lib/erlang/lib
AMQP
• Advanced Message Queuing Protocol
• Enterprise quality protocol developed by
  several financial institutions
• Includes wire protocol to ensure language
  neutral
• RabbitMQ implements AMQP 0.8
Installing AMQP for Ruby


$ sudo gem install eventmachine
$ git clone git://github.com/tmm1/amqp.git
$ cd amqp && rake gem && sudo gem install
  amqp-<version>.gem
Nanite
         NA                         A   Nanite
   Thin                                 Actor
          a M                       M
   App
         ni Q                       Q   Nanite
  Server
         te P                       P   Actor

Nanite Mappers   m e s s a g e s   Nanite Agents

         NA                         A   Nanite
  Ruby a M                          M   Actor
  Script ni Q                       Q   Nanite
         te P                       P   Actor
Nanite Mappers
• Control and track work
• Unlimited number can be run that get
  updates from mapper exchange
 •   mapper exchange itself is just a heartbeat and
     registration MQ

• Run either inside Rails/MERB app (on Thin)
  or via command line
Nanite Agents
• Do the work
• A given nanite agent can have multiple
  actors
• Scale by adding more agents
• Pings the mapper exchange every
  @ping_time seconds to report health
Nanite Actors

class Manor < Nanite::Actor
 expose :name

 def name(vars)
  # Do something interesting here
  :result => “RubyManor”
 end
end

Nanite::Dispatcher.register(Manor.new)
Agent directory
        structure
+ myagent
 + actors
  - manor.rb
 + files
 - init.rb
 - config.yml
Agent config.yml
---
:vhost: /nanite   # Allow multiple agents with different queues [compulsory]
:user: nanite     # Username for queue
:pass: testing    # Password for queue
:identity: barney # Can be auto-generated but useful to send work to specific
                  # agents
:file_root: path   # where to store any transfered files
:format: marshal # or :json


# Additional options include host and port. All options can be passed into
# nanite command so can avoid config file if want
Getting Started...

• Start RabbitMQ
 •   /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq-
     server

• On first run nanite/bin/rabbitconf
 • Sets up RabbitMQ with a vhost and users
     for that vhost (more on this later)
Starting agents
• cd <agentdir> && <nanite basedir>/bin/
  nanite
• I’ve been using:
  cd <agentdir> && <nanite basedir>/bin/
  nanite -t <identity> &
• Could be managed better through a
  daemon/monitoring system though
Offloading work to
        Nanite
• Use the following code:
  Nanite.request(callable, params = ‘’,   options =
  {}) {|res| # use res to do something}


 •   callable is the actor and method - eg ‘/manor/
     name’

 •   params are parameters for the callable method -
     eg ‘2008’

 •   options includes timeout, target and the routing
     algorithm - more on this later
Interfaces into the
         Mappers
• Via console:
 • nanite/nanite-mapper -i
• Via command line:
 • See nanite/examples/cli.rb
• Via Rails/MERB app:
 • See next slide....
Rails/MERB & Nanite
# Updates the user
def update
 ...
 if (@user.save)
    Nanite.request(‘/updates/twitter’, ‘georgio_1999’) {|res|
      # This block won’t execute until the event fires
      @user.status = res[:status]
      @user.save
    }
 end
end

def ajax_call
 # Must use database for state and not Nanite job
end
Allocation of work
• The pings are used by the mappers to find
  the healthiest nanite agents
• If a nanite agent doesn’t ping inside a
  window it is removed from the active list
  (until it does ping again)
  • perfect for busy or error hit nanites
• The default routing algorithm is based on
  server load
Routing options
• In options argument of        Nanite.request   you
  can choose:

 • selector
   •   :least_loaded (default), :all, :random, :rr

 • target - use this to target a specific agent
 • timeout - override the default timeout
    (60s)
Custom algorithms
• Nanites report state with their ping
  •   By default this is the server load

• Can override this by adding code to agent
  init.rb
  •   Nanite.status_proc = lambda
      { MyApp.some_statistic_indicating_load }


  •   Must be comparable

• Can use with existing routing algorithms or
  create own more complex ones
File transfers
• Nanite can handle file transmission
 • Agents subscribe (for all actors) in init.rb
    or Actors subscribe individually:
    •   Nanite.subscribe_to_files(domain)


  • Mappers send via:
    •   Nanite.broadcast_file(filepath, opts)


    •   where opts can contain :domain
        and :destination (destination filename)
The beauty of JSON
• Nanite is built on top of AMQP, so if the
  queue items are serialised using JSON...
  • Then Nanite isn’t needed at the agent
    side
  • Any AMQP implementing daemon can
    read message and respond
  • Useful for legacy code (or legacy people)
Understanding Security
• Security is implemented using RabbitMQ’s
  vhosts and username/passwords
• The username/password is defined in
  config.yml for each nanite
• Need to configure RabbitMQ using the
  rabbitmqctl command

• Generally one vhost per application
Let’s play
• cd <git_resources>/nanite/examples/myagent
• ../../bin/nanite -t <identity> -h <host>
    •      Let’s use fullname for identity - eg georgepalmer

    • Should see something like:
  # subscribing to file broadcasts for foobar

  # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/gems.rb

  # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/mock.rb

  # quot;advertise_servicesquot;

  # [quot;/gems/listquot;, quot;/mock/listquot;]
Resources

• RabbitMQ: http://www.rabbitmq.com
• Ruby AMQP with RabbitMQ tutorial:
  http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html




• Nanite: http://github.com/ezmobius/nanite
    •     Doc isn’t great, code is very readable

More Related Content

Similar to Rubymanor - Nanite talk

XMPP & AMQP
XMPP & AMQPXMPP & AMQP
XMPP & AMQP
voluntas
 
Automated Performance Testing With J Meter And Maven
Automated  Performance  Testing With  J Meter And  MavenAutomated  Performance  Testing With  J Meter And  Maven
Automated Performance Testing With J Meter And Maven
PerconaPerformance
 
Openshift: Deployments for the rest of us
Openshift: Deployments for the rest of usOpenshift: Deployments for the rest of us
Openshift: Deployments for the rest of us
Anurag Patel
 

Similar to Rubymanor - Nanite talk (20)

Security testing with gauntlt
Security testing with gauntltSecurity testing with gauntlt
Security testing with gauntlt
 
Merb + Nanite
Merb + NaniteMerb + Nanite
Merb + Nanite
 
Zen map
Zen mapZen map
Zen map
 
Be Mean to Your Code
Be Mean to Your CodeBe Mean to Your Code
Be Mean to Your Code
 
Nmap
NmapNmap
Nmap
 
AMS Node Meetup December presentation Phusion Passenger
AMS Node Meetup December presentation Phusion PassengerAMS Node Meetup December presentation Phusion Passenger
AMS Node Meetup December presentation Phusion Passenger
 
The Current State of Asynchronous Processing With Ruby
The Current State of Asynchronous Processing With RubyThe Current State of Asynchronous Processing With Ruby
The Current State of Asynchronous Processing With Ruby
 
Network Mapper (NMAP)
Network Mapper (NMAP)Network Mapper (NMAP)
Network Mapper (NMAP)
 
Nmap
NmapNmap
Nmap
 
Combining the strength of erlang and Ruby
Combining the strength of erlang and RubyCombining the strength of erlang and Ruby
Combining the strength of erlang and Ruby
 
Combining the Strengths or Erlang and Ruby
Combining the Strengths or Erlang and RubyCombining the Strengths or Erlang and Ruby
Combining the Strengths or Erlang and Ruby
 
Neutron Network Namespaces and IPtables--A Technical Deep Dive
Neutron Network Namespaces and IPtables--A Technical Deep DiveNeutron Network Namespaces and IPtables--A Technical Deep Dive
Neutron Network Namespaces and IPtables--A Technical Deep Dive
 
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
 
XMPP & AMQP
XMPP & AMQPXMPP & AMQP
XMPP & AMQP
 
Automated Performance Testing With J Meter And Maven
Automated  Performance  Testing With  J Meter And  MavenAutomated  Performance  Testing With  J Meter And  Maven
Automated Performance Testing With J Meter And Maven
 
Nmap basics-1198948509608024-3
Nmap basics-1198948509608024-3Nmap basics-1198948509608024-3
Nmap basics-1198948509608024-3
 
Openshift: Deployments for the rest of us
Openshift: Deployments for the rest of usOpenshift: Deployments for the rest of us
Openshift: Deployments for the rest of us
 
Null Delhi chapter - Feb 2019
Null Delhi chapter - Feb 2019Null Delhi chapter - Feb 2019
Null Delhi chapter - Feb 2019
 
Recon with Nmap
Recon with Nmap Recon with Nmap
Recon with Nmap
 
The Art of Grey-Box Attack
The Art of Grey-Box AttackThe Art of Grey-Box Attack
The Art of Grey-Box Attack
 

Recently uploaded

Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
FIDO Alliance
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
panagenda
 

Recently uploaded (20)

TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024TopCryptoSupers 12thReport OrionX May2024
TopCryptoSupers 12thReport OrionX May2024
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdfIntroduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
Introduction to FDO and How It works Applications _ Richard at FIDO Alliance.pdf
 
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
1111 ChatGPT Prompts PDF Free Download - Prompts for ChatGPT
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The InsideCollecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
Collecting & Temporal Analysis of Behavioral Web Data - Tales From The Inside
 
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
Choosing the Right FDO Deployment Model for Your Application _ Geoffrey at In...
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
Using IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & IrelandUsing IESVE for Room Loads Analysis - UK & Ireland
Using IESVE for Room Loads Analysis - UK & Ireland
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Your enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4jYour enemies use GenAI too - staying ahead of fraud with Neo4j
Your enemies use GenAI too - staying ahead of fraud with Neo4j
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
ADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptxADP Passwordless Journey Case Study.pptx
ADP Passwordless Journey Case Study.pptx
 
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdfWhere to Learn More About FDO _ Richard at FIDO Alliance.pdf
Where to Learn More About FDO _ Richard at FIDO Alliance.pdf
 
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
Easier, Faster, and More Powerful – Alles Neu macht der Mai -Wir durchleuchte...
 
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdfThe Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
 

Rubymanor - Nanite talk

  • 2. Background processing • Work should be moved to the background to stop application server load • This keeps website responsive • Useful for: images, videos, web services, slow database queries....anything that isn’t quick!
  • 3.
  • 4.
  • 5. Current background offerings • Fork into background of rails process • eg spawn, run_later • Record to database (or file) and background daemon picks up from there • eg background_job, delayed_job • Fork onto some kind of queue • eg backgroundRB, beanstalk, starling with workling
  • 6. Introducing Nanite • Developed by EngineYard • Gives presence - we know what’s available • Handles failure • Instant scalability • Event based architecture
  • 7. Pre-reqs • Erlang • Not R12B5 (the version installed by MacPorts) • RabbitMQ • Untar to erlang lib directory - most likely /usr/local/lib/erlang/lib
  • 8. AMQP • Advanced Message Queuing Protocol • Enterprise quality protocol developed by several financial institutions • Includes wire protocol to ensure language neutral • RabbitMQ implements AMQP 0.8
  • 9. Installing AMQP for Ruby $ sudo gem install eventmachine $ git clone git://github.com/tmm1/amqp.git $ cd amqp && rake gem && sudo gem install amqp-<version>.gem
  • 10.
  • 11. Nanite NA A Nanite Thin Actor a M M App ni Q Q Nanite Server te P P Actor Nanite Mappers m e s s a g e s Nanite Agents NA A Nanite Ruby a M M Actor Script ni Q Q Nanite te P P Actor
  • 12. Nanite Mappers • Control and track work • Unlimited number can be run that get updates from mapper exchange • mapper exchange itself is just a heartbeat and registration MQ • Run either inside Rails/MERB app (on Thin) or via command line
  • 13. Nanite Agents • Do the work • A given nanite agent can have multiple actors • Scale by adding more agents • Pings the mapper exchange every @ping_time seconds to report health
  • 14. Nanite Actors class Manor < Nanite::Actor expose :name def name(vars) # Do something interesting here :result => “RubyManor” end end Nanite::Dispatcher.register(Manor.new)
  • 15. Agent directory structure + myagent + actors - manor.rb + files - init.rb - config.yml
  • 16. Agent config.yml --- :vhost: /nanite # Allow multiple agents with different queues [compulsory] :user: nanite # Username for queue :pass: testing # Password for queue :identity: barney # Can be auto-generated but useful to send work to specific # agents :file_root: path # where to store any transfered files :format: marshal # or :json # Additional options include host and port. All options can be passed into # nanite command so can avoid config file if want
  • 17.
  • 18. Getting Started... • Start RabbitMQ • /usr/local/lib/erlang/lib/rabbitmq/sbin/rabbitmq- server • On first run nanite/bin/rabbitconf • Sets up RabbitMQ with a vhost and users for that vhost (more on this later)
  • 19. Starting agents • cd <agentdir> && <nanite basedir>/bin/ nanite • I’ve been using: cd <agentdir> && <nanite basedir>/bin/ nanite -t <identity> & • Could be managed better through a daemon/monitoring system though
  • 20. Offloading work to Nanite • Use the following code: Nanite.request(callable, params = ‘’, options = {}) {|res| # use res to do something} • callable is the actor and method - eg ‘/manor/ name’ • params are parameters for the callable method - eg ‘2008’ • options includes timeout, target and the routing algorithm - more on this later
  • 21. Interfaces into the Mappers • Via console: • nanite/nanite-mapper -i • Via command line: • See nanite/examples/cli.rb • Via Rails/MERB app: • See next slide....
  • 22. Rails/MERB & Nanite # Updates the user def update ... if (@user.save) Nanite.request(‘/updates/twitter’, ‘georgio_1999’) {|res| # This block won’t execute until the event fires @user.status = res[:status] @user.save } end end def ajax_call # Must use database for state and not Nanite job end
  • 23.
  • 24. Allocation of work • The pings are used by the mappers to find the healthiest nanite agents • If a nanite agent doesn’t ping inside a window it is removed from the active list (until it does ping again) • perfect for busy or error hit nanites • The default routing algorithm is based on server load
  • 25. Routing options • In options argument of Nanite.request you can choose: • selector • :least_loaded (default), :all, :random, :rr • target - use this to target a specific agent • timeout - override the default timeout (60s)
  • 26. Custom algorithms • Nanites report state with their ping • By default this is the server load • Can override this by adding code to agent init.rb • Nanite.status_proc = lambda { MyApp.some_statistic_indicating_load } • Must be comparable • Can use with existing routing algorithms or create own more complex ones
  • 27.
  • 28. File transfers • Nanite can handle file transmission • Agents subscribe (for all actors) in init.rb or Actors subscribe individually: • Nanite.subscribe_to_files(domain) • Mappers send via: • Nanite.broadcast_file(filepath, opts) • where opts can contain :domain and :destination (destination filename)
  • 29.
  • 30. The beauty of JSON • Nanite is built on top of AMQP, so if the queue items are serialised using JSON... • Then Nanite isn’t needed at the agent side • Any AMQP implementing daemon can read message and respond • Useful for legacy code (or legacy people)
  • 31.
  • 32. Understanding Security • Security is implemented using RabbitMQ’s vhosts and username/passwords • The username/password is defined in config.yml for each nanite • Need to configure RabbitMQ using the rabbitmqctl command • Generally one vhost per application
  • 33.
  • 34. Let’s play • cd <git_resources>/nanite/examples/myagent • ../../bin/nanite -t <identity> -h <host> • Let’s use fullname for identity - eg georgepalmer • Should see something like: # subscribing to file broadcasts for foobar # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/gems.rb # loading actor: /Users/georgepalmer/work/git_resources/nanite/examples/myagent/actors/mock.rb # quot;advertise_servicesquot; # [quot;/gems/listquot;, quot;/mock/listquot;]
  • 35. Resources • RabbitMQ: http://www.rabbitmq.com • Ruby AMQP with RabbitMQ tutorial: http://hopper.squarespace.com/blog/2008/7/22/simple-amqp-library-for-ruby.html • Nanite: http://github.com/ezmobius/nanite • Doc isn’t great, code is very readable