Background Processing with Nanite

  • 3,726 views
Uploaded on

A short presentation I gave to my team regarding Nanite and the benefits of using it for background processing.

A short presentation I gave to my team regarding Nanite and the benefits of using it for background processing.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,726
On Slideshare
0
From Embeds
0
Number of Embeds
3

Actions

Shares
Downloads
135
Comments
0
Likes
7

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide











  • BackgrounDRb
    BackgroundJob
    Starling with Workling
  • BackgrounDRb
    BackgroundJob
    Starling with Workling
  • BackgrounDRb
    BackgroundJob
    Starling with Workling

  • It’s highly scalable and provides failover. Workers can easily be removed and added therefore adding to the reliability and availability of the application. It relies on message queueing and provides load balancing.
  • It’s highly scalable and provides failover. Workers can easily be removed and added therefore adding to the reliability and availability of the application. It relies on message queueing and provides load balancing.
































  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents







  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on













  • By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organisations in a timely manner.
    The \"exchange\" receives messages from publisher applications and routes these to \"message queues\", based on arbitrary criteria - usually message properties or content.
    The \"message queue\" stores messages until they can be safely processed by a consuming client application (or multiple applications).
    The \"binding\" defines the relationship between a message queue and an exchange and provides the message routing criteria.

    Standard proposed by industrial banks
    High performance
    Nanite implements AMQP
  • By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organisations in a timely manner.
    The \"exchange\" receives messages from publisher applications and routes these to \"message queues\", based on arbitrary criteria - usually message properties or content.
    The \"message queue\" stores messages until they can be safely processed by a consuming client application (or multiple applications).
    The \"binding\" defines the relationship between a message queue and an exchange and provides the message routing criteria.

    Standard proposed by industrial banks
    High performance
    Nanite implements AMQP
  • By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organisations in a timely manner.
    The \"exchange\" receives messages from publisher applications and routes these to \"message queues\", based on arbitrary criteria - usually message properties or content.
    The \"message queue\" stores messages until they can be safely processed by a consuming client application (or multiple applications).
    The \"binding\" defines the relationship between a message queue and an exchange and provides the message routing criteria.

    Standard proposed by industrial banks
    High performance
    Nanite implements AMQP
  • By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organisations in a timely manner.
    The \"exchange\" receives messages from publisher applications and routes these to \"message queues\", based on arbitrary criteria - usually message properties or content.
    The \"message queue\" stores messages until they can be safely processed by a consuming client application (or multiple applications).
    The \"binding\" defines the relationship between a message queue and an exchange and provides the message routing criteria.

    Standard proposed by industrial banks
    High performance
    Nanite implements AMQP





Transcript

  • 1. Background Processing with Nanite By Jennifer Davis
  • 2. Overview
  • 3. Overview
  • 4. Overview • Nanite
  • 5. Overview • Nanite • RabbitMQ/AMQP
  • 6. Overview • Nanite • RabbitMQ/AMQP • Code Example
  • 7. Overview • Nanite • RabbitMQ/AMQP • Code Example • Questions
  • 8. Overview
  • 9. Background Processing
  • 10. Background Processing
  • 11. Background Processing Q. Don’t we already do background processing with BackgrounDRb?
  • 12. Background Processing Q. Don’t we already do background processing with BackgrounDRb? A. Yes, but it is not scalable for our needs.
  • 13. Background Processing
  • 14. Background Processing
  • 15. Background Processing Q. Why Nanite?
  • 16. Background Processing Q. Why Nanite? A. Although other methods of background processing exist, Nanite is more suited to our needs.
  • 17. Background Processing
  • 18. Nanite
  • 19. Nanite
  • 20. Nanite “A Self Assembling Cluster of Ruby Daemons” - Ezra Zygmuntowicz
  • 21. Nanite
  • 22. Nanite
  • 23. Nanite Composed of Mappers and Agents
  • 24. Nanite Composed of Mappers and Agents Thin Mapper
  • 25. Nanite Composed of Mappers and Agents Thin Mapper Shell Mapper
  • 26. Nanite Composed of Mappers and Agents Thin A Mapper M Message Queue Q (RabbitMQ) P Shell Mapper
  • 27. Nanite Composed of Mappers and Agents Agent Thin Actor A Mapper Actor M Message Actor Queue Q (RabbitMQ) P Shell Mapper
  • 28. Nanite Composed of Mappers and Agents Agent Thin Actor A Mapper Actor M Message Actor Queue Q (RabbitMQ) P Agent Shell Actor Mapper Actor Actor
  • 29. Nanite Composed of Mappers and Agents Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
  • 30. Nanite Composed of Mappers and Agents Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
  • 31. Nanite
  • 32. Nanite Mappers
  • 33. Nanite Mappers • Can have one or many
  • 34. Nanite Mappers • Can have one or many • Run within Rails or the shell
  • 35. Nanite Mappers • Can have one or many • Run within Rails or the shell • Subscribed to the Mapper Exchange
  • 36. Nanite Mappers • Can have one or many • Run within Rails or the shell • Subscribed to the Mapper Exchange • Adds/Removes agents based on agent availability
  • 37. Nanite
  • 38. Nanite
  • 39. Nanite Agents
  • 40. Nanite Agents • Can have one or many
  • 41. Nanite Agents • Can have one or many • Ping the Mapper Exchange every @ping_time seconds
  • 42. Nanite Agents • Can have one or many • Ping the Mapper Exchange every @ping_time seconds • A single agent can have many actor classes
  • 43. Nanite
  • 44. Nanite
  • 45. Nanite Selectors for load balancing
  • 46. Nanite Selectors for load balancing • :least_loaded
  • 47. Nanite Selectors for load balancing • :least_loaded • :rr
  • 48. Nanite Selectors for load balancing • :least_loaded • :rr • :random
  • 49. Nanite Selectors for load balancing • :least_loaded • :rr • :random • :all
  • 50. Nanite
  • 51. Nanite Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
  • 52. Nanite Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
  • 53. Nanite Ping back status Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
  • 54. Nanite Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
  • 55. Nanite
  • 56. Nanite Nanite requires
  • 57. Nanite Nanite requires • Erlang
  • 58. Nanite Nanite requires • Erlang • Nanite ruby gem
  • 59. Nanite Nanite requires • Erlang • Nanite ruby gem • EventMachine
  • 60. Nanite Nanite requires • Erlang • Nanite ruby gem • EventMachine • AMQP
  • 61. Nanite Nanite requires • Erlang • Nanite ruby gem • EventMachine • AMQP • RabbitMQ
  • 62. Nanite
  • 63. AMQP
  • 64. AMQP
  • 65. AMQP Advanced Messaging Queue Protocol
  • 66. AMQP Advanced Messaging Queue Protocol • An open Internet Protocol for Business Messaging
  • 67. AMQP Advanced Messaging Queue Protocol • An open Internet Protocol for Business Messaging • Provides a standard for messaging middleware
  • 68. AMQP
  • 69. RabbitMQ
  • 70. RabbitMQ
  • 71. RabbitMQ • An AMQP Messaging Broker
  • 72. RabbitMQ • An AMQP Messaging Broker • Written in Erlang
  • 73. RabbitMQ • An AMQP Messaging Broker • Written in Erlang • Uses the OTP (Open Telecom Platform) system
  • 74. RabbitMQ • An AMQP Messaging Broker • Written in Erlang • Uses the OTP (Open Telecom Platform) system • Is stable - Achieved 99.9999999% availability
  • 75. RabbitMQ
  • 76. Code Example
  • 77. Review
  • 78. Review • Why background processing is important to us • What Nanite is and how to use it • A little background on RabbitMQ
  • 79. Review
  • 80. Review • Why background processing is important to us • What Nanite is and how to use it • A little background on RabbitMQ
  • 81. Resources
  • 82. Resources • Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/ nanite/tree/master/
  • 83. Resources • Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/ nanite/tree/master/ • George Palmer’s RubyManor Nanite presentation at http://rubymanor.org/ videos/nanite/
  • 84. Resources • Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/ nanite/tree/master/ • George Palmer’s RubyManor Nanite presentation at http://rubymanor.org/ videos/nanite/ • RabbitMQ - Open Source Enterprise Messaging at http:// www.rabbitmq.com/
  • 85. Resources • Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/ nanite/tree/master/ • George Palmer’s RubyManor Nanite presentation at http://rubymanor.org/ videos/nanite/ • RabbitMQ - Open Source Enterprise Messaging at http:// www.rabbitmq.com/ • AMQP - Advanced Message Queuing Protocol at http://jira.amqp.org/ confluence/display/AMQP/Advanced+Message+Queuing+Protocol
  • 86. Resources
  • 87. Questions?