Background
Processing
   with Nanite




                 By Jennifer Davis
Overview
Overview
Overview

• Nanite
Overview

• Nanite
• RabbitMQ/AMQP
Overview

• Nanite
• RabbitMQ/AMQP
• Code Example
Overview

• Nanite
• RabbitMQ/AMQP
• Code Example
• Questions
Overview
Background Processing
Background Processing
Background Processing

Q. Don’t we already do background
processing with BackgrounDRb?
Background Processing

Q. Don’t we already do background
processing with BackgrounDRb?
A. Yes, but it is not scalable for ...
Background Processing
Background Processing
Background Processing

Q. Why Nanite?
Background Processing

Q. Why Nanite?
A. Although other methods of background
processing exist, Nanite is more suited to
o...
Background Processing
Nanite
Nanite
Nanite

“A Self Assembling Cluster of Ruby Daemons”
                                - Ezra Zygmuntowicz
Nanite
Nanite
Nanite
Composed of Mappers and Agents
Nanite
         Composed of Mappers and Agents
 Thin

Mapper
Nanite
         Composed of Mappers and Agents
 Thin

Mapper




 Shell

Mapper
Nanite
         Composed of Mappers and Agents
 Thin

           A
Mapper
           M          Message
                  ...
Nanite
         Composed of Mappers and Agents
                                          Agent
 Thin
                     ...
Nanite
         Composed of Mappers and Agents
                                          Agent
 Thin
                     ...
Nanite
         Composed of Mappers and Agents
                                          Agent
 Thin
                     ...
Nanite
         Composed of Mappers and Agents
                                          Agent
 Thin
                     ...
Nanite
Nanite
Mappers
Nanite
Mappers
• Can have one or many
Nanite
Mappers
• Can have one or many
• Run within Rails or the shell
Nanite
Mappers
• Can have one or many
• Run within Rails or the shell
• Subscribed to the Mapper Exchange
Nanite
Mappers
• Can have one or many
• Run within Rails or the shell
• Subscribed to the Mapper Exchange
• Adds/Removes a...
Nanite
Nanite
Nanite
Agents
Nanite
Agents
• Can have one or many
Nanite
Agents
• Can have one or many
• Ping the Mapper Exchange every
   @ping_time seconds
Nanite
Agents
• Can have one or many
• Ping the Mapper Exchange every
   @ping_time seconds
• A single agent can have many...
Nanite
Nanite
Nanite
Selectors for load balancing
Nanite
Selectors for load balancing
• :least_loaded
Nanite
Selectors for load balancing
• :least_loaded
• :rr
Nanite
Selectors for load balancing
• :least_loaded
• :rr
• :random
Nanite
Selectors for load balancing
• :least_loaded
• :rr
• :random
• :all
Nanite
Nanite
                               Agent
 Thin
                               Actor
         A                 A
Mapper...
Nanite
                               Agent
 Thin
                               Actor
         A                 A
Mapper...
Nanite
                           Ping back status   Agent
 Thin
                                              Actor
     ...
Nanite
                               Agent
 Thin
                               Actor
         A                 A
Mapper...
Nanite
Nanite
Nanite requires
Nanite
Nanite requires
• Erlang
Nanite
Nanite requires
• Erlang
• Nanite ruby gem
Nanite
Nanite requires
• Erlang
• Nanite ruby gem
• EventMachine
Nanite
Nanite requires
• Erlang
• Nanite ruby gem
• EventMachine
• AMQP
Nanite
Nanite requires
• Erlang
• Nanite ruby gem
• EventMachine
• AMQP
• RabbitMQ
Nanite
AMQP
AMQP
AMQP
Advanced Messaging Queue Protocol
AMQP
Advanced Messaging Queue Protocol
• An open Internet Protocol for Business
   Messaging
AMQP
Advanced Messaging Queue Protocol
• An open Internet Protocol for Business
   Messaging
• Provides a standard for mes...
AMQP
RabbitMQ
RabbitMQ
RabbitMQ
• An AMQP Messaging Broker
RabbitMQ
• An AMQP Messaging Broker
• Written in Erlang
RabbitMQ
• An AMQP Messaging Broker
• Written in Erlang
• Uses the OTP (Open Telecom Platform)
  system
RabbitMQ
• An AMQP Messaging Broker
• Written in Erlang
• Uses the OTP (Open Telecom Platform)
  system
• Is stable - Achi...
RabbitMQ
Code Example
Review
Review

• Why background processing is important
  to us
• What Nanite is and how to use it
• A little background on Rabbi...
Review
Review

• Why background processing is important
  to us
• What Nanite is and how to use it
• A little background on Rabbi...
Resources
Resources
•   Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/
    nanite/tree/master/
Resources
•   Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/
    nanite/tree/master/

•   George Palm...
Resources
•   Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/
    nanite/tree/master/

•   George Palm...
Resources
•   Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/
    nanite/tree/master/

•   George Palm...
Resources
Questions?
Upcoming SlideShare
Loading in …5
×

Background Processing with Nanite

4,129 views

Published on

A short presentation I gave to my team regarding Nanite and the benefits of using it for background processing.

0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,129
On SlideShare
0
From Embeds
0
Number of Embeds
239
Actions
Shares
0
Downloads
138
Comments
0
Likes
7
Embeds 0
No embeds

No notes for slide











  • BackgrounDRb
    BackgroundJob
    Starling with Workling
  • BackgrounDRb
    BackgroundJob
    Starling with Workling
  • BackgrounDRb
    BackgroundJob
    Starling with Workling

  • It’s highly scalable and provides failover. Workers can easily be removed and added therefore adding to the reliability and availability of the application. It relies on message queueing and provides load balancing.
  • It’s highly scalable and provides failover. Workers can easily be removed and added therefore adding to the reliability and availability of the application. It relies on message queueing and provides load balancing.
































  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents


  • The control nodes for Nanite
    Right now we only have one
    All mappers are copied the ping times from the agents







  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on

  • The default status advertised is the load average as a float. Used for determining the default request dispatching based on the least loaded server.
    The status can be changed as long as it’s something that’s comparable
    Selectors include :rr, :random, :all, :least_loaded (default)

    Return state - default is the load of the server they’re currently running on













  • By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organisations in a timely manner.
    The \"exchange\" receives messages from publisher applications and routes these to \"message queues\", based on arbitrary criteria - usually message properties or content.
    The \"message queue\" stores messages until they can be safely processed by a consuming client application (or multiple applications).
    The \"binding\" defines the relationship between a message queue and an exchange and provides the message routing criteria.

    Standard proposed by industrial banks
    High performance
    Nanite implements AMQP
  • By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organisations in a timely manner.
    The \"exchange\" receives messages from publisher applications and routes these to \"message queues\", based on arbitrary criteria - usually message properties or content.
    The \"message queue\" stores messages until they can be safely processed by a consuming client application (or multiple applications).
    The \"binding\" defines the relationship between a message queue and an exchange and provides the message routing criteria.

    Standard proposed by industrial banks
    High performance
    Nanite implements AMQP
  • By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organisations in a timely manner.
    The \"exchange\" receives messages from publisher applications and routes these to \"message queues\", based on arbitrary criteria - usually message properties or content.
    The \"message queue\" stores messages until they can be safely processed by a consuming client application (or multiple applications).
    The \"binding\" defines the relationship between a message queue and an exchange and provides the message routing criteria.

    Standard proposed by industrial banks
    High performance
    Nanite implements AMQP
  • By complying to the AMQP standard, middleware products written for different platforms and in different languages can send messages to one another. AMQP addresses the problem of transporting value-bearing messages across and between organisations in a timely manner.
    The \"exchange\" receives messages from publisher applications and routes these to \"message queues\", based on arbitrary criteria - usually message properties or content.
    The \"message queue\" stores messages until they can be safely processed by a consuming client application (or multiple applications).
    The \"binding\" defines the relationship between a message queue and an exchange and provides the message routing criteria.

    Standard proposed by industrial banks
    High performance
    Nanite implements AMQP





  • Background Processing with Nanite

    1. 1. Background Processing with Nanite By Jennifer Davis
    2. 2. Overview
    3. 3. Overview
    4. 4. Overview • Nanite
    5. 5. Overview • Nanite • RabbitMQ/AMQP
    6. 6. Overview • Nanite • RabbitMQ/AMQP • Code Example
    7. 7. Overview • Nanite • RabbitMQ/AMQP • Code Example • Questions
    8. 8. Overview
    9. 9. Background Processing
    10. 10. Background Processing
    11. 11. Background Processing Q. Don’t we already do background processing with BackgrounDRb?
    12. 12. Background Processing Q. Don’t we already do background processing with BackgrounDRb? A. Yes, but it is not scalable for our needs.
    13. 13. Background Processing
    14. 14. Background Processing
    15. 15. Background Processing Q. Why Nanite?
    16. 16. Background Processing Q. Why Nanite? A. Although other methods of background processing exist, Nanite is more suited to our needs.
    17. 17. Background Processing
    18. 18. Nanite
    19. 19. Nanite
    20. 20. Nanite “A Self Assembling Cluster of Ruby Daemons” - Ezra Zygmuntowicz
    21. 21. Nanite
    22. 22. Nanite
    23. 23. Nanite Composed of Mappers and Agents
    24. 24. Nanite Composed of Mappers and Agents Thin Mapper
    25. 25. Nanite Composed of Mappers and Agents Thin Mapper Shell Mapper
    26. 26. Nanite Composed of Mappers and Agents Thin A Mapper M Message Queue Q (RabbitMQ) P Shell Mapper
    27. 27. Nanite Composed of Mappers and Agents Agent Thin Actor A Mapper Actor M Message Actor Queue Q (RabbitMQ) P Shell Mapper
    28. 28. Nanite Composed of Mappers and Agents Agent Thin Actor A Mapper Actor M Message Actor Queue Q (RabbitMQ) P Agent Shell Actor Mapper Actor Actor
    29. 29. Nanite Composed of Mappers and Agents Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
    30. 30. Nanite Composed of Mappers and Agents Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
    31. 31. Nanite
    32. 32. Nanite Mappers
    33. 33. Nanite Mappers • Can have one or many
    34. 34. Nanite Mappers • Can have one or many • Run within Rails or the shell
    35. 35. Nanite Mappers • Can have one or many • Run within Rails or the shell • Subscribed to the Mapper Exchange
    36. 36. Nanite Mappers • Can have one or many • Run within Rails or the shell • Subscribed to the Mapper Exchange • Adds/Removes agents based on agent availability
    37. 37. Nanite
    38. 38. Nanite
    39. 39. Nanite Agents
    40. 40. Nanite Agents • Can have one or many
    41. 41. Nanite Agents • Can have one or many • Ping the Mapper Exchange every @ping_time seconds
    42. 42. Nanite Agents • Can have one or many • Ping the Mapper Exchange every @ping_time seconds • A single agent can have many actor classes
    43. 43. Nanite
    44. 44. Nanite
    45. 45. Nanite Selectors for load balancing
    46. 46. Nanite Selectors for load balancing • :least_loaded
    47. 47. Nanite Selectors for load balancing • :least_loaded • :rr
    48. 48. Nanite Selectors for load balancing • :least_loaded • :rr • :random
    49. 49. Nanite Selectors for load balancing • :least_loaded • :rr • :random • :all
    50. 50. Nanite
    51. 51. Nanite Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
    52. 52. Nanite Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
    53. 53. Nanite Ping back status Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
    54. 54. Nanite Agent Thin Actor A A Mapper Actor M M Message Actor Queue Q Q (RabbitMQ) P P Agent Shell Actor Mapper Actor Actor
    55. 55. Nanite
    56. 56. Nanite Nanite requires
    57. 57. Nanite Nanite requires • Erlang
    58. 58. Nanite Nanite requires • Erlang • Nanite ruby gem
    59. 59. Nanite Nanite requires • Erlang • Nanite ruby gem • EventMachine
    60. 60. Nanite Nanite requires • Erlang • Nanite ruby gem • EventMachine • AMQP
    61. 61. Nanite Nanite requires • Erlang • Nanite ruby gem • EventMachine • AMQP • RabbitMQ
    62. 62. Nanite
    63. 63. AMQP
    64. 64. AMQP
    65. 65. AMQP Advanced Messaging Queue Protocol
    66. 66. AMQP Advanced Messaging Queue Protocol • An open Internet Protocol for Business Messaging
    67. 67. AMQP Advanced Messaging Queue Protocol • An open Internet Protocol for Business Messaging • Provides a standard for messaging middleware
    68. 68. AMQP
    69. 69. RabbitMQ
    70. 70. RabbitMQ
    71. 71. RabbitMQ • An AMQP Messaging Broker
    72. 72. RabbitMQ • An AMQP Messaging Broker • Written in Erlang
    73. 73. RabbitMQ • An AMQP Messaging Broker • Written in Erlang • Uses the OTP (Open Telecom Platform) system
    74. 74. RabbitMQ • An AMQP Messaging Broker • Written in Erlang • Uses the OTP (Open Telecom Platform) system • Is stable - Achieved 99.9999999% availability
    75. 75. RabbitMQ
    76. 76. Code Example
    77. 77. Review
    78. 78. Review • Why background processing is important to us • What Nanite is and how to use it • A little background on RabbitMQ
    79. 79. Review
    80. 80. Review • Why background processing is important to us • What Nanite is and how to use it • A little background on RabbitMQ
    81. 81. Resources
    82. 82. Resources • Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/ nanite/tree/master/
    83. 83. Resources • Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/ nanite/tree/master/ • George Palmer’s RubyManor Nanite presentation at http://rubymanor.org/ videos/nanite/
    84. 84. Resources • Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/ nanite/tree/master/ • George Palmer’s RubyManor Nanite presentation at http://rubymanor.org/ videos/nanite/ • RabbitMQ - Open Source Enterprise Messaging at http:// www.rabbitmq.com/
    85. 85. Resources • Ezra Zygmuntowicz’s Nanite on Github at http://github.com/ezmobius/ nanite/tree/master/ • George Palmer’s RubyManor Nanite presentation at http://rubymanor.org/ videos/nanite/ • RabbitMQ - Open Source Enterprise Messaging at http:// www.rabbitmq.com/ • AMQP - Advanced Message Queuing Protocol at http://jira.amqp.org/ confluence/display/AMQP/Advanced+Message+Queuing+Protocol
    86. 86. Resources
    87. 87. Questions?

    ×