Just Do It Later...Procrastination in Ruby Philly.Rb – August 9, 2011
Hi I'm Rob <ul><li>Twitter: iotr
Github:  https://github.com/robdimarco/
Email: rob @ 416 software . com
More:  http://www.innovationontherun.com
This Presentation:  https://github.com/robdimarco/deferred_processing_talk </li></ul>
Out of Stream Processing
Always Looking to Speed Up User Experience
One Way to Get Faster..Do Less
Targets for Deferral <ul><li>Sending email
Image resizing
HTTP/FTP downloads
Updating indexes
Batch import / exports
Analytics </li></ul>
 
<ul>Thread.fork {do_stuff} </ul>
Using Threads <ul><li>Easy... But </li><ul><li>Process cannot exit before thread
Work kept in same process
Need to control thread count </li></ul><li>Use when </li><ul><li>Runtime makes sense within current process
Shared access to data is desired
Want communication with other threads </li></ul></ul>
<ul>pid = fork {do_stuff} Process.detach pid # Maybe?? </ul>
Using Forks <ul><li>Easy...But </li><ul><li>Need to control processes
Need to control output </li></ul><li>Use if </li><ul><li>Separate resources desired
Runtime outside of main process but on same machine </li></ul></ul>
Let's Take It Up A Notch Queueing!!!
“Queues” Queue Request Response Push Pop
Delayed::Job
Delayed::Job <ul><li>Brought to you by Shopify
Your code creates a Delayed::Job persisted object
Worker will go through and work off jobs </li></ul>
Installation <ul><li>Gemfile </li><ul><li>gem 'delayed_job' </li></ul><li>script/rails generate delayed_job </li><ul><li>C...
Adds script/delayed_job </li></ul><li>rake db:migrate </li></ul>
Using  delay <ul><li>Any method can be converted into a delayed job by calling  delay
User.find(1).follow(User.find(2)
becomes
User.find(1).delay.follow!(User.find(2)) </li></ul>
Using  enqueue <ul><li>Can create an object that defines a perform method
Enqueue with
Delayed::Job.enqueue(MyJob.new) </li></ul>
Delayed::Job Configuration <ul><li>On delay/enqueue </li><ul><li>:priority – a number
:run_at – a time </li></ul><li>As defaults </li><ul><li>sleep_delay
max_attempts
max_run_time
destroy_failed_jobs </li></ul></ul>
Persistence <ul><li>Saved to a data store
By default, uses ActiveRecord, other frameworks available
Stores job off using YAML </li><ul><li>!! Model objects may have been changed between enqueuing and processing !! </li></u...
Processing The Jobs <ul><li>Start the script/delayed_job </li><ul><li>Can pass in # of workers and priority
Upcoming SlideShare
Loading in …5
×

Deferred Processing in Ruby - Philly rb - August 2011

2,351 views
2,202 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,351
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
12
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Deferred Processing in Ruby - Philly rb - August 2011

  1. 1. Just Do It Later...Procrastination in Ruby Philly.Rb – August 9, 2011
  2. 2. Hi I'm Rob <ul><li>Twitter: iotr
  3. 3. Github: https://github.com/robdimarco/
  4. 4. Email: rob @ 416 software . com
  5. 5. More: http://www.innovationontherun.com
  6. 6. This Presentation: https://github.com/robdimarco/deferred_processing_talk </li></ul>
  7. 7. Out of Stream Processing
  8. 8. Always Looking to Speed Up User Experience
  9. 9. One Way to Get Faster..Do Less
  10. 10. Targets for Deferral <ul><li>Sending email
  11. 11. Image resizing
  12. 12. HTTP/FTP downloads
  13. 13. Updating indexes
  14. 14. Batch import / exports
  15. 15. Analytics </li></ul>
  16. 17. <ul>Thread.fork {do_stuff} </ul>
  17. 18. Using Threads <ul><li>Easy... But </li><ul><li>Process cannot exit before thread
  18. 19. Work kept in same process
  19. 20. Need to control thread count </li></ul><li>Use when </li><ul><li>Runtime makes sense within current process
  20. 21. Shared access to data is desired
  21. 22. Want communication with other threads </li></ul></ul>
  22. 23. <ul>pid = fork {do_stuff} Process.detach pid # Maybe?? </ul>
  23. 24. Using Forks <ul><li>Easy...But </li><ul><li>Need to control processes
  24. 25. Need to control output </li></ul><li>Use if </li><ul><li>Separate resources desired
  25. 26. Runtime outside of main process but on same machine </li></ul></ul>
  26. 27. Let's Take It Up A Notch Queueing!!!
  27. 28. “Queues” Queue Request Response Push Pop
  28. 29. Delayed::Job
  29. 30. Delayed::Job <ul><li>Brought to you by Shopify
  30. 31. Your code creates a Delayed::Job persisted object
  31. 32. Worker will go through and work off jobs </li></ul>
  32. 33. Installation <ul><li>Gemfile </li><ul><li>gem 'delayed_job' </li></ul><li>script/rails generate delayed_job </li><ul><li>Creates migration
  33. 34. Adds script/delayed_job </li></ul><li>rake db:migrate </li></ul>
  34. 35. Using delay <ul><li>Any method can be converted into a delayed job by calling delay
  35. 36. User.find(1).follow(User.find(2)
  36. 37. becomes
  37. 38. User.find(1).delay.follow!(User.find(2)) </li></ul>
  38. 39. Using enqueue <ul><li>Can create an object that defines a perform method
  39. 40. Enqueue with
  40. 41. Delayed::Job.enqueue(MyJob.new) </li></ul>
  41. 42. Delayed::Job Configuration <ul><li>On delay/enqueue </li><ul><li>:priority – a number
  42. 43. :run_at – a time </li></ul><li>As defaults </li><ul><li>sleep_delay
  43. 44. max_attempts
  44. 45. max_run_time
  45. 46. destroy_failed_jobs </li></ul></ul>
  46. 47. Persistence <ul><li>Saved to a data store
  47. 48. By default, uses ActiveRecord, other frameworks available
  48. 49. Stores job off using YAML </li><ul><li>!! Model objects may have been changed between enqueuing and processing !! </li></ul></ul>
  49. 50. Processing The Jobs <ul><li>Start the script/delayed_job </li><ul><li>Can pass in # of workers and priority
  50. 51. Uses daemons library </li></ul><li>Workers lock jobs by editing the object in the database
  51. 52. Errors are stored alongside the job </li></ul>
  52. 53. Operating Delayed::Job <ul><li>Need same code that you used to enqueue to process
  53. 54. One solitary queue with priorities, cannot create individual queues
  54. 55. Have had problem with workers processing long jobs not dying
  55. 56. Big queues can lead to database contention </li></ul>
  56. 57. What's To Like <ul><li>Simple installation
  57. 58. Minimally invasive to object model </li><ul><li>Any object can be queued </li></ul><li>Numeric priorities </li></ul>
  58. 59. Maybe Not Right For You If... <ul><li>Want multiple queues
  59. 60. Want “out of the box” GUI
  60. 61. Big queues
  61. 62. Lots of workers
  62. 63. Want to process outside your code base </li></ul>
  63. 64. Resque
  64. 65. Resque <ul><li>Brought to you by GitHub
  65. 66. Uses Redis to persist queues
  66. 67. Rake task for starting a worker
  67. 68. Sinatra app for monitoring queues, jobs, and workers. </li></ul>
  68. 69. Installation <ul><li>Install Redis
  69. 70. Gemfile </li><ul><li>gem 'resque'
  70. 71. require 'resque' </li></ul><li>Configuration </li><ul><li>config/initializers/resque.rb
  71. 72. config/resque.yml </li></ul></ul>
  72. 73. Enqueue <ul><li>Class that has class methods </li><ul><li>perform – Method to do the work
  73. 74. queue – The name of the queue to use (symbol/string) </li></ul><li>Enqueue with Resque.enqueue(class, *args) </li></ul>
  74. 75. More on Methods <ul><li>Arguments to enqueue are serialized to JSON </li><ul><li>Pro tip...only send in objects that can be encoded/decoded to JSON </li></ul><li>queue can be class attribute or class method
  75. 76. perform takes in arguments sent to enqueue </li></ul>
  76. 77. Resque Admin <ul><li>Sinatra application
  77. 78. Standalone or mounted in your application
  78. 79. Good snapshot of what is happening </li></ul>
  79. 80. About Redis <ul><li>Redis is an open source, advanced key-value store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets and sorted sets.
  80. 81. In-memory data set </li><ul><li>Can persist to disk </li></ul><li>Easy m aster-slave replication </li></ul>
  81. 82. Resque Uses Redis For... <ul><li>Storing list of queues in a set
  82. 83. Creating a list of jobs for each queue
  83. 84. Tracking workers
  84. 85. Maintaining stats </li></ul>
  85. 86. Enqueuing a User Follow <ul><li>Add queue and perform methods to the User class
  86. 87. Enqueue the job
  87. 88. Resque.enqueue User, 2, 1 </li></ul>
  88. 89. Process <ul>echo “require 'resque/tasks'” > lib/tasks/resque.rake QUEUE=* rake environment resque:work <li>Forks off child to do actual work </li><ul><li>Makes it easier to stop/start
  89. 90. Looks to see if you are using Ruby EE </li></ul></ul>
  90. 91. What's To Like <ul><li>Fast on/off queue
  91. 92. Large queues
  92. 93. Multiple queues
  93. 94. Nice admin tool
  94. 95. Workers resistant to job quirks </li></ul>
  95. 96. Maybe Not Right For You If... <ul><li>Want priorities
  96. 97. Do not want Redis
  97. 98. Do not want queues in RAM
  98. 99. Do not want JSON serialization
  99. 100. Want to manually query the job queue </li></ul>
  100. 101. Amazon SQS
  101. 102. What is SQS <ul><li>Web-based Message Queue </li><ul><li>Language agnostic </li></ul><li>On Amazon's Cloud </li><ul><li>High Availability
  102. 103. Scalable </li></ul><li>Queue level security </li></ul>
  103. 104. Using SQS <ul><li>Requires AWS Account </li><ul><li>Currently 100,000 messages / month free
  104. 105. $.01 / 10,000 messages after that </li></ul><li>All interactions over SOAP queries
  105. 106. Using RightAws gem to abstract this away </li></ul>
  106. 107. How to Queue Up Jobs <ul>sqs = RightAws::SqsGen2.new( ENV['ACCESS_KEY'], ENV['SECRET_KEY']) q = sqs.queue('my_queue_name') q.push('my data') </ul>
  107. 108. Processing <ul>sqs = RightAws::SqsGen2.new( ENV['ACCESS_KEY'], ENV['SECRET_KEY']) q = sqs.queue('my_queue_name') loop;process(q.pop);end </ul>
  108. 109. What's To Like <ul><li>Easy to process out of your application code </li><ul><li>E.g. processing needs to happen on Windows </li></ul><li>Easy to process out of your data center
  109. 110. Easy to add/remove queues
  110. 111. No additional services to manage </li></ul>
  111. 112. Maybe Not Right For You If... <ul><li>You do not want to figure out the serialization / de-serialization
  112. 113. You do not want to handle worker process management
  113. 114. You want to minimize processing time / job. </li></ul>
  114. 115. AMQP Messaging - The Big Gun
  115. 116. AMQP <ul><li>Advanced Message Queue Protocol
  116. 117. Protocol defining how messaging clients and brokers will interact
  117. 118. Open, c ross-vendor specification </li></ul>
  118. 119. ruby-amqp <ul><li>Requires EventMachine
  119. 120. Connects to an AMQP-compliant server </li><ul><li>Rabbit MQ
  120. 121. Apache Qpid </li></ul><li>Will need to specify host, channel, queue, and message </li></ul>
  121. 122. Look Into AMQP-Based System If <ul><li>Needs extend beyond just simple queue </li><ul><li>Publish/subscribe
  122. 123. Transactional messaging </li></ul><li>High volume / fast processing
  123. 124. Cross-systems </li></ul>
  124. 125. Other Options <ul><li>Beanstalkd </li><ul><li>Standalone server </li></ul><li>Starling </li><ul><li>From twitter, speaks memcached </li></ul><li>SimpleWorker </li><ul><li>SaaS job processing http://www.simpleworker.com/ </li></ul><li>Cloud Crowd </li><ul><li>Map-reduce framework </li></ul></ul>
  125. 126. Key Links <ul><li>Source Code </li><ul><li>https://github.com/robdimarco/deferred_processing_talk </li></ul><li>Delayed::Job </li><ul><li>https://github.com/collectiveidea/delayed_job </li></ul><li>Resque </li><ul><li>https://github.com/defunkt/resque
  126. 127. https://github.com/blog/542-introducing-resque </li></ul></ul>
  127. 128. More Links <ul><li>SQS </li><ul><li>http://aws.amazon.com/sqs/
  128. 129. https://github.com/rightscale/right_aws </li></ul><li>AMQP </li><ul><li>http://rdoc.info/github/ruby-amqp/amqp/master/file/docs/GettingStarted.textile
  129. 130. http://www.amqp.org/ </li></ul><li>More </li><ul><li>http://ruby-toolbox.com/categories/queueing.html
  130. 131. http://kr.github.com/beanstalkd/ </li></ul></ul>

×