Your SlideShare is downloading. ×
0
Background Jobs with      Resque           Jon Homan       Apprentice @ Obtiva       http://jonhoman.com           @jonhoman
Agenda• Resque• Examples• Real world use• Plugins• Alternate Implementations• When to use Resque
Background Jobs• Remote API calls• Uploading images• Number crunching• Updating RSS feeds
Resque     https://github.com/defunkt/resque• Open-source library for creating background  jobs• Redis backend• Support fo...
More Resque Features• Monitor with god or monit• Distributed processing• Priorities• Persistent queues• Memory leak resist...
resque-web
Redis              http://redis.io/• Key-value datastore• data structure server• constant time pushes and pops• atomic ope...
All “Foreground”class FeedsController def create   feed = Feed.create!   feed.fetch   redirect_to feed_path(feed) endendcl...
Queue up creationclass FeedsController def create  feed = Feed.create!  Queue.push(feed) # won’t block UI  redirect_to fee...
Now with Resqueclass FeedsController def create  feed = Feed.create!  Resque.enqueue(FeedCreator, feed.id)  redirect_to fe...
How to process jobsclass FeedCreator @queue = :feed_creation def perform(feed_id)   feed = Feed.find(feed_id)   rss = RSS:...
Workers?• Long-running Rake tasks• Look for jobs every n seconds
How to start workers$ rake resque:work$ QUEUE=feed rake resque:work$ QUEUE=feed, twitter rake resque:work$ QUEUE=* rake re...
Testing w/ RSpecit "stores the name of the feed" do FeedCreator.perform(feed.id) feed.reload.title.should == "Feedend
Testing w/ RSpecit "creates Items for every item in the feed" do FeedCreator.perform(feed.id) feed.reload.items.count.shou...
Testing w/ RSpecit “parses the feed” do RSS::Parser.should_receive(:parse) FeedCreator.perform(feed.id)end
Real world uses• GitHub• Groupon•   lifeStreams
GitHub35 different types of jobs:  • Warming caches  • Counting disk usage  • Building tarballs  • Building Rubygems  • Bu...
Groupon• Resque for background jobs • Notifications • Lockable worker• Redis for semaphores
lifeStreams• Process updates to blog feeds• Publish updates to Facebook, Twitter, Email
Plugins• resque_spec• Resque Heroku Autoscaler• resque multi step• ResqueMailer• resque-retry• and at least 25 others
ResqueSpec    https://github.com/leshill/resque_spec• Resque plugin that adds resque-specific  matchers• have_queued• have_...
Examplesit “adds a feed to the creation queue” do post :create, :feed FeedCreator.should have_queued(feed.id)endit “schedu...
Resque Heroku         Autoscaler• Scales Heroku workers• Start worker if none running• Stops worker after jobs are processed
RHArequire resque/plugins/resque_heroku_autoscalerclass TestJob  extend Resque::Plugins::HerokuAutoscaler  @queue = :test ...
Resque Mailer• Asynchronously deliver email in Rails
Resque Mailerclass MyMailer < ActionMailer::Base  include Resque::MailerendMyMailer.subject_email(params).deliverQUEUE=mai...
resque-retry• Retry, delay and exponential backoff support
resque-retryrequire resque-retryclass ExampleRetryJob  extend Resque::Plugins::Retry  @queue = :example_queue  @retry_limi...
resque-retry              no delay, 1m, 10m,   1h,    3h,    6h@backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
Alternate           Implementations•   C•   Java•   Scala•   .NET•   Python•   PHP•   node.js
jesque// Add a job to the queuefinal Job job = new Job("TestAction",    new Object[]{ 1, 2, true, "test", Arrays.asList("i...
Resque?• Multiple queues• Potentially HUGE queues• Admin UI• Expecting chaos/failures
DelayedJob?• Numeric priorities• Queue is small• Add whole objects to queue• No need to setup Redis
Do your homework
I didn’t (kinda)
Resourcesgithub.com/defunkt/resqueredis.ioredistogo.com
Questions?
Background Jobs with Resque
Upcoming SlideShare
Loading in...5
×

Background Jobs with Resque

7,725

Published on

As more and more web applications integrate with third-party APIs and other external data, processing those external resources in the background more and more important. A simple job runner is a great start, however as your load increases, you very quickly outgrew that simplistic queuing system. We will cover where getting started using Resque and Redis, how to test your jobs, when it makes sense to use Resque, implementations of Resque in other languages, and look how I've used Resque.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,725
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
68
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • \n
  • Software consulting company in Chicago, IL.\n\nPrimarily RoR, but have mobile, Java, .NET project from time to time\n\nAlways looking for good developers. Interested? Find me after the talk\n
  • \n
  • Useful for cpu- and network-intensive tasks\n\nProcesses that run on an interval\n
  • queue examples: \n* one for each email type; \n* one for updating feeds, one for posting to twitter\n\n
  • priority is assigned per-worker; specified by queue\n\nMemory leak -&gt; worker forks a child process for every job\n
  • \n
  • Resque uses Redis as its database backend\n\ndata structure: string, hash, list, set\n\nlinked list gives great performance for insert/retrieve at head and tail\n\natomic operations: one action won&amp;#x2019;t stomp on another\n
  • UI will block for each feed creation\nunder load, consume a ton of resources (each call to the controller forks a process???)\n
  • \n
  • enqueue takes a Ruby class and any number of arguments after that\n\nargs passed as JSON; can&amp;#x2019;t pass entire objects, pass an id\n
  • \n
  • 5 seconds is the default\n
  • \n
  • processors are POROs so testing is pretty straightforward\ntest side effects\n
  • \n
  • \n
  • \n
  • copies from Resque&amp;#x2019;s README\n
  • semaphore - ensures a deal is still able to sold\n\natomic value of deal sales remaining across processess\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Ethan&amp;#x2019;s geekfest on nosql db&amp;#x2019;s - choose the right solution for the job\n\n
  • I picked Resque as a learning tool\n\nBreakable Toys\n
  • \n
  • \n
  • Transcript of "Background Jobs with Resque"

    1. 1. Background Jobs with Resque Jon Homan Apprentice @ Obtiva http://jonhoman.com @jonhoman
    2. 2. Agenda• Resque• Examples• Real world use• Plugins• Alternate Implementations• When to use Resque
    3. 3. Background Jobs• Remote API calls• Uploading images• Number crunching• Updating RSS feeds
    4. 4. Resque https://github.com/defunkt/resque• Open-source library for creating background jobs• Redis backend• Support for multiple queues• Queues are processed by workers
    5. 5. More Resque Features• Monitor with god or monit• Distributed processing• Priorities• Persistent queues• Memory leak resistant• Web interface
    6. 6. resque-web
    7. 7. Redis http://redis.io/• Key-value datastore• data structure server• constant time pushes and pops• atomic operations
    8. 8. All “Foreground”class FeedsController def create feed = Feed.create! feed.fetch redirect_to feed_path(feed) endendclass Feed def fetch rss = RSS::Parse.parse(url, false) feed.title = rss.title endend
    9. 9. Queue up creationclass FeedsController def create feed = Feed.create! Queue.push(feed) # won’t block UI redirect_to feed_path(feed) endendclass Queue def push(feed) queue << feed endend
    10. 10. Now with Resqueclass FeedsController def create feed = Feed.create! Resque.enqueue(FeedCreator, feed.id) redirect_to feed_path(feed) endend
    11. 11. How to process jobsclass FeedCreator @queue = :feed_creation def perform(feed_id) feed = Feed.find(feed_id) rss = RSS::Parser.parse(feed.url,false) feed.title = rss.title # create all items in the rss endend
    12. 12. Workers?• Long-running Rake tasks• Look for jobs every n seconds
    13. 13. How to start workers$ rake resque:work$ QUEUE=feed rake resque:work$ QUEUE=feed, twitter rake resque:work$ QUEUE=* rake resque:work$ COUNT=3 QUEUE=* rake resque:work
    14. 14. Testing w/ RSpecit "stores the name of the feed" do FeedCreator.perform(feed.id) feed.reload.title.should == "Feedend
    15. 15. Testing w/ RSpecit "creates Items for every item in the feed" do FeedCreator.perform(feed.id) feed.reload.items.count.should == 5end
    16. 16. Testing w/ RSpecit “parses the feed” do RSS::Parser.should_receive(:parse) FeedCreator.perform(feed.id)end
    17. 17. Real world uses• GitHub• Groupon• lifeStreams
    18. 18. GitHub35 different types of jobs: • Warming caches • Counting disk usage • Building tarballs • Building Rubygems • Building graphs • Updating search index
    19. 19. Groupon• Resque for background jobs • Notifications • Lockable worker• Redis for semaphores
    20. 20. lifeStreams• Process updates to blog feeds• Publish updates to Facebook, Twitter, Email
    21. 21. Plugins• resque_spec• Resque Heroku Autoscaler• resque multi step• ResqueMailer• resque-retry• and at least 25 others
    22. 22. ResqueSpec https://github.com/leshill/resque_spec• Resque plugin that adds resque-specific matchers• have_queued• have_scheduled• have_queue_size_of
    23. 23. Examplesit “adds a feed to the creation queue” do post :create, :feed FeedCreator.should have_queued(feed.id)endit “schedules a feed update” do feed.update FeedUpdater.should have_scheduled(feed.id)end
    24. 24. Resque Heroku Autoscaler• Scales Heroku workers• Start worker if none running• Stops worker after jobs are processed
    25. 25. RHArequire resque/plugins/resque_heroku_autoscalerclass TestJob extend Resque::Plugins::HerokuAutoscaler @queue = :test def perform ...awesome stuff... endend
    26. 26. Resque Mailer• Asynchronously deliver email in Rails
    27. 27. Resque Mailerclass MyMailer < ActionMailer::Base include Resque::MailerendMyMailer.subject_email(params).deliverQUEUE=mailer rake environment resque:work
    28. 28. resque-retry• Retry, delay and exponential backoff support
    29. 29. resque-retryrequire resque-retryclass ExampleRetryJob extend Resque::Plugins::Retry @queue = :example_queue @retry_limit = 3 @retry_delay = 60 def self.perform(*args) # your magic/heavy lifting goes here. endend
    30. 30. resque-retry no delay, 1m, 10m, 1h, 3h, 6h@backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
    31. 31. Alternate Implementations• C• Java• Scala• .NET• Python• PHP• node.js
    32. 32. jesque// Add a job to the queuefinal Job job = new Job("TestAction", new Object[]{ 1, 2, true, "test", Arrays.asList("inner",4ß)});final Client client = new ClientImpl(config);client.enqueue("foo", job);client.end();// Start a worker to run jobs from the queuefinal Worker worker = new WorkerImpl(config, Arrays.asList("foo"), Arrays.asList(TestAction.class));final Thread workerThread = new Thread(worker);workerThread.start();
    33. 33. Resque?• Multiple queues• Potentially HUGE queues• Admin UI• Expecting chaos/failures
    34. 34. DelayedJob?• Numeric priorities• Queue is small• Add whole objects to queue• No need to setup Redis
    35. 35. Do your homework
    36. 36. I didn’t (kinda)
    37. 37. Resourcesgithub.com/defunkt/resqueredis.ioredistogo.com
    38. 38. Questions?
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×