Background Jobs with Resque


Published on

As more and more web applications integrate with third-party APIs and other external data, processing those external resources in the background more and more important. A simple job runner is a great start, however as your load increases, you very quickly outgrew that simplistic queuing system. We will cover where getting started using Resque and Redis, how to test your jobs, when it makes sense to use Resque, implementations of Resque in other languages, and look how I've used Resque.

Published in: Technology
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • \n
  • Software consulting company in Chicago, IL.\n\nPrimarily RoR, but have mobile, Java, .NET project from time to time\n\nAlways looking for good developers. Interested? Find me after the talk\n
  • \n
  • Useful for cpu- and network-intensive tasks\n\nProcesses that run on an interval\n
  • queue examples: \n* one for each email type; \n* one for updating feeds, one for posting to twitter\n\n
  • priority is assigned per-worker; specified by queue\n\nMemory leak -> worker forks a child process for every job\n
  • \n
  • Resque uses Redis as its database backend\n\ndata structure: string, hash, list, set\n\nlinked list gives great performance for insert/retrieve at head and tail\n\natomic operations: one action won’t stomp on another\n
  • UI will block for each feed creation\nunder load, consume a ton of resources (each call to the controller forks a process???)\n
  • \n
  • enqueue takes a Ruby class and any number of arguments after that\n\nargs passed as JSON; can’t pass entire objects, pass an id\n
  • \n
  • 5 seconds is the default\n
  • \n
  • processors are POROs so testing is pretty straightforward\ntest side effects\n
  • \n
  • \n
  • \n
  • copies from Resque’s README\n
  • semaphore - ensures a deal is still able to sold\n\natomic value of deal sales remaining across processess\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Ethan’s geekfest on nosql db’s - choose the right solution for the job\n\n
  • I picked Resque as a learning tool\n\nBreakable Toys\n
  • \n
  • \n
  • Background Jobs with Resque

    1. 1. Background Jobs with Resque Jon Homan Apprentice @ Obtiva @jonhoman
    2. 2. Agenda• Resque• Examples• Real world use• Plugins• Alternate Implementations• When to use Resque
    3. 3. Background Jobs• Remote API calls• Uploading images• Number crunching• Updating RSS feeds
    4. 4. Resque• Open-source library for creating background jobs• Redis backend• Support for multiple queues• Queues are processed by workers
    5. 5. More Resque Features• Monitor with god or monit• Distributed processing• Priorities• Persistent queues• Memory leak resistant• Web interface
    6. 6. resque-web
    7. 7. Redis• Key-value datastore• data structure server• constant time pushes and pops• atomic operations
    8. 8. All “Foreground”class FeedsController def create feed = Feed.create! feed.fetch redirect_to feed_path(feed) endendclass Feed def fetch rss = RSS::Parse.parse(url, false) feed.title = rss.title endend
    9. 9. Queue up creationclass FeedsController def create feed = Feed.create! Queue.push(feed) # won’t block UI redirect_to feed_path(feed) endendclass Queue def push(feed) queue << feed endend
    10. 10. Now with Resqueclass FeedsController def create feed = Feed.create! Resque.enqueue(FeedCreator, redirect_to feed_path(feed) endend
    11. 11. How to process jobsclass FeedCreator @queue = :feed_creation def perform(feed_id) feed = Feed.find(feed_id) rss = RSS::Parser.parse(feed.url,false) feed.title = rss.title # create all items in the rss endend
    12. 12. Workers?• Long-running Rake tasks• Look for jobs every n seconds
    13. 13. How to start workers$ rake resque:work$ QUEUE=feed rake resque:work$ QUEUE=feed, twitter rake resque:work$ QUEUE=* rake resque:work$ COUNT=3 QUEUE=* rake resque:work
    14. 14. Testing w/ RSpecit "stores the name of the feed" do FeedCreator.perform( feed.reload.title.should == "Feedend
    15. 15. Testing w/ RSpecit "creates Items for every item in the feed" do FeedCreator.perform( feed.reload.items.count.should == 5end
    16. 16. Testing w/ RSpecit “parses the feed” do RSS::Parser.should_receive(:parse) FeedCreator.perform(
    17. 17. Real world uses• GitHub• Groupon• lifeStreams
    18. 18. GitHub35 different types of jobs: • Warming caches • Counting disk usage • Building tarballs • Building Rubygems • Building graphs • Updating search index
    19. 19. Groupon• Resque for background jobs • Notifications • Lockable worker• Redis for semaphores
    20. 20. lifeStreams• Process updates to blog feeds• Publish updates to Facebook, Twitter, Email
    21. 21. Plugins• resque_spec• Resque Heroku Autoscaler• resque multi step• ResqueMailer• resque-retry• and at least 25 others
    22. 22. ResqueSpec• Resque plugin that adds resque-specific matchers• have_queued• have_scheduled• have_queue_size_of
    23. 23. Examplesit “adds a feed to the creation queue” do post :create, :feed FeedCreator.should have_queued( “schedules a feed update” do feed.update FeedUpdater.should have_scheduled(
    24. 24. Resque Heroku Autoscaler• Scales Heroku workers• Start worker if none running• Stops worker after jobs are processed
    25. 25. RHArequire resque/plugins/resque_heroku_autoscalerclass TestJob extend Resque::Plugins::HerokuAutoscaler @queue = :test def perform ...awesome stuff... endend
    26. 26. Resque Mailer• Asynchronously deliver email in Rails
    27. 27. Resque Mailerclass MyMailer < ActionMailer::Base include Resque::MailerendMyMailer.subject_email(params).deliverQUEUE=mailer rake environment resque:work
    28. 28. resque-retry• Retry, delay and exponential backoff support
    29. 29. resque-retryrequire resque-retryclass ExampleRetryJob extend Resque::Plugins::Retry @queue = :example_queue @retry_limit = 3 @retry_delay = 60 def self.perform(*args) # your magic/heavy lifting goes here. endend
    30. 30. resque-retry no delay, 1m, 10m, 1h, 3h, 6h@backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
    31. 31. Alternate Implementations• C• Java• Scala• .NET• Python• PHP• node.js
    32. 32. jesque// Add a job to the queuefinal Job job = new Job("TestAction", new Object[]{ 1, 2, true, "test", Arrays.asList("inner",4ß)});final Client client = new ClientImpl(config);client.enqueue("foo", job);client.end();// Start a worker to run jobs from the queuefinal Worker worker = new WorkerImpl(config, Arrays.asList("foo"), Arrays.asList(TestAction.class));final Thread workerThread = new Thread(worker);workerThread.start();
    33. 33. Resque?• Multiple queues• Potentially HUGE queues• Admin UI• Expecting chaos/failures
    34. 34. DelayedJob?• Numeric priorities• Queue is small• Add whole objects to queue• No need to setup Redis
    35. 35. Do your homework
    36. 36. I didn’t (kinda)
    37. 37.
    38. 38. Questions?