Background Jobs with Resque
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Background Jobs with Resque

on

  • 7,786 views

As more and more web applications integrate with third-party APIs and other external data, processing those external resources in the background more and more important. A simple job runner is a great ...

As more and more web applications integrate with third-party APIs and other external data, processing those external resources in the background more and more important. A simple job runner is a great start, however as your load increases, you very quickly outgrew that simplistic queuing system. We will cover where getting started using Resque and Redis, how to test your jobs, when it makes sense to use Resque, implementations of Resque in other languages, and look how I've used Resque.

Statistics

Views

Total Views
7,786
Views on SlideShare
7,774
Embed Views
12

Actions

Likes
4
Downloads
55
Comments
0

1 Embed 12

https://twitter.com 12

Accessibility

Categories

Upload Details

Uploaded via as Apple Keynote

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • \n
  • Software consulting company in Chicago, IL.\n\nPrimarily RoR, but have mobile, Java, .NET project from time to time\n\nAlways looking for good developers. Interested? Find me after the talk\n
  • \n
  • Useful for cpu- and network-intensive tasks\n\nProcesses that run on an interval\n
  • queue examples: \n* one for each email type; \n* one for updating feeds, one for posting to twitter\n\n
  • priority is assigned per-worker; specified by queue\n\nMemory leak -> worker forks a child process for every job\n
  • \n
  • Resque uses Redis as its database backend\n\ndata structure: string, hash, list, set\n\nlinked list gives great performance for insert/retrieve at head and tail\n\natomic operations: one action won’t stomp on another\n
  • UI will block for each feed creation\nunder load, consume a ton of resources (each call to the controller forks a process???)\n
  • \n
  • enqueue takes a Ruby class and any number of arguments after that\n\nargs passed as JSON; can’t pass entire objects, pass an id\n
  • \n
  • 5 seconds is the default\n
  • \n
  • processors are POROs so testing is pretty straightforward\ntest side effects\n
  • \n
  • \n
  • \n
  • copies from Resque’s README\n
  • semaphore - ensures a deal is still able to sold\n\natomic value of deal sales remaining across processess\n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • \n
  • Ethan’s geekfest on nosql db’s - choose the right solution for the job\n\n
  • I picked Resque as a learning tool\n\nBreakable Toys\n
  • \n
  • \n

Background Jobs with Resque Presentation Transcript

  • 1. Background Jobs with Resque Jon Homan Apprentice @ Obtiva http://jonhoman.com @jonhoman
  • 2. Agenda• Resque• Examples• Real world use• Plugins• Alternate Implementations• When to use Resque
  • 3. Background Jobs• Remote API calls• Uploading images• Number crunching• Updating RSS feeds
  • 4. Resque https://github.com/defunkt/resque• Open-source library for creating background jobs• Redis backend• Support for multiple queues• Queues are processed by workers
  • 5. More Resque Features• Monitor with god or monit• Distributed processing• Priorities• Persistent queues• Memory leak resistant• Web interface
  • 6. resque-web
  • 7. Redis http://redis.io/• Key-value datastore• data structure server• constant time pushes and pops• atomic operations
  • 8. All “Foreground”class FeedsController def create feed = Feed.create! feed.fetch redirect_to feed_path(feed) endendclass Feed def fetch rss = RSS::Parse.parse(url, false) feed.title = rss.title endend
  • 9. Queue up creationclass FeedsController def create feed = Feed.create! Queue.push(feed) # won’t block UI redirect_to feed_path(feed) endendclass Queue def push(feed) queue << feed endend
  • 10. Now with Resqueclass FeedsController def create feed = Feed.create! Resque.enqueue(FeedCreator, feed.id) redirect_to feed_path(feed) endend
  • 11. How to process jobsclass FeedCreator @queue = :feed_creation def perform(feed_id) feed = Feed.find(feed_id) rss = RSS::Parser.parse(feed.url,false) feed.title = rss.title # create all items in the rss endend
  • 12. Workers?• Long-running Rake tasks• Look for jobs every n seconds
  • 13. How to start workers$ rake resque:work$ QUEUE=feed rake resque:work$ QUEUE=feed, twitter rake resque:work$ QUEUE=* rake resque:work$ COUNT=3 QUEUE=* rake resque:work
  • 14. Testing w/ RSpecit "stores the name of the feed" do FeedCreator.perform(feed.id) feed.reload.title.should == "Feedend
  • 15. Testing w/ RSpecit "creates Items for every item in the feed" do FeedCreator.perform(feed.id) feed.reload.items.count.should == 5end
  • 16. Testing w/ RSpecit “parses the feed” do RSS::Parser.should_receive(:parse) FeedCreator.perform(feed.id)end
  • 17. Real world uses• GitHub• Groupon• lifeStreams
  • 18. GitHub35 different types of jobs: • Warming caches • Counting disk usage • Building tarballs • Building Rubygems • Building graphs • Updating search index
  • 19. Groupon• Resque for background jobs • Notifications • Lockable worker• Redis for semaphores
  • 20. lifeStreams• Process updates to blog feeds• Publish updates to Facebook, Twitter, Email
  • 21. Plugins• resque_spec• Resque Heroku Autoscaler• resque multi step• ResqueMailer• resque-retry• and at least 25 others
  • 22. ResqueSpec https://github.com/leshill/resque_spec• Resque plugin that adds resque-specific matchers• have_queued• have_scheduled• have_queue_size_of
  • 23. Examplesit “adds a feed to the creation queue” do post :create, :feed FeedCreator.should have_queued(feed.id)endit “schedules a feed update” do feed.update FeedUpdater.should have_scheduled(feed.id)end
  • 24. Resque Heroku Autoscaler• Scales Heroku workers• Start worker if none running• Stops worker after jobs are processed
  • 25. RHArequire resque/plugins/resque_heroku_autoscalerclass TestJob extend Resque::Plugins::HerokuAutoscaler @queue = :test def perform ...awesome stuff... endend
  • 26. Resque Mailer• Asynchronously deliver email in Rails
  • 27. Resque Mailerclass MyMailer < ActionMailer::Base include Resque::MailerendMyMailer.subject_email(params).deliverQUEUE=mailer rake environment resque:work
  • 28. resque-retry• Retry, delay and exponential backoff support
  • 29. resque-retryrequire resque-retryclass ExampleRetryJob extend Resque::Plugins::Retry @queue = :example_queue @retry_limit = 3 @retry_delay = 60 def self.perform(*args) # your magic/heavy lifting goes here. endend
  • 30. resque-retry no delay, 1m, 10m, 1h, 3h, 6h@backoff_strategy = [0, 60, 600, 3600, 10800, 21600]
  • 31. Alternate Implementations• C• Java• Scala• .NET• Python• PHP• node.js
  • 32. jesque// Add a job to the queuefinal Job job = new Job("TestAction", new Object[]{ 1, 2, true, "test", Arrays.asList("inner",4ß)});final Client client = new ClientImpl(config);client.enqueue("foo", job);client.end();// Start a worker to run jobs from the queuefinal Worker worker = new WorkerImpl(config, Arrays.asList("foo"), Arrays.asList(TestAction.class));final Thread workerThread = new Thread(worker);workerThread.start();
  • 33. Resque?• Multiple queues• Potentially HUGE queues• Admin UI• Expecting chaos/failures
  • 34. DelayedJob?• Numeric priorities• Queue is small• Add whole objects to queue• No need to setup Redis
  • 35. Do your homework
  • 36. I didn’t (kinda)
  • 37. Resourcesgithub.com/defunkt/resqueredis.ioredistogo.com
  • 38. Questions?