Sidekiq Basics

Lyon.rb
12 feb 2014
Kurt Sussman
@neophiliac
What is Sidekiq?

●

Async job manager

●

Released by Mike Perham in January 2012

●

Developed for theclymb.com

●

Drop...
Async challenges

●

Adds complexity (redis, sidekiq daemon)

●

Workers must be thread-safe and idempotent

●

Timing-rel...
Thread-safety
●

●
●

●

Uses all shared variables in an atomic way,
unless each is allocated to a specific instance
of th...
Idempotency

●

●

An idempotent operation is one that has no
additional effect if it is called more than once
with the sa...
Why Sidekiq?
●

●

Sidekiq is faster, uses fewer resources, and is
well-supported
TheClymb claims 100% uptime for sidekiq*...
Submit a job
Mailer.welcome(@user.email)
becomes

MailWelcomeWorker.perform_async(@user.id)
Schedule a job
●

#perform_in(interval, *args)

●

Worker.perform_in(5.days, @user.id)

●

#perform_at(timestamp, *args)

...
Testing

●

require 'sidekiq/testing'

●

Sidekiq::Testing.fake! (the default)

●

Worker.drain # force processing of queu...
Testbed

●

github.com/neophiliac/sidekiq-example

●

Disclaimer: not a real app!

●

git diff master sidekiq

●

Uses for...
Further reading...
●

https://github.com/mperham/sidekiq/wiki/_pages

●

http://www.mikeperham.com/2014/01/04/the-clymb-20...
Upcoming SlideShare
Loading in …5
×

Lyonrb 12feb2014-sidekiq

818 views
664 views

Published on

What you need to know to start integrating Mike Perham's Sidekiq threaded work queue with your Rails app.

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
818
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Resque forks a complete copy of parent
    Sidekiq spawns threads, uses actor model
  • Shared mutable state == risky
    Rebuild context in the worker; assume nothing
    Constants aren't!
    Pay special attention to anything global
    ENV
    Constants
  • Retry interval and count are configurable
  • For real work, people report that one sidekiq daemon can do the work of 8-25 resque workers in far less memory
  • Use ID isntead of data for 2 reasons:
    Less data to marshall/unmarshall
    Get to test whether it still exists (and needs to be sent) when the job is processed
  • Lyonrb 12feb2014-sidekiq

    1. 1. Sidekiq Basics Lyon.rb 12 feb 2014 Kurt Sussman @neophiliac
    2. 2. What is Sidekiq? ● Async job manager ● Released by Mike Perham in January 2012 ● Developed for theclymb.com ● Drop-in replacement for Resque (almost) ● Uses Celluloid gem for concurrency
    3. 3. Async challenges ● Adds complexity (redis, sidekiq daemon) ● Workers must be thread-safe and idempotent ● Timing-related bugs are difficult to reproduce
    4. 4. Thread-safety ● ● ● ● Uses all shared variables in an atomic way, unless each is allocated to a specific instance of the worker Does not call non-thread-safe methods Does not use APIs or database in a non-atomic way The G{I,V}L does NOT make Ruby thread-safe
    5. 5. Idempotency ● ● An idempotent operation is one that has no additional effect if it is called more than once with the same input parameters Sidekiq will retry jobs if there is an error
    6. 6. Why Sidekiq? ● ● Sidekiq is faster, uses fewer resources, and is well-supported TheClymb claims 100% uptime for sidekiq* Name Database delayed_job SQL Resque redis Sidekiq redis Threads 1 1 many * but they restart sidekiq daemons every day
    7. 7. Submit a job Mailer.welcome(@user.email) becomes MailWelcomeWorker.perform_async(@user.id)
    8. 8. Schedule a job ● #perform_in(interval, *args) ● Worker.perform_in(5.days, @user.id) ● #perform_at(timestamp, *args) ● Worker.perform_at(5.days.from_now, @user.id)
    9. 9. Testing ● require 'sidekiq/testing' ● Sidekiq::Testing.fake! (the default) ● Worker.drain # force processing of queued jobs ● Sidekiq::Testing.inline! ● Workers are POROs
    10. 10. Testbed ● github.com/neophiliac/sidekiq-example ● Disclaimer: not a real app! ● git diff master sidekiq ● Uses foreman to start redis and sidekiq ● Includes (some) rspec tests
    11. 11. Further reading... ● https://github.com/mperham/sidekiq/wiki/_pages ● http://www.mikeperham.com/2014/01/04/the-clymb-2013/ ● https://github.com/neophiliac/sidekiq-example ● ● http://www.sitepoint.com/comparing-background-processinglibraries-sidekiq/ https://github.com/JustinLove/autoscaler

    ×