Lightning talk at Redisconf

1. Qless – Powerful Queues
Dan Lecocq
http://bit.ly/redisconf-qless

2. About SEOmoz

3. Martin Luther Time

4. Dropped Jobs, Flaky Workers
- We’re on the cloud
- Jobs are long-running
- Jobs are sparse
Please forgive my memegenerator abuse!

5. Debugging, Tracking, Insight
- Squeaky wheels
- Finding misbehaving hosts
- How long are jobs taking?

6. Scheduling, Dependencies, Operations
- Crawls run once a week
- Customer site transient outages
- Multiple campaigns, same domain
- Empower help team to help

7. A New Hope

8. Language Agnostic
github.com/seomoz/qless
github.com/seomoz/qless-py
bitbucket.org/nuclon/qless-perl
github.com/seomoz/qless-core
Releasing Soon!
We’ve also played with Node.js and C++ bindings

9. Job Anatomy
Job ID and Type Priority, Flagging, Move
Searchable Tags
JSON Blob Data History
Straight out of the web app

10. Web App and Operational Insight

11. Strong Guarantees
- Completely client managed
- Strong job completion guarantees
- Decommissioning workers is easy
- Flaky workers affect efficiency, not correctness

12. Code Example

13. Gnomes!
For those unfamiliar with this meme: wiki/Gnomes_(South_Park)

14. Python Client
# In gnomes.py
class GnomesJob(object):
# This would be invoked when a GnomesJob is popped off the 'underpants' queue
@staticmethod
def underpants(job):
# 1) Collect Underpants
job['foo'] # Reference job data
# Complete and advance to the next step, 'unknown’
job.complete('unknown')
@staticmethod
def unknown(job):
# 2) ?
...
# Complete and advance to the next step, 'profit’
job.complete('profit')
@staticmethod
def profit(job):
# 3) Profit
...
# Complete the job
job.complete()

15. Running A Worker
qless-py-worker [-h] [--host HOST] [-w WORKERS] [-g GREENLETS]
[-p PATH] [-q QUEUE] [-m IMPORT] [-i INTERVAL]
[-n NAME] [-d WORKDIR] [-r] [-v]
qless-py-worker --host "my-qless-host"
-q underpants
-q unknown
-q profit
-w 20
-d /path/to/sandboxes
-v
# Now in python, add some jobs:
import qless
from qless import GnomesJob
client = qless.Client(...)
client.put('underpants', GnomesJob, {'foo': 'bar'})

16. Shameless Plug
bit.ly/work-at-seomoz

17. Dan Lecocq, Software Ninja | @danlecocq
dan@seomoz.org
Slides: http://bit.ly/redisconf-qless
Apply: http://bit.ly/work-at-seomoz
Follow Up!

18. Where My Images At?
1) “Angry Guy” from http://dinkerson.wordpress.com/
http://dinkerson.files.wordpress.com/2012/02/angry-
guy.jpg
2) “Underpants Gnomes” from
http://www.tumblr.com/tagged/underpants-gnomes
3) “Switchboards” from
http://markc1.typepad.com/.a/6a00d83451bb2969e201310f
8060a1970c-800wi
Thanks for the images!