• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Why Task Queues - ComoRichWeb
 

Why Task Queues - ComoRichWeb

on

  • 5,213 views

We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque. ...

We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.

Finally, we wrap up with a quick example of a task queue in PHP using Redis.

https://github.com/bryanhelmig/phqueue

Statistics

Views

Total Views
5,213
Views on SlideShare
5,205
Embed Views
8

Actions

Likes
9
Downloads
38
Comments
0

6 Embeds 8

https://twitter.com 3
http://www.hanrss.com 1
https://twimg0-a.akamaihd.net 1
https://si0.twimg.com 1
http://abulte2.overblog.com 1
http://www.docshut.com 1

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike LicenseCC Attribution-NonCommercial-ShareAlike License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Why Task Queues - ComoRichWeb Why Task Queues - ComoRichWeb Presentation Transcript

    • Why Task Queues inPython, Ruby and More! ~ a talk by Bryan Helmig
    • task queue noun ˈtask ˈkyüa system for parallel execution of discretetasks in a non-blocking fashion. celery, resque, or home-grown...
    • broker noun broh-kerthe middle man holding the tasks(messages) themselves. rabbitmq, gearman, redis, etc...
    • producer noun pruh-doo-serthe code that places the tasks to beexecuted later in the broker. your application code!
    • consumer noun kuhn-soo-meralso called the worker, takes tasks fromthe broker and perform them. usually a daemon under supervision.
    • imagine this...your app: posting a new message triggers“new message” emails to all your friends.
    • def new_message(request): user = get_user_or_404(request) message = request.POST.get(message, None) if not message: raise Http404 user.save_new_message(message) for friend in user.friends.all(): friend.send_email(message) return redirect(reverse(dashboard))
    • def new_message(request): user = get_user_or_404(request) message = request.POST.get(message, None) if not message: raise Http404 user.save_new_message(message) for friend in user.friends.all(): friend.send_email(message) return redirect(reverse(dashboard))
    • the problem:that works good for like 0-6 friends... butwhat if you have 100,000? or more? willyour user ever get a response?
    • bad idea #1: ignoremake your users wait through your longrequest/response cycle. your users are important, right?
    • bad idea #2: ajaxreturn the page fast with JS code that callsanother script in the browser’s background. duplicate calls & http cycles are not cool.
    • bad idea #3: cronjobmake a email_friends table with user_id &message column. cron every 5/10 minutes. backlogs will destroy you.
    • good idea: queuesthe task to potentially email thousands ofusers is put into a queue to be dealt withlater, leaving you to return the response.
    • @taskdef alert_friends(user_id, message): user = User.objects.get(id=user_id) for friend in user.friends.all(): friend.send_email(message)def new_message(request): user = get_user_or_404(request) message = request.POST.get(message, None) if not message: raise Http404 user.save_new_message(message) alert_friends.delay(user.id, message) return redirect(reverse(dashboard))
    • RULE #1: adding a task to a queue should befaster than performing the task itself.
    • RULE #2:you should consume tasks faster than you produce them. if not, add more workers.
    • queue libraries:1. celery (python) earlier...2. resque (ruby) up next...3. build your own grand finalé!
    • class MessageSend def self.perform(user_id, message) user = User.find(user_id) user.friends.each do |friend| friend.send_email(message) end endendclass MessageController < ActionController::Base def new_message render(file: public/404.html, status: 404) unless params[:message] current_user.save_new_message(params[:message]) Resque.enqueue(MessageSend, current_user.id, params[:message]) redirect_to dashboard_url endend
    • let’s build our own!and let’s do it with redis, in php and in less than 50 lines of code! (not including the task code)
    • quick redis aside:so, redis is a key-value store. values canbe strings, blobs, lists or hashes. we’ll uselists and RPUSH and LPOP.
    • step #1: task runneri’m envisioning this as a Task class withvarious methods named after each task.each method accepts an array $params.
    • class Tasks { public function email_friends($params) { $user_id = $params->user_id; $message = $params->message; # get $friends from a db query... $friends = array(paul@example.com, john@example.com); foreach ($friends as $friend) { echo "Fake email ".$friend. " with ".$message."n"; } }}
    • step #2: workeri think an infinite loop with a blockingredis BLPOP will work. then it needs to runthe right method on the Task object.
    • include_once redis.php; # sets up $redis, $queueinclude_once tasks.php;$tasks = new Tasks();while (true) { # BLPOP will block until it gets an item.. $data = $redis->blpop($queue, 0); $obj = json_decode($data[1]); # grab json... $method = $obj->task_name; $params = $obj->params; # calls the task: Task->$task_name($params)... call_user_func(array($tasks, $method), $params);}
    • step #3: run it!we will need to run the worker: ➜ php worker.php
    • step #4: producerlet’s make a handy add_task function foreasy placement of tasks into the queue.
    • include_once redis.php; # sets up $redis & $queuefunction add_task($task_name, $params) { global $redis, $queue; $data = Array(task_name => $task_name, params => $params); $json = json_encode($data) $redis->rpush($queue, $json);}# an example of our task api...add_task(email_friends, array(user_id => 1234, message => I just bought a car!));
    • step #5: watch ita command line that runs a worker isstandard. use something like supervisordor god to run it & monitor it.
    • things missing:our php example does not: store returnresults, handle errors, route tasks,degrade gracefully, log activity, etc...
    • the endquestion time!