Why Task Queues - ComoRichWeb

7,538
-1

Published on

We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.

Finally, we wrap up with a quick example of a task queue in PHP using Redis.

https://github.com/bryanhelmig/phqueue

Why Task Queues - ComoRichWeb

  1. 1. Why Task Queues inPython, Ruby and More! ~ a talk by Bryan Helmig
  2. 2. task queue noun ˈtask ˈkyüa system for parallel execution of discretetasks in a non-blocking fashion. celery, resque, or home-grown...
  3. 3. broker noun broh-kerthe middle man holding the tasks(messages) themselves. rabbitmq, gearman, redis, etc...
  4. 4. producer noun pruh-doo-serthe code that places the tasks to beexecuted later in the broker. your application code!
  5. 5. consumer noun kuhn-soo-meralso called the worker, takes tasks fromthe broker and perform them. usually a daemon under supervision.
  6. 6. imagine this...your app: posting a new message triggers“new message” emails to all your friends.
  7. 7. def new_message(request): user = get_user_or_404(request) message = request.POST.get(message, None) if not message: raise Http404 user.save_new_message(message) for friend in user.friends.all(): friend.send_email(message) return redirect(reverse(dashboard))
  8. 8. def new_message(request): user = get_user_or_404(request) message = request.POST.get(message, None) if not message: raise Http404 user.save_new_message(message) for friend in user.friends.all(): friend.send_email(message) return redirect(reverse(dashboard))
  9. 9. the problem:that works good for like 0-6 friends... butwhat if you have 100,000? or more? willyour user ever get a response?
  10. 10. bad idea #1: ignoremake your users wait through your longrequest/response cycle. your users are important, right?
  11. 11. bad idea #2: ajaxreturn the page fast with JS code that callsanother script in the browser’s background. duplicate calls & http cycles are not cool.
  12. 12. bad idea #3: cronjobmake a email_friends table with user_id &message column. cron every 5/10 minutes. backlogs will destroy you.
  13. 13. good idea: queuesthe task to potentially email thousands ofusers is put into a queue to be dealt withlater, leaving you to return the response.
  14. 14. @taskdef alert_friends(user_id, message): user = User.objects.get(id=user_id) for friend in user.friends.all(): friend.send_email(message)def new_message(request): user = get_user_or_404(request) message = request.POST.get(message, None) if not message: raise Http404 user.save_new_message(message) alert_friends.delay(user.id, message) return redirect(reverse(dashboard))
  15. 15. RULE #1: adding a task to a queue should befaster than performing the task itself.
  16. 16. RULE #2:you should consume tasks faster than you produce them. if not, add more workers.
  17. 17. queue libraries:1. celery (python) earlier...2. resque (ruby) up next...3. build your own grand finalé!
  18. 18. class MessageSend def self.perform(user_id, message) user = User.find(user_id) user.friends.each do |friend| friend.send_email(message) end endendclass MessageController < ActionController::Base def new_message render(file: public/404.html, status: 404) unless params[:message] current_user.save_new_message(params[:message]) Resque.enqueue(MessageSend, current_user.id, params[:message]) redirect_to dashboard_url endend
  19. 19. let’s build our own!and let’s do it with redis, in php and in less than 50 lines of code! (not including the task code)
  20. 20. quick redis aside:so, redis is a key-value store. values canbe strings, blobs, lists or hashes. we’ll uselists and RPUSH and LPOP.
  21. 21. step #1: task runneri’m envisioning this as a Task class withvarious methods named after each task.each method accepts an array $params.
  22. 22. class Tasks { public function email_friends($params) { $user_id = $params->user_id; $message = $params->message; # get $friends from a db query... $friends = array(paul@example.com, john@example.com); foreach ($friends as $friend) { echo "Fake email ".$friend. " with ".$message."n"; } }}
  23. 23. step #2: workeri think an infinite loop with a blockingredis BLPOP will work. then it needs to runthe right method on the Task object.
  24. 24. include_once redis.php; # sets up $redis, $queueinclude_once tasks.php;$tasks = new Tasks();while (true) { # BLPOP will block until it gets an item.. $data = $redis->blpop($queue, 0); $obj = json_decode($data[1]); # grab json... $method = $obj->task_name; $params = $obj->params; # calls the task: Task->$task_name($params)... call_user_func(array($tasks, $method), $params);}
  25. 25. step #3: run it!we will need to run the worker: ➜ php worker.php
  26. 26. step #4: producerlet’s make a handy add_task function foreasy placement of tasks into the queue.
  27. 27. include_once redis.php; # sets up $redis & $queuefunction add_task($task_name, $params) { global $redis, $queue; $data = Array(task_name => $task_name, params => $params); $json = json_encode($data) $redis->rpush($queue, $json);}# an example of our task api...add_task(email_friends, array(user_id => 1234, message => I just bought a car!));
  28. 28. step #5: watch ita command line that runs a worker isstandard. use something like supervisordor god to run it & monitor it.
  29. 29. things missing:our php example does not: store returnresults, handle errors, route tasks,degrade gracefully, log activity, etc...
  30. 30. the endquestion time!
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×