Your SlideShare is downloading. ×
Why Task Queues - ComoRichWeb
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Why Task Queues - ComoRichWeb

5,752

Published on

We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque. …

We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.

Finally, we wrap up with a quick example of a task queue in PHP using Redis.

https://github.com/bryanhelmig/phqueue

0 Comments
16 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
5,752
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
45
Comments
0
Likes
16
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Why Task Queues inPython, Ruby and More! ~ a talk by Bryan Helmig
  • 2. task queue noun ˈtask ˈkyüa system for parallel execution of discretetasks in a non-blocking fashion. celery, resque, or home-grown...
  • 3. broker noun broh-kerthe middle man holding the tasks(messages) themselves. rabbitmq, gearman, redis, etc...
  • 4. producer noun pruh-doo-serthe code that places the tasks to beexecuted later in the broker. your application code!
  • 5. consumer noun kuhn-soo-meralso called the worker, takes tasks fromthe broker and perform them. usually a daemon under supervision.
  • 6. imagine this...your app: posting a new message triggers“new message” emails to all your friends.
  • 7. def new_message(request): user = get_user_or_404(request) message = request.POST.get(message, None) if not message: raise Http404 user.save_new_message(message) for friend in user.friends.all(): friend.send_email(message) return redirect(reverse(dashboard))
  • 8. def new_message(request): user = get_user_or_404(request) message = request.POST.get(message, None) if not message: raise Http404 user.save_new_message(message) for friend in user.friends.all(): friend.send_email(message) return redirect(reverse(dashboard))
  • 9. the problem:that works good for like 0-6 friends... butwhat if you have 100,000? or more? willyour user ever get a response?
  • 10. bad idea #1: ignoremake your users wait through your longrequest/response cycle. your users are important, right?
  • 11. bad idea #2: ajaxreturn the page fast with JS code that callsanother script in the browser’s background. duplicate calls & http cycles are not cool.
  • 12. bad idea #3: cronjobmake a email_friends table with user_id &message column. cron every 5/10 minutes. backlogs will destroy you.
  • 13. good idea: queuesthe task to potentially email thousands ofusers is put into a queue to be dealt withlater, leaving you to return the response.
  • 14. @taskdef alert_friends(user_id, message): user = User.objects.get(id=user_id) for friend in user.friends.all(): friend.send_email(message)def new_message(request): user = get_user_or_404(request) message = request.POST.get(message, None) if not message: raise Http404 user.save_new_message(message) alert_friends.delay(user.id, message) return redirect(reverse(dashboard))
  • 15. RULE #1: adding a task to a queue should befaster than performing the task itself.
  • 16. RULE #2:you should consume tasks faster than you produce them. if not, add more workers.
  • 17. queue libraries:1. celery (python) earlier...2. resque (ruby) up next...3. build your own grand finalé!
  • 18. class MessageSend def self.perform(user_id, message) user = User.find(user_id) user.friends.each do |friend| friend.send_email(message) end endendclass MessageController < ActionController::Base def new_message render(file: public/404.html, status: 404) unless params[:message] current_user.save_new_message(params[:message]) Resque.enqueue(MessageSend, current_user.id, params[:message]) redirect_to dashboard_url endend
  • 19. let’s build our own!and let’s do it with redis, in php and in less than 50 lines of code! (not including the task code)
  • 20. quick redis aside:so, redis is a key-value store. values canbe strings, blobs, lists or hashes. we’ll uselists and RPUSH and LPOP.
  • 21. step #1: task runneri’m envisioning this as a Task class withvarious methods named after each task.each method accepts an array $params.
  • 22. class Tasks { public function email_friends($params) { $user_id = $params->user_id; $message = $params->message; # get $friends from a db query... $friends = array(paul@example.com, john@example.com); foreach ($friends as $friend) { echo "Fake email ".$friend. " with ".$message."n"; } }}
  • 23. step #2: workeri think an infinite loop with a blockingredis BLPOP will work. then it needs to runthe right method on the Task object.
  • 24. include_once redis.php; # sets up $redis, $queueinclude_once tasks.php;$tasks = new Tasks();while (true) { # BLPOP will block until it gets an item.. $data = $redis->blpop($queue, 0); $obj = json_decode($data[1]); # grab json... $method = $obj->task_name; $params = $obj->params; # calls the task: Task->$task_name($params)... call_user_func(array($tasks, $method), $params);}
  • 25. step #3: run it!we will need to run the worker: ➜ php worker.php
  • 26. step #4: producerlet’s make a handy add_task function foreasy placement of tasks into the queue.
  • 27. include_once redis.php; # sets up $redis & $queuefunction add_task($task_name, $params) { global $redis, $queue; $data = Array(task_name => $task_name, params => $params); $json = json_encode($data) $redis->rpush($queue, $json);}# an example of our task api...add_task(email_friends, array(user_id => 1234, message => I just bought a car!));
  • 28. step #5: watch ita command line that runs a worker isstandard. use something like supervisordor god to run it & monitor it.
  • 29. things missing:our php example does not: store returnresults, handle errors, route tasks,degrade gracefully, log activity, etc...
  • 30. the endquestion time!

×