Successfully reported this slideshow.

Why Task Queues - ComoRichWeb

42

Share

Loading in …3
×
1 of 30
1 of 30

Why Task Queues - ComoRichWeb

42

Share

Download to read offline

We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.

Finally, we wrap up with a quick example of a task queue in PHP using Redis.

https://github.com/bryanhelmig/phqueue

We start with why you should use task queues. Then we show a few straightforward examples with Python and Celery and Ruby and Resque.

Finally, we wrap up with a quick example of a task queue in PHP using Redis.

https://github.com/bryanhelmig/phqueue

More Related Content

Related Books

Free with a 14 day trial from Scribd

See all

Why Task Queues - ComoRichWeb

  1. 1. Why Task Queues in Python, Ruby and More! ~ a talk by Bryan Helmig
  2. 2. task queue noun ˈtask ˈkyü a system for parallel execution of discrete tasks in a non-blocking fashion. celery, resque, or home-grown...
  3. 3. broker noun broh-ker the middle man holding the tasks (messages) themselves. rabbitmq, gearman, redis, etc...
  4. 4. producer noun pruh-doo-ser the code that places the tasks to be executed later in the broker. your application code!
  5. 5. consumer noun kuhn-soo-mer also called the worker, takes tasks from the broker and perform them. usually a daemon under supervision.
  6. 6. imagine this... your app: posting a new message triggers “new message” emails to all your friends.
  7. 7. def new_message(request): user = get_user_or_404(request) message = request.POST.get('message', None) if not message: raise Http404 user.save_new_message(message) for friend in user.friends.all(): friend.send_email(message) return redirect(reverse('dashboard'))
  8. 8. def new_message(request): user = get_user_or_404(request) message = request.POST.get('message', None) if not message: raise Http404 user.save_new_message(message) for friend in user.friends.all(): friend.send_email(message) return redirect(reverse('dashboard'))
  9. 9. the problem: that works good for like 0-6 friends... but what if you have 100,000? or more? will your user ever get a response?
  10. 10. bad idea #1: ignore make your users wait through your long request/response cycle. your users are important, right?
  11. 11. bad idea #2: ajax return the page fast with JS code that calls another script in the browser’s background. duplicate calls & http cycles are not cool.
  12. 12. bad idea #3: cronjob make a email_friends table with user_id & message column. cron every 5/10 minutes. backlogs will destroy you.
  13. 13. good idea: queues the task to potentially email thousands of users is put into a queue to be dealt with later, leaving you to return the response.
  14. 14. @task def alert_friends(user_id, message): user = User.objects.get(id=user_id) for friend in user.friends.all(): friend.send_email(message) def new_message(request): user = get_user_or_404(request) message = request.POST.get('message', None) if not message: raise Http404 user.save_new_message(message) alert_friends.delay(user.id, message) return redirect(reverse('dashboard'))
  15. 15. RULE #1: adding a task to a queue should be faster than performing the task itself.
  16. 16. RULE #2: you should consume tasks faster than you produce them. if not, add more workers.
  17. 17. queue libraries: 1. celery (python) earlier... 2. resque (ruby) up next... 3. build your own grand finalé!
  18. 18. class MessageSend def self.perform(user_id, message) user = User.find(user_id) user.friends.each do |friend| friend.send_email(message) end end end class MessageController < ActionController::Base def new_message render(file: 'public/404.html', status: 404) unless params[:message] current_user.save_new_message(params[:message]) Resque.enqueue(MessageSend, current_user.id, params[:message]) redirect_to dashboard_url end end
  19. 19. let’s build our own! and let’s do it with redis, in php and in less than 50 lines of code! (not including the task code)
  20. 20. quick redis aside: so, redis is a key-value store. values can be strings, blobs, lists or hashes. we’ll use lists and RPUSH and LPOP.
  21. 21. step #1: task runner i’m envisioning this as a Task class with various methods named after each task. each method accepts an array $params.
  22. 22. class Tasks { public function email_friends($params) { $user_id = $params->user_id; $message = $params->message; # get $friends from a db query... $friends = array('paul@example.com', 'john@example.com'); foreach ($friends as $friend) { echo "Fake email ".$friend. " with ".$message."n"; } } }
  23. 23. step #2: worker i think an infinite loop with a blocking redis BLPOP will work. then it needs to run the right method on the Task object.
  24. 24. include_once 'redis.php'; # sets up $redis, $queue include_once 'tasks.php'; $tasks = new Tasks(); while (true) { # BLPOP will block until it gets an item.. $data = $redis->blpop($queue, 0); $obj = json_decode($data[1]); # grab json... $method = $obj->task_name; $params = $obj->params; # calls the task: Task->$task_name($params)... call_user_func(array($tasks, $method), $params); }
  25. 25. step #3: run it! we will need to run the worker: ➜ php worker.php
  26. 26. step #4: producer let’s make a handy add_task function for easy placement of tasks into the queue.
  27. 27. include_once 'redis.php'; # sets up $redis & $queue function add_task($task_name, $params) { global $redis, $queue; $data = Array('task_name' => $task_name, 'params' => $params); $json = json_encode($data) $redis->rpush($queue, $json); } # an example of our task api... add_task('email_friends', array('user_id' => 1234, 'message' => 'I just bought a car!') );
  28. 28. step #5: watch it a command line that runs a worker is standard. use something like supervisord or god to run it & monitor it.
  29. 29. things missing: our php example does not: store return results, handle errors, route tasks, degrade gracefully, log activity, etc...
  30. 30. the end question time!

×