Job Queue - web is more than request and response

4,880 views

Published on

Job queue presentation for PHP meetup bangalore on 24th April 2010

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,880
On SlideShare
0
From Embeds
0
Number of Embeds
2,455
Actions
Shares
0
Downloads
46
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Job Queue - web is more than request and response

  1. 1. JOB QUEUE web is more than request and response Abhinav Lal
  2. 2. WHO AM I <ul><li>CTO at Naabo </li></ul><ul><li>TurboDoc.in </li></ul><ul><li>Cooking PHP for 4 years </li></ul>
  3. 3. Job Queue <ul><li>In computer science, message queues and mailboxes are software-engineering components used for interprocess communication, or for inter-thread communication within the same process. </li></ul><ul><li>http://en.wikipedia.org/wiki/Message_queue </li></ul><ul><li>Any long running work which does not require user feedback should be done in background. </li></ul>
  4. 4. Order processing without job queue
  5. 5. Order processing with job queue
  6. 6. Jobs to run in background <ul><ul><li>Sending email </li></ul></ul><ul><ul><li>Fetching RSS feed periodically </li></ul></ul><ul><ul><li>Interacting with other application API </li></ul></ul><ul><ul><li>Complex report generation </li></ul></ul><ul><ul><li>Log analysis </li></ul></ul><ul><ul><li>cleaning file system </li></ul></ul><ul><ul><ul><li>periodically or after certain action. </li></ul></ul></ul>
  7. 7. But I can use cron for that <ul><li>Asynchronous </li></ul><ul><li>Low latency </li></ul><ul><li>Reliable </li></ul><ul><li>Scalable </li></ul>
  8. 8. Options out there
  9. 9. Options out there
  10. 10. Gearman = Manager <ul><li>Developed by Danga (People behind memcache) </li></ul><ul><li>Open source C daemon </li></ul><ul><li>Native PHP extension </li></ul><ul><li>Persistence – drizzle, postgres, memcache, sqllite </li></ul><ul><li>Distributed </li></ul><ul><li>No single point of failure </li></ul>
  11. 11. Beanstalkd <ul><li>Open source C daemon </li></ul><ul><li>Fast </li></ul><ul><li>Persistence – bin log file </li></ul><ul><li>Distributed </li></ul><ul><li>PHP library – pheanstalk, beanstalk </li></ul><ul><li>Simple protocol </li></ul>
  12. 12. Simple setup Client script Job Server Worker script
  13. 13. Beanstalkd
  14. 14. Beanstalkd client
  15. 15. Beanstalkd Worker
  16. 16. Beanstalkd Goodness <ul><li>Named Tubes – max 200 bytes </li></ul><ul><li>Priority </li></ul><ul><li>Delayed job </li></ul><ul><li>Bury </li></ul><ul><li>Kick </li></ul>
  17. 17. Advantages <ul><li>Speed up work </li></ul><ul><li>Parallel and asynchronous work </li></ul><ul><li>Scales easily </li></ul><ul><li>Call functionality in other programming language </li></ul><ul><li>Does not block your apache process. </li></ul>
  18. 18. Running worker script <ul><li>Use supervisord </li></ul><ul><li>Gearman Manager </li></ul><ul><li>Gear up </li></ul><ul><li>Check for memory leaks in worker scripts </li></ul><ul><li>Restart your worker scripts after certain number of runs. </li></ul>
  19. 19. <ul><li>Questions </li></ul>Abhinav Lal Blog: abhinavlal.wordpress.com Twitter: @abhinavlal Slideshare: slideshare.net/abhinavlal

×