Gearman - Job Queue


Published on

Presentation about the Gearman job queue and examples using PHP

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Gearman - Job Queue

  1. 1. Gearman 'The manager' ” since it dispatches jobs to be done, but does not do anything useful itself.”
  2. 2. Presentation done using info from... others...
  3. 3. Scalable Solutions.. <ul><li>More Hardware </li></ul><ul><li>Caching </li></ul><ul><li>Precalculated Data </li></ul><ul><li>Load Balancing </li></ul><ul><li>Multi-tier application </li></ul><ul><li>Job Queue </li></ul>
  4. 4. History <ul><li>Created by Danga Interactive. </li></ul><ul><li>Some company that developed Memcache. </li></ul><ul><li>Original implementation in perl (2005). </li></ul><ul><li>2008 rewriteen in C by Brian Aker </li></ul><ul><li>PHP Extension by James Luedke </li></ul>
  5. 5. Used by Digg: 45+ Servers, 400K jobs/day Yahoo: 60+ servers, 6M jobs/day And many others..
  6. 6. Installing Compiling: tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install Starting server: $ gearmand -d Pecl extension: tar xzf gearman-X.Y.tgz cd gearman-X.Y phpize ./configure make make install To add to the php.ini: extension=&quot;;
  7. 7. Terminology <ul><li>Client: Create jobs to be run and send them to a job server. </li></ul><ul><li>Worker: Run jobs given from the job server. </li></ul><ul><li>Job Server: Handle the job queue form clients to workers. </li></ul>
  8. 8. “ A massively distributed, massively fault tolerant fork mechanism.” - Joe Stump, SimpleGeo Gearman is...
  9. 9. <ul><li>Open Source. </li></ul><ul><li>Simple & Fast. </li></ul><ul><li>Multi-language. </li></ul><ul><li>Flexible application design. </li></ul><ul><li>Load Balancing. </li></ul><ul><li>No single point of failure. </li></ul>Features
  10. 10. Client Worker Job Server Job Server Client Client Client Worker Worker Worker
  11. 11. <ul><li>Memory </li></ul><ul><li>Memcached </li></ul><ul><li>Mysql/Drizzle </li></ul><ul><li>PostgreSQL </li></ul><ul><li>SQLite </li></ul><ul><li>Tokio Cabinet </li></ul>Queue Options
  12. 12. Foreground (synchronus) Or Background (asynchronus)
  13. 13. Fishpond _Controller_Front::getResource('gearman') ->getGearmanClient() ->doBackground(&quot;updateCompetitorPrice&quot;, $this->_barcode); ->do(&quot;updateCompetitorPrice&quot;, $this->_barcode); <ul><li>GearmanClient::do() - Run a single task and return a result </li></ul><ul><li>GearmanClient::doLow() - Run a single low priority task </li></ul><ul><li>GearmanClient::doBackground() - Run a task in the background </li></ul><ul><li>GearmanClient::doHighBackground() - Run a high priority task in the background </li></ul><ul><li>GearmanClient::doLowBackground() - Run a low priority task in the background </li></ul>Gearman Client
  14. 14. <ul><li>Scatter / Gather. </li></ul><ul><li>Map / Reduce. </li></ul><ul><li>Asynchronus Queues. </li></ul><ul><li>Pipeline Processing. </li></ul>Strategies
  15. 15. Scatter / Gather Price Calculation Image Resize Recomendations Product Detail Client
  16. 16. $client = Fishpond_Controller_Front::getResource('gearman') ->getGearmanClient(); //adding gearman tasks $client->addTask(&quot;getProductDetail&quot;, $barcode); $client->addTask(&quot;getPrice&quot;, $barcode); $client->addTask(&quot;resizeImage&quot;, serialize($barcode,100,100)); $client->addTask(&quot;getRecomendations&quot;, $barcode); //callbacks to know when this finish $client->setCompleteCallback(array($this, &quot;complete&quot;)); //runing tasks $client->runTasks(); /** * Callback when task is complete * */ public function complete($task) { $data = $task->data(); }
  17. 17. <ul><li>GearmanClient::addTaskHigh() - Add a high priority task to run in parallel </li></ul><ul><li>GearmanClient::addTaskLow() - Add a low priority task to run in parallel </li></ul><ul><li>GearmanClient::addTaskBackground() - Add a background task to be run in parallel </li></ul><ul><li>GearmanClient::addTaskHighBackground() - Add a high priority background task to be run in parallel </li></ul><ul><li>GearmanClient::addTaskLowBackground() - Add a low priority background task to be run in parallel </li></ul><ul><li>GearmanClient::runTasks() - Run a list of tasks in parallel </li></ul>Task Methods
  18. 18. GearmanClient::setDataCallback() - Callback function when there is a data packet for a task GearmanClient::setCompleteCallback() - Set a function to be called on task completion GearmanClient::setCreatedCallback() - Set a callback for when a task is queued. GearmanClient::setExceptionCallback() - Set a callback for worker exceptions. GearmanClient::setFailCallback() - Set callback for job failure. GearmanClient::setStatusCallback() - Set a callback for collecting task status. GearmanClient::setWarningCallback() - Set a callback for worker warnings. GearmanClient::setWorkloadCallback() - Set a callback for accepting incremental data updates Client Callback
  19. 19. <ul><li>Concurrent tasks with different workers. </li></ul><ul><li>All tasks run in the time for longest running. </li></ul><ul><li>Must have enough workers available. </li></ul>Scatter / Gather
  20. 20. Map/Reduce Client Task T Task T-0 Task T-3 Task T-2 Task T-1 Task T-00 Task T-02 Task T-01
  21. 21. Asynchronous Queues <ul><li>No everyting need inmediate procesing.. </li></ul><ul><li>Competitor pricing. </li></ul><ul><li>Emails. </li></ul><ul><li>Whole price engine. </li></ul><ul><li>Loging. </li></ul><ul><li>Etc. </li></ul>Example: $gearmanClient = Fishpond_Controller_Front::getResource('gearman')->getGearmanClient(); $gearmanClient->doBackground(&quot;updateCompetitorPrice&quot;, $this->_barcode);
  22. 22. Loging <VirtualHost *:80> ServerName DocumentRoot /var/www/ CustomLog “| gearman -n -f looger” common (client) </VirtualHost>
  23. 23. Pipeline Procesing Client Task T Output Worker Operation 3 Worker Operation 2 Worker Operation 1
  24. 24. Questions ?