Gearmanpresentation 110308165409-phpapp01


Published on


Published in: Education, Technology, Business
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Gearmanpresentation 110308165409-phpapp01

  1. 1. Gearman The manager ”since it dispatches jobs to be done,but does not do anything useful itself.”
  2. 2. Presentation done using info from...
  3. 3. Scalable Solutions..  More Hardware  Caching  Precalculated Data  Load Balancing  Multi-tier application  Job Queue
  4. 4. History Created by Danga Interactive. Some company that developed Memcache. Original implementation in perl (2005). 2008 rewriteen in C by Brian Aker PHP Extension by James Luedke
  5. 5. Used byDigg: 45+ Servers, 400K jobs/dayYahoo: 60+ servers, 6M jobs/dayAnd many others..
  6. 6. InstallingCompiling:tar xzf gearmand-X.Y.tar.gzcd gearmand-X.Y./configuremakemake installStarting server:$ gearmand -dPecl extension:tar xzf gearman-X.Y.tgzcd gearman-X.Yphpize./configuremakemake installTo add to the php.ini:extension=""
  7. 7. Terminology Client: Create jobs to be run and send them to a job server. Worker: Run jobs given from the job server. Job Server: Handle the job queue form clients to workers.
  8. 8. Gearman is...“A massively distributed,massively fault tolerantfork mechanism.”- Joe Stump, SimpleGeo
  9. 9. Features Open Source. Simple & Fast. Multi-language. Flexible application design. Load Balancing. No single point of failure.
  10. 10. Client Client Client Client Job Server Job ServerWorker Worker Worker Worker
  11. 11. Queue Options  Memory  Memcached  Mysql/Drizzle  PostgreSQL  SQLite  Tokio Cabinet
  12. 12. Foreground(synchronus) Or Background(asynchronus)
  13. 13. Gearman Client Fishpond_Controller_Front::getResource(gearman) ->getGearmanClient() ->doBackground("updateCompetitorPrice", $this->_barcode); ->do("updateCompetitorPrice", $this->_barcode); GearmanClient::do() - Run a single task and return a result GearmanClient::doLow() - Run a single low priority task GearmanClient::doBackground() - Run a task in the background GearmanClient::doHighBackground() - Run a high priority task in the background GearmanClient::doLowBackground() - Run a low priority task in the background
  14. 14. Strategies Scatter / Gather. Map / Reduce. Asynchronus Queues. Pipeline Processing.
  15. 15. Scatter / Gather ClientProduct Detail Recomendations Price Calculation Image Resize
  16. 16. $client = Fishpond_Controller_Front::getResource(gearman) ->getGearmanClient(); //adding gearman tasks $client->addTask("getProductDetail", $barcode); $client->addTask("getPrice", $barcode); $client->addTask("resizeImage", serialize($barcode,100,100)); $client->addTask("getRecomendations", $barcode); //callbacks to know when this finish $client->setCompleteCallback(array($this, "complete")); //runing tasks $client->runTasks();/** * Callback when task is complete * */public function complete($task){ $data = $task->data();}
  17. 17. Task Methods GearmanClient::addTaskHigh() - Add a high priority task to run in parallel GearmanClient::addTaskLow() - Add a low priority task to run in parallel GearmanClient::addTaskBackground() - Add a background task to be run in parallel GearmanClient::addTaskHighBackground() - Add a high priority background task to be run in parallel GearmanClient::addTaskLowBackground() - Add a low priority background task to be run in parallel GearmanClient::runTasks() - Run a list of tasks in parallel
  18. 18. Client CallbackGearmanClient::setDataCallback() - Callback function when there is a data packet for a taskGearmanClient::setCompleteCallback() - Set a function to be called on task completionGearmanClient::setCreatedCallback() - Set a callback for when a task is queued.GearmanClient::setExceptionCallback() - Set a callback for worker exceptions.GearmanClient::setFailCallback() - Set callback for job failure.GearmanClient::setStatusCallback() - Set a callback for collecting task status.GearmanClient::setWarningCallback() - Set a callback for worker warnings.GearmanClient::setWorkloadCallback() - Set a callback for accepting incremental data updates
  19. 19. Scatter / Gather Concurrent tasks with different workers. All tasks run in the time for longest running. Must have enough workers available.
  20. 20. Map/Reduce Client Task TTask T-0 Task T-1 Task T-2 Task T-3Task T-00 Task T-01 Task T-02
  21. 21. Asynchronous Queues No everyting need inmediate procesing..  Competitor pricing.  Emails.  Whole price engine.  Loging.  Etc.Example:$gearmanClient = Fishpond_Controller_Front::getResource(gearman)->getGearmanClient();$gearmanClient->doBackground("updateCompetitorPrice", $this->_barcode);
  22. 22. Loging<VirtualHost *:80> ServerName DocumentRoot /var/www/ CustomLog “| gearman -n -f looger” common (client)</VirtualHost>
  23. 23. Pipeline Procesing Client Task T Worker Worker WorkerOperation 1 Operation 2 Operation 3 Output
  24. 24. Questions ?