Gearman - Job Queue
Upcoming SlideShare
Loading in...5

Gearman - Job Queue



Presentation about the Gearman job queue and examples using PHP

Presentation about the Gearman job queue and examples using PHP



Total Views
Slideshare-icon Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Gearman - Job Queue Gearman - Job Queue Presentation Transcript

    • Gearman 'The manager' ” since it dispatches jobs to be done, but does not do anything useful itself.”
    • Presentation done using info from... others...
    • Scalable Solutions..
      • More Hardware
      • Caching
      • Precalculated Data
      • Load Balancing
      • Multi-tier application
      • Job Queue
    • History
      • Created by Danga Interactive.
      • Some company that developed Memcache.
      • Original implementation in perl (2005).
      • 2008 rewriteen in C by Brian Aker
      • PHP Extension by James Luedke
    • Used by Digg: 45+ Servers, 400K jobs/day Yahoo: 60+ servers, 6M jobs/day And many others..
    • Installing Compiling: tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install Starting server: $ gearmand -d Pecl extension: tar xzf gearman-X.Y.tgz cd gearman-X.Y phpize ./configure make make install To add to the php.ini: extension=""
    • Terminology
      • Client: Create jobs to be run and send them to a job server.
      • Worker: Run jobs given from the job server.
      • Job Server: Handle the job queue form clients to workers.
    • “ A massively distributed, massively fault tolerant fork mechanism.” - Joe Stump, SimpleGeo Gearman is...
      • Open Source.
      • Simple & Fast.
      • Multi-language.
      • Flexible application design.
      • Load Balancing.
      • No single point of failure.
    • Client Worker Job Server Job Server Client Client Client Worker Worker Worker
      • Memory
      • Memcached
      • Mysql/Drizzle
      • PostgreSQL
      • SQLite
      • Tokio Cabinet
      Queue Options
    • Foreground (synchronus) Or Background (asynchronus)
    • Fishpond _Controller_Front::getResource('gearman') ->getGearmanClient() ->doBackground("updateCompetitorPrice", $this->_barcode); ->do("updateCompetitorPrice", $this->_barcode);
      • GearmanClient::do() - Run a single task and return a result
      • GearmanClient::doLow() - Run a single low priority task
      • GearmanClient::doBackground() - Run a task in the background
      • GearmanClient::doHighBackground() - Run a high priority task in the background
      • GearmanClient::doLowBackground() - Run a low priority task in the background
      Gearman Client
      • Scatter / Gather.
      • Map / Reduce.
      • Asynchronus Queues.
      • Pipeline Processing.
    • Scatter / Gather Price Calculation Image Resize Recomendations Product Detail Client
    • $client = Fishpond_Controller_Front::getResource('gearman') ->getGearmanClient(); //adding gearman tasks $client->addTask("getProductDetail", $barcode); $client->addTask("getPrice", $barcode); $client->addTask("resizeImage", serialize($barcode,100,100)); $client->addTask("getRecomendations", $barcode); //callbacks to know when this finish $client->setCompleteCallback(array($this, "complete")); //runing tasks $client->runTasks(); /** * Callback when task is complete * */ public function complete($task) { $data = $task->data(); }
      • GearmanClient::addTaskHigh() - Add a high priority task to run in parallel
      • GearmanClient::addTaskLow() - Add a low priority task to run in parallel
      • GearmanClient::addTaskBackground() - Add a background task to be run in parallel
      • GearmanClient::addTaskHighBackground() - Add a high priority background task to be run in parallel
      • GearmanClient::addTaskLowBackground() - Add a low priority background task to be run in parallel
      • GearmanClient::runTasks() - Run a list of tasks in parallel
      Task Methods
    • GearmanClient::setDataCallback() - Callback function when there is a data packet for a task GearmanClient::setCompleteCallback() - Set a function to be called on task completion GearmanClient::setCreatedCallback() - Set a callback for when a task is queued. GearmanClient::setExceptionCallback() - Set a callback for worker exceptions. GearmanClient::setFailCallback() - Set callback for job failure. GearmanClient::setStatusCallback() - Set a callback for collecting task status. GearmanClient::setWarningCallback() - Set a callback for worker warnings. GearmanClient::setWorkloadCallback() - Set a callback for accepting incremental data updates Client Callback
      • Concurrent tasks with different workers.
      • All tasks run in the time for longest running.
      • Must have enough workers available.
      Scatter / Gather
    • Map/Reduce Client Task T Task T-0 Task T-3 Task T-2 Task T-1 Task T-00 Task T-02 Task T-01
    • Asynchronous Queues
      • No everyting need inmediate procesing..
      • Competitor pricing.
      • Emails.
      • Whole price engine.
      • Loging.
      • Etc.
      Example: $gearmanClient = Fishpond_Controller_Front::getResource('gearman')->getGearmanClient(); $gearmanClient->doBackground("updateCompetitorPrice", $this->_barcode);
    • Loging <VirtualHost *:80> ServerName DocumentRoot /var/www/ CustomLog “| gearman -n -f looger” common (client) </VirtualHost>
    • Pipeline Procesing Client Task T Output Worker Operation 3 Worker Operation 2 Worker Operation 1
    • Questions ?