Gearman - Job Queue
Upcoming SlideShare
Loading in...5

Gearman - Job Queue



Presentation about the Gearman job queue and examples using PHP

Presentation about the Gearman job queue and examples using PHP



Total Views
Views on SlideShare
Embed Views



0 Embeds 0

No embeds



Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

Gearman - Job Queue Gearman - Job Queue Presentation Transcript

  • Gearman 'The manager' ” since it dispatches jobs to be done, but does not do anything useful itself.”
  • Presentation done using info from... others...
  • Scalable Solutions..
    • More Hardware
    • Caching
    • Precalculated Data
    • Load Balancing
    • Multi-tier application
    • Job Queue
  • History
    • Created by Danga Interactive.
    • Some company that developed Memcache.
    • Original implementation in perl (2005).
    • 2008 rewriteen in C by Brian Aker
    • PHP Extension by James Luedke
  • Used by Digg: 45+ Servers, 400K jobs/day Yahoo: 60+ servers, 6M jobs/day And many others..
  • Installing Compiling: tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install Starting server: $ gearmand -d Pecl extension: tar xzf gearman-X.Y.tgz cd gearman-X.Y phpize ./configure make make install To add to the php.ini: extension=""
  • Terminology
    • Client: Create jobs to be run and send them to a job server.
    • Worker: Run jobs given from the job server.
    • Job Server: Handle the job queue form clients to workers.
  • “ A massively distributed, massively fault tolerant fork mechanism.” - Joe Stump, SimpleGeo Gearman is...
    • Open Source.
    • Simple & Fast.
    • Multi-language.
    • Flexible application design.
    • Load Balancing.
    • No single point of failure.
  • Client Worker Job Server Job Server Client Client Client Worker Worker Worker
    • Memory
    • Memcached
    • Mysql/Drizzle
    • PostgreSQL
    • SQLite
    • Tokio Cabinet
    Queue Options
  • Foreground (synchronus) Or Background (asynchronus)
  • Fishpond _Controller_Front::getResource('gearman') ->getGearmanClient() ->doBackground("updateCompetitorPrice", $this->_barcode); ->do("updateCompetitorPrice", $this->_barcode);
    • GearmanClient::do() - Run a single task and return a result
    • GearmanClient::doLow() - Run a single low priority task
    • GearmanClient::doBackground() - Run a task in the background
    • GearmanClient::doHighBackground() - Run a high priority task in the background
    • GearmanClient::doLowBackground() - Run a low priority task in the background
    Gearman Client
    • Scatter / Gather.
    • Map / Reduce.
    • Asynchronus Queues.
    • Pipeline Processing.
  • Scatter / Gather Price Calculation Image Resize Recomendations Product Detail Client
  • $client = Fishpond_Controller_Front::getResource('gearman') ->getGearmanClient(); //adding gearman tasks $client->addTask("getProductDetail", $barcode); $client->addTask("getPrice", $barcode); $client->addTask("resizeImage", serialize($barcode,100,100)); $client->addTask("getRecomendations", $barcode); //callbacks to know when this finish $client->setCompleteCallback(array($this, "complete")); //runing tasks $client->runTasks(); /** * Callback when task is complete * */ public function complete($task) { $data = $task->data(); }
    • GearmanClient::addTaskHigh() - Add a high priority task to run in parallel
    • GearmanClient::addTaskLow() - Add a low priority task to run in parallel
    • GearmanClient::addTaskBackground() - Add a background task to be run in parallel
    • GearmanClient::addTaskHighBackground() - Add a high priority background task to be run in parallel
    • GearmanClient::addTaskLowBackground() - Add a low priority background task to be run in parallel
    • GearmanClient::runTasks() - Run a list of tasks in parallel
    Task Methods
  • GearmanClient::setDataCallback() - Callback function when there is a data packet for a task GearmanClient::setCompleteCallback() - Set a function to be called on task completion GearmanClient::setCreatedCallback() - Set a callback for when a task is queued. GearmanClient::setExceptionCallback() - Set a callback for worker exceptions. GearmanClient::setFailCallback() - Set callback for job failure. GearmanClient::setStatusCallback() - Set a callback for collecting task status. GearmanClient::setWarningCallback() - Set a callback for worker warnings. GearmanClient::setWorkloadCallback() - Set a callback for accepting incremental data updates Client Callback
    • Concurrent tasks with different workers.
    • All tasks run in the time for longest running.
    • Must have enough workers available.
    Scatter / Gather
  • Map/Reduce Client Task T Task T-0 Task T-3 Task T-2 Task T-1 Task T-00 Task T-02 Task T-01
  • Asynchronous Queues
    • No everyting need inmediate procesing..
    • Competitor pricing.
    • Emails.
    • Whole price engine.
    • Loging.
    • Etc.
    Example: $gearmanClient = Fishpond_Controller_Front::getResource('gearman')->getGearmanClient(); $gearmanClient->doBackground("updateCompetitorPrice", $this->_barcode);
  • Loging <VirtualHost *:80> ServerName DocumentRoot /var/www/ CustomLog “| gearman -n -f looger” common (client) </VirtualHost>
  • Pipeline Procesing Client Task T Output Worker Operation 3 Worker Operation 2 Worker Operation 1
  • Questions ?