Gearman - Job Queue
Upcoming SlideShare
Loading in...5
×
 

Gearman - Job Queue

on

  • 7,707 views

Presentation about the Gearman job queue and examples using PHP

Presentation about the Gearman job queue and examples using PHP

Statistics

Views

Total Views
7,707
Views on SlideShare
7,707
Embed Views
0

Actions

Likes
10
Downloads
91
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Gearman - Job Queue Gearman - Job Queue Presentation Transcript

  • Gearman 'The manager' ” since it dispatches jobs to be done, but does not do anything useful itself.”
  • Presentation done using info from... http://www.slideshare.net/pcdinh/gearman-and-asynchronous-processing-in-php-applications-6135047 http://assets.en.oreilly.com/1/event/45/The%20Gearman%20Cookbook%20Presentation.pdf http://www.gearman.org http://nz.php.net/manual/en/book.gearman.php others...
  • Scalable Solutions..
    • More Hardware
    • Caching
    • Precalculated Data
    • Load Balancing
    • Multi-tier application
    • Job Queue
  • History
    • Created by Danga Interactive.
    • Some company that developed Memcache.
    • Original implementation in perl (2005).
    • 2008 rewriteen in C by Brian Aker
    • PHP Extension by James Luedke
  • Used by Digg: 45+ Servers, 400K jobs/day Yahoo: 60+ servers, 6M jobs/day And many others..
  • Installing Compiling: tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install Starting server: $ gearmand -d Pecl extension: tar xzf gearman-X.Y.tgz cd gearman-X.Y phpize ./configure make make install To add to the php.ini: extension="gearman.so"
  • Terminology
    • Client: Create jobs to be run and send them to a job server.
    • Worker: Run jobs given from the job server.
    • Job Server: Handle the job queue form clients to workers.
  • “ A massively distributed, massively fault tolerant fork mechanism.” - Joe Stump, SimpleGeo Gearman is...
    • Open Source.
    • Simple & Fast.
    • Multi-language.
    • Flexible application design.
    • Load Balancing.
    • No single point of failure.
    Features
  • Client Worker Job Server Job Server Client Client Client Worker Worker Worker
    • Memory
    • Memcached
    • Mysql/Drizzle
    • PostgreSQL
    • SQLite
    • Tokio Cabinet
    Queue Options
  • Foreground (synchronus) Or Background (asynchronus)
  • Fishpond _Controller_Front::getResource('gearman') ->getGearmanClient() ->doBackground("updateCompetitorPrice", $this->_barcode); ->do("updateCompetitorPrice", $this->_barcode);
    • GearmanClient::do() - Run a single task and return a result
    • GearmanClient::doLow() - Run a single low priority task
    • GearmanClient::doBackground() - Run a task in the background
    • GearmanClient::doHighBackground() - Run a high priority task in the background
    • GearmanClient::doLowBackground() - Run a low priority task in the background
    Gearman Client
    • Scatter / Gather.
    • Map / Reduce.
    • Asynchronus Queues.
    • Pipeline Processing.
    Strategies
  • Scatter / Gather Price Calculation Image Resize Recomendations Product Detail Client
  • $client = Fishpond_Controller_Front::getResource('gearman') ->getGearmanClient(); //adding gearman tasks $client->addTask("getProductDetail", $barcode); $client->addTask("getPrice", $barcode); $client->addTask("resizeImage", serialize($barcode,100,100)); $client->addTask("getRecomendations", $barcode); //callbacks to know when this finish $client->setCompleteCallback(array($this, "complete")); //runing tasks $client->runTasks(); /** * Callback when task is complete * */ public function complete($task) { $data = $task->data(); }
    • GearmanClient::addTaskHigh() - Add a high priority task to run in parallel
    • GearmanClient::addTaskLow() - Add a low priority task to run in parallel
    • GearmanClient::addTaskBackground() - Add a background task to be run in parallel
    • GearmanClient::addTaskHighBackground() - Add a high priority background task to be run in parallel
    • GearmanClient::addTaskLowBackground() - Add a low priority background task to be run in parallel
    • GearmanClient::runTasks() - Run a list of tasks in parallel
    Task Methods
  • GearmanClient::setDataCallback() - Callback function when there is a data packet for a task GearmanClient::setCompleteCallback() - Set a function to be called on task completion GearmanClient::setCreatedCallback() - Set a callback for when a task is queued. GearmanClient::setExceptionCallback() - Set a callback for worker exceptions. GearmanClient::setFailCallback() - Set callback for job failure. GearmanClient::setStatusCallback() - Set a callback for collecting task status. GearmanClient::setWarningCallback() - Set a callback for worker warnings. GearmanClient::setWorkloadCallback() - Set a callback for accepting incremental data updates Client Callback
    • Concurrent tasks with different workers.
    • All tasks run in the time for longest running.
    • Must have enough workers available.
    Scatter / Gather
  • Map/Reduce Client Task T Task T-0 Task T-3 Task T-2 Task T-1 Task T-00 Task T-02 Task T-01
  • Asynchronous Queues
    • No everyting need inmediate procesing..
    • Competitor pricing.
    • Emails.
    • Whole price engine.
    • Loging.
    • Etc.
    Example: $gearmanClient = Fishpond_Controller_Front::getResource('gearman')->getGearmanClient(); $gearmanClient->doBackground("updateCompetitorPrice", $this->_barcode);
  • Loging <VirtualHost *:80> ServerName example.com DocumentRoot /var/www/ CustomLog “| gearman -n -f looger” common (client) </VirtualHost>
  • Pipeline Procesing Client Task T Output Worker Operation 3 Worker Operation 2 Worker Operation 1
  • Questions ?