Your SlideShare is downloading. ×
Gearman - Job Queue
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Introducing the official SlideShare app

Stunning, full-screen experience for iPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Gearman - Job Queue

7,401
views

Published on

Presentation about the Gearman job queue and examples using PHP

Presentation about the Gearman job queue and examples using PHP


0 Comments
11 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,401
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
99
Comments
0
Likes
11
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Gearman 'The manager' ” since it dispatches jobs to be done, but does not do anything useful itself.”
  • 2. Presentation done using info from... http://www.slideshare.net/pcdinh/gearman-and-asynchronous-processing-in-php-applications-6135047 http://assets.en.oreilly.com/1/event/45/The%20Gearman%20Cookbook%20Presentation.pdf http://www.gearman.org http://nz.php.net/manual/en/book.gearman.php others...
  • 3. Scalable Solutions..
    • More Hardware
    • Caching
    • Precalculated Data
    • Load Balancing
    • Multi-tier application
    • Job Queue
  • 4. History
    • Created by Danga Interactive.
    • Some company that developed Memcache.
    • Original implementation in perl (2005).
    • 2008 rewriteen in C by Brian Aker
    • PHP Extension by James Luedke
  • 5. Used by Digg: 45+ Servers, 400K jobs/day Yahoo: 60+ servers, 6M jobs/day And many others..
  • 6. Installing Compiling: tar xzf gearmand-X.Y.tar.gz cd gearmand-X.Y ./configure make make install Starting server: $ gearmand -d Pecl extension: tar xzf gearman-X.Y.tgz cd gearman-X.Y phpize ./configure make make install To add to the php.ini: extension="gearman.so"
  • 7. Terminology
    • Client: Create jobs to be run and send them to a job server.
    • Worker: Run jobs given from the job server.
    • Job Server: Handle the job queue form clients to workers.
  • 8. “ A massively distributed, massively fault tolerant fork mechanism.” - Joe Stump, SimpleGeo Gearman is...
  • 9.
    • Open Source.
    • Simple & Fast.
    • Multi-language.
    • Flexible application design.
    • Load Balancing.
    • No single point of failure.
    Features
  • 10. Client Worker Job Server Job Server Client Client Client Worker Worker Worker
  • 11.
    • Memory
    • Memcached
    • Mysql/Drizzle
    • PostgreSQL
    • SQLite
    • Tokio Cabinet
    Queue Options
  • 12. Foreground (synchronus) Or Background (asynchronus)
  • 13. Fishpond _Controller_Front::getResource('gearman') ->getGearmanClient() ->doBackground("updateCompetitorPrice", $this->_barcode); ->do("updateCompetitorPrice", $this->_barcode);
    • GearmanClient::do() - Run a single task and return a result
    • GearmanClient::doLow() - Run a single low priority task
    • GearmanClient::doBackground() - Run a task in the background
    • GearmanClient::doHighBackground() - Run a high priority task in the background
    • GearmanClient::doLowBackground() - Run a low priority task in the background
    Gearman Client
  • 14.
    • Scatter / Gather.
    • Map / Reduce.
    • Asynchronus Queues.
    • Pipeline Processing.
    Strategies
  • 15. Scatter / Gather Price Calculation Image Resize Recomendations Product Detail Client
  • 16. $client = Fishpond_Controller_Front::getResource('gearman') ->getGearmanClient(); //adding gearman tasks $client->addTask("getProductDetail", $barcode); $client->addTask("getPrice", $barcode); $client->addTask("resizeImage", serialize($barcode,100,100)); $client->addTask("getRecomendations", $barcode); //callbacks to know when this finish $client->setCompleteCallback(array($this, "complete")); //runing tasks $client->runTasks(); /** * Callback when task is complete * */ public function complete($task) { $data = $task->data(); }
  • 17.
    • GearmanClient::addTaskHigh() - Add a high priority task to run in parallel
    • GearmanClient::addTaskLow() - Add a low priority task to run in parallel
    • GearmanClient::addTaskBackground() - Add a background task to be run in parallel
    • GearmanClient::addTaskHighBackground() - Add a high priority background task to be run in parallel
    • GearmanClient::addTaskLowBackground() - Add a low priority background task to be run in parallel
    • GearmanClient::runTasks() - Run a list of tasks in parallel
    Task Methods
  • 18. GearmanClient::setDataCallback() - Callback function when there is a data packet for a task GearmanClient::setCompleteCallback() - Set a function to be called on task completion GearmanClient::setCreatedCallback() - Set a callback for when a task is queued. GearmanClient::setExceptionCallback() - Set a callback for worker exceptions. GearmanClient::setFailCallback() - Set callback for job failure. GearmanClient::setStatusCallback() - Set a callback for collecting task status. GearmanClient::setWarningCallback() - Set a callback for worker warnings. GearmanClient::setWorkloadCallback() - Set a callback for accepting incremental data updates Client Callback
  • 19.
    • Concurrent tasks with different workers.
    • All tasks run in the time for longest running.
    • Must have enough workers available.
    Scatter / Gather
  • 20. Map/Reduce Client Task T Task T-0 Task T-3 Task T-2 Task T-1 Task T-00 Task T-02 Task T-01
  • 21. Asynchronous Queues
    • No everyting need inmediate procesing..
    • Competitor pricing.
    • Emails.
    • Whole price engine.
    • Loging.
    • Etc.
    Example: $gearmanClient = Fishpond_Controller_Front::getResource('gearman')->getGearmanClient(); $gearmanClient->doBackground("updateCompetitorPrice", $this->_barcode);
  • 22. Loging <VirtualHost *:80> ServerName example.com DocumentRoot /var/www/ CustomLog “| gearman -n -f looger” common (client) </VirtualHost>
  • 23. Pipeline Procesing Client Task T Output Worker Operation 3 Worker Operation 2 Worker Operation 1
  • 24. Questions ?