0
Gearman                 From the Workers PerspectiveTuesday, June 21, 2011
/usr/bin/whoami                            Brian Aker                            CTO/Founder Data Differential            ...
What is Gearman?Tuesday, June 21, 2011
“The way I like to think of Gearman is a               massively distributed fork mechanism”                          -Joe...
resize_image()                   do (“resize_image”)   {                                           …                      ...
!"#$%#&()#*+       !                       !Tuesday, June 21, 2011
!"#$%&$()&$*&+,-./           !   01"2)3+&$4$3)&51)6-5+3$7+12"-&$&8&5+9           !   :45-14.$."43$64.47;)7<               ...
Server                         Provides Asynchronous and Synchronous Requests                         Restarts Work       ...
Client                         # Create our client object.                         $gmclient= new GearmanClient();        ...
Worker                         # Create our worker object.                         $gmw= new GearmanWorker();             ...
Worker Function                          function reverse_fn($job)                          {                           $w...
Lots of functions...                          $gmw->addFunction("resize", "resize_fn");                          $gmw->add...
Function            gearman_return_t fetch_url(gearman_job_st *job, void*)            {              const char *workload=...
Worker Return                         GEARMAN_SUCCESS                         GEARMAN_FATAL                         GEARMA...
reduce() {…}                         map(list[…],                             reduce());   map() {…}                      ...
Map @#$@# ?Tuesday, June 21, 2011
find()       Partitioning                                   find()                         {A...K}                         {...
Partitioning               gearman_return_t split_worker(gearman_job_st *job, void* /* context */)               {        ...
$result      Aggregation                                        $result                           + result                ...
Aggregation              gearman_return_t cat_aggregator (gearman_aggregator_st *,                                        ...
Do we have to                           partition?                          (What other tricks exist!)Tuesday, June 21, 2011
Pipeline                         Store()        Resize()   Publish()Tuesday, June 21, 2011
CPU?    poll()Tuesday, June 21, 2011
Tuesday, June 21, 2011
multiple languagesTuesday, June 21, 2011
Namespaces                         Foo::resize_image()   Acme::resize_image()                           {                 ...
Future                         0.22 Released                         Custom Logging Plugins                         Client...
• Gearman.info (up to date)     • Gearman.org (...)     • http://launchpad.net/gearmand/     • twitter: brianaker     • bl...
Upcoming SlideShare
Loading in...5
×

Gearman, from the worker's perspective

3,332

Published on

Slides from my talk at Open Source Bridges

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,332
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
63
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Transcript of "Gearman, from the worker's perspective"

  1. 1. Gearman From the Workers PerspectiveTuesday, June 21, 2011
  2. 2. /usr/bin/whoami Brian Aker CTO/Founder Data Differential Previously MySQL, Slashdot, Sun MicrosystemsTuesday, June 21, 2011
  3. 3. What is Gearman?Tuesday, June 21, 2011
  4. 4. “The way I like to think of Gearman is a massively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMugTuesday, June 21, 2011
  5. 5. resize_image() do (“resize_image”) { … return $image; }Tuesday, June 21, 2011
  6. 6. !"#$%#&()#*+ ! !Tuesday, June 21, 2011
  7. 7. !"#$%&$()&$*&+,-./ ! 01"2)3+&$4$3)&51)6-5+3$7+12"-&$&8&5+9 ! :45-14.$."43$64.47;)7< " ="1>+1&$41+$7"5),)+3$473$4&>$,"1$#"1>?$7"5$,"1;+3 ! @-.5)A.47<-4<+$)75+<145)"7 ! B)&51)6-5+$C1";+&&)7< " 0"&&)6.8$;."&+1$5"$3454 ! D87;(1"7"-&$473$4&87;(1"7"-&$E-+-+& ! !Tuesday, June 21, 2011
  8. 8. Server Provides Asynchronous and Synchronous Requests Restarts Work Durable Requests Gearman Protocol/HTTP Epoch Scheduling LoggingTuesday, June 21, 2011
  9. 9. Client # Create our client object. $gmclient= new GearmanClient(); # Add default server (localhost). $gmclient->addServer(); $result= $gmclient->do("reverse", "Hello!"); echo "Success: $resultn";Tuesday, June 21, 2011
  10. 10. Worker # Create our worker object. $gmw= new GearmanWorker(); # Add default server (localhost). $gmw->addServer(); $gmw->addFunction("reverse", "reverse_fn"); while ($gmworker->work()) {…}Tuesday, June 21, 2011
  11. 11. Worker Function function reverse_fn($job) { $workload= $job->workload(); $result= strrev($workload); return $result; }Tuesday, June 21, 2011
  12. 12. Lots of functions... $gmw->addFunction("resize", "resize_fn"); $gmw->addFunction("grep", "grep_fn"); $gmw->addFunction("fetch_url", "fetch_url");Tuesday, June 21, 2011
  13. 13. Function gearman_return_t fetch_url(gearman_job_st *job, void*) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100); … gearman_job_send_data(job, chunk, sizeofchunk); … gearman_job_send_status(job, 50,100); … if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  14. 14. Worker Return GEARMAN_SUCCESS GEARMAN_FATAL GEARMAN_ERROR GEARMAN_SHUTDOWNTuesday, June 21, 2011
  15. 15. reduce() {…} map(list[…], reduce()); map() {…} reduce() {…} reduce() {…}Tuesday, June 21, 2011
  16. 16. Map @#$@# ?Tuesday, June 21, 2011
  17. 17. find() Partitioning find() {A...K} {L...Q} {R...Z} find()Tuesday, June 21, 2011
  18. 18. Partitioning gearman_return_t split_worker(gearman_job_st *job, void* /* context */) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); const char *chunk_begin= workload; for (size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin= workload +x +1; } } return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  19. 19. $result Aggregation $result + result + result + result $result = sum resultTuesday, June 21, 2011
  20. 20. Aggregation gearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result) { std::string string_value; do { gearman_result_st *result_ptr= gearman_task_result(task); string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task))); gearman_result_store_value(result, string_value.c_str(), string_value.size()); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  21. 21. Do we have to partition? (What other tricks exist!)Tuesday, June 21, 2011
  22. 22. Pipeline Store() Resize() Publish()Tuesday, June 21, 2011
  23. 23. CPU? poll()Tuesday, June 21, 2011
  24. 24. Tuesday, June 21, 2011
  25. 25. multiple languagesTuesday, June 21, 2011
  26. 26. Namespaces Foo::resize_image() Acme::resize_image() { { … … return $image; return $image; } }Tuesday, June 21, 2011
  27. 27. Future 0.22 Released Custom Logging Plugins Client/Worker Configuration Extended Administrative Commands SSL Job Result Cache Uplift!Tuesday, June 21, 2011
  28. 28. • Gearman.info (up to date) • Gearman.org (...) • http://launchpad.net/gearmand/ • twitter: brianaker • blog: blog.krow.netTuesday, June 21, 2011
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×