Gearman, from the worker's perspective

3,720 views
3,593 views

Published on

Slides from my talk at Open Source Bridges

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
3,720
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
64
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Gearman, from the worker's perspective

  1. 1. Gearman From the Workers PerspectiveTuesday, June 21, 2011
  2. 2. /usr/bin/whoami Brian Aker CTO/Founder Data Differential Previously MySQL, Slashdot, Sun MicrosystemsTuesday, June 21, 2011
  3. 3. What is Gearman?Tuesday, June 21, 2011
  4. 4. “The way I like to think of Gearman is a massively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMugTuesday, June 21, 2011
  5. 5. resize_image() do (“resize_image”) { … return $image; }Tuesday, June 21, 2011
  6. 6. !"#$%#&()#*+ ! !Tuesday, June 21, 2011
  7. 7. !"#$%&$()&$*&+,-./ ! 01"2)3+&$4$3)&51)6-5+3$7+12"-&$&8&5+9 ! :45-14.$."43$64.47;)7< " ="1>+1&$41+$7"5),)+3$473$4&>$,"1$#"1>?$7"5$,"1;+3 ! @-.5)A.47<-4<+$)75+<145)"7 ! B)&51)6-5+$C1";+&&)7< " 0"&&)6.8$;."&+1$5"$3454 ! D87;(1"7"-&$473$4&87;(1"7"-&$E-+-+& ! !Tuesday, June 21, 2011
  8. 8. Server Provides Asynchronous and Synchronous Requests Restarts Work Durable Requests Gearman Protocol/HTTP Epoch Scheduling LoggingTuesday, June 21, 2011
  9. 9. Client # Create our client object. $gmclient= new GearmanClient(); # Add default server (localhost). $gmclient->addServer(); $result= $gmclient->do("reverse", "Hello!"); echo "Success: $resultn";Tuesday, June 21, 2011
  10. 10. Worker # Create our worker object. $gmw= new GearmanWorker(); # Add default server (localhost). $gmw->addServer(); $gmw->addFunction("reverse", "reverse_fn"); while ($gmworker->work()) {…}Tuesday, June 21, 2011
  11. 11. Worker Function function reverse_fn($job) { $workload= $job->workload(); $result= strrev($workload); return $result; }Tuesday, June 21, 2011
  12. 12. Lots of functions... $gmw->addFunction("resize", "resize_fn"); $gmw->addFunction("grep", "grep_fn"); $gmw->addFunction("fetch_url", "fetch_url");Tuesday, June 21, 2011
  13. 13. Function gearman_return_t fetch_url(gearman_job_st *job, void*) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100); … gearman_job_send_data(job, chunk, sizeofchunk); … gearman_job_send_status(job, 50,100); … if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  14. 14. Worker Return GEARMAN_SUCCESS GEARMAN_FATAL GEARMAN_ERROR GEARMAN_SHUTDOWNTuesday, June 21, 2011
  15. 15. reduce() {…} map(list[…], reduce()); map() {…} reduce() {…} reduce() {…}Tuesday, June 21, 2011
  16. 16. Map @#$@# ?Tuesday, June 21, 2011
  17. 17. find() Partitioning find() {A...K} {L...Q} {R...Z} find()Tuesday, June 21, 2011
  18. 18. Partitioning gearman_return_t split_worker(gearman_job_st *job, void* /* context */) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); const char *chunk_begin= workload; for (size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin= workload +x +1; } } return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  19. 19. $result Aggregation $result + result + result + result $result = sum resultTuesday, June 21, 2011
  20. 20. Aggregation gearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result) { std::string string_value; do { gearman_result_st *result_ptr= gearman_task_result(task); string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task))); gearman_result_store_value(result, string_value.c_str(), string_value.size()); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  21. 21. Do we have to partition? (What other tricks exist!)Tuesday, June 21, 2011
  22. 22. Pipeline Store() Resize() Publish()Tuesday, June 21, 2011
  23. 23. CPU? poll()Tuesday, June 21, 2011
  24. 24. Tuesday, June 21, 2011
  25. 25. multiple languagesTuesday, June 21, 2011
  26. 26. Namespaces Foo::resize_image() Acme::resize_image() { { … … return $image; return $image; } }Tuesday, June 21, 2011
  27. 27. Future 0.22 Released Custom Logging Plugins Client/Worker Configuration Extended Administrative Commands SSL Job Result Cache Uplift!Tuesday, June 21, 2011
  28. 28. • Gearman.info (up to date) • Gearman.org (...) • http://launchpad.net/gearmand/ • twitter: brianaker • blog: blog.krow.netTuesday, June 21, 2011

×