Your SlideShare is downloading. ×
Gearman, from the worker's perspective
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Gearman, from the worker's perspective

3,274

Published on

Slides from my talk at Open Source Bridges

Slides from my talk at Open Source Bridges

Published in: Technology
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
3,274
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
62
Comments
0
Likes
9
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Gearman From the Workers PerspectiveTuesday, June 21, 2011
  • 2. /usr/bin/whoami Brian Aker CTO/Founder Data Differential Previously MySQL, Slashdot, Sun MicrosystemsTuesday, June 21, 2011
  • 3. What is Gearman?Tuesday, June 21, 2011
  • 4. “The way I like to think of Gearman is a massively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMugTuesday, June 21, 2011
  • 5. resize_image() do (“resize_image”) { … return $image; }Tuesday, June 21, 2011
  • 6. !"#$%#&()#*+ ! !Tuesday, June 21, 2011
  • 7. !"#$%&$()&$*&+,-./ ! 01"2)3+&$4$3)&51)6-5+3$7+12"-&$&8&5+9 ! :45-14.$."43$64.47;)7< " ="1>+1&$41+$7"5),)+3$473$4&>$,"1$#"1>?$7"5$,"1;+3 ! @-.5)A.47<-4<+$)75+<145)"7 ! B)&51)6-5+$C1";+&&)7< " 0"&&)6.8$;."&+1$5"$3454 ! D87;(1"7"-&$473$4&87;(1"7"-&$E-+-+& ! !Tuesday, June 21, 2011
  • 8. Server Provides Asynchronous and Synchronous Requests Restarts Work Durable Requests Gearman Protocol/HTTP Epoch Scheduling LoggingTuesday, June 21, 2011
  • 9. Client # Create our client object. $gmclient= new GearmanClient(); # Add default server (localhost). $gmclient->addServer(); $result= $gmclient->do("reverse", "Hello!"); echo "Success: $resultn";Tuesday, June 21, 2011
  • 10. Worker # Create our worker object. $gmw= new GearmanWorker(); # Add default server (localhost). $gmw->addServer(); $gmw->addFunction("reverse", "reverse_fn"); while ($gmworker->work()) {…}Tuesday, June 21, 2011
  • 11. Worker Function function reverse_fn($job) { $workload= $job->workload(); $result= strrev($workload); return $result; }Tuesday, June 21, 2011
  • 12. Lots of functions... $gmw->addFunction("resize", "resize_fn"); $gmw->addFunction("grep", "grep_fn"); $gmw->addFunction("fetch_url", "fetch_url");Tuesday, June 21, 2011
  • 13. Function gearman_return_t fetch_url(gearman_job_st *job, void*) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100); … gearman_job_send_data(job, chunk, sizeofchunk); … gearman_job_send_status(job, 50,100); … if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  • 14. Worker Return GEARMAN_SUCCESS GEARMAN_FATAL GEARMAN_ERROR GEARMAN_SHUTDOWNTuesday, June 21, 2011
  • 15. reduce() {…} map(list[…], reduce()); map() {…} reduce() {…} reduce() {…}Tuesday, June 21, 2011
  • 16. Map @#$@# ?Tuesday, June 21, 2011
  • 17. find() Partitioning find() {A...K} {L...Q} {R...Z} find()Tuesday, June 21, 2011
  • 18. Partitioning gearman_return_t split_worker(gearman_job_st *job, void* /* context */) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); const char *chunk_begin= workload; for (size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin= workload +x +1; } } return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  • 19. $result Aggregation $result + result + result + result $result = sum resultTuesday, June 21, 2011
  • 20. Aggregation gearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result) { std::string string_value; do { gearman_result_st *result_ptr= gearman_task_result(task); string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task))); gearman_result_store_value(result, string_value.c_str(), string_value.size()); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  • 21. Do we have to partition? (What other tricks exist!)Tuesday, June 21, 2011
  • 22. Pipeline Store() Resize() Publish()Tuesday, June 21, 2011
  • 23. CPU? poll()Tuesday, June 21, 2011
  • 24. Tuesday, June 21, 2011
  • 25. multiple languagesTuesday, June 21, 2011
  • 26. Namespaces Foo::resize_image() Acme::resize_image() { { … … return $image; return $image; } }Tuesday, June 21, 2011
  • 27. Future 0.22 Released Custom Logging Plugins Client/Worker Configuration Extended Administrative Commands SSL Job Result Cache Uplift!Tuesday, June 21, 2011
  • 28. • Gearman.info (up to date) • Gearman.org (...) • http://launchpad.net/gearmand/ • twitter: brianaker • blog: blog.krow.netTuesday, June 21, 2011

×