Your SlideShare is downloading. ×
  • Like
Gearman, from the worker's perspective
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Gearman, from the worker's perspective

  • 3,227 views
Published

Slides from my talk at Open Source Bridges

Slides from my talk at Open Source Bridges

Published in Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,227
On SlideShare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
62
Comments
0
Likes
9

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Gearman From the Workers PerspectiveTuesday, June 21, 2011
  • 2. /usr/bin/whoami Brian Aker CTO/Founder Data Differential Previously MySQL, Slashdot, Sun MicrosystemsTuesday, June 21, 2011
  • 3. What is Gearman?Tuesday, June 21, 2011
  • 4. “The way I like to think of Gearman is a massively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMugTuesday, June 21, 2011
  • 5. resize_image() do (“resize_image”) { … return $image; }Tuesday, June 21, 2011
  • 6. !"#$%#&()#*+ ! !Tuesday, June 21, 2011
  • 7. !"#$%&$()&$*&+,-./ ! 01"2)3+&$4$3)&51)6-5+3$7+12"-&$&8&5+9 ! :45-14.$."43$64.47;)7< " ="1>+1&$41+$7"5),)+3$473$4&>$,"1$#"1>?$7"5$,"1;+3 ! @-.5)A.47<-4<+$)75+<145)"7 ! B)&51)6-5+$C1";+&&)7< " 0"&&)6.8$;."&+1$5"$3454 ! D87;(1"7"-&$473$4&87;(1"7"-&$E-+-+& ! !Tuesday, June 21, 2011
  • 8. Server Provides Asynchronous and Synchronous Requests Restarts Work Durable Requests Gearman Protocol/HTTP Epoch Scheduling LoggingTuesday, June 21, 2011
  • 9. Client # Create our client object. $gmclient= new GearmanClient(); # Add default server (localhost). $gmclient->addServer(); $result= $gmclient->do("reverse", "Hello!"); echo "Success: $resultn";Tuesday, June 21, 2011
  • 10. Worker # Create our worker object. $gmw= new GearmanWorker(); # Add default server (localhost). $gmw->addServer(); $gmw->addFunction("reverse", "reverse_fn"); while ($gmworker->work()) {…}Tuesday, June 21, 2011
  • 11. Worker Function function reverse_fn($job) { $workload= $job->workload(); $result= strrev($workload); return $result; }Tuesday, June 21, 2011
  • 12. Lots of functions... $gmw->addFunction("resize", "resize_fn"); $gmw->addFunction("grep", "grep_fn"); $gmw->addFunction("fetch_url", "fetch_url");Tuesday, June 21, 2011
  • 13. Function gearman_return_t fetch_url(gearman_job_st *job, void*) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100); … gearman_job_send_data(job, chunk, sizeofchunk); … gearman_job_send_status(job, 50,100); … if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  • 14. Worker Return GEARMAN_SUCCESS GEARMAN_FATAL GEARMAN_ERROR GEARMAN_SHUTDOWNTuesday, June 21, 2011
  • 15. reduce() {…} map(list[…], reduce()); map() {…} reduce() {…} reduce() {…}Tuesday, June 21, 2011
  • 16. Map @#$@# ?Tuesday, June 21, 2011
  • 17. find() Partitioning find() {A...K} {L...Q} {R...Z} find()Tuesday, June 21, 2011
  • 18. Partitioning gearman_return_t split_worker(gearman_job_st *job, void* /* context */) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); const char *chunk_begin= workload; for (size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin= workload +x +1; } } return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  • 19. $result Aggregation $result + result + result + result $result = sum resultTuesday, June 21, 2011
  • 20. Aggregation gearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result) { std::string string_value; do { gearman_result_st *result_ptr= gearman_task_result(task); string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task))); gearman_result_store_value(result, string_value.c_str(), string_value.size()); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  • 21. Do we have to partition? (What other tricks exist!)Tuesday, June 21, 2011
  • 22. Pipeline Store() Resize() Publish()Tuesday, June 21, 2011
  • 23. CPU? poll()Tuesday, June 21, 2011
  • 24. Tuesday, June 21, 2011
  • 25. multiple languagesTuesday, June 21, 2011
  • 26. Namespaces Foo::resize_image() Acme::resize_image() { { … … return $image; return $image; } }Tuesday, June 21, 2011
  • 27. Future 0.22 Released Custom Logging Plugins Client/Worker Configuration Extended Administrative Commands SSL Job Result Cache Uplift!Tuesday, June 21, 2011
  • 28. • Gearman.info (up to date) • Gearman.org (...) • http://launchpad.net/gearmand/ • twitter: brianaker • blog: blog.krow.netTuesday, June 21, 2011