Gearman, from the worker's perspective

  • 3,190 views
Uploaded on

Slides from my talk at Open Source Bridges

Slides from my talk at Open Source Bridges

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
3,190
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
61
Comments
0
Likes
9

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Gearman From the Workers PerspectiveTuesday, June 21, 2011
  • 2. /usr/bin/whoami Brian Aker CTO/Founder Data Differential Previously MySQL, Slashdot, Sun MicrosystemsTuesday, June 21, 2011
  • 3. What is Gearman?Tuesday, June 21, 2011
  • 4. “The way I like to think of Gearman is a massively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMugTuesday, June 21, 2011
  • 5. resize_image() do (“resize_image”) { … return $image; }Tuesday, June 21, 2011
  • 6. !"#$%#&()#*+ ! !Tuesday, June 21, 2011
  • 7. !"#$%&$()&$*&+,-./ ! 01"2)3+&$4$3)&51)6-5+3$7+12"-&$&8&5+9 ! :45-14.$."43$64.47;)7< " ="1>+1&$41+$7"5),)+3$473$4&>$,"1$#"1>?$7"5$,"1;+3 ! @-.5)A.47<-4<+$)75+<145)"7 ! B)&51)6-5+$C1";+&&)7< " 0"&&)6.8$;."&+1$5"$3454 ! D87;(1"7"-&$473$4&87;(1"7"-&$E-+-+& ! !Tuesday, June 21, 2011
  • 8. Server Provides Asynchronous and Synchronous Requests Restarts Work Durable Requests Gearman Protocol/HTTP Epoch Scheduling LoggingTuesday, June 21, 2011
  • 9. Client # Create our client object. $gmclient= new GearmanClient(); # Add default server (localhost). $gmclient->addServer(); $result= $gmclient->do("reverse", "Hello!"); echo "Success: $resultn";Tuesday, June 21, 2011
  • 10. Worker # Create our worker object. $gmw= new GearmanWorker(); # Add default server (localhost). $gmw->addServer(); $gmw->addFunction("reverse", "reverse_fn"); while ($gmworker->work()) {…}Tuesday, June 21, 2011
  • 11. Worker Function function reverse_fn($job) { $workload= $job->workload(); $result= strrev($workload); return $result; }Tuesday, June 21, 2011
  • 12. Lots of functions... $gmw->addFunction("resize", "resize_fn"); $gmw->addFunction("grep", "grep_fn"); $gmw->addFunction("fetch_url", "fetch_url");Tuesday, June 21, 2011
  • 13. Function gearman_return_t fetch_url(gearman_job_st *job, void*) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100); … gearman_job_send_data(job, chunk, sizeofchunk); … gearman_job_send_status(job, 50,100); … if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  • 14. Worker Return GEARMAN_SUCCESS GEARMAN_FATAL GEARMAN_ERROR GEARMAN_SHUTDOWNTuesday, June 21, 2011
  • 15. reduce() {…} map(list[…], reduce()); map() {…} reduce() {…} reduce() {…}Tuesday, June 21, 2011
  • 16. Map @#$@# ?Tuesday, June 21, 2011
  • 17. find() Partitioning find() {A...K} {L...Q} {R...Z} find()Tuesday, June 21, 2011
  • 18. Partitioning gearman_return_t split_worker(gearman_job_st *job, void* /* context */) { const char *workload= gearman_job_workload(job); size_t workload_size= gearman_job_workload_size(job); const char *chunk_begin= workload; for (size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin= workload +x +1; } } return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  • 19. $result Aggregation $result + result + result + result $result = sum resultTuesday, June 21, 2011
  • 20. Aggregation gearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result) { std::string string_value; do { gearman_result_st *result_ptr= gearman_task_result(task); string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task))); gearman_result_store_value(result, string_value.c_str(), string_value.size()); return GEARMAN_SUCCESS; }Tuesday, June 21, 2011
  • 21. Do we have to partition? (What other tricks exist!)Tuesday, June 21, 2011
  • 22. Pipeline Store() Resize() Publish()Tuesday, June 21, 2011
  • 23. CPU? poll()Tuesday, June 21, 2011
  • 24. Tuesday, June 21, 2011
  • 25. multiple languagesTuesday, June 21, 2011
  • 26. Namespaces Foo::resize_image() Acme::resize_image() { { … … return $image; return $image; } }Tuesday, June 21, 2011
  • 27. Future 0.22 Released Custom Logging Plugins Client/Worker Configuration Extended Administrative Commands SSL Job Result Cache Uplift!Tuesday, June 21, 2011
  • 28. • Gearman.info (up to date) • Gearman.org (...) • http://launchpad.net/gearmand/ • twitter: brianaker • blog: blog.krow.netTuesday, June 21, 2011