Your SlideShare is downloading. ×
Gearmam, from the_worker's_perspective copy
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Gearmam, from the_worker's_perspective copy


Published on

Published in: Technology

1 Like
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide
  • There is no difference if one of the actors is using one operating system rather than the same one used by the server. Actually, clients can get specific advantage of this architecture, by requesting tasks that are not available in their operating system but can be easily performed in the OS of one of the workers.
  • Language, the greatest divider in the recent technology, is not an obstacle anymore. Clients can keep coding in the language they are most familiar with, and the workers will use libraries and classes that are only available to specific languages, without need for the clients to be involved with the nitpicks of such complex systems.
  • Transcript

    • 1. GearmanFrom the Workers Perspective
    • 2. /usr/bin/whoami• Brian Aker• HP Fellow• Previously MySQL, Slashdot, Sun Microsystems
    • 3. What is Gearman?
    • 4. “The way I like to think of Gearman is amassively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMug
    • 5. resize_image()do {(“resize_image”) … return $image; }
    • 6. (Livejournal)
    • 7. ServerProvides Asynchronous and Synchronous RequestsRestarts WorkDurable Requests (MySQL, Postgres,...)Gearman Protocol/HTTPEpoch SchedulingLogging(also available, native Servers in Java, Erlang, and Perl)
    • 8. Client# Create our client object.$gmclient= newGearmanClient();# Add default server(localhost).$gmclient->addServer();$result= $gmclient->do("reverse", "Hello!");echo "Success: $resultn";
    • 9. Worker# Create our worker object.$gmw= newGearmanWorker();# Add default server(localhost).$gmw->addServer();$gmw->addFunction("reverse","reverse_fn");while ($gmworker->work()){…}
    • 10. Worker Function function reverse_fn($job){ $workload= $job- >workload(); $result= strrev($workload); return $result;}
    • 11. Lots of functions...$gmw->addFunction("resize", "resize_fn");$gmw->addFunction("grep", "grep_fn");$gmw->addFunction("fetch_url", "fetch_url");
    • 12. Functiongearman_return_t fetch_url(gearman_job_st *job, void*){ const char *workload=gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100);… gearman_job_send_data(job, chunk, sizeofchunk);… gearman_job_send_status(job, 50,100);… if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS;}
    • 14. Your Client request() Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, ServerFault Tolerant Worker API (C, PHP, Perl, Python, Java, ...) Your Worker function()
    • 15. Your Clientresize_image(“ request()…”); Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, Provided by ServerFault Tolerant Gearman Worker API resize_image() (C, PHP, Perl, Python, Java, ...) { …; return resized; Your Worker } function()
    • 16. CPU? event()
    • 17. multiple languages
    • 18. Connectors?• C/C++• PHP• Python• Java• MySQL and Postgres• ....
    • 19. Other items of interest?• Work status requests.• Chunked Data.• Exception Handling.• Up to 4gig message sizes.• Threaded server.• Coalescence (the stealth killer feature)
    • 20. NamespacesFoo::resize_image() Acme::resize_image() { { … … return $image; return $image;} }
    • 21. Better Map Reduce?
    • 22. reduce() {…}map(list[…], map() reduce()); {…} reduce() {…} reduce() {…}
    • 23. Map @#$@# ?
    • 24. find()Partitioning find() {A...K} {L...Q} {R...Z} find()
    • 25. Partitioninggearman_return_t split_worker(gearman_job_st *job, void* /* context */){ constchar *workload= gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); const char *chunk_begin= workload; for(size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin=workload +x +1; } } return GEARMAN_SUCCESS;}
    • 26. $resuAggregation lt $resu lt + result + result + result $resu lt = sum result
    • 27. Aggregationgearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result){ std::stringstring_value; do { gearman_result_st *result_ptr=gearman_task_result(task);string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task)));gearman_result_store_value(result, string_value.c_str(), string_value.size()); returnGEARMAN_SUCCESS;}
    • 28. Do we have to partition? (What other tricks exist!)
    • 29. PipelineStore() Resize() Publish()
    • 30. Future0.32 ReleasedCustom Logging PluginsClient/Worker ConfigurationExtended Administrative CommandsSSLStatus lookup via Unique IdentifierJob Result CacheUplift!
    • 31. Gearman.inf o• (...)•• twitter: brianaker• blog: