0
GearmanFrom the Workers Perspective
/usr/bin/whoami• Brian Aker• HP Fellow• Previously MySQL, Slashdot, Sun Microsystems
What is Gearman?
“The way I like to think of Gearman is amassively distributed fork mechanism”          -Joe Stump, Digg                  “...
resize_image()do                   {(“resize_image”)       …                       return $image;                   }
(Livejournal)
ServerProvides Asynchronous and Synchronous RequestsRestarts WorkDurable Requests (MySQL, Postgres,...)Gearman Protocol/HT...
Client# Create our client object.$gmclient= newGearmanClient();# Add default server(localhost).$gmclient->addServer();$res...
Worker# Create our worker object.$gmw= newGearmanWorker();# Add default server(localhost).$gmw->addServer();$gmw->addFunct...
Worker Function function reverse_fn($job){ $workload= $job- >workload(); $result= strrev($workload); return $result;}
Lots of functions...$gmw->addFunction("resize", "resize_fn");$gmw->addFunction("grep", "grep_fn");$gmw->addFunction("fetch...
Functiongearman_return_t fetch_url(gearman_job_st *job, void*){ const char *workload=gearman_job_workload(job); size_t wor...
Worker Return  GEARMAN_SUCCESS  GEARMAN_FATAL  GEARMAN_ERROR  GEARMAN_SHUTDOWN
Your Client                         request()                          Client API                    (C, PHP, Perl, Python...
Your Clientresize_image(“            request()…”);                           Client API                     (C, PHP, Perl,...
CPU?   event()
multiple languages
Connectors?• C/C++• PHP• Python• Java• MySQL and Postgres• ....
Other items of         interest?• Work status requests.• Chunked Data.• Exception Handling.• Up to 4gig message sizes.• Th...
NamespacesFoo::resize_image()   Acme::resize_image()  {                     {    …                     …    return $image;...
Better Map Reduce?
reduce()                                      {…}map(list[…],                   map()      reduce());                   {…...
Map @#$@# ?
find()Partitioning                     find()           {A...K}           {L...Q}           {R...Z}   find()
Partitioninggearman_return_t split_worker(gearman_job_st *job, void* /* context */){ constchar *workload= gearman_job_work...
$resuAggregation             lt                      $resu                        lt         + result         + result    ...
Aggregationgearman_return_t cat_aggregator (gearman_aggregator_st *,                                 gearman_task_st *task...
Do we have to  partition? (What other tricks exist!)
PipelineStore()        Resize()   Publish()
Future0.32 ReleasedCustom Logging PluginsClient/Worker ConfigurationExtended Administrative CommandsSSLStatus lookup via U...
Gearman.inf                                       o• Gearman.org (...)• http://launchpad.net/gearmand/• twitter: brianaker...
Gearmam, from the_worker's_perspective copy
Gearmam, from the_worker's_perspective copy
Gearmam, from the_worker's_perspective copy
Gearmam, from the_worker's_perspective copy
Gearmam, from the_worker's_perspective copy
Gearmam, from the_worker's_perspective copy
Gearmam, from the_worker's_perspective copy
Upcoming SlideShare
Loading in...5
×

Gearmam, from the_worker's_perspective copy

809

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
809
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
12
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • There is no difference if one of the actors is using one operating system rather than the same one used by the server. Actually, clients can get specific advantage of this architecture, by requesting tasks that are not available in their operating system but can be easily performed in the OS of one of the workers.
  • Language, the greatest divider in the recent technology, is not an obstacle anymore. Clients can keep coding in the language they are most familiar with, and the workers will use libraries and classes that are only available to specific languages, without need for the clients to be involved with the nitpicks of such complex systems.
  • Transcript of "Gearmam, from the_worker's_perspective copy"

    1. 1. GearmanFrom the Workers Perspective
    2. 2. /usr/bin/whoami• Brian Aker• HP Fellow• Previously MySQL, Slashdot, Sun Microsystems
    3. 3. What is Gearman?
    4. 4. “The way I like to think of Gearman is amassively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMug
    5. 5. resize_image()do {(“resize_image”) … return $image; }
    6. 6. (Livejournal)
    7. 7. ServerProvides Asynchronous and Synchronous RequestsRestarts WorkDurable Requests (MySQL, Postgres,...)Gearman Protocol/HTTPEpoch SchedulingLogging(also available, native Servers in Java, Erlang, and Perl)
    8. 8. Client# Create our client object.$gmclient= newGearmanClient();# Add default server(localhost).$gmclient->addServer();$result= $gmclient->do("reverse", "Hello!");echo "Success: $resultn";
    9. 9. Worker# Create our worker object.$gmw= newGearmanWorker();# Add default server(localhost).$gmw->addServer();$gmw->addFunction("reverse","reverse_fn");while ($gmworker->work()){…}
    10. 10. Worker Function function reverse_fn($job){ $workload= $job- >workload(); $result= strrev($workload); return $result;}
    11. 11. Lots of functions...$gmw->addFunction("resize", "resize_fn");$gmw->addFunction("grep", "grep_fn");$gmw->addFunction("fetch_url", "fetch_url");
    12. 12. Functiongearman_return_t fetch_url(gearman_job_st *job, void*){ const char *workload=gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100);… gearman_job_send_data(job, chunk, sizeofchunk);… gearman_job_send_status(job, 50,100);… if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS;}
    13. 13. Worker Return GEARMAN_SUCCESS GEARMAN_FATAL GEARMAN_ERROR GEARMAN_SHUTDOWN
    14. 14. Your Client request() Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, ServerFault Tolerant Worker API (C, PHP, Perl, Python, Java, ...) Your Worker function()
    15. 15. Your Clientresize_image(“ request()…”); Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, Provided by ServerFault Tolerant Gearman Worker API resize_image() (C, PHP, Perl, Python, Java, ...) { …; return resized; Your Worker } function()
    16. 16. CPU? event()
    17. 17. multiple languages
    18. 18. Connectors?• C/C++• PHP• Python• Java• MySQL and Postgres• ....
    19. 19. Other items of interest?• Work status requests.• Chunked Data.• Exception Handling.• Up to 4gig message sizes.• Threaded server.• Coalescence (the stealth killer feature)
    20. 20. NamespacesFoo::resize_image() Acme::resize_image() { { … … return $image; return $image;} }
    21. 21. Better Map Reduce?
    22. 22. reduce() {…}map(list[…], map() reduce()); {…} reduce() {…} reduce() {…}
    23. 23. Map @#$@# ?
    24. 24. find()Partitioning find() {A...K} {L...Q} {R...Z} find()
    25. 25. Partitioninggearman_return_t split_worker(gearman_job_st *job, void* /* context */){ constchar *workload= gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); const char *chunk_begin= workload; for(size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin=workload +x +1; } } return GEARMAN_SUCCESS;}
    26. 26. $resuAggregation lt $resu lt + result + result + result $resu lt = sum result
    27. 27. Aggregationgearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result){ std::stringstring_value; do { gearman_result_st *result_ptr=gearman_task_result(task);string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task)));gearman_result_store_value(result, string_value.c_str(), string_value.size()); returnGEARMAN_SUCCESS;}
    28. 28. Do we have to partition? (What other tricks exist!)
    29. 29. PipelineStore() Resize() Publish()
    30. 30. Future0.32 ReleasedCustom Logging PluginsClient/Worker ConfigurationExtended Administrative CommandsSSLStatus lookup via Unique IdentifierJob Result CacheUplift!
    31. 31. Gearman.inf o• Gearman.org (...)• http://launchpad.net/gearmand/• twitter: brianaker• blog: blog.krow.net
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×