• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Gearmam, from the_worker's_perspective copy
 

Gearmam, from the_worker's_perspective copy

on

  • 855 views

 

Statistics

Views

Total Views
855
Views on SlideShare
855
Embed Views
0

Actions

Likes
1
Downloads
11
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • There is no difference if one of the actors is using one operating system rather than the same one used by the server. Actually, clients can get specific advantage of this architecture, by requesting tasks that are not available in their operating system but can be easily performed in the OS of one of the workers.
  • Language, the greatest divider in the recent technology, is not an obstacle anymore. Clients can keep coding in the language they are most familiar with, and the workers will use libraries and classes that are only available to specific languages, without need for the clients to be involved with the nitpicks of such complex systems.

Gearmam, from the_worker's_perspective copy Gearmam, from the_worker's_perspective copy Presentation Transcript

  • GearmanFrom the Workers Perspective
  • /usr/bin/whoami• Brian Aker• HP Fellow• Previously MySQL, Slashdot, Sun Microsystems
  • What is Gearman?
  • “The way I like to think of Gearman is amassively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMug
  • resize_image()do {(“resize_image”) … return $image; }
  • (Livejournal)
  • ServerProvides Asynchronous and Synchronous RequestsRestarts WorkDurable Requests (MySQL, Postgres,...)Gearman Protocol/HTTPEpoch SchedulingLogging(also available, native Servers in Java, Erlang, and Perl)
  • Client# Create our client object.$gmclient= newGearmanClient();# Add default server(localhost).$gmclient->addServer();$result= $gmclient->do("reverse", "Hello!");echo "Success: $resultn";
  • Worker# Create our worker object.$gmw= newGearmanWorker();# Add default server(localhost).$gmw->addServer();$gmw->addFunction("reverse","reverse_fn");while ($gmworker->work()){…}
  • Worker Function function reverse_fn($job){ $workload= $job- >workload(); $result= strrev($workload); return $result;}
  • Lots of functions...$gmw->addFunction("resize", "resize_fn");$gmw->addFunction("grep", "grep_fn");$gmw->addFunction("fetch_url", "fetch_url");
  • Functiongearman_return_t fetch_url(gearman_job_st *job, void*){ const char *workload=gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100);… gearman_job_send_data(job, chunk, sizeofchunk);… gearman_job_send_status(job, 50,100);… if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS;}
  • Worker Return GEARMAN_SUCCESS GEARMAN_FATAL GEARMAN_ERROR GEARMAN_SHUTDOWN
  • Your Client request() Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, ServerFault Tolerant Worker API (C, PHP, Perl, Python, Java, ...) Your Worker function()
  • Your Clientresize_image(“ request()…”); Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, Provided by ServerFault Tolerant Gearman Worker API resize_image() (C, PHP, Perl, Python, Java, ...) { …; return resized; Your Worker } function()
  • CPU? event()
  • multiple languages
  • Connectors?• C/C++• PHP• Python• Java• MySQL and Postgres• ....
  • Other items of interest?• Work status requests.• Chunked Data.• Exception Handling.• Up to 4gig message sizes.• Threaded server.• Coalescence (the stealth killer feature)
  • NamespacesFoo::resize_image() Acme::resize_image() { { … … return $image; return $image;} }
  • Better Map Reduce?
  • reduce() {…}map(list[…], map() reduce()); {…} reduce() {…} reduce() {…}
  • Map @#$@# ?
  • find()Partitioning find() {A...K} {L...Q} {R...Z} find()
  • Partitioninggearman_return_t split_worker(gearman_job_st *job, void* /* context */){ constchar *workload= gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); const char *chunk_begin= workload; for(size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin=workload +x +1; } } return GEARMAN_SUCCESS;}
  • $resuAggregation lt $resu lt + result + result + result $resu lt = sum result
  • Aggregationgearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result){ std::stringstring_value; do { gearman_result_st *result_ptr=gearman_task_result(task);string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task)));gearman_result_store_value(result, string_value.c_str(), string_value.size()); returnGEARMAN_SUCCESS;}
  • Do we have to partition? (What other tricks exist!)
  • PipelineStore() Resize() Publish()
  • Future0.32 ReleasedCustom Logging PluginsClient/Worker ConfigurationExtended Administrative CommandsSSLStatus lookup via Unique IdentifierJob Result CacheUplift!
  • Gearman.inf o• Gearman.org (...)• http://launchpad.net/gearmand/• twitter: brianaker• blog: blog.krow.net