Gearmam, from the_worker's_perspective copy

  • 735 views
Uploaded on

 

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
735
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
11
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • There is no difference if one of the actors is using one operating system rather than the same one used by the server. Actually, clients can get specific advantage of this architecture, by requesting tasks that are not available in their operating system but can be easily performed in the OS of one of the workers.
  • Language, the greatest divider in the recent technology, is not an obstacle anymore. Clients can keep coding in the language they are most familiar with, and the workers will use libraries and classes that are only available to specific languages, without need for the clients to be involved with the nitpicks of such complex systems.

Transcript

  • 1. GearmanFrom the Workers Perspective
  • 2. /usr/bin/whoami• Brian Aker• HP Fellow• Previously MySQL, Slashdot, Sun Microsystems
  • 3. What is Gearman?
  • 4. “The way I like to think of Gearman is amassively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMug
  • 5. resize_image()do {(“resize_image”) … return $image; }
  • 6. (Livejournal)
  • 7. ServerProvides Asynchronous and Synchronous RequestsRestarts WorkDurable Requests (MySQL, Postgres,...)Gearman Protocol/HTTPEpoch SchedulingLogging(also available, native Servers in Java, Erlang, and Perl)
  • 8. Client# Create our client object.$gmclient= newGearmanClient();# Add default server(localhost).$gmclient->addServer();$result= $gmclient->do("reverse", "Hello!");echo "Success: $resultn";
  • 9. Worker# Create our worker object.$gmw= newGearmanWorker();# Add default server(localhost).$gmw->addServer();$gmw->addFunction("reverse","reverse_fn");while ($gmworker->work()){…}
  • 10. Worker Function function reverse_fn($job){ $workload= $job- >workload(); $result= strrev($workload); return $result;}
  • 11. Lots of functions...$gmw->addFunction("resize", "resize_fn");$gmw->addFunction("grep", "grep_fn");$gmw->addFunction("fetch_url", "fetch_url");
  • 12. Functiongearman_return_t fetch_url(gearman_job_st *job, void*){ const char *workload=gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100);… gearman_job_send_data(job, chunk, sizeofchunk);… gearman_job_send_status(job, 50,100);… if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS;}
  • 13. Worker Return GEARMAN_SUCCESS GEARMAN_FATAL GEARMAN_ERROR GEARMAN_SHUTDOWN
  • 14. Your Client request() Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, ServerFault Tolerant Worker API (C, PHP, Perl, Python, Java, ...) Your Worker function()
  • 15. Your Clientresize_image(“ request()…”); Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, Provided by ServerFault Tolerant Gearman Worker API resize_image() (C, PHP, Perl, Python, Java, ...) { …; return resized; Your Worker } function()
  • 16. CPU? event()
  • 17. multiple languages
  • 18. Connectors?• C/C++• PHP• Python• Java• MySQL and Postgres• ....
  • 19. Other items of interest?• Work status requests.• Chunked Data.• Exception Handling.• Up to 4gig message sizes.• Threaded server.• Coalescence (the stealth killer feature)
  • 20. NamespacesFoo::resize_image() Acme::resize_image() { { … … return $image; return $image;} }
  • 21. Better Map Reduce?
  • 22. reduce() {…}map(list[…], map() reduce()); {…} reduce() {…} reduce() {…}
  • 23. Map @#$@# ?
  • 24. find()Partitioning find() {A...K} {L...Q} {R...Z} find()
  • 25. Partitioninggearman_return_t split_worker(gearman_job_st *job, void* /* context */){ constchar *workload= gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); const char *chunk_begin= workload; for(size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin=workload +x +1; } } return GEARMAN_SUCCESS;}
  • 26. $resuAggregation lt $resu lt + result + result + result $resu lt = sum result
  • 27. Aggregationgearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result){ std::stringstring_value; do { gearman_result_st *result_ptr=gearman_task_result(task);string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task)));gearman_result_store_value(result, string_value.c_str(), string_value.size()); returnGEARMAN_SUCCESS;}
  • 28. Do we have to partition? (What other tricks exist!)
  • 29. PipelineStore() Resize() Publish()
  • 30. Future0.32 ReleasedCustom Logging PluginsClient/Worker ConfigurationExtended Administrative CommandsSSLStatus lookup via Unique IdentifierJob Result CacheUplift!
  • 31. Gearman.inf o• Gearman.org (...)• http://launchpad.net/gearmand/• twitter: brianaker• blog: blog.krow.net