Gearmam, from the_worker's_perspective copy

916 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
916
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • There is no difference if one of the actors is using one operating system rather than the same one used by the server. Actually, clients can get specific advantage of this architecture, by requesting tasks that are not available in their operating system but can be easily performed in the OS of one of the workers.
  • Language, the greatest divider in the recent technology, is not an obstacle anymore. Clients can keep coding in the language they are most familiar with, and the workers will use libraries and classes that are only available to specific languages, without need for the clients to be involved with the nitpicks of such complex systems.
  • Gearmam, from the_worker's_perspective copy

    1. 1. GearmanFrom the Workers Perspective
    2. 2. /usr/bin/whoami• Brian Aker• HP Fellow• Previously MySQL, Slashdot, Sun Microsystems
    3. 3. What is Gearman?
    4. 4. “The way I like to think of Gearman is amassively distributed fork mechanism” -Joe Stump, Digg “The Not Mechanical Turk” -Don MacAskill, SmugMug
    5. 5. resize_image()do {(“resize_image”) … return $image; }
    6. 6. (Livejournal)
    7. 7. ServerProvides Asynchronous and Synchronous RequestsRestarts WorkDurable Requests (MySQL, Postgres,...)Gearman Protocol/HTTPEpoch SchedulingLogging(also available, native Servers in Java, Erlang, and Perl)
    8. 8. Client# Create our client object.$gmclient= newGearmanClient();# Add default server(localhost).$gmclient->addServer();$result= $gmclient->do("reverse", "Hello!");echo "Success: $resultn";
    9. 9. Worker# Create our worker object.$gmw= newGearmanWorker();# Add default server(localhost).$gmw->addServer();$gmw->addFunction("reverse","reverse_fn");while ($gmworker->work()){…}
    10. 10. Worker Function function reverse_fn($job){ $workload= $job- >workload(); $result= strrev($workload); return $result;}
    11. 11. Lots of functions...$gmw->addFunction("resize", "resize_fn");$gmw->addFunction("grep", "grep_fn");$gmw->addFunction("fetch_url", "fetch_url");
    12. 12. Functiongearman_return_t fetch_url(gearman_job_st *job, void*){ const char *workload=gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); gearman_job_send_status(job, 0, 100);… gearman_job_send_data(job, chunk, sizeofchunk);… gearman_job_send_status(job, 50,100);… if (issue_warning) gearman_job_warning(job, “Im sorry, Dave. Im afraid I cant do that.”, size); return GEARMAN_SUCCESS;}
    13. 13. Worker Return GEARMAN_SUCCESS GEARMAN_FATAL GEARMAN_ERROR GEARMAN_SHUTDOWN
    14. 14. Your Client request() Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, ServerFault Tolerant Worker API (C, PHP, Perl, Python, Java, ...) Your Worker function()
    15. 15. Your Clientresize_image(“ request()…”); Client API (C, PHP, Perl, Python, Java,Drizzle, ...)Network,Highly Available, Provided by ServerFault Tolerant Gearman Worker API resize_image() (C, PHP, Perl, Python, Java, ...) { …; return resized; Your Worker } function()
    16. 16. CPU? event()
    17. 17. multiple languages
    18. 18. Connectors?• C/C++• PHP• Python• Java• MySQL and Postgres• ....
    19. 19. Other items of interest?• Work status requests.• Chunked Data.• Exception Handling.• Up to 4gig message sizes.• Threaded server.• Coalescence (the stealth killer feature)
    20. 20. NamespacesFoo::resize_image() Acme::resize_image() { { … … return $image; return $image;} }
    21. 21. Better Map Reduce?
    22. 22. reduce() {…}map(list[…], map() reduce()); {…} reduce() {…} reduce() {…}
    23. 23. Map @#$@# ?
    24. 24. find()Partitioning find() {A...K} {L...Q} {R...Z} find()
    25. 25. Partitioninggearman_return_t split_worker(gearman_job_st *job, void* /* context */){ constchar *workload= gearman_job_workload(job); size_t workload_size=gearman_job_workload_size(job); const char *chunk_begin= workload; for(size_t x= 0; x < workload_size; x++) { if (workload[x] == 0 or workload[x] == ) { gearman_job_send_data(job, chunk_begin, workload +x -chunk_begin); chunk_begin=workload +x +1; } } return GEARMAN_SUCCESS;}
    26. 26. $resuAggregation lt $resu lt + result + result + result $resu lt = sum result
    27. 27. Aggregationgearman_return_t cat_aggregator (gearman_aggregator_st *, gearman_task_st *task, gearman_result_st *result){ std::stringstring_value; do { gearman_result_st *result_ptr=gearman_task_result(task);string_value.append(gearman_result_value(result_ptr), gearman_result_size(result_ptr)); } while ((task= gearman_next(task)));gearman_result_store_value(result, string_value.c_str(), string_value.size()); returnGEARMAN_SUCCESS;}
    28. 28. Do we have to partition? (What other tricks exist!)
    29. 29. PipelineStore() Resize() Publish()
    30. 30. Future0.32 ReleasedCustom Logging PluginsClient/Worker ConfigurationExtended Administrative CommandsSSLStatus lookup via Unique IdentifierJob Result CacheUplift!
    31. 31. Gearman.inf o• Gearman.org (...)• http://launchpad.net/gearmand/• twitter: brianaker• blog: blog.krow.net

    ×