Your SlideShare is downloading. ×
Distributed Applications with Perl & Gearman
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Distributed Applications with Perl & Gearman

2,146
views

Published on

Gearman is a framework for writing distributed applications across many nodes. It allows you to do work in parallel, load balance processes and write applications across several programming languages. …

Gearman is a framework for writing distributed applications across many nodes. It allows you to do work in parallel, load balance processes and write applications across several programming languages. In this presentation we'll learn how to get started writing Gearman-powered applications in Perl.

Published in: Technology

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,146
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
29
Comments
0
Likes
3
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Distributed Applications With Perl & Gearman Issac Goldstand issac@itsgoodconsulting.com ABOUT THE PRESENTOR http://linkedin.com/in/margol
  • 2. 60 seconds before we get started…
  • 3. Why Distributed?  Allow   horizontal scaling of compute nodes Normal resources (CPU, RAM, Disk) Other resources (specialized HW, SW)  True asynchronous worker processing  Cross-Language support  Redundant application availability
  • 4. Gearman Architecture Worker Worker Server Client Server (Cluster) Client Server Worker Worker Pool
  • 5. Example 1 – Video Processing   User uploads a video Server processes the video     Server must transcode the video to several pre-set resolutions/codecs Server must extract sample still images Server must run speech-to-text recognition to extract subtitles (closed captions) Once all of that is completed, server must update the video metadata to contain all of the newly available data and metadata
  • 6. Example 2 – Map/Reduce Style Big Data  User searches for information Server must search catalog 1  Server must search catalog 2 …  Server must search catalog n   Server must return combined search results to user
  • 7. Example 3 – AntiVirus Scanner  User uploads a file Server must scan with McAfee  Server must scan with Norton 360  Server must scan with Sophos …  Server must scan with Engine XYZ   Server returns scan results to user
  • 8. Gearman Performance Stats  Collected in October 2013 (https://groups.google.com/forum/#!topic/gearman/ror1rd6EGX0)  DealNews – 40 foreground tasks/sec (2 datacenters, 4 servers, 350 workers)  Etsy – 1000 tasks/sec (2 datacenters, 4 servers, 40 workers)  Shazam – 10K tasks/sec  From my own experience – 30 tasks/sec (1 datacenter, 1 server, 240 workers)
  • 9. Gearman in Perl  Gearman / Gearman::Server (PP)  Gearman::XS (“official” libgearman)  AnyEvent::Gearman and AnyEvent::Gearman::Client (not the same)  They aren’t 100% up-to-date  They aren’t 100% feature-compatible  The first two are both (individually) good for 90% of use cases (in my personal experience )
  • 10. Gearman in other languages         C/C++ - official libgearman+ CLI PHP – GearmanManager – well maintained framework, PHP_Gearman (PECL), Net_Gearman (Pure PHP) Node.JS .NET/C# JAVA Python UDFs – MySQL, PostgreSQL, Drizzle Write your own to implement the Gearman binary protocol
  • 11. Creating a worker use strict; use warnings; use Gearman::Worker; my $worker = Gearman::Worker->new; $worker->job_servers('127.0.0.1:4730'); $worker->register_function('say_hello', &hello); $worker->work ; # will never return sub hello { my $arg = $_[0]->arg; return "Hello, $argn"; }
  • 12. Testing the worker [issac@localhost ~]$ gearman -f say_hello 'world' Hello, world [issac@localhost ~]$
  • 13. Writing a client use strict; use warnings; use Gearman::Client; my $client = Gearman::Client->new; $client->job_servers('127.0.0.1:4730'); print ${$client->do_task(say_hello => 'world')}; # do_task returns a reference to the response
  • 14. Writing an asynchronous client use strict; use warnings; use Gearman::Client; my $client = Gearman::Client->new; $client->job_servers('127.0.0.1:4730'); my $tasks = $client->new_task_set; $tasks->add_task(say_hello => 'world', { on_complete => &done }); $tasks->wait; sub done { print ${$_[0]}; }
  • 15. Worker response (packet) types  SUCCESS  FAIL  STATUS  DATA  EXCEPTION*  WARNING*
  • 16. A more sophisticated worker 1/3 use use use use strict; warnings; Gearman::XS qw(:constants); Gearman::XS::Worker; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server('127.0.0.1'); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } $worker->add_options(GEARMAN_WORKER_NON_BLOCKING); $worker->set_timeout(500);
  • 17. A more sophisticated worker 2/3 $ret = $worker->add_function("hello", 3, &hello, {}); $ret = $worker->add_function("uhoh", 3, &uhoh, {}); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } my $go = 1; $SIG{TERM} = sub {print "Caught SIGTERMn";$go = 0;}; while ($go) { my $ret = $worker->work(); # will return after 500ms since we set timeout + nonblocking mode above if (!($ret == GEARMAN_SUCCESS || $ret == GEARMAN_IO_WAIT || $ret == GEARMAN_NO_JOBS)) { printf(STDERR "%sn", $worker->error()); } $worker->wait(); }
  • 18. A more sophisticated worker 3/3 sub hello { my $job = shift; $job->send_status(1,2); my $string = "Hello, “.$job->workload()."n"; $job->send_status(2,2); return $string; } sub uhoh{ my $job = shift; $job->send_warning("uh oh"); $job->send_data($job->workload() . "n"); $job->send_fail(); }
  • 19. Testing the (slightly) sophisticated worker [issac@localhost ~]$ gearman -f hello 'world' 50% Complete 100% Complete Hello, world [issac@localhost ~]$ gearman -f uhoh 'world' uh ohworld Job failed [issac@localhost ~]$
  • 20. A more sophisticated client 1/3 use use use use strict; warnings; Gearman::XS qw(:constants); Gearman::XS::Client; my $client = Gearman::XS::Client->new; my $task; my $ret = $client->add_server('127.0.0.1'); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); }
  • 21. A more sophisticated client 2/3 $client->set_complete_fn(sub { my $task = shift; print "COMPLETE: " . $task->data() . "n"; return GEARMAN_SUCCESS; }); $client->set_data_fn(sub { print "DATA: " . $_[0]->data() . "n"; return GEARMAN_SUCCESS; }); $client->set_warning_fn(sub { print "WARNING: " . $_[0]->data() . "n"; return GEARMAN_SUCCESS; });
  • 22. A more sophisticated client 3/3 $client->set_fail_fn(sub { print "FAIL: " . $_[0]->function_name() . "n"; return GEARMAN_SUCCESS; }); $client->set_status_fn(sub { print "STATUS: " . $_[0]->numerator() . "/" . $_[0]->denominator() . "n"; return GEARMAN_SUCCESS; }); ($ret, $task) = $client->add_task("hello", "world"); ($ret, $task) = $client->add_task("uhoh", "it broke"); $ret = $client->run_tasks();
  • 23. A more sophisticated client (output) [issac@localhost ~]$ perl asclient.pl STATUS: 1/2 STATUS: 2/2 COMPLETE: Hello, world WARNING: uh oh DATA: it broke FAIL: uhoh [issac@localhost ~]$
  • 24. That’s All, Folks! Issac Goldstand issac@itsgoodconsulting.com Link To Slides http://www.itsgoodconsulting.com/blog/issac-presenting-distributed-apps-with-gearman-at-telaviv-pm/

×