Distributed Applications with Perl & Gearman

  • 1,891 views
Uploaded on

Gearman is a framework for writing distributed applications across many nodes. It allows you to do work in parallel, load balance processes and write applications across several programming languages. …

Gearman is a framework for writing distributed applications across many nodes. It allows you to do work in parallel, load balance processes and write applications across several programming languages. In this presentation we'll learn how to get started writing Gearman-powered applications in Perl.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,891
On Slideshare
0
From Embeds
0
Number of Embeds
2

Actions

Shares
Downloads
28
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Distributed Applications With Perl & Gearman Issac Goldstand issac@itsgoodconsulting.com ABOUT THE PRESENTOR http://linkedin.com/in/margol
  • 2. 60 seconds before we get started…
  • 3. Why Distributed?  Allow   horizontal scaling of compute nodes Normal resources (CPU, RAM, Disk) Other resources (specialized HW, SW)  True asynchronous worker processing  Cross-Language support  Redundant application availability
  • 4. Gearman Architecture Worker Worker Server Client Server (Cluster) Client Server Worker Worker Pool
  • 5. Example 1 – Video Processing   User uploads a video Server processes the video     Server must transcode the video to several pre-set resolutions/codecs Server must extract sample still images Server must run speech-to-text recognition to extract subtitles (closed captions) Once all of that is completed, server must update the video metadata to contain all of the newly available data and metadata
  • 6. Example 2 – Map/Reduce Style Big Data  User searches for information Server must search catalog 1  Server must search catalog 2 …  Server must search catalog n   Server must return combined search results to user
  • 7. Example 3 – AntiVirus Scanner  User uploads a file Server must scan with McAfee  Server must scan with Norton 360  Server must scan with Sophos …  Server must scan with Engine XYZ   Server returns scan results to user
  • 8. Gearman Performance Stats  Collected in October 2013 (https://groups.google.com/forum/#!topic/gearman/ror1rd6EGX0)  DealNews – 40 foreground tasks/sec (2 datacenters, 4 servers, 350 workers)  Etsy – 1000 tasks/sec (2 datacenters, 4 servers, 40 workers)  Shazam – 10K tasks/sec  From my own experience – 30 tasks/sec (1 datacenter, 1 server, 240 workers)
  • 9. Gearman in Perl  Gearman / Gearman::Server (PP)  Gearman::XS (“official” libgearman)  AnyEvent::Gearman and AnyEvent::Gearman::Client (not the same)  They aren’t 100% up-to-date  They aren’t 100% feature-compatible  The first two are both (individually) good for 90% of use cases (in my personal experience )
  • 10. Gearman in other languages         C/C++ - official libgearman+ CLI PHP – GearmanManager – well maintained framework, PHP_Gearman (PECL), Net_Gearman (Pure PHP) Node.JS .NET/C# JAVA Python UDFs – MySQL, PostgreSQL, Drizzle Write your own to implement the Gearman binary protocol
  • 11. Creating a worker use strict; use warnings; use Gearman::Worker; my $worker = Gearman::Worker->new; $worker->job_servers('127.0.0.1:4730'); $worker->register_function('say_hello', &hello); $worker->work ; # will never return sub hello { my $arg = $_[0]->arg; return "Hello, $argn"; }
  • 12. Testing the worker [issac@localhost ~]$ gearman -f say_hello 'world' Hello, world [issac@localhost ~]$
  • 13. Writing a client use strict; use warnings; use Gearman::Client; my $client = Gearman::Client->new; $client->job_servers('127.0.0.1:4730'); print ${$client->do_task(say_hello => 'world')}; # do_task returns a reference to the response
  • 14. Writing an asynchronous client use strict; use warnings; use Gearman::Client; my $client = Gearman::Client->new; $client->job_servers('127.0.0.1:4730'); my $tasks = $client->new_task_set; $tasks->add_task(say_hello => 'world', { on_complete => &done }); $tasks->wait; sub done { print ${$_[0]}; }
  • 15. Worker response (packet) types  SUCCESS  FAIL  STATUS  DATA  EXCEPTION*  WARNING*
  • 16. A more sophisticated worker 1/3 use use use use strict; warnings; Gearman::XS qw(:constants); Gearman::XS::Worker; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server('127.0.0.1'); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } $worker->add_options(GEARMAN_WORKER_NON_BLOCKING); $worker->set_timeout(500);
  • 17. A more sophisticated worker 2/3 $ret = $worker->add_function("hello", 3, &hello, {}); $ret = $worker->add_function("uhoh", 3, &uhoh, {}); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } my $go = 1; $SIG{TERM} = sub {print "Caught SIGTERMn";$go = 0;}; while ($go) { my $ret = $worker->work(); # will return after 500ms since we set timeout + nonblocking mode above if (!($ret == GEARMAN_SUCCESS || $ret == GEARMAN_IO_WAIT || $ret == GEARMAN_NO_JOBS)) { printf(STDERR "%sn", $worker->error()); } $worker->wait(); }
  • 18. A more sophisticated worker 3/3 sub hello { my $job = shift; $job->send_status(1,2); my $string = "Hello, “.$job->workload()."n"; $job->send_status(2,2); return $string; } sub uhoh{ my $job = shift; $job->send_warning("uh oh"); $job->send_data($job->workload() . "n"); $job->send_fail(); }
  • 19. Testing the (slightly) sophisticated worker [issac@localhost ~]$ gearman -f hello 'world' 50% Complete 100% Complete Hello, world [issac@localhost ~]$ gearman -f uhoh 'world' uh ohworld Job failed [issac@localhost ~]$
  • 20. A more sophisticated client 1/3 use use use use strict; warnings; Gearman::XS qw(:constants); Gearman::XS::Client; my $client = Gearman::XS::Client->new; my $task; my $ret = $client->add_server('127.0.0.1'); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); }
  • 21. A more sophisticated client 2/3 $client->set_complete_fn(sub { my $task = shift; print "COMPLETE: " . $task->data() . "n"; return GEARMAN_SUCCESS; }); $client->set_data_fn(sub { print "DATA: " . $_[0]->data() . "n"; return GEARMAN_SUCCESS; }); $client->set_warning_fn(sub { print "WARNING: " . $_[0]->data() . "n"; return GEARMAN_SUCCESS; });
  • 22. A more sophisticated client 3/3 $client->set_fail_fn(sub { print "FAIL: " . $_[0]->function_name() . "n"; return GEARMAN_SUCCESS; }); $client->set_status_fn(sub { print "STATUS: " . $_[0]->numerator() . "/" . $_[0]->denominator() . "n"; return GEARMAN_SUCCESS; }); ($ret, $task) = $client->add_task("hello", "world"); ($ret, $task) = $client->add_task("uhoh", "it broke"); $ret = $client->run_tasks();
  • 23. A more sophisticated client (output) [issac@localhost ~]$ perl asclient.pl STATUS: 1/2 STATUS: 2/2 COMPLETE: Hello, world WARNING: uh oh DATA: it broke FAIL: uhoh [issac@localhost ~]$
  • 24. That’s All, Folks! Issac Goldstand issac@itsgoodconsulting.com Link To Slides http://www.itsgoodconsulting.com/blog/issac-presenting-distributed-apps-with-gearman-at-telaviv-pm/