Distributed
Applications With
Perl & Gearman
Issac Goldstand
issac@itsgoodconsulting.com
ABOUT THE PRESENTOR

http://linke...
60 seconds before we get started…
Why Distributed?
 Allow



horizontal scaling of compute nodes

Normal resources (CPU, RAM, Disk)
Other resources (spec...
Gearman Architecture
Worker
Worker

Server

Client

Server (Cluster)

Client

Server

Worker

Worker Pool
Example 1 – Video Processing



User uploads a video
Server processes the video






Server must transcode the vide...
Example 2 – Map/Reduce Style Big Data
 User

searches for information

Server must search catalog 1
 Server must search ...
Example 3 – AntiVirus Scanner


User uploads a file
Server must scan with McAfee
 Server must scan with Norton 360
 Ser...
Gearman Performance Stats
 Collected

in October 2013 (https://groups.google.com/forum/#!topic/gearman/ror1rd6EGX0)
 Dea...
Gearman in Perl
 Gearman

/ Gearman::Server (PP)
 Gearman::XS (“official” libgearman)
 AnyEvent::Gearman and AnyEvent::...
Gearman in other languages










C/C++ - official libgearman+ CLI
PHP – GearmanManager – well maintained
frame...
Creating a worker
use strict;
use warnings;
use Gearman::Worker;
my $worker = Gearman::Worker->new;
$worker->job_servers('...
Testing the worker
[issac@localhost ~]$ gearman -f say_hello 'world'
Hello, world
[issac@localhost ~]$
Writing a client
use strict;
use warnings;
use Gearman::Client;
my $client = Gearman::Client->new;
$client->job_servers('1...
Writing an asynchronous client
use strict;
use warnings;
use Gearman::Client;
my $client = Gearman::Client->new;
$client->...
Worker response (packet) types
 SUCCESS
 FAIL
 STATUS
 DATA
 EXCEPTION*

 WARNING*
A more sophisticated worker 1/3
use
use
use
use

strict;
warnings;
Gearman::XS qw(:constants);
Gearman::XS::Worker;

my $w...
A more sophisticated worker 2/3
$ret = $worker->add_function("hello", 3, &hello, {});
$ret = $worker->add_function("uhoh",...
A more sophisticated worker 3/3
sub hello {
my $job = shift;
$job->send_status(1,2);
my $string = "Hello, “.$job->workload...
Testing the (slightly) sophisticated worker
[issac@localhost ~]$ gearman -f hello 'world'
50% Complete
100% Complete
Hello...
A more sophisticated client 1/3
use
use
use
use

strict;
warnings;
Gearman::XS qw(:constants);
Gearman::XS::Client;

my $c...
A more sophisticated client 2/3
$client->set_complete_fn(sub {
my $task = shift;
print "COMPLETE: " . $task->data() . "n";...
A more sophisticated client 3/3
$client->set_fail_fn(sub {
print "FAIL: " . $_[0]->function_name() . "n";
return GEARMAN_S...
A more sophisticated client (output)
[issac@localhost ~]$ perl asclient.pl
STATUS: 1/2
STATUS: 2/2
COMPLETE: Hello, world
...
That’s All, Folks!
Issac Goldstand
issac@itsgoodconsulting.com

Link To Slides

http://www.itsgoodconsulting.com/blog/issa...
Upcoming SlideShare
Loading in...5
×

Distributed Applications with Perl & Gearman

2,436

Published on

Gearman is a framework for writing distributed applications across many nodes. It allows you to do work in parallel, load balance processes and write applications across several programming languages. In this presentation we'll learn how to get started writing Gearman-powered applications in Perl.

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,436
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
30
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "Distributed Applications with Perl & Gearman"

  1. 1. Distributed Applications With Perl & Gearman Issac Goldstand issac@itsgoodconsulting.com ABOUT THE PRESENTOR http://linkedin.com/in/margol
  2. 2. 60 seconds before we get started…
  3. 3. Why Distributed?  Allow   horizontal scaling of compute nodes Normal resources (CPU, RAM, Disk) Other resources (specialized HW, SW)  True asynchronous worker processing  Cross-Language support  Redundant application availability
  4. 4. Gearman Architecture Worker Worker Server Client Server (Cluster) Client Server Worker Worker Pool
  5. 5. Example 1 – Video Processing   User uploads a video Server processes the video     Server must transcode the video to several pre-set resolutions/codecs Server must extract sample still images Server must run speech-to-text recognition to extract subtitles (closed captions) Once all of that is completed, server must update the video metadata to contain all of the newly available data and metadata
  6. 6. Example 2 – Map/Reduce Style Big Data  User searches for information Server must search catalog 1  Server must search catalog 2 …  Server must search catalog n   Server must return combined search results to user
  7. 7. Example 3 – AntiVirus Scanner  User uploads a file Server must scan with McAfee  Server must scan with Norton 360  Server must scan with Sophos …  Server must scan with Engine XYZ   Server returns scan results to user
  8. 8. Gearman Performance Stats  Collected in October 2013 (https://groups.google.com/forum/#!topic/gearman/ror1rd6EGX0)  DealNews – 40 foreground tasks/sec (2 datacenters, 4 servers, 350 workers)  Etsy – 1000 tasks/sec (2 datacenters, 4 servers, 40 workers)  Shazam – 10K tasks/sec  From my own experience – 30 tasks/sec (1 datacenter, 1 server, 240 workers)
  9. 9. Gearman in Perl  Gearman / Gearman::Server (PP)  Gearman::XS (“official” libgearman)  AnyEvent::Gearman and AnyEvent::Gearman::Client (not the same)  They aren’t 100% up-to-date  They aren’t 100% feature-compatible  The first two are both (individually) good for 90% of use cases (in my personal experience )
  10. 10. Gearman in other languages         C/C++ - official libgearman+ CLI PHP – GearmanManager – well maintained framework, PHP_Gearman (PECL), Net_Gearman (Pure PHP) Node.JS .NET/C# JAVA Python UDFs – MySQL, PostgreSQL, Drizzle Write your own to implement the Gearman binary protocol
  11. 11. Creating a worker use strict; use warnings; use Gearman::Worker; my $worker = Gearman::Worker->new; $worker->job_servers('127.0.0.1:4730'); $worker->register_function('say_hello', &hello); $worker->work ; # will never return sub hello { my $arg = $_[0]->arg; return "Hello, $argn"; }
  12. 12. Testing the worker [issac@localhost ~]$ gearman -f say_hello 'world' Hello, world [issac@localhost ~]$
  13. 13. Writing a client use strict; use warnings; use Gearman::Client; my $client = Gearman::Client->new; $client->job_servers('127.0.0.1:4730'); print ${$client->do_task(say_hello => 'world')}; # do_task returns a reference to the response
  14. 14. Writing an asynchronous client use strict; use warnings; use Gearman::Client; my $client = Gearman::Client->new; $client->job_servers('127.0.0.1:4730'); my $tasks = $client->new_task_set; $tasks->add_task(say_hello => 'world', { on_complete => &done }); $tasks->wait; sub done { print ${$_[0]}; }
  15. 15. Worker response (packet) types  SUCCESS  FAIL  STATUS  DATA  EXCEPTION*  WARNING*
  16. 16. A more sophisticated worker 1/3 use use use use strict; warnings; Gearman::XS qw(:constants); Gearman::XS::Worker; my $worker = new Gearman::XS::Worker; my $ret = $worker->add_server('127.0.0.1'); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); exit(1); } $worker->add_options(GEARMAN_WORKER_NON_BLOCKING); $worker->set_timeout(500);
  17. 17. A more sophisticated worker 2/3 $ret = $worker->add_function("hello", 3, &hello, {}); $ret = $worker->add_function("uhoh", 3, &uhoh, {}); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $worker->error()); } my $go = 1; $SIG{TERM} = sub {print "Caught SIGTERMn";$go = 0;}; while ($go) { my $ret = $worker->work(); # will return after 500ms since we set timeout + nonblocking mode above if (!($ret == GEARMAN_SUCCESS || $ret == GEARMAN_IO_WAIT || $ret == GEARMAN_NO_JOBS)) { printf(STDERR "%sn", $worker->error()); } $worker->wait(); }
  18. 18. A more sophisticated worker 3/3 sub hello { my $job = shift; $job->send_status(1,2); my $string = "Hello, “.$job->workload()."n"; $job->send_status(2,2); return $string; } sub uhoh{ my $job = shift; $job->send_warning("uh oh"); $job->send_data($job->workload() . "n"); $job->send_fail(); }
  19. 19. Testing the (slightly) sophisticated worker [issac@localhost ~]$ gearman -f hello 'world' 50% Complete 100% Complete Hello, world [issac@localhost ~]$ gearman -f uhoh 'world' uh ohworld Job failed [issac@localhost ~]$
  20. 20. A more sophisticated client 1/3 use use use use strict; warnings; Gearman::XS qw(:constants); Gearman::XS::Client; my $client = Gearman::XS::Client->new; my $task; my $ret = $client->add_server('127.0.0.1'); if ($ret != GEARMAN_SUCCESS) { printf(STDERR "%sn", $client->error()); exit(1); }
  21. 21. A more sophisticated client 2/3 $client->set_complete_fn(sub { my $task = shift; print "COMPLETE: " . $task->data() . "n"; return GEARMAN_SUCCESS; }); $client->set_data_fn(sub { print "DATA: " . $_[0]->data() . "n"; return GEARMAN_SUCCESS; }); $client->set_warning_fn(sub { print "WARNING: " . $_[0]->data() . "n"; return GEARMAN_SUCCESS; });
  22. 22. A more sophisticated client 3/3 $client->set_fail_fn(sub { print "FAIL: " . $_[0]->function_name() . "n"; return GEARMAN_SUCCESS; }); $client->set_status_fn(sub { print "STATUS: " . $_[0]->numerator() . "/" . $_[0]->denominator() . "n"; return GEARMAN_SUCCESS; }); ($ret, $task) = $client->add_task("hello", "world"); ($ret, $task) = $client->add_task("uhoh", "it broke"); $ret = $client->run_tasks();
  23. 23. A more sophisticated client (output) [issac@localhost ~]$ perl asclient.pl STATUS: 1/2 STATUS: 2/2 COMPLETE: Hello, world WARNING: uh oh DATA: it broke FAIL: uhoh [issac@localhost ~]$
  24. 24. That’s All, Folks! Issac Goldstand issac@itsgoodconsulting.com Link To Slides http://www.itsgoodconsulting.com/blog/issac-presenting-distributed-apps-with-gearman-at-telaviv-pm/
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×