Gearman is a framework for writing distributed applications across many nodes. It allows you to do work in parallel, load balance processes and write applications across several programming languages. In this presentation we'll learn how to get started writing Gearman-powered applications in Perl.

1. Distributed
Applications With
Perl & Gearman
Issac Goldstand
issac@itsgoodconsulting.com
ABOUT THE PRESENTOR
http://linkedin.com/in/margol

2. 60 seconds before we get started…

3. Why Distributed?
 Allow


horizontal scaling of compute nodes
Normal resources (CPU, RAM, Disk)
Other resources (specialized HW, SW)
 True
asynchronous worker processing
 Cross-Language support
 Redundant application availability

4. Gearman Architecture
Worker
Worker
Server
Client
Server (Cluster)
Client
Server
Worker
Worker Pool

5. Example 1 – Video Processing


User uploads a video
Server processes the video




Server must transcode the video to
several pre-set resolutions/codecs
Server must extract sample still images
Server must run speech-to-text
recognition to extract subtitles (closed
captions)
Once all of that is completed, server
must update the video metadata to
contain all of the newly available
data and metadata

6. Example 2 – Map/Reduce Style Big Data
 User
searches for information
Server must search catalog 1
 Server must search catalog 2
…
 Server must search catalog n

 Server
must return combined
search results to user

7. Example 3 – AntiVirus Scanner

User uploads a file
Server must scan with McAfee
 Server must scan with Norton 360
 Server must scan with Sophos
…
 Server must scan with Engine XYZ


Server returns scan results to user

8. Gearman Performance Stats
 Collected
in October 2013 (https://groups.google.com/forum/#!topic/gearman/ror1rd6EGX0)
 DealNews – 40 foreground tasks/sec (2 datacenters, 4
servers, 350 workers)
 Etsy – 1000 tasks/sec (2 datacenters, 4 servers, 40 workers)
 Shazam – 10K tasks/sec
 From
my own experience – 30 tasks/sec (1 datacenter, 1
server, 240 workers)

9. Gearman in Perl
 Gearman
/ Gearman::Server (PP)
 Gearman::XS (“official” libgearman)
 AnyEvent::Gearman and AnyEvent::Gearman::Client (not
the same)
 They aren’t 100% up-to-date
 They aren’t 100% feature-compatible
 The first two are both (individually) good for 90% of use
cases (in my personal experience )

10. Gearman in other languages








C/C++ - official libgearman+ CLI
PHP – GearmanManager – well maintained
framework, PHP_Gearman (PECL), Net_Gearman (Pure PHP)
Node.JS
.NET/C#
JAVA
Python
UDFs – MySQL, PostgreSQL, Drizzle
Write your own to implement the Gearman binary protocol

11. Creating a worker
use strict;
use warnings;
use Gearman::Worker;
my $worker = Gearman::Worker->new;
$worker->job_servers('127.0.0.1:4730');
$worker->register_function('say_hello', &hello);
$worker->work ; # will never return
sub hello {
my $arg = $_[0]->arg;
return "Hello, $argn";
}

12. Testing the worker
[issac@localhost ~]$ gearman -f say_hello 'world'
Hello, world
[issac@localhost ~]$

13. Writing a client
use strict;
use warnings;
use Gearman::Client;
my $client = Gearman::Client->new;
$client->job_servers('127.0.0.1:4730');
print ${$client->do_task(say_hello => 'world')};
# do_task returns a reference to the response

14. Writing an asynchronous client
use strict;
use warnings;
use Gearman::Client;
my $client = Gearman::Client->new;
$client->job_servers('127.0.0.1:4730');
my $tasks = $client->new_task_set;
$tasks->add_task(say_hello => 'world', {
on_complete => &done
});
$tasks->wait;
sub done {
print ${$_[0]};
}

15. Worker response (packet) types
 SUCCESS
 FAIL
 STATUS
 DATA
 EXCEPTION*
 WARNING*

16. A more sophisticated worker 1/3
use
use
use
use
strict;
warnings;
Gearman::XS qw(:constants);
Gearman::XS::Worker;
my $worker = new Gearman::XS::Worker;
my $ret = $worker->add_server('127.0.0.1');
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $worker->error());
exit(1);
}
$worker->add_options(GEARMAN_WORKER_NON_BLOCKING);
$worker->set_timeout(500);

17. A more sophisticated worker 2/3
$ret = $worker->add_function("hello", 3, &hello, {});
$ret = $worker->add_function("uhoh", 3, &uhoh, {});
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $worker->error());
}
my $go = 1;
$SIG{TERM} = sub {print "Caught SIGTERMn";$go = 0;};
while ($go) {
my $ret = $worker->work(); # will return after 500ms since we set timeout + nonblocking mode above
if (!($ret == GEARMAN_SUCCESS ||
$ret == GEARMAN_IO_WAIT ||
$ret == GEARMAN_NO_JOBS)) {
printf(STDERR "%sn", $worker->error());
}
$worker->wait();
}

18. A more sophisticated worker 3/3
sub hello {
my $job = shift;
$job->send_status(1,2);
my $string = "Hello, “.$job->workload()."n";
$job->send_status(2,2);
return $string;
}
sub uhoh{
my $job = shift;
$job->send_warning("uh oh");
$job->send_data($job->workload() . "n");
$job->send_fail();
}

19. Testing the (slightly) sophisticated worker
[issac@localhost ~]$ gearman -f hello 'world'
50% Complete
100% Complete
Hello, world
[issac@localhost ~]$ gearman -f uhoh 'world'
uh ohworld
Job failed
[issac@localhost ~]$

20. A more sophisticated client 1/3
use
use
use
use
strict;
warnings;
Gearman::XS qw(:constants);
Gearman::XS::Client;
my $client = Gearman::XS::Client->new;
my $task;
my $ret = $client->add_server('127.0.0.1');
if ($ret != GEARMAN_SUCCESS) {
printf(STDERR "%sn", $client->error());
exit(1);
}

21. A more sophisticated client 2/3
$client->set_complete_fn(sub {
my $task = shift;
print "COMPLETE: " . $task->data() . "n";
return GEARMAN_SUCCESS;
});
$client->set_data_fn(sub {
print "DATA: " . $_[0]->data() . "n";
return GEARMAN_SUCCESS;
});
$client->set_warning_fn(sub {
print "WARNING: " . $_[0]->data() . "n";
return GEARMAN_SUCCESS;
});

22. A more sophisticated client 3/3
$client->set_fail_fn(sub {
print "FAIL: " . $_[0]->function_name() . "n";
return GEARMAN_SUCCESS;
});
$client->set_status_fn(sub {
print "STATUS: " . $_[0]->numerator() .
"/" . $_[0]->denominator() . "n";
return GEARMAN_SUCCESS;
});
($ret, $task) = $client->add_task("hello", "world");
($ret, $task) = $client->add_task("uhoh", "it broke");
$ret = $client->run_tasks();

23. A more sophisticated client (output)
[issac@localhost ~]$ perl asclient.pl
STATUS: 1/2
STATUS: 2/2
COMPLETE: Hello, world
WARNING: uh oh
DATA: it broke
FAIL: uhoh
[issac@localhost ~]$

24. That’s All, Folks!
Issac Goldstand
issac@itsgoodconsulting.com
Link To Slides
http://www.itsgoodconsulting.com/blog/issac-presenting-distributed-apps-with-gearman-at-telaviv-pm/