Gearman and Perl

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Ad

Dispatching jobs
with Gearman

How to
farm out jobs

Using Gearman
in Perl

Using Gearman
in Perl

gearman.org

The name "Gearman" was chosen as an
anagram for "Manager," "since it dispatches
jobs to be done, but does not do anything
...

Originally written in Perl

Rewritten in C

Clients in
Perl, PHP, Phython,
Java, C# (.NET)

Clients in
Perl, PHP, Phython,
Java, C# (.NET),
even MySQL and PostgreSQL

WTF?

Part I

Architecture

Application
Job

Application
Job (or task)

Application
Job
gearmand

Application
Job
gearmand
Job
Worker

Application
Job
gearmand
Job Response
Worker

Application

gearmand

Worker

Worker

Worker

Application

gearmand

gearmand

Worker

Worker

Application
Application

gearmand

gearmand

Worker

Worker

Workers are scalable

Run any number you need

Run on remote servers

Application does not care

Application only talks with
gearmand server

Application only talks with
gearmand server
(one or more)

One or more application

One or more application
Applications throw jobs

One or more job servers

One or more job servers
Job servers dispatch jobs

One or more workers

One or more workers
(clones or different)

One or more workers
Workers do jobs

One or more workers
Workers do jobs
and may issue new jobs

Scalable also means
redundant

Scalable also means
redundant and reliable

Scalable also means
redundant and reliable*

* when it works properly

Different
approaches
to the architecture

Synchronous
or
asynchronous

Worker either generates
a response or works silently

Synchronous

Application
Job Response
gearmand
Job Response
Worker

Asynchronous

Application
Job Response
gearmand
Job Response
Worker

With a response

Application
Job Response
gearmand
Job Response
Worker

With no response

Application
Job
gearmand
Job “I’m done”
Worker

With no response

Application
Job
gearmand
Job “I’m done”
Worker
Resu...

Installation

Installation
1. Job server
2. Client libraries
3. Worker libraries

Installation
1. Job server C
2. Client libraries Perl
3. Worker libraries

Installation
1. Job server C
2. Client libraries Perl XS
3. Worker libraries

Installing job server

https://launchpad.net/gearmand

$ ./configure
$ make
# make install

Installing Perl modules

# cpan Gearman
or

# cpan Gearman::Client
# cpan Gearman::Worker

Installing Perl modules

# cpan Gearman
or

# cpan Gearman::XS

Starting gearmand deamon

$ gearmand ‐d

Starting gearmand deamon

$ gearmand ‐d
or
$ gearmand ‐d   
  ‐L 127.0.0.1  
  ‐p 4730

Starting worker(s)

$ ./worker.pl &

Starting worker(s)

$ ./worker.pl &
or
$ ./workerA.pl &
$ ./workerB.pl &

Starting worker(s)

$ ./worker.pl &
or
$ ./workerA.pl &
$ ./workerB.pl &
$ ./workerA.pl &
$ ./wo...

Running client

$ ./client.pl

Worker example
#!/usr/bin/perl
use v5.10;
use strict;
use Gearman::Worker;

my $worker = new Gearman::Worker;
$worker‐>jo...

Worker example

my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo ...

Worker example

my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo ...

Worker example

my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo ...

Worker example

my $worker = new Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo ...

Worker example
#!/usr/bin/perl
use v5.10;
use strict;
use Gearman::Worker;

my $worker = new Gearman::Worker;
$worker‐>jo...

Job method

sub echo {
    my $job = shift;
    say '+1';
    return $job‐>arg + 1;
}

Receive data

Job method

sub echo {
    my $job = shift;
    say '+1';
    return $job‐>arg + 1;
}

Evaluate response

Client example
#!/usr/bin/perl
use v5.10;
use strict;
use Gearman::Client;

my $client = new Gearman::Client;
$client->jo...

$Client example for (1..10000) { my $result_ref = $client->do_task('echo', 10); say $$result_ref; } ...$

Asynchronous client example
use v5.10;
use strict;
use Gearman::Client;

my $client = new Gearman::Client;
$client->job_s...

$Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task( 'echo' => 10, { on_compl...$

$Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task( 'echo' => 10, { on_compl...$

$Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task( 'echo' => 10, { on_compl...$

$Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task( 'echo' => 10, { on_compl...$

Asynchronous client example

sub completed {
my $result = shift;
say $$result;
}

Use result somehow

XS worker example
use v5.10;
use strict;
use Gearman::XS::Worker;

my $worker = new Gearman::XS::Worker;
$worker->add_ser...

XS client example
use v5.10;
use strict;
use Gearman::XS::Client;

my $client = new Gearman::XS::Client;
$client->add_ser...

Async XS client example
use v5.10;
use strict;
use Gearman::XS::Client;

my $client = new Gearman::XS::Client;
$client->a...

Monitoring job server

TCP protocol

Connect with telnet

$ telnet localhost 4730

$ telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

$ telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
status

$ telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
status
echo2 1 1 ...

echo2 1 1 2
echo 5 2 5
.

Known worker methods

echo2 1 1 2
echo 5 2 5
.

Jobs running

echo2 1 1 2
echo 5 2 5
.

Jobs queued

echo2 1 1 2
echo 5 2 5
.

Workers available

echo2 1 1 2
echo 5 2 5
.

Part III

Step by step
example

worker.pl
for(...) {
  ... worker.pl
} worker.pl
worker.pl

Live demo

__END__

Andrew Shitov
andy@shitov.ru