Gearman and Perl

Dispatching jobs with Gearman

How to farm out jobs

Using Gearman in Perl

gearman.org

The name "Gearman" was chosen as an anagram for "Manager," "since it dispatches jobs to be done, but does not do anything useful itself"

Originally written in Perl

Rewritten in C

Clients in Perl, PHP, Phython, Java, C# (.NET)

Clients in Perl, PHP, Phython, Java, C# (.NET), even MySQL and PostgreSQL

Part I Architecture

Application Job

Application Job (or task)

Application Job gearmand

Application Job gearmand Job Worker

Application Job gearmand Job Response Worker

Application gearmand Worker Worker Worker

Application gearmand gearmand Worker Worker

Application Application gearmand gearmand Worker Worker

Workers are scalable

Run any number you need

Run on remote servers

Application does not care

Application only talks with gearmand server

Application only talks with gearmand server (one or more)

One or more application

One or more application Applications throw jobs

One or more job servers

One or more job servers Job servers dispatch jobs

One or more workers

One or more workers (clones or different)

One or more workers Workers do jobs

One or more workers Workers do jobs and may issue new jobs

Scalable also means redundant

Scalable also means redundant and reliable

Scalable also means redundant and reliable* * when it works properly

Different approaches to the architecture

Synchronous or asynchronous

Worker either generates a response or works silently

Synchronous Application Job Response gearmand Job Response Worker

Asynchronous Application Job Response gearmand Job Response Worker

With a response Application Job Response gearmand Job Response Worker

With no response Application Job gearmand Job “I’m done” Worker

With no response Application Job gearmand Job “I’m done” Worker Result Database

Installation

Installation 1. Job server 2. Client libraries 3. Worker libraries

Installation 1. Job server C 2. Client libraries Perl 3. Worker libraries

Installation 1. Job server C 2. Client libraries Perl XS 3. Worker libraries

Installing job server https://launchpad.net/gearmand $ ./configure $ make # make install

Installing Perl modules # cpan Gearman or # cpan Gearman::Client # cpan Gearman::Worker

Installing Perl modules # cpan Gearman or # cpan Gearman::XS

Starting gearmand deamon $ gearmand ‐d

Starting gearmand deamon $ gearmand ‐d or $ gearmand ‐d      ‐L 127.0.0.1     ‐p 4730

Starting worker(s) $ ./worker.pl &

Starting worker(s) $ ./worker.pl & or $ ./workerA.pl & $ ./workerB.pl &

Starting worker(s) $ ./worker.pl & or $ ./workerA.pl & $ ./workerB.pl & $ ./workerA.pl & $ ./workerB.pl &

Running client $ ./client.pl

Worker example #!/usr/bin/perl use v5.10; use strict; use Gearman::Worker; my $worker = new Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo => &echo); $worker‐>work while 1; sub echo {     my $job = shift;     say '+1';     return $job‐>arg + 1; }

Worker example my $worker = new Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo => &echo); $worker‐>work while 1;

Worker example my $worker = new Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo => &echo); $worker‐>work while 1; Connect to the server

Worker example my $worker = new Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo => &echo); $worker‐>work while 1; Register worker method

Worker example my $worker = new Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo => &echo); $worker‐>work while 1; Wait for incoming jobs

Worker example #!/usr/bin/perl use v5.10; use strict; use Gearman::Worker; my $worker = new Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo => &echo); $worker‐>work while 1; sub echo {     my $job = shift;     say '+1';     return $job‐>arg + 1; }

Job method sub echo {     my $job = shift;     say '+1';     return $job‐>arg + 1; } Receive data

Job method sub echo {     my $job = shift;     say '+1';     return $job‐>arg + 1; } Evaluate response

Client example #!/usr/bin/perl use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('127.0.0.1:4730'); for (1..10000) { my $result_ref = $client->do_task('echo', 10); say $$result_ref; }

Client example for (1..10000) { my $result_ref = $client->do_task('echo', 10); say $$result_ref; } Issue jobs

Asynchronous client example use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('example.com:4730'); my $tasks = $client->new_task_set; $tasks->add_task( 'echo' => 10, { on_complete => &completed } ); $tasks->wait; sub completed { my $result = shift; say $$result; }

Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task( 'echo' => 10, { on_complete => &completed } ); $tasks->wait; Get task set placeholder

Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task( 'echo' => 10, { on_complete => &completed } ); $tasks->wait; Place the task

Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task( 'echo' => 10, { on_complete => &completed } ); $tasks->wait; and register callback method

Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task( 'echo' => 10, { on_complete => &completed } ); $tasks->wait; Wait until the job is done

Asynchronous client example sub completed { my $result = shift; say $$result; } Use result somehow

XS worker example use v5.10; use strict; use Gearman::XS::Worker; my $worker = new Gearman::XS::Worker; $worker->add_server('127.0.0.1', 4730); $worker->add_function('echo', 0, &echo, undef); $worker->work() while 1; sub echo { my $job = shift; return $job->workload() + 1; }

XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret, $result) = $client->do('echo', '15'); # arg as string! say $result;

Async XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret1, $task1) = $client->add_task('echo', '15'); my ($ret2, $task2) = $client->add_task('echo', '51'); $client->run_tasks();

Monitoring job server

TCP protocol

Connect with telnet

$ telnet localhost 4730

$ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'.

$ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status

$ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status echo2 1 1 2 echo 5 2 5 .

echo2 1 1 2 echo 5 2 5 .

Known worker methods echo2 1 1 2 echo 5 2 5 .

Jobs running echo2 1 1 2 echo 5 2 5 .

Jobs queued echo2 1 1 2 echo 5 2 5 .

Workers available echo2 1 1 2 echo 5 2 5 .

Part III Step by step example

worker.pl for(...) {   ... worker.pl } worker.pl worker.pl

Live demo

__END__ Andrew Shitov andy@shitov.ru

http://onperl.ru	624
http://niceperl.blogspot.com	255
http://www.onperl.ru	67
http://sayperl.org	56
http://www.slideshare.net	23
http://shafiqissani.wordpress.com	23
http://www.sayperl.org	21
http://planetperl.ru	7
http://hghltd.yandex.net	4
http://xss.yandex.net	2
http://planetperl.ru.	1
https://shafiqissani.wordpress.com	1
http://static.slidesharecdn.com	1
http://niceperl.blogspot.com.br	1

Gearman and Perl

by andy.sh on Jan 30, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

14 Embeds 1,086

Statistics

Gearman and Perl — Presentation Transcript