• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Gearman and Perl
 

Gearman and Perl

on

  • 10,008 views

An introduction of what are Gearman and its architecture, and how to use it to dispatch jobs in Perl.

An introduction of what are Gearman and its architecture, and how to use it to dispatch jobs in Perl.

Statistics

Views

Total Views
10,008
Views on SlideShare
8,337
Embed Views
1,671

Actions

Likes
7
Downloads
124
Comments
0

22 Embeds 1,671

http://onperl.ru 893
http://niceperl.blogspot.com 262
http://www.onperl.ru 189
http://www.scoop.it 108
http://shafiqissani.wordpress.com 90
http://sayperl.org 56
http://www.slideshare.net 23
http://www.sayperl.org 21
http://planetperl.ru 7
http://hghltd.yandex.net 5
http://niceperl.blogspot.in 2
http://niceperl.blogspot.com.es 2
http://niceperl.blogspot.co.il 2
http://niceperl.blogspot.de 2
http://xss.yandex.net 2
http://niceperl.blogspot.com.br 1
http://planetperl.ru. 1
https://shafiqissani.wordpress.com 1
http://niceperl.blogspot.co.uk 1
http://niceperl.blogspot.ru 1
http://static.slidesharecdn.com 1
http://news.google.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Gearman and Perl Gearman and Perl Presentation Transcript

    • Dispatching jobs with Gearman
    • or
    • How to farm out jobs
    • or
    • Using Gearman in Perl
    • Using Gearman in Perl
    • gearman.org
    • The name "Gearman" was chosen as an anagram for "Manager," "since it dispatches jobs to be done, but does not do anything useful itself"
    • Originally written in Perl
    • Rewritten in C
    • Clients in Perl, PHP, Phython, Java, C# (.NET)
    • Clients in Perl, PHP, Phython, Java, C# (.NET), even MySQL and PostgreSQL
    • WTF?
    • Part I Architecture
    • Application Job
    • Application Job (or task)
    • Application Job gearmand
    • Application Job gearmand Job Worker
    • Application Job gearmand Job Response Worker
    • Application gearmand Worker Worker Worker
    • Application gearmand gearmand Worker Worker
    • Application Application gearmand gearmand Worker Worker
    • Workers are scalable
    • Run any number you need
    • Run on remote servers
    • Application does not care
    • Application only talks with gearmand server
    • Application only talks with gearmand server (one or more)
    • One or more application
    • One or more application Applications throw jobs
    • One or more job servers
    • One or more job servers Job servers dispatch jobs
    • One or more workers
    • One or more workers (clones or different)
    • One or more workers Workers do jobs
    • One or more workers Workers do jobs and may issue new jobs
    • Scalable also means redundant
    • Scalable also means redundant and reliable
    • Scalable also means redundant and reliable* * when it works properly
    • Different approaches to the architecture
    • Synchronous or asynchronous
    • Worker either generates a response or works silently
    • Synchronous Application Job Response gearmand Job Response Worker
    • Asynchronous Application Job Response gearmand Job Response Worker
    • With a response Application Job Response gearmand Job Response Worker
    • With no response Application Job gearmand Job “I’m done” Worker
    • With no response Application Job gearmand Job “I’m done” Worker Result Database
    • Installation
    • Installation 1. Job server 2. Client libraries 3. Worker libraries
    • Installation 1. Job server C 2. Client libraries Perl 3. Worker libraries
    • Installation 1. Job server C 2. Client libraries Perl XS 3. Worker libraries
    • Installing job server https://launchpad.net/gearmand $
./configure $
make #
make
install
    • Installing Perl modules #
cpan
Gearman or #
cpan
Gearman::Client #
cpan
Gearman::Worker
    • Installing Perl modules #
cpan
Gearman or #
cpan
Gearman::XS
    • Starting gearmand deamon $
gearmand
‐d
    • Starting gearmand deamon $
gearmand
‐d or $
gearmand
‐d


 

‐L
127.0.0.1

 

‐p
4730
    • Starting worker(s) $
./worker.pl
&
    • Starting worker(s) $
./worker.pl
& or $
./workerA.pl
& $
./workerB.pl
&
    • Starting worker(s) $
./worker.pl
& or $
./workerA.pl
& $
./workerB.pl
& $
./workerA.pl
& $
./workerB.pl
&
    • Running client $
./client.pl
    • Worker example #!/usr/bin/perl use
v5.10; use
strict; use
Gearman::Worker; my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; }
    • Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1;
    • Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Connect to the server
    • Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Register worker method
    • Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Wait for incoming jobs
    • Worker example #!/usr/bin/perl use
v5.10; use
strict; use
Gearman::Worker; my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; }
    • Job method sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; } Receive data
    • Job method sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; } Evaluate response
    • Client example #!/usr/bin/perl use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('127.0.0.1:4730'); for (1..10000) {     my $result_ref = $client->do_task('echo', 10);     say $$result_ref; }
    • Client example for (1..10000) {     my $result_ref = $client->do_task('echo', 10);     say $$result_ref; } Issue jobs
    • Asynchronous client example use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('example.com:4730'); my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; sub completed {     my $result = shift;     say $$result; }
    • Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Get task set placeholder
    • Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Place the task
    • Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; and register callback method
    • Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Wait until the job is done
    • Asynchronous client example sub completed {     my $result = shift;     say $$result; } Use result somehow
    • XS worker example use v5.10; use strict; use Gearman::XS::Worker; my $worker = new Gearman::XS::Worker; $worker->add_server('127.0.0.1', 4730); $worker->add_function('echo', 0, &echo, undef); $worker->work() while 1; sub echo {     my $job = shift;     return $job->workload() + 1; }
    • XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret, $result) = $client->do('echo', '15'); # arg as string! say $result;
    • Async XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret1, $task1) = $client->add_task('echo', '15'); my ($ret2, $task2) = $client->add_task('echo', '51'); $client->run_tasks();
    • Monitoring job server
    • TCP protocol
    • Connect with telnet
    • $ telnet localhost 4730
    • $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'.
    • $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status
    • $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status echo2   1      1       2 echo    5      2       5 .
    • echo2   1      1       2 echo    5      2       5 .
    • Known worker methods echo2   1      1       2 echo    5      2       5 .
    • Jobs running echo2   1      1       2 echo    5      2       5 .
    • Jobs queued echo2   1      1       2 echo    5      2       5 .
    • Workers available echo2   1      1       2 echo    5      2       5 .
    • Part III Step by step example
    • worker.pl for(...)
{ 

... worker.pl } worker.pl worker.pl
    • Live demo
    • __END__ Andrew Shitov andy@shitov.ru