Your SlideShare is downloading. ×
0
Dispatching jobs
 with Gearman
or
How to
farm out jobs
or
Using Gearman
    in Perl
Using Gearman
    in Perl
gearman.org
The name "Gearman" was chosen as an
anagram for "Manager," "since it dispatches
jobs to be done, but does not do anything
...
Originally written in Perl
Rewritten in C
Clients in
Perl, PHP, Phython,
  Java, C# (.NET)
Clients in
    Perl, PHP, Phython,
      Java, C# (.NET),
even MySQL and PostgreSQL
WTF?
Part I

Architecture
Application
Job
Application
Job (or task)
Application
Job
      gearmand
Application
Job
      gearmand
Job
       Worker
Application
Job
      gearmand
Job                 Response
       Worker
Application


gearmand


 Worker

   Worker

      Worker
Application


gearmand

  gearmand


  Worker

    Worker
Application
  Application

gearmand

  gearmand


  Worker

    Worker
Workers are scalable
Run any number you need
Run on remote servers
Application does not care
Application only talks with
    gearmand server
Application only talks with
    gearmand server
      (one or more)
One or more application
One or more application
   Applications throw jobs
One or more job servers
One or more job servers
   Job servers dispatch jobs
One or more workers
One or more workers
(clones or different)
One or more workers
    Workers do jobs
One or more workers
     Workers do jobs
  and may issue new jobs
Scalable also means
     redundant
Scalable also means
redundant and reliable
Scalable also means
redundant and reliable*



* when it works properly
Different
    approaches
to the architecture
Synchronous
     or
asynchronous
Worker either generates
a response or works silently
Synchronous

      Application
Job                 Response
      gearmand
Job                 Response
       Worker
Asynchronous

      Application
Job                 Response
      gearmand
Job                 Response
       Worker
With a response

      Application
Job                 Response
      gearmand
Job                 Response
       Worker
With no response

      Application
Job
      gearmand
Job                 “I’m done”
       Worker
With no response

      Application
Job
      gearmand
Job                 “I’m done”
       Worker
                  Resu...
Installation
Installation
1. Job server
2. Client libraries
3. Worker libraries
Installation
1. Job server       C
2. Client libraries Perl
3. Worker libraries
Installation
1. Job server       C
2. Client libraries Perl XS
3. Worker libraries
Installing job server

https://launchpad.net/gearmand



  $
./configure
  $
make
  #
make
install
Installing Perl modules


     #
cpan
Gearman
or

     #
cpan
Gearman::Client
     #
cpan
Gearman::Worker
Installing Perl modules


     #
cpan
Gearman
or

     #
cpan
Gearman::XS
Starting gearmand deamon


  $
gearmand
‐d
Starting gearmand deamon


     $
gearmand
‐d
or
     $
gearmand
‐d



     

‐L
127.0.0.1


     

‐p
4730
Starting worker(s)


$
./worker.pl
&
Starting worker(s)


     $
./worker.pl
&
or
     $
./workerA.pl
&
     $
./workerB.pl
&
Starting worker(s)


     $
./worker.pl
&
or
     $
./workerA.pl
&
     $
./workerB.pl
&
     $
./workerA.pl
&
     $
./wo...
Running client


$
./client.pl
Worker example
#!/usr/bin/perl
use
v5.10;
use
strict;
use
Gearman::Worker;


my
$worker
=
new
Gearman::Worker;
$worker‐>jo...
Worker example

my
$worker
=
new
Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo
...
Worker example

my
$worker
=
new
Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo
...
Worker example

my
$worker
=
new
Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo
...
Worker example

my
$worker
=
new
Gearman::Worker;
$worker‐>job_servers('127.0.0.1:4730');
$worker‐>register_function(echo
...
Worker example
#!/usr/bin/perl
use
v5.10;
use
strict;
use
Gearman::Worker;


my
$worker
=
new
Gearman::Worker;
$worker‐>jo...
Job method

sub
echo
{




my
$job
=
shift;




say
'+1';




return
$job‐>arg
+
1;
}




                 Receive data
Job method

sub
echo
{




my
$job
=
shift;




say
'+1';




return
$job‐>arg
+
1;
}




               Evaluate response
Client example
#!/usr/bin/perl
use v5.10;
use strict;
use Gearman::Client;


my $client = new Gearman::Client;
$client->jo...
Client example

for (1..10000) {
    my $result_ref =
        $client->do_task('echo', 10);
    say $$result_ref;
}



   ...
Asynchronous client example
use v5.10;
use strict;
use Gearman::Client;


my $client = new Gearman::Client;
$client->job_s...
Asynchronous client example

my $tasks = $client->new_task_set;
$tasks->add_task(
    'echo' => 10,
    {
        on_compl...
Asynchronous client example

my $tasks = $client->new_task_set;
$tasks->add_task(
    'echo' => 10,
    {
        on_compl...
Asynchronous client example

my $tasks = $client->new_task_set;
$tasks->add_task(
    'echo' => 10,
    {
        on_compl...
Asynchronous client example

my $tasks = $client->new_task_set;
$tasks->add_task(
    'echo' => 10,
    {
        on_compl...
Asynchronous client example

sub completed {
    my $result = shift;
    say $$result;
}




            Use result somehow
XS worker example
use v5.10;
use strict;
use Gearman::XS::Worker;


my $worker = new Gearman::XS::Worker;
$worker->add_ser...
XS client example
use v5.10;
use strict;
use Gearman::XS::Client;


my $client = new Gearman::XS::Client;
$client->add_ser...
Async XS client example
use v5.10;
use strict;
use Gearman::XS::Client;


my $client = new Gearman::XS::Client;
$client->a...
Monitoring job server
TCP protocol
Connect with telnet
$ telnet localhost 4730
$ telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
$ telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
status
$ telnet localhost 4730
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
status
echo2   1      1     ...
echo2   1      1       2
echo    5      2       5
.
Known worker methods




echo2   1      1       2
echo    5      2       5
.
Jobs running




echo2   1      1       2
echo    5      2       5
.
Jobs queued




echo2   1      1       2
echo    5      2       5
.
Workers available




echo2   1      1       2
echo    5      2       5
.
Part III

Step by step
example
worker.pl
for(...)
{


...        worker.pl
}            worker.pl
             worker.pl
Live demo
__END__

Andrew Shitov
andy@shitov.ru
Gearman and Perl
Gearman and Perl
Gearman and Perl
Gearman and Perl
Gearman and Perl
Gearman and Perl
Upcoming SlideShare
Loading in...5
×

Gearman and Perl

8,766

Published on

An introduction of what are Gearman and its architecture, and how to use it to dispatch jobs in Perl.

Published in: Technology, Business
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
8,766
On Slideshare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
130
Comments
0
Likes
9
Embeds 0
No embeds

No notes for slide

Transcript of "Gearman and Perl"

  1. 1. Dispatching jobs with Gearman
  2. 2. or
  3. 3. How to farm out jobs
  4. 4. or
  5. 5. Using Gearman in Perl
  6. 6. Using Gearman in Perl
  7. 7. gearman.org
  8. 8. The name "Gearman" was chosen as an anagram for "Manager," "since it dispatches jobs to be done, but does not do anything useful itself"
  9. 9. Originally written in Perl
  10. 10. Rewritten in C
  11. 11. Clients in Perl, PHP, Phython, Java, C# (.NET)
  12. 12. Clients in Perl, PHP, Phython, Java, C# (.NET), even MySQL and PostgreSQL
  13. 13. WTF?
  14. 14. Part I Architecture
  15. 15. Application Job
  16. 16. Application Job (or task)
  17. 17. Application Job gearmand
  18. 18. Application Job gearmand Job Worker
  19. 19. Application Job gearmand Job Response Worker
  20. 20. Application gearmand Worker Worker Worker
  21. 21. Application gearmand gearmand Worker Worker
  22. 22. Application Application gearmand gearmand Worker Worker
  23. 23. Workers are scalable
  24. 24. Run any number you need
  25. 25. Run on remote servers
  26. 26. Application does not care
  27. 27. Application only talks with gearmand server
  28. 28. Application only talks with gearmand server (one or more)
  29. 29. One or more application
  30. 30. One or more application Applications throw jobs
  31. 31. One or more job servers
  32. 32. One or more job servers Job servers dispatch jobs
  33. 33. One or more workers
  34. 34. One or more workers (clones or different)
  35. 35. One or more workers Workers do jobs
  36. 36. One or more workers Workers do jobs and may issue new jobs
  37. 37. Scalable also means redundant
  38. 38. Scalable also means redundant and reliable
  39. 39. Scalable also means redundant and reliable* * when it works properly
  40. 40. Different approaches to the architecture
  41. 41. Synchronous or asynchronous
  42. 42. Worker either generates a response or works silently
  43. 43. Synchronous Application Job Response gearmand Job Response Worker
  44. 44. Asynchronous Application Job Response gearmand Job Response Worker
  45. 45. With a response Application Job Response gearmand Job Response Worker
  46. 46. With no response Application Job gearmand Job “I’m done” Worker
  47. 47. With no response Application Job gearmand Job “I’m done” Worker Result Database
  48. 48. Installation
  49. 49. Installation 1. Job server 2. Client libraries 3. Worker libraries
  50. 50. Installation 1. Job server C 2. Client libraries Perl 3. Worker libraries
  51. 51. Installation 1. Job server C 2. Client libraries Perl XS 3. Worker libraries
  52. 52. Installing job server https://launchpad.net/gearmand $
./configure $
make #
make
install
  53. 53. Installing Perl modules #
cpan
Gearman or #
cpan
Gearman::Client #
cpan
Gearman::Worker
  54. 54. Installing Perl modules #
cpan
Gearman or #
cpan
Gearman::XS
  55. 55. Starting gearmand deamon $
gearmand
‐d
  56. 56. Starting gearmand deamon $
gearmand
‐d or $
gearmand
‐d


 

‐L
127.0.0.1

 

‐p
4730
  57. 57. Starting worker(s) $
./worker.pl
&
  58. 58. Starting worker(s) $
./worker.pl
& or $
./workerA.pl
& $
./workerB.pl
&
  59. 59. Starting worker(s) $
./worker.pl
& or $
./workerA.pl
& $
./workerB.pl
& $
./workerA.pl
& $
./workerB.pl
&
  60. 60. Running client $
./client.pl
  61. 61. Worker example #!/usr/bin/perl use
v5.10; use
strict; use
Gearman::Worker; my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; }
  62. 62. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1;
  63. 63. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Connect to the server
  64. 64. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Register worker method
  65. 65. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Wait for incoming jobs
  66. 66. Worker example #!/usr/bin/perl use
v5.10; use
strict; use
Gearman::Worker; my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; }
  67. 67. Job method sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; } Receive data
  68. 68. Job method sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; } Evaluate response
  69. 69. Client example #!/usr/bin/perl use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('127.0.0.1:4730'); for (1..10000) {     my $result_ref = $client->do_task('echo', 10);     say $$result_ref; }
  70. 70. Client example for (1..10000) {     my $result_ref = $client->do_task('echo', 10);     say $$result_ref; } Issue jobs
  71. 71. Asynchronous client example use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('example.com:4730'); my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; sub completed {     my $result = shift;     say $$result; }
  72. 72. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Get task set placeholder
  73. 73. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Place the task
  74. 74. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; and register callback method
  75. 75. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Wait until the job is done
  76. 76. Asynchronous client example sub completed {     my $result = shift;     say $$result; } Use result somehow
  77. 77. XS worker example use v5.10; use strict; use Gearman::XS::Worker; my $worker = new Gearman::XS::Worker; $worker->add_server('127.0.0.1', 4730); $worker->add_function('echo', 0, &echo, undef); $worker->work() while 1; sub echo {     my $job = shift;     return $job->workload() + 1; }
  78. 78. XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret, $result) = $client->do('echo', '15'); # arg as string! say $result;
  79. 79. Async XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret1, $task1) = $client->add_task('echo', '15'); my ($ret2, $task2) = $client->add_task('echo', '51'); $client->run_tasks();
  80. 80. Monitoring job server
  81. 81. TCP protocol
  82. 82. Connect with telnet
  83. 83. $ telnet localhost 4730
  84. 84. $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'.
  85. 85. $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status
  86. 86. $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status echo2   1      1       2 echo    5      2       5 .
  87. 87. echo2   1      1       2 echo    5      2       5 .
  88. 88. Known worker methods echo2   1      1       2 echo    5      2       5 .
  89. 89. Jobs running echo2   1      1       2 echo    5      2       5 .
  90. 90. Jobs queued echo2   1      1       2 echo    5      2       5 .
  91. 91. Workers available echo2   1      1       2 echo    5      2       5 .
  92. 92. Part III Step by step example
  93. 93. worker.pl for(...)
{ 

... worker.pl } worker.pl worker.pl
  94. 94. Live demo
  95. 95. __END__ Andrew Shitov andy@shitov.ru
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×