Gearman and Perl

10,447 views
10,136 views

Published on

An introduction of what are Gearman and its architecture, and how to use it to dispatch jobs in Perl.

Published in: Technology, Business

Gearman and Perl

  1. 1. Dispatching jobs with Gearman
  2. 2. or
  3. 3. How to farm out jobs
  4. 4. or
  5. 5. Using Gearman in Perl
  6. 6. Using Gearman in Perl
  7. 7. gearman.org
  8. 8. The name "Gearman" was chosen as an anagram for "Manager," "since it dispatches jobs to be done, but does not do anything useful itself"
  9. 9. Originally written in Perl
  10. 10. Rewritten in C
  11. 11. Clients in Perl, PHP, Phython, Java, C# (.NET)
  12. 12. Clients in Perl, PHP, Phython, Java, C# (.NET), even MySQL and PostgreSQL
  13. 13. WTF?
  14. 14. Part I Architecture
  15. 15. Application Job
  16. 16. Application Job (or task)
  17. 17. Application Job gearmand
  18. 18. Application Job gearmand Job Worker
  19. 19. Application Job gearmand Job Response Worker
  20. 20. Application gearmand Worker Worker Worker
  21. 21. Application gearmand gearmand Worker Worker
  22. 22. Application Application gearmand gearmand Worker Worker
  23. 23. Workers are scalable
  24. 24. Run any number you need
  25. 25. Run on remote servers
  26. 26. Application does not care
  27. 27. Application only talks with gearmand server
  28. 28. Application only talks with gearmand server (one or more)
  29. 29. One or more application
  30. 30. One or more application Applications throw jobs
  31. 31. One or more job servers
  32. 32. One or more job servers Job servers dispatch jobs
  33. 33. One or more workers
  34. 34. One or more workers (clones or different)
  35. 35. One or more workers Workers do jobs
  36. 36. One or more workers Workers do jobs and may issue new jobs
  37. 37. Scalable also means redundant
  38. 38. Scalable also means redundant and reliable
  39. 39. Scalable also means redundant and reliable* * when it works properly
  40. 40. Different approaches to the architecture
  41. 41. Synchronous or asynchronous
  42. 42. Worker either generates a response or works silently
  43. 43. Synchronous Application Job Response gearmand Job Response Worker
  44. 44. Asynchronous Application Job Response gearmand Job Response Worker
  45. 45. With a response Application Job Response gearmand Job Response Worker
  46. 46. With no response Application Job gearmand Job “I’m done” Worker
  47. 47. With no response Application Job gearmand Job “I’m done” Worker Result Database
  48. 48. Installation
  49. 49. Installation 1. Job server 2. Client libraries 3. Worker libraries
  50. 50. Installation 1. Job server C 2. Client libraries Perl 3. Worker libraries
  51. 51. Installation 1. Job server C 2. Client libraries Perl XS 3. Worker libraries
  52. 52. Installing job server https://launchpad.net/gearmand $
./configure $
make #
make
install
  53. 53. Installing Perl modules #
cpan
Gearman or #
cpan
Gearman::Client #
cpan
Gearman::Worker
  54. 54. Installing Perl modules #
cpan
Gearman or #
cpan
Gearman::XS
  55. 55. Starting gearmand deamon $
gearmand
‐d
  56. 56. Starting gearmand deamon $
gearmand
‐d or $
gearmand
‐d


 

‐L
127.0.0.1

 

‐p
4730
  57. 57. Starting worker(s) $
./worker.pl
&
  58. 58. Starting worker(s) $
./worker.pl
& or $
./workerA.pl
& $
./workerB.pl
&
  59. 59. Starting worker(s) $
./worker.pl
& or $
./workerA.pl
& $
./workerB.pl
& $
./workerA.pl
& $
./workerB.pl
&
  60. 60. Running client $
./client.pl
  61. 61. Worker example #!/usr/bin/perl use
v5.10; use
strict; use
Gearman::Worker; my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; }
  62. 62. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1;
  63. 63. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Connect to the server
  64. 64. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Register worker method
  65. 65. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Wait for incoming jobs
  66. 66. Worker example #!/usr/bin/perl use
v5.10; use
strict; use
Gearman::Worker; my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; }
  67. 67. Job method sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; } Receive data
  68. 68. Job method sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; } Evaluate response
  69. 69. Client example #!/usr/bin/perl use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('127.0.0.1:4730'); for (1..10000) {     my $result_ref = $client->do_task('echo', 10);     say $$result_ref; }
  70. 70. Client example for (1..10000) {     my $result_ref = $client->do_task('echo', 10);     say $$result_ref; } Issue jobs
  71. 71. Asynchronous client example use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('example.com:4730'); my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; sub completed {     my $result = shift;     say $$result; }
  72. 72. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Get task set placeholder
  73. 73. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Place the task
  74. 74. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; and register callback method
  75. 75. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Wait until the job is done
  76. 76. Asynchronous client example sub completed {     my $result = shift;     say $$result; } Use result somehow
  77. 77. XS worker example use v5.10; use strict; use Gearman::XS::Worker; my $worker = new Gearman::XS::Worker; $worker->add_server('127.0.0.1', 4730); $worker->add_function('echo', 0, &echo, undef); $worker->work() while 1; sub echo {     my $job = shift;     return $job->workload() + 1; }
  78. 78. XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret, $result) = $client->do('echo', '15'); # arg as string! say $result;
  79. 79. Async XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret1, $task1) = $client->add_task('echo', '15'); my ($ret2, $task2) = $client->add_task('echo', '51'); $client->run_tasks();
  80. 80. Monitoring job server
  81. 81. TCP protocol
  82. 82. Connect with telnet
  83. 83. $ telnet localhost 4730
  84. 84. $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'.
  85. 85. $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status
  86. 86. $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status echo2   1      1       2 echo    5      2       5 .
  87. 87. echo2   1      1       2 echo    5      2       5 .
  88. 88. Known worker methods echo2   1      1       2 echo    5      2       5 .
  89. 89. Jobs running echo2   1      1       2 echo    5      2       5 .
  90. 90. Jobs queued echo2   1      1       2 echo    5      2       5 .
  91. 91. Workers available echo2   1      1       2 echo    5      2       5 .
  92. 92. Part III Step by step example
  93. 93. worker.pl for(...)
{ 

... worker.pl } worker.pl worker.pl
  94. 94. Live demo
  95. 95. __END__ Andrew Shitov andy@shitov.ru

×