Gearman and Perl

  • 8,375 views
Uploaded on

An introduction of what are Gearman and its architecture, and how to use it to dispatch jobs in Perl.

An introduction of what are Gearman and its architecture, and how to use it to dispatch jobs in Perl.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
8,375
On Slideshare
0
From Embeds
0
Number of Embeds
9

Actions

Shares
Downloads
126
Comments
0
Likes
7

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Dispatching jobs with Gearman
  • 2. or
  • 3. How to farm out jobs
  • 4. or
  • 5. Using Gearman in Perl
  • 6. Using Gearman in Perl
  • 7. gearman.org
  • 8. The name "Gearman" was chosen as an anagram for "Manager," "since it dispatches jobs to be done, but does not do anything useful itself"
  • 9. Originally written in Perl
  • 10. Rewritten in C
  • 11. Clients in Perl, PHP, Phython, Java, C# (.NET)
  • 12. Clients in Perl, PHP, Phython, Java, C# (.NET), even MySQL and PostgreSQL
  • 13. WTF?
  • 14. Part I Architecture
  • 15. Application Job
  • 16. Application Job (or task)
  • 17. Application Job gearmand
  • 18. Application Job gearmand Job Worker
  • 19. Application Job gearmand Job Response Worker
  • 20. Application gearmand Worker Worker Worker
  • 21. Application gearmand gearmand Worker Worker
  • 22. Application Application gearmand gearmand Worker Worker
  • 23. Workers are scalable
  • 24. Run any number you need
  • 25. Run on remote servers
  • 26. Application does not care
  • 27. Application only talks with gearmand server
  • 28. Application only talks with gearmand server (one or more)
  • 29. One or more application
  • 30. One or more application Applications throw jobs
  • 31. One or more job servers
  • 32. One or more job servers Job servers dispatch jobs
  • 33. One or more workers
  • 34. One or more workers (clones or different)
  • 35. One or more workers Workers do jobs
  • 36. One or more workers Workers do jobs and may issue new jobs
  • 37. Scalable also means redundant
  • 38. Scalable also means redundant and reliable
  • 39. Scalable also means redundant and reliable* * when it works properly
  • 40. Different approaches to the architecture
  • 41. Synchronous or asynchronous
  • 42. Worker either generates a response or works silently
  • 43. Synchronous Application Job Response gearmand Job Response Worker
  • 44. Asynchronous Application Job Response gearmand Job Response Worker
  • 45. With a response Application Job Response gearmand Job Response Worker
  • 46. With no response Application Job gearmand Job “I’m done” Worker
  • 47. With no response Application Job gearmand Job “I’m done” Worker Result Database
  • 48. Installation
  • 49. Installation 1. Job server 2. Client libraries 3. Worker libraries
  • 50. Installation 1. Job server C 2. Client libraries Perl 3. Worker libraries
  • 51. Installation 1. Job server C 2. Client libraries Perl XS 3. Worker libraries
  • 52. Installing job server https://launchpad.net/gearmand $
./configure $
make #
make
install
  • 53. Installing Perl modules #
cpan
Gearman or #
cpan
Gearman::Client #
cpan
Gearman::Worker
  • 54. Installing Perl modules #
cpan
Gearman or #
cpan
Gearman::XS
  • 55. Starting gearmand deamon $
gearmand
‐d
  • 56. Starting gearmand deamon $
gearmand
‐d or $
gearmand
‐d


 

‐L
127.0.0.1

 

‐p
4730
  • 57. Starting worker(s) $
./worker.pl
&
  • 58. Starting worker(s) $
./worker.pl
& or $
./workerA.pl
& $
./workerB.pl
&
  • 59. Starting worker(s) $
./worker.pl
& or $
./workerA.pl
& $
./workerB.pl
& $
./workerA.pl
& $
./workerB.pl
&
  • 60. Running client $
./client.pl
  • 61. Worker example #!/usr/bin/perl use
v5.10; use
strict; use
Gearman::Worker; my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; }
  • 62. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1;
  • 63. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Connect to the server
  • 64. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Register worker method
  • 65. Worker example my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; Wait for incoming jobs
  • 66. Worker example #!/usr/bin/perl use
v5.10; use
strict; use
Gearman::Worker; my
$worker
=
new
Gearman::Worker; $worker‐>job_servers('127.0.0.1:4730'); $worker‐>register_function(echo
=>
&echo); $worker‐>work
while
1; sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; }
  • 67. Job method sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; } Receive data
  • 68. Job method sub
echo
{ 



my
$job
=
shift; 



say
'+1'; 



return
$job‐>arg
+
1; } Evaluate response
  • 69. Client example #!/usr/bin/perl use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('127.0.0.1:4730'); for (1..10000) {     my $result_ref = $client->do_task('echo', 10);     say $$result_ref; }
  • 70. Client example for (1..10000) {     my $result_ref = $client->do_task('echo', 10);     say $$result_ref; } Issue jobs
  • 71. Asynchronous client example use v5.10; use strict; use Gearman::Client; my $client = new Gearman::Client; $client->job_servers('example.com:4730'); my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; sub completed {     my $result = shift;     say $$result; }
  • 72. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Get task set placeholder
  • 73. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Place the task
  • 74. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; and register callback method
  • 75. Asynchronous client example my $tasks = $client->new_task_set; $tasks->add_task(     'echo' => 10,     {         on_complete => &completed     } ); $tasks->wait; Wait until the job is done
  • 76. Asynchronous client example sub completed {     my $result = shift;     say $$result; } Use result somehow
  • 77. XS worker example use v5.10; use strict; use Gearman::XS::Worker; my $worker = new Gearman::XS::Worker; $worker->add_server('127.0.0.1', 4730); $worker->add_function('echo', 0, &echo, undef); $worker->work() while 1; sub echo {     my $job = shift;     return $job->workload() + 1; }
  • 78. XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret, $result) = $client->do('echo', '15'); # arg as string! say $result;
  • 79. Async XS client example use v5.10; use strict; use Gearman::XS::Client; my $client = new Gearman::XS::Client; $client->add_server('127.0.0.1', 4730); my ($ret1, $task1) = $client->add_task('echo', '15'); my ($ret2, $task2) = $client->add_task('echo', '51'); $client->run_tasks();
  • 80. Monitoring job server
  • 81. TCP protocol
  • 82. Connect with telnet
  • 83. $ telnet localhost 4730
  • 84. $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'.
  • 85. $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status
  • 86. $ telnet localhost 4730 Trying 127.0.0.1... Connected to localhost. Escape character is '^]'. status echo2   1      1       2 echo    5      2       5 .
  • 87. echo2   1      1       2 echo    5      2       5 .
  • 88. Known worker methods echo2   1      1       2 echo    5      2       5 .
  • 89. Jobs running echo2   1      1       2 echo    5      2       5 .
  • 90. Jobs queued echo2   1      1       2 echo    5      2       5 .
  • 91. Workers available echo2   1      1       2 echo    5      2       5 .
  • 92. Part III Step by step example
  • 93. worker.pl for(...)
{ 

... worker.pl } worker.pl worker.pl
  • 94. Live demo
  • 95. __END__ Andrew Shitov andy@shitov.ru