Writing Prefork Workers / Servers

Writing Prefork Workers / Servers Cybozu Labs, Inc. Kazuho Oku

Job workers: the application area
Essential for large-scale webapps
to communicate with other services
for synchronizing data between storages
for resizing images, …
Oct 15 2010 Writing Prefork Workers / Servers

Job workers: prefork vs. event-driven
Prefork-based
Good for writing application servers, transcoders (image resizing, etc.), DB-based job workers
+ easy to write and to maintain
− consumes more memory (improved by CoW)

Job Workers: prefork vs. event-driven (2)
Event-driven-based
good for writing chat server / client (irc), comet-based applications
+ easy to implement interaction between the tasks (connections)
+ consumes less memory
− difficult to write and to maintain
− most modules cannot be called asynchronously

Job Workers: prefork vs. event-driven (3)
Using prefork at first is generally good
and then switch to an event-driven-based approach if performance matters
unless you are writing a server that implements interaction between the clients

Agenda
Parallel::Prefork
for managing prefork’ed processes
Server::Starter
for hot-deploying servers
Parallel::Scoreboard
for monitoring prefork-based workers / servers

Parallel::Prefork

Parallel::ForkManager – the good old way
# taken from the POD
my $pm = new Parallel::ForkManager($MAX_PROCESSES);
foreach $data (@all_data) {
# Forks and returns the pid for the child:
$pm->start and next;
... do some work with $data in the child process ...
$pm->finish; # Terminates the child process
}

Parallel::ForkManager – the problem
No support for shutdown
designed for operating on already-existing data in parallel, not for receiving and handling data concurrently
child processes are not killed by parent
no clean shutdown or restart

Parallel::Prefork – a signal-savvy manager
my $pm = Parallel::Prefork->new({
max_workers => $MAX_PROCESSES,
trap_signals => {
TERM => 'TERM', # send TERM to children when parent gets TERM
},
);
while ( $pm->signal_received ne 'TERM' ) {
... do some work within the child process ...
$pm->finish;
}
$pm->wait_all_children(); # wait for all children to exit

Parallel::Prefork – writing a Gearman worker
my $pm = Parallel::Prefork->new(...);
my $worker = Gearman::Worker->new;
...
while ($pm->signal_received ne 'TERM') {
# gracefully exit the child process when parent delegates TERM
my $stop_worker = undef;
local $SIG{TERM} = sub { $stop_worker = 1 };
$worker->work(stop_if => sub { $stop_worker });
$pm->finish;
}

Parallel::Prefork – writing a prefork server
my $listen_sock = IO::Socket::INET->new(
LocalAddr => $hostport,
Listen => Socket::SOMAXCONN,
ReuseAddr => 1,
);
while (my $socket = $listen_sock->accept) {
... communicate with the client ...
}
$pm->finish;
}

Parallel::Prefork – advanced topics
Graceful reconfiguration
possible, but often unnecessary, since…
for workers, short downtime is acceptable
… , or we can run multiple instances of worker processes to hide the downtime
for servers, we need hot-deploy to update code
and the same technique can be used for changing configuration
Changing # of worker processes
in general a wrong idea :-p

Parallel::Prefork – graceful reconfiguration
my $pm = Parallel::Prefork->new({
max_workers => $MAX_PROCESSES,
trap_signals => {
TERM => 'TERM', # send TERM to children when parent gets TERM
HUP => 'TERM', # send TERM to chlidren on graceful reconf.
},
);
reload_config() if $pm->signal_received eq 'HUP';
... do some work within the child process ...
$pm->finish;
}

Parallel::Prefork – dynamic scaling
my $pm = Parallel::Prefork::SpareWorkers->new({
max_workers => 40,
min_spare_workers => 5,
max_spare_workers => 10,
...
);
while ($pm->signal_recieived ne 'TERM') {
... # setup signal handlers, etc.
while (my $sock = $listener->accept()) {
$pm->set_state('A'); # set state of the worker proc. to active
...
$pm->set_state(Parallel::Prefork::SpareWorkers::STATUS_IDLE);
}
$pm->finish();
}
...

Server::Starter

Hot deployment
what is it?
upgrading web application without restarting the application server
the goals
no downtime
no resource leaks
fail-safe

Old techniques to restart a webapp. server
restart the interpreter (mod_perl)
pros: graceful
cons: XS may cause resource leaks, service-down on deployment failure, cannot implement in pure-perl
bind to unix socket (FastCGI)
pros: graceful, fail-safe
cons: only useful for local-machine communication

Old techniques to restart a webapp. server (2)
exec(myself) (Net::Server)
pros: graceful, pure-perl
cons: file descriptor leaks, service-down on deployment failure

The restart method of Server::Starter
a superdaemon for hot-deploying TCP servers
superdaemon binds to TCP ports, then spawns the application server
Oct 15 2010 Writing Prefork Workers / Servers listen spawn app. servers SIGHUP accept app. logic fork & exec accept app. logic SIGTERM accept app. logic fork & exec

Server::Starter – no downtime
listening socket shared by old and new generation app. servers
old app. servers receive SIGTERM after new servers start

Server::Starter – no resource leaks
no chance of resource leaks
every generation of app. servers spawned from superdaemon

Server::Starter – fail safe
old app. server retired if and only if the new app. server starts up successfully
service continues even if the updated app. server fails to start, in cases like missing modules, etc.
a good practice is to do self-testing in the app. server before starting to serve client connections
is also an efficient way to preload modules

Server::Starter – the code
# only change the code that listens to a port
+ if (defined $ENV{SERVER_STARTER_PORT}) {
+ ($hostport, my $fd) = %{Server::Starter::server_ports};
+ $hostport = “0.0.0.0:$_”
+ unless $hostport =~ /:/;
+ $listen_sock = IO::Socket::INET->new(Proto => 'tcp');
+ $listen_sock->fdopen($fd, 'w’)
+ or die "failed to bind to listening socket:$!";
+ } else {
$listen_sock = IO::Socket::INET->new(
LocalAddr => $hostport,
Listen => Socket::SOMAXCONN,
ReuseAddr => 1,
);
+ }

Server::Starter – integration w. daemontools
The “run” script:
#! /bin/sh
exec start_server --port=80 -- my_server.pl
To start the server:
svc –u /service/my_server
To stop the server:
svc –d /service/my_server
To gracefully restart the updated version of the server:
svc –h /service/my_server

Parallel::Scoreboard

Monitoring the workers / servers
is essential for…
resource provisioning
fault management / fixing bugs
but how?
load average is not an answer
the bottleneck might not be CPU or disk I/O
logging is good for tracking down bugs, but difficult to use for monitoring

Parallel::Scoreboard – the caveats
a building block for monitoring processes
for visualization (like the mod_status of Apache)
for creating automated monitoring system
flexible
any number of processes can be monitored
any information can be stored for monitoring
any process can monitor any set of processes
no relationship between the monitoring processes and monitored processes is necessary

Parallel::Scoreboard – under the hood
one file per monitored process
<base_dir>/status_<pid>
uses flock for GC
monitored process locks its status file while alive
monitoring process removes unlocked status files
uses checksum for detecting r/w collision
monitoring process retries on collision

Parallel::Scoreboard – monitored process
# prepare
my $scoreboard = Parallel::Scoreboard->new(
base_dir => '/tmp/scoreboard',
);
# set the initial status. Note that the "update” method should only
# be called from worker processes
$scoreboard->update('initializing');
# the main loop
while (1) {
$scoreboard->update('waiting for task');
my $task = get_task();
$scoreboard->update('handling ' . $task->{id});
handle_task($task);
}
$scoreboard->update('exitting');

Parallel::Scoreboard – integrating with P::Prefork
my $scoreboard = Parallel::Scoreboard->new(...);
my $worker = Gearman::Worker->new;
$worker->register_function(handle_job => sub {
$scoreboard->update('handling ' . ...);
try {
... handle the job ...
} finally {
$scoreboard->update('idle');
}
});
$scoreboard->update(‘idle’); # just entered the child process
...

Parallel::Scoreboard – monitoring process
# prepare
my $scoreboard = Parallel::Scoreboard->new(
base_dir => '/tmp/scoreboard',
);
# read the status, and print
my $stats = $scoreboard->read_all();
for my $pid (sort { $a <=> $b } keys %$stats) {
print "PID:$pid is $stats->{$pid} ”;
}

Parallel::Scoreboard – monitoring by HTTP
Parallel::Scoreboard::PSGI::App
mod_status-like display
Parallel::Scoreboard::PSGI::App::JSON
scoreboard sent out in JSON
expects the status to be string or a JSON array / object
useful for gathering scoreboard of many machines
also useful for auto-scaling
automatically power-up / down the app. servers depending on the output of the scoreboards

Parallel::Scoreboard – monitoring in HTTP (2)
plackup --port 8080 –e 'use Parallel::Scoreboard::PSGI::App::JSON;
Parallel::Scoreboard::PSGI::App::JSON->new(
scoreboard => Parallel::Scoreboard->new(
base_dir => "/tmp/my_scoreboard",
),
)->to_app'

Summary

Summary
use Parallel::Prefork when writing job workers / servers
use Server::Starter when hot-deployment is necessary
use Parallel::Scoreboard to create monitors for your job workers / servers

http://developer.cybozu.co.jp	2228
http://developer-old.cybozu.co.jp	11
http://webcache.googleusercontent.com	10
http://translate.googleusercontent.com	4
http://anonymouse.org	2
http://reblog.rot13.org	1
http://rssreader.isolinear.jp	1
http://dashboard.bloglines.com	1

Writing Prefork Workers / Servers

by Kazuho Oku on Oct 15, 2010

Accessibility

Categories

Upload Details

Usage Rights

8 Embeds 2,258

Statistics

Writing Prefork Workers / Servers Presentation Transcript