Unix Programming with Perl 2

Unix Programming with Perl 2 DeNA Co., Ltd. Kazuho Oku

Writing correct code
tests aren’t enough
tests don’t ensure that the code is correct
writing correct code requires…
knowledge of perl and knowledge of the OS
Oct 15 2011 Unix Programming with Perl 2

Last Year’s Talk
Covered these aspects of Unix programming using Perl
$! and Errno
how to evaluate the errors
file handles
their internals
interaction w. fork(2)
Unix signals
various signals and how to handle them
writing cancellable code

Last Year’s Slides
http://www.slideshare.net/kazuho/unix-programming-with-perl

Today’s Talk
will cover more advanced topics
inter-process communication
Unix signals and race condition

IPC::Open3

IPC::Open3 - Pipe and Deadlock
# check syntax errors in perl scripts by using perl –c
my @cmd = (
$^X,
(map { "-I$_" } grep { !ref $_ } @INC),
'-c',
$file,
);
my $pid = open3(my $cin, my $cout, my $cerr, @cmd);
while (waitpid($pid, 0) != $pid) {}
ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file);
http://d.hatena.ne.jp/tokuhirom/20100813/1281666615

Q. generally speaking, this code might block. But when?

A1. blocks if the child process reads from STDIN

Close STDIN of the child before calling waitpid
close $cin;

A1. blocks if the child process reads from STDIN
A2. blocks if the child process writes long data to STDOUT or STDERR

Size of the pipe buffer
Pipe has size limit
cannot write infinitely
unless the other peer reads from pipe
But actually, how large is the size limit?

Size of the pipe buffer (2)
Checking the size of the pipe buffer
for (my $sz = 1; ; $sz++) {
my @cmd = (
$^X, '-e', qq(print "1"x$sz),
);
my $pid = open3(my $cin, my $cout, 0, @cmd)
or die $!;
print "size: $szn";
}

Size of the pipe buffer (3)
Size of the pipe buffer was…
Linux 2.6.32 (x86-64): 65,536 bytes
Mac OS X 10.6: 16,384 bytes
Size may vary
old versions of Linux: 4,096 bytes
TCP streams and Unix sockets have configurable buffer size as well

How should we avoid deadlocks?
if we do not need the output of the child process (ex. perl –c), could this be the right answer?
close $cout;
close $cerr;

No. The child process may get killed while trying to write to the output streams closed by the parent process
close $cout;
close $cerr;
# if child process tries to write at this moment, it
# will be killed by SIGPIPE
# and as a result, $? may become different

So how about this? we read all data from the child process
while (<$cout>) {}
while (<$cerr>) {}

No. Will deadlock if the child process writes more bytes than the pipe buffer size to STDERR
while (<$cout>) {} # may enter deadlock at this point
while (<$cerr>) {}

Tip: passing undef as CHLD_ERR will send all output to CHLD_OUT
my $pid = open3(my $cin, my $cout, undef , @cmd);
while (<$cout>) {}

IPC::Open3 – Better to use temporary files
Advice: don’t use pipes, use temporary files (unless you need to read the output of the child process while it is running)
my $cout = File::Temp->new();
my $pid = do {
local *COUT = $cout;
open3(my $cin, ’>&COUT’, 0, @cmd);
} or die $!;
seek($cout, 0, SEEK_SET) or die $!; # seek to the start
...

IPC::Open3 vs. open |-
Q. Why use IPC::Open3 instead of open ’-| cmd > tmpfile’?
A. to skip the shell invocation
for speed and security

IPC::Open3 vs. open |- (2)
Escaping for open |- is difficult
# using IPC::Open3
my @cmd = (
$prog, ’-e’, $arg2,
);
my $pid = open3(my $cin, my $cout, 0, @cmd)
or die $!;
# using open |-
my $cmd = ”$prog –e $arg”; # need to escape $arg
open my $fh, ’|-’,”$cmd > ” . $tempfh->filename
or die $!;

Avoid shell invocation
Shell invocation is evil
ex. iT○ns upgrade accidentally removes user files
failed to quote usernames with a whitespace
Direct invocation is safer
system($args) => system(@args)
open | => IPC::Open2 or IPC::Open3

Rewriting open |- using IPC::Open3
# the original
open(my $fh,’|-’, $cmd)
or die $!;
print $fh ”hellon”;
close $fh;
# TODO: check $?
# using IPC::Open3
my $pid = open3(my $fh,’>&STDOUT’, ’>&STDERR’, @cmd)
or die $!;
print $fh ”hellon”;
close $fh;
# TODO: check $?

Rewriting open -| using IPC::Open3
# the original
open(my $fh,’-|’, $cmd)
or die $!;
my $line = <$fh>;
close $fh;
# TODO: check $?
# using IPC::Open3
my $pid = open3(my $cin, my $fh, ’>&STDERR’, @cmd)
or die $!;
close $cin;
my $line = <$fh>;
close $fh;
# TODO: check $?

Signal and Race Condition

How to sleep until receiving a signal?
Is the code correct?
my $gothup = 0;
local $SIG{HUP} = sub { $gothup++ };
while (! $gothup) {
sleep(30 * 60); # sleep for 30 minutes
do_cleanup(); # do some periodical tasks
}
print ”SIGHUP!n”;

How to sleep until receiving a signal? (2)
a race condition exists
my $gothup = 0;
while (! $gothup) {
# What happens if a signal is arrives here?
# sleep() will sleep 30 minutes since it never
# gets interrupted by the signal
}

Use POSIX::pselect
SIGHUP is blocked outside of pselect (and thus no race conditions)
my $blocked = POSIX::SigSet->new(SIGHUP);
my $unblocked = POSIX::SigSet->new();
my $gothup = 0;
sigprocmask(SIG_BLOCK, $blocked, $unblocked);
while (! $gothup) {
pselect(undef, undef, undef, 30 * 60, $unblocked);
}

The problem of pselect
Pselect has race condition on many environments, implemented like…
sub select {
my ($rset, $wset, $eset, $secs, $mask) = @_;
my $oldmask = POSIX::SigSet->new();
sigprocmask(SIG_SETMASK, $mask, $oldmask); $oldmask);
my $ret = select($rset, $wset, $eset, $secs);
sigprocmask(SIG_SETMASK, $oldmask);
$ret;
}
osx has the problem, glibc on linux does not have the problem but bionic (android) has the problem, …

Using eval & die does not solve the problem
my $blocked = POSIX::SigSet->new(SIGHUP);
my $unblocked = POSIX::SigSet->new();
sigprocmask(SIG_BLOCK, $blocked, $unblocked);
$SIG{HUP} = sub { die ’sighup:’ };
while (! $gothup) {
{
local $@;
eval {
sigprocmask(SIG_SETMASK, $unblocked);
# what if the signal is delivered at the very moment the
# perl interpreter calls sleep(3)?
};
$gothup = $@ =~ /^sighup:/;
sigprocmask(SIG_SETMASK, $blocked);
}
}

The fix – call syswrite on signal
# set unsafe signal handler (see perldoc perlipc) that converts a
# signal to a message (Q. proper error handling as advised in this
# year and last year’s slides are missing. Can you name them?)
socketpair(my $sig_rdr, my $sig_wtr, AF_UNIX, SOCK_STREAM, PF_UNSPEC)
or die $!;
POSIX::sigaction(SIGHUP, POSIX::SigAction->new(sub {
syswrite($sig_wtr, "1", 1) == 1 or die $!;
}
}));
while (1) {
my $rfds = ''; vec($rfds, fileno($sig_rdr), 1) = 1;
if (select($rfds, undef, undef, 30 * 60) > 0) {
sysread($sig_rdr, my $buf, 1) == 1 or die $!;
last;
}
}

Summary

Summary
buffer size is not infinite, be aware of deadlocks on inter-process / network communication
avoid shell invocation, use system(@args) or IPC::Open3
be careful of race conditions when handling Unix signals

http://blog.kazuhooku.com	707
http://yapcasia.org	123
http://paper.li	17
http://us-w1.rockmelt.com	2
http://translate.googleusercontent.com	1

Unix Programming with Perl 2

by kazuho on Oct 14, 2011

Accessibility

Categories

Tags

Upload Details

Usage Rights

5 Embeds 850

Statistics

Unix Programming with Perl 2 — Presentation Transcript