Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Unix Programming with Perl 2

7,956 views

Published on

Techniques for writing good code in perl running on Unix platform.

Published in: Technology

Unix Programming with Perl 2

  1. 1. Unix Programming with Perl 2 DeNA Co., Ltd. Kazuho Oku
  2. 2. Writing correct code <ul><li>tests aren’t enough </li></ul><ul><ul><li>tests don’t ensure that the code is correct </li></ul></ul><ul><li>writing correct code requires… </li></ul><ul><ul><li>knowledge of perl and knowledge of the OS </li></ul></ul>Oct 15 2011 Unix Programming with Perl 2
  3. 3. Last Year’s Talk <ul><li>Covered these aspects of Unix programming using Perl </li></ul><ul><ul><li>$! and Errno </li></ul></ul><ul><ul><ul><li>how to evaluate the errors </li></ul></ul></ul><ul><ul><li>file handles </li></ul></ul><ul><ul><ul><li>their internals </li></ul></ul></ul><ul><ul><ul><li>interaction w. fork(2) </li></ul></ul></ul><ul><ul><li>Unix signals </li></ul></ul><ul><ul><ul><li>various signals and how to handle them </li></ul></ul></ul><ul><ul><ul><li>writing cancellable code </li></ul></ul></ul>Oct 15 2011 Unix Programming with Perl 2
  4. 4. Last Year’s Slides <ul><li>http://www.slideshare.net/kazuho/unix-programming-with-perl </li></ul>Oct 15 2011 Unix Programming with Perl 2
  5. 5. Today’s Talk <ul><li>will cover more advanced topics </li></ul><ul><ul><li>inter-process communication </li></ul></ul><ul><ul><li>Unix signals and race condition </li></ul></ul>Oct 15 2011 Unix Programming with Perl 2
  6. 6. <ul><li>IPC::Open3 </li></ul>Oct 15 2011 Unix Programming with Perl 2
  7. 7. IPC::Open3 - Pipe and Deadlock <ul><li># check syntax errors in perl scripts by using perl –c </li></ul><ul><li>my @cmd = ( </li></ul><ul><li>$^X, </li></ul><ul><li>(map { &quot;-I$_&quot; } grep { !ref $_ } @INC), </li></ul><ul><li>'-c', </li></ul><ul><li>$file, </li></ul><ul><li>); </li></ul><ul><li>my $pid = open3(my $cin, my $cout, my $cerr, @cmd); </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul><ul><li>ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file); </li></ul><ul><li>http://d.hatena.ne.jp/tokuhirom/20100813/1281666615 </li></ul>Oct 15 2011 Unix Programming with Perl 2
  8. 8. IPC::Open3 - Pipe and Deadlock <ul><li>Q. generally speaking, this code might block. But when? </li></ul><ul><li>my $pid = open3(my $cin, my $cout, my $cerr, @cmd); </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul>Oct 15 2011 Unix Programming with Perl 2
  9. 9. IPC::Open3 - Pipe and Deadlock <ul><li>Q. generally speaking, this code might block. But when? </li></ul><ul><li>my $pid = open3(my $cin, my $cout, my $cerr, @cmd); </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul><ul><li>A1. blocks if the child process reads from STDIN </li></ul>Oct 15 2011 Unix Programming with Perl 2
  10. 10. IPC::Open3 - Pipe and Deadlock <ul><li>Close STDIN of the child before calling waitpid </li></ul><ul><li>my $pid = open3(my $cin, my $cout, my $cerr, @cmd); </li></ul><ul><li>close $cin; </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul>Oct 15 2011 Unix Programming with Perl 2
  11. 11. IPC::Open3 - Pipe and Deadlock <ul><li>Q. generally speaking, this code might block. But when? </li></ul><ul><li>my $pid = open3(my $cin, my $cout, my $cerr, @cmd); </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul><ul><li>A1. blocks if the child process reads from STDIN </li></ul><ul><li>A2. blocks if the child process writes long data to STDOUT or STDERR </li></ul>Oct 15 2011 Unix Programming with Perl 2
  12. 12. Size of the pipe buffer <ul><li>Pipe has size limit </li></ul><ul><ul><li>cannot write infinitely </li></ul></ul><ul><ul><li>unless the other peer reads from pipe </li></ul></ul><ul><li>But actually, how large is the size limit? </li></ul>Oct 15 2011 Unix Programming with Perl 2
  13. 13. Size of the pipe buffer (2) <ul><li>Checking the size of the pipe buffer </li></ul><ul><li>for (my $sz = 1; ; $sz++) { </li></ul><ul><li>my @cmd = ( </li></ul><ul><li>$^X, '-e', qq(print &quot;1&quot;x$sz), </li></ul><ul><li>); </li></ul><ul><li>my $pid = open3(my $cin, my $cout, 0, @cmd) </li></ul><ul><li>or die $!; </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul><ul><li>print &quot;size: $szn&quot;; </li></ul><ul><li>} </li></ul>Oct 15 2011 Unix Programming with Perl 2
  14. 14. Size of the pipe buffer (3) <ul><li>Size of the pipe buffer was… </li></ul><ul><ul><li>Linux 2.6.32 (x86-64): 65,536 bytes </li></ul></ul><ul><ul><li>Mac OS X 10.6: 16,384 bytes </li></ul></ul><ul><li>Size may vary </li></ul><ul><ul><li>old versions of Linux: 4,096 bytes </li></ul></ul><ul><li>TCP streams and Unix sockets have configurable buffer size as well </li></ul>Oct 15 2011 Unix Programming with Perl 2
  15. 15. IPC::Open3 - Pipe and Deadlock <ul><li>How should we avoid deadlocks? </li></ul><ul><ul><li>if we do not need the output of the child process (ex. perl –c), could this be the right answer? </li></ul></ul><ul><li>my $pid = open3(my $cin, my $cout, my $cerr, @cmd); </li></ul><ul><li>close $cout; </li></ul><ul><li>close $cerr; </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul><ul><li>ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file); </li></ul>Oct 15 2011 Unix Programming with Perl 2
  16. 16. IPC::Open3 - Pipe and Deadlock <ul><li>No. The child process may get killed while trying to write to the output streams closed by the parent process </li></ul><ul><li>my $pid = open3(my $cin, my $cout, my $cerr, @cmd); </li></ul><ul><li>close $cout; </li></ul><ul><li>close $cerr; </li></ul><ul><li># if child process tries to write at this moment, it </li></ul><ul><li># will be killed by SIGPIPE </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul><ul><li># and as a result, $? may become different </li></ul><ul><li>ok((WIFEXITED($?) && WEXITSTATUS($?) == 0), $file); </li></ul>Oct 15 2011 Unix Programming with Perl 2
  17. 17. IPC::Open3 - Pipe and Deadlock <ul><li>So how about this? we read all data from the child process </li></ul><ul><li>my $pid = open3(my $cin, my $cout, my $cerr, @cmd); </li></ul><ul><li>while (<$cout>) {} </li></ul><ul><li>while (<$cerr>) {} </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul>Oct 15 2011 Unix Programming with Perl 2
  18. 18. IPC::Open3 - Pipe and Deadlock <ul><li>No. Will deadlock if the child process writes more bytes than the pipe buffer size to STDERR </li></ul><ul><li>my $pid = open3(my $cin, my $cout, my $cerr, @cmd); </li></ul><ul><li>while (<$cout>) {} # may enter deadlock at this point </li></ul><ul><li>while (<$cerr>) {} </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul>Oct 15 2011 Unix Programming with Perl 2
  19. 19. IPC::Open3 - Pipe and Deadlock <ul><li>Tip: passing undef as CHLD_ERR will send all output to CHLD_OUT </li></ul><ul><li>my $pid = open3(my $cin, my $cout, undef , @cmd); </li></ul><ul><li>while (<$cout>) {} </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul>Oct 15 2011 Unix Programming with Perl 2
  20. 20. IPC::Open3 – Better to use temporary files <ul><li>Advice: don’t use pipes, use temporary files (unless you need to read the output of the child process while it is running) </li></ul><ul><li>my $cout = File::Temp->new(); </li></ul><ul><li>my $pid = do { </li></ul><ul><li>local *COUT = $cout; </li></ul><ul><li>open3(my $cin, ’>&COUT’, 0, @cmd); </li></ul><ul><li>} or die $!; </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul><ul><li>seek($cout, 0, SEEK_SET) or die $!; # seek to the start </li></ul><ul><li>... </li></ul>Oct 15 2011 Unix Programming with Perl 2
  21. 21. IPC::Open3 vs. open |- <ul><li>Q. Why use IPC::Open3 instead of open ’-| cmd > tmpfile’? </li></ul><ul><li>A. to skip the shell invocation </li></ul><ul><ul><li>for speed and security </li></ul></ul>Oct 15 2011 Unix Programming with Perl 2
  22. 22. IPC::Open3 vs. open |- (2) <ul><li>Escaping for open |- is difficult </li></ul><ul><ul><li># using IPC::Open3 </li></ul></ul><ul><ul><li>my @cmd = ( </li></ul></ul><ul><ul><li>$prog, ’-e’, $arg2, </li></ul></ul><ul><ul><li>); </li></ul></ul><ul><ul><li>my $pid = open3(my $cin, my $cout, 0, @cmd) </li></ul></ul><ul><ul><li>or die $!; </li></ul></ul><ul><ul><li># using open |- </li></ul></ul><ul><ul><li>my $cmd = ”$prog –e $arg”; # need to escape $arg </li></ul></ul><ul><ul><li>open my $fh, ’|-’,”$cmd > ” . $tempfh->filename </li></ul></ul><ul><ul><li>or die $!; </li></ul></ul>Oct 15 2011 Unix Programming with Perl 2
  23. 23. Avoid shell invocation <ul><li>Shell invocation is evil </li></ul><ul><ul><li>ex. iT○ns upgrade accidentally removes user files </li></ul></ul><ul><ul><ul><li>failed to quote usernames with a whitespace </li></ul></ul></ul><ul><li>Direct invocation is safer </li></ul><ul><ul><li>system($args) => system(@args) </li></ul></ul><ul><ul><li>open | => IPC::Open2 or IPC::Open3 </li></ul></ul>Oct 15 2011 Unix Programming with Perl 2
  24. 24. Rewriting open |- using IPC::Open3 <ul><li># the original </li></ul><ul><li>open(my $fh,’|-’, $cmd) </li></ul><ul><li>or die $!; </li></ul><ul><li>print $fh ”hellon”; </li></ul><ul><li>close $fh; </li></ul><ul><li># TODO: check $? </li></ul><ul><li># using IPC::Open3 </li></ul><ul><li>my $pid = open3(my $fh,’>&STDOUT’, ’>&STDERR’, @cmd) </li></ul><ul><li>or die $!; </li></ul><ul><li>print $fh ”hellon”; </li></ul><ul><li>close $fh; </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul><ul><li># TODO: check $? </li></ul>Oct 15 2011 Unix Programming with Perl 2
  25. 25. Rewriting open -| using IPC::Open3 <ul><li># the original </li></ul><ul><li>open(my $fh,’-|’, $cmd) </li></ul><ul><li>or die $!; </li></ul><ul><li>my $line = <$fh>; </li></ul><ul><li>close $fh; </li></ul><ul><li># TODO: check $? </li></ul><ul><li># using IPC::Open3 </li></ul><ul><li>my $pid = open3(my $cin, my $fh, ’>&STDERR’, @cmd) </li></ul><ul><li>or die $!; </li></ul><ul><li>close $cin; </li></ul><ul><li>my $line = <$fh>; </li></ul><ul><li>close $fh; </li></ul><ul><li>while (waitpid($pid, 0) != $pid) {} </li></ul><ul><li># TODO: check $? </li></ul>Oct 15 2011 Unix Programming with Perl 2
  26. 26. <ul><li>Signal and Race Condition </li></ul>Oct 15 2011 Unix Programming with Perl 2
  27. 27. How to sleep until receiving a signal? <ul><li>Is the code correct? </li></ul><ul><li>my $gothup = 0; </li></ul><ul><li>local $SIG{HUP} = sub { $gothup++ }; </li></ul><ul><li>while (! $gothup) { </li></ul><ul><li>sleep(30 * 60); # sleep for 30 minutes </li></ul><ul><li>do_cleanup(); # do some periodical tasks </li></ul><ul><li>} </li></ul><ul><li>print ”SIGHUP!n”; </li></ul>Oct 15 2011 Unix Programming with Perl 2
  28. 28. How to sleep until receiving a signal? (2) <ul><li>a race condition exists </li></ul><ul><li>my $gothup = 0; </li></ul><ul><li>local $SIG{HUP} = sub { $gothup++ }; </li></ul><ul><li>while (! $gothup) { </li></ul><ul><li># What happens if a signal is arrives here? </li></ul><ul><li># sleep() will sleep 30 minutes since it never </li></ul><ul><li># gets interrupted by the signal </li></ul><ul><li>sleep(30 * 60); # sleep for 30 minutes </li></ul><ul><li>do_cleanup(); # do some periodical tasks </li></ul><ul><li>} </li></ul><ul><li>print ”SIGHUP!n”; </li></ul>Oct 15 2011 Unix Programming with Perl 2
  29. 29. Use POSIX::pselect <ul><li>SIGHUP is blocked outside of pselect (and thus no race conditions) </li></ul><ul><li>my $blocked = POSIX::SigSet->new(SIGHUP); </li></ul><ul><li>my $unblocked = POSIX::SigSet->new(); </li></ul><ul><li>my $gothup = 0; </li></ul><ul><li>local $SIG{HUP} = sub { $gothup++ }; </li></ul><ul><li>sigprocmask(SIG_BLOCK, $blocked, $unblocked); </li></ul><ul><li>while (! $gothup) { </li></ul><ul><li>pselect(undef, undef, undef, 30 * 60, $unblocked); </li></ul><ul><li>do_cleanup(); # do some periodical tasks </li></ul><ul><li>} </li></ul><ul><li>print ”SIGHUP!n”; </li></ul>Oct 15 2011 Unix Programming with Perl 2
  30. 30. The problem of pselect <ul><li>Pselect has race condition on many environments, implemented like… </li></ul><ul><li>sub select { </li></ul><ul><li>my ($rset, $wset, $eset, $secs, $mask) = @_; </li></ul><ul><li>my $oldmask = POSIX::SigSet->new(); </li></ul><ul><li>sigprocmask(SIG_SETMASK, $mask, $oldmask); $oldmask); </li></ul><ul><li>my $ret = select($rset, $wset, $eset, $secs); </li></ul><ul><li>sigprocmask(SIG_SETMASK, $oldmask); </li></ul><ul><li>$ret; </li></ul><ul><li>} </li></ul><ul><ul><li>osx has the problem, glibc on linux does not have the problem but bionic (android) has the problem, … </li></ul></ul>Oct 15 2011 Unix Programming with Perl 2
  31. 31. Using eval & die does not solve the problem <ul><li>my $blocked = POSIX::SigSet->new(SIGHUP); </li></ul><ul><li>my $unblocked = POSIX::SigSet->new(); </li></ul><ul><li>sigprocmask(SIG_BLOCK, $blocked, $unblocked); </li></ul><ul><li>$SIG{HUP} = sub { die ’sighup:’ }; </li></ul><ul><li>while (! $gothup) { </li></ul><ul><li>{ </li></ul><ul><li>local $@; </li></ul><ul><li>eval { </li></ul><ul><li>sigprocmask(SIG_SETMASK, $unblocked); </li></ul><ul><li># what if the signal is delivered at the very moment the </li></ul><ul><li># perl interpreter calls sleep(3)? </li></ul><ul><li>sleep(30 * 60); # sleep for 30 minutes </li></ul><ul><li>}; </li></ul><ul><li>$gothup = $@ =~ /^sighup:/; </li></ul><ul><li>sigprocmask(SIG_SETMASK, $blocked); </li></ul><ul><li>} </li></ul><ul><li>do_cleanup(); # do some periodical tasks </li></ul><ul><li>} </li></ul>Oct 15 2011 Unix Programming with Perl 2
  32. 32. The fix – call syswrite on signal <ul><li># set unsafe signal handler (see perldoc perlipc) that converts a </li></ul><ul><li># signal to a message (Q. proper error handling as advised in this </li></ul><ul><li># year and last year’s slides are missing. Can you name them?) </li></ul><ul><li>socketpair(my $sig_rdr, my $sig_wtr, AF_UNIX, SOCK_STREAM, PF_UNSPEC) </li></ul><ul><li>or die $!; </li></ul><ul><li>POSIX::sigaction(SIGHUP, POSIX::SigAction->new(sub { </li></ul><ul><li>syswrite($sig_wtr, &quot;1&quot;, 1) == 1 or die $!; </li></ul><ul><li>} </li></ul><ul><li>})); </li></ul><ul><li>while (1) { </li></ul><ul><li>my $rfds = ''; vec($rfds, fileno($sig_rdr), 1) = 1; </li></ul><ul><li>if (select($rfds, undef, undef, 30 * 60) > 0) { </li></ul><ul><li>sysread($sig_rdr, my $buf, 1) == 1 or die $!; </li></ul><ul><li>last; </li></ul><ul><li>} </li></ul><ul><li>do_cleanup(); # do some periodical tasks </li></ul><ul><li>} </li></ul>Oct 15 2011 Unix Programming with Perl 2
  33. 33. <ul><li>Summary </li></ul>Oct 15 2011 Unix Programming with Perl 2
  34. 34. Summary <ul><li>buffer size is not infinite, be aware of deadlocks on inter-process / network communication </li></ul><ul><li>avoid shell invocation, use system(@args) or IPC::Open3 </li></ul><ul><li>be careful of race conditions when handling Unix signals </li></ul>Oct 15 2011 Unix Programming with Perl 2

×