Unix Programming with Perl

37,049 views

Published on

Published in: Technology
2 Comments
11 Likes
Statistics
Notes
  • i saw a much Better PPT on ThesisScientist.com on the same Topic
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • More than 5000 registered IT consultants and Corporates.Search for IT online training Providers at http://www.todaycourses.com
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total views
37,049
On SlideShare
0
From Embeds
0
Number of Embeds
6,386
Actions
Shares
0
Downloads
181
Comments
2
Likes
11
Embeds 0
No embeds

No notes for slide

Unix Programming with Perl

  1. 1. Unix Programming with Perl Cybozu Labs, Inc. Kazuho Oku
  2. 2. Writing correct code <ul><li>tests aren’t enough </li></ul><ul><ul><li>tests don’t ensure that the code is correct </li></ul></ul><ul><li>writing correct code requires… </li></ul><ul><ul><li>knowledge of perl and knowledge of the OS </li></ul></ul><ul><li>the presentation covers the ascpects of unix programming using perl, including </li></ul><ul><ul><li>errno </li></ul></ul><ul><ul><li>fork and filehandles </li></ul></ul><ul><ul><li>Unix signals </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  3. 3. <ul><li>Errno </li></ul>Oct 16 2010 Unix Programming with Perl
  4. 4. The right way to “create a dir if not exists” <ul><li>Is this OK? </li></ul><ul><li>if (! -d $dir) { </li></ul><ul><li>mkdir $dir </li></ul><ul><li>or die &quot;failed to create dir:$dir:$!&quot;; </li></ul><ul><li>} </li></ul>Oct 16 2010 Unix Programming with Perl
  5. 5. The right way to “create a dir if not exists” (2) <ul><li>No! </li></ul><ul><li>if (! -d $dir) { </li></ul><ul><li># what if another process created a dir </li></ul><ul><li># while we are HERE? </li></ul><ul><li>mkdir $dir </li></ul><ul><li>or die &quot;failed to create dir:$dir:$!&quot;; </li></ul><ul><li>} </li></ul>Oct 16 2010 Unix Programming with Perl
  6. 6. The right way to “create a dir if not exists” (3) <ul><li>The right way is to check the cause of the error when mkdir fails </li></ul>Oct 16 2010 Unix Programming with Perl
  7. 7. The right way to “create a dir if not exists” (4) <ul><li>So, is this OK? </li></ul><ul><li>if (mkdir $dir) { </li></ul><ul><li># ok, dir created </li></ul><ul><li>} elsif ($! =~ /File exists/) { </li></ul><ul><li># ok, directory exists </li></ul><ul><li>} else { </li></ul><ul><li>die &quot;failed to create dir:$dir:$!&quot;; </li></ul><ul><li>} </li></ul>Oct 16 2010 Unix Programming with Perl
  8. 8. The right way to “create a dir if not exists” (5) <ul><li>No! The message stored in $! depends on OS and / or locale. </li></ul><ul><li>if (mkdir $dir) { </li></ul><ul><li># ok, dir created </li></ul><ul><li>} elsif ($! =~ /File exists/) { </li></ul><ul><li># ok, directory exists </li></ul><ul><li>} else { </li></ul><ul><li>die &quot;failed to create dir:$dir:$!&quot;; </li></ul><ul><li>} </li></ul>Oct 16 2010 Unix Programming with Perl
  9. 9. The right way to “create a dir if not exists” (6) <ul><li>The right way is to use Errno. </li></ul><ul><li>use Errno (); </li></ul><ul><li>if (mkdir $dir) { </li></ul><ul><li># ok, created dir </li></ul><ul><li>} elsif ($! == Errno::EEXIST) { </li></ul><ul><li># ok, already exists </li></ul><ul><li>} else { </li></ul><ul><li>die &quot;failed to create dir:$dir:$!&quot;; </li></ul><ul><li>} </li></ul>Oct 16 2010 Unix Programming with Perl
  10. 10. $! and Errno <ul><li>$! is a dualvar </li></ul><ul><ul><li>is a number in numeric context (ex. 17) </li></ul></ul><ul><ul><ul><li>equiv. to the errno global in C </li></ul></ul></ul><ul><ul><li>is a string in string context (ex. “File exists”) </li></ul></ul><ul><ul><ul><li>equiv. to strerror(errno) in C </li></ul></ul></ul><ul><li>the Errno module </li></ul><ul><ul><li>list of constants (numbers) that errno ($! in numeric context) may take </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  11. 11. How to find the Errnos <ul><li>perldoc -f mkdir doesn’t include a list of errnos it might return </li></ul><ul><li>see man 2 mkdir </li></ul><ul><ul><li>man mkdir will show the man page of the mkdir command </li></ul></ul><ul><ul><li>specify section 2 for system calls </li></ul></ul><ul><ul><li>specify section 3 for C library calls </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  12. 12. How to find the Errnos <ul><li>errnos on man include those defined by POSIX and OS-specific constants </li></ul><ul><li>the POSIX spec. can be found at opengroup.org, etc. </li></ul><ul><ul><li>http://www.opengroup.org/onlinepubs/000095399/ </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  13. 13. <ul><li>Fork and filehandles </li></ul>Oct 16 2010 Unix Programming with Perl
  14. 14. Filehandles aren’t cloned by fork <ul><li>fork clones the memory image </li></ul><ul><ul><li>uses CoW (Copy-on-Write) for optimization </li></ul></ul><ul><li>fork does not clone file handles </li></ul><ul><ul><li>only increments the refcount to the file handle in the OS </li></ul></ul><ul><ul><ul><li>the file is left open until both the parent and child closes the file </li></ul></ul></ul><ul><ul><li>seek position and lock states are shared between the processes </li></ul></ul><ul><ul><li>the same for TCP / UDP sockets </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  15. 15. File handles aren’t cloned by fork (2) Oct 16 2010 Unix Programming with Perl memory Parent Process Operating System File System lock owner, etc. File memory Another Process memory Child Process fork seek pos. / lock state, etc. File Control Info. open(file) seek pos. / lock state, etc. File Control Info. open(file)
  16. 16. Examples of resource collisions due to fork <ul><li>FAQ </li></ul><ul><ul><li>“ The SQLite database becomes corrupt” </li></ul></ul><ul><ul><li>“ MySQL reports malformed packet” </li></ul></ul><ul><li>mostly due to sharing a single DBI connection created before calling fork </li></ul><ul><ul><li>SQLite uses file locks for access control </li></ul></ul><ul><ul><ul><li>file lock needed for each process, however after fork the lock is shared between the processes </li></ul></ul></ul><ul><ul><li>in the case of MySQL a single TCP (or unix) connection is shared between the processes </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  17. 17. Examples of resource collisions due to fork (2) <ul><li>The wrong code… </li></ul><ul><li>my $dbh = DBI->connect(...); </li></ul><ul><li>my $pid = fork; </li></ul><ul><li>if ($pid == 0) { </li></ul><ul><li># child process </li></ul><ul><li>$dbi->do(...); </li></ul><ul><li>} else { </li></ul><ul><li># parent process </li></ul><ul><li>$dbi->do(...); </li></ul>Oct 16 2010 Unix Programming with Perl
  18. 18. How to avoid resource collisions after fork <ul><li>close the file handle in the child process (or in the parent) right after fork </li></ul><ul><ul><li>only the refcount will be decremented. lock states / seek positions do not change </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  19. 19. How to avoid collisions after fork (DBI) <ul><li>undef $dbh in the child process doesn’t work </li></ul><ul><ul><li>since the child process will run things such as unlocking and / or rollbacks on the shared DBI connection </li></ul></ul><ul><ul><li>the connection needs to be closed, without running such operations </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  20. 20. How to avoid collisions after fork (DBI) (2) <ul><li>the answer: use InactiveDestroy </li></ul><ul><li>my $pid = fork; </li></ul><ul><li>if ($pid == 0) { </li></ul><ul><li># child process </li></ul><ul><li>$dbh->{InactiveDestroy} = 1; </li></ul><ul><li>undef $dbh; </li></ul><ul><li>... </li></ul><ul><li>} </li></ul>Oct 16 2010 Unix Programming with Perl
  21. 21. How to avoid collisions after fork (DBI) (3) <ul><li>if fork is called deep inside a module and can’t be modified, then… </li></ul><ul><li># thanks to tokuhirom, kazeburo </li></ul><ul><li>BEGIN { </li></ul><ul><li>no strict qw(refs); </li></ul><ul><li>no warnings qw(redefine); </li></ul><ul><li>*CORE::GLOBAL::fork = sub { </li></ul><ul><li>my $pid = CORE::fork; </li></ul><ul><li>if ($pid == 0) { # do the cleanup for child process </li></ul><ul><li>$dbh->{InactiveDestroy} = 1; </li></ul><ul><li>undef $dbh; </li></ul><ul><li>} </li></ul><ul><li>$pid; </li></ul><ul><li>}; </li></ul><ul><li>} </li></ul>Oct 16 2010 Unix Programming with Perl
  22. 22. How to avoid collisions after fork (DBI) (4) <ul><li>other ways to change the behavior of fork </li></ul><ul><ul><li>POSIX::AtFork (gfx) </li></ul></ul><ul><ul><ul><li>Perl wrapper for pthread_atfork </li></ul></ul></ul><ul><ul><ul><li>can change the behavior of fork(2) called within XS </li></ul></ul></ul><ul><ul><li>forks.pm (rybskej) </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  23. 23. Close filehandles before calling exec <ul><li>file handles (file descriptors) are passed to the new process created by exec </li></ul><ul><ul><li>some tools (setlock of daemontools, Server::Starter) rely on the feature </li></ul></ul><ul><li>OTOH, it is a good practice to close the file handles that needn’t be passed to the exec’ed process, to avoid child process from accidentially using them </li></ul>Oct 16 2010 Unix Programming with Perl
  24. 24. Close file handles before calling exec (2) <ul><li>my $pid = fork; </li></ul><ul><li>if ($pid == 0) { </li></ul><ul><li># child process, close filehandles </li></ul><ul><li>$dbh->{InactiveDestroy} = 1; </li></ul><ul><li>undef $dbh; </li></ul><ul><li>exec ...; </li></ul><ul><li>... </li></ul>Oct 16 2010 Unix Programming with Perl
  25. 25. Close file handles before calling exec (3) <ul><li>Some OS’es have O_CLOEXEC flag </li></ul><ul><ul><li>designates the file descriptors to be closed when exec(2) is being called </li></ul></ul><ul><ul><li>is OS-dependent </li></ul></ul><ul><ul><ul><li>linux supports the flag, OSX doesn’t </li></ul></ul></ul><ul><ul><li>not usable from perl? </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  26. 26. <ul><li>Unix Signals </li></ul>Oct 16 2010 Unix Programming with Perl
  27. 27. SIGPIPE <ul><li>“ my network application suddenly dies without saying anything” </li></ul><ul><li>often due to not catching SIGPIPE </li></ul><ul><ul><li>a signal sent when failing to write to a filehandle </li></ul></ul><ul><ul><ul><li>ex. when the socket is closed by peer </li></ul></ul></ul><ul><ul><li>the default behavior is to kill the process </li></ul></ul><ul><li>solution: $SIG{PIPE} = 'IGNORE'; </li></ul><ul><ul><li>downside: you should consult the return value of print, etc. to check if the writes succeeded </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  28. 28. Using alarm <ul><li>alarm can be used (together with SIG{ALRM}, EINTR) to handle timeouts </li></ul><ul><li>local $SIG{ALRM} = sub {}; </li></ul><ul><li>alarm($timeout); </li></ul><ul><li>my $len = $sock->read(my $buf, $maxlen); </li></ul><ul><li>if (! defined($len) && $! == Errno::EINTR) { </li></ul><ul><li>warn 'timeout’; </li></ul><ul><li>return; </li></ul><ul><li>} </li></ul>Oct 16 2010 Unix Programming with Perl
  29. 29. Pros and cons of using alarm <ul><li>+ can be used to timeout almost all system calls (that may block) </li></ul><ul><li>− the timeout set by alarm(2) is a process-wide global (and so is $SIG{ALRM}) </li></ul><ul><ul><li>use of select (or IO::Select) is preferable for network access </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  30. 30. Writing cancellable code <ul><li>typical use-case: run forever until receiving a signal, and gracefully shutdown </li></ul><ul><ul><li>ex. Gearman::Worker </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  31. 31. Writing cancellable code (2) <ul><li>make your module cancellable </li></ul><ul><ul><li>- my $len = $sock->read(my $buf, $maxlen); </li></ul></ul><ul><ul><li>+ my $len; </li></ul></ul><ul><ul><li>+ { </li></ul></ul><ul><ul><li>+ $len = $sock->read(my $buf, $maxlen); </li></ul></ul><ul><ul><li>+ if (! defined($len) && $! == Errno::EINTR) { </li></ul></ul><ul><ul><li>+ return if $self->{cancel_requested}; </li></ul></ul><ul><ul><li>+ redo; </li></ul></ul><ul><ul><li>+ } </li></ul></ul><ul><ul><li>+ } </li></ul></ul><ul><ul><li>... </li></ul></ul><ul><ul><li>+ sub request_cancel { </li></ul></ul><ul><ul><li>+ my $self = shift; </li></ul></ul><ul><ul><li>+ $self->{cancel_requested} = 1; </li></ul></ul><ul><ul><li>+ } </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  32. 32. Writing cancellable code (3) <ul><li>The code that cancels the operation on SIGTERM </li></ul><ul><ul><li>$SIG{TERM} = sub { $my_module->request_cancel }; </li></ul></ul><ul><ul><li>$my_module->run_forever(); </li></ul></ul><ul><li>Or the caller may use alarm to set timeout </li></ul><ul><ul><li>$SIG{ALRM} = sub { $my_module->request_cancel }; </li></ul></ul><ul><ul><li>alarm(100); </li></ul></ul><ul><ul><li>$my_module->run_forever(); </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  33. 33. Proc::Wait3 <ul><li>built-in wait() and waitpid() does not return when receiving signals </li></ul><ul><ul><li>use Proc::Wait3 instead </li></ul></ul>Oct 16 2010 Unix Programming with Perl
  34. 34. <ul><li>Further reading </li></ul>Oct 16 2010 Unix Programming with Perl
  35. 35. Further reading <ul><li>this presentation is based on my memo for an article on WEB+DB PRESS </li></ul><ul><ul><li>if you have any questions, having problems on Unix programming, please let me know </li></ul></ul>Oct 16 2010 Unix Programming with Perl

×