Python & Perl: Unix/Linux Pipeline

919 views

Published on

Python & Perl: Unix/Linux Pipeline

Published in: Software
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
919
On SlideShare
0
From Embeds
0
Number of Embeds
372
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Python & Perl: Unix/Linux Pipeline

  1. 1. Python & Perl File Processing in Perl: Part 02 Unix/Linux Pipeline Vladimir Kulyukinwww.youtube.com/vkedco www.vkedco.blogspot.com
  2. 2. Outline ● Sorting Files ● Pipeswww.youtube.com/vkedco www.vkedco.blogspot.com
  3. 3. Sorting Fileswww.youtube.com/vkedco www.vkedco.blogspot.com
  4. 4. Sorting Files ● Files can be sorted in PL ● Remember that PL sort is by default sorts its input as ASCII strings ● There are two ways to sort the contents of a file: – Slurp and Sort – Use Filehandleswww.youtube.com/vkedco www.vkedco.blogspot.com
  5. 5. Slurp & Sort as Numbers use strict; use warnings; my $infile = shift; ## 1. open input file handles open(FH, <, $infile) or die $!; ## 2. slurped lines into list my @lines = <FH>; ## 3. sort slurped lines as numbers print sort { $a <=> $b } @lines;www.youtube.com/vkedco www.vkedco.blogspot.com
  6. 6. Slurp & Sort as Strings use strict; use warnings; my $infile = shift; ## 1. open input file handles open(FH, <, $infile) or die $!; ## 2. slurped lines into list my @lines = <FH>; ## 3. sort slurped lines as strings print sort { $a cmp $b } @lines;www.youtube.com/vkedco www.vkedco.blogspot.com
  7. 7. Sort as Numbers with Filehandles my $infile = shift(); my $outfile = shift(); ## open input and output file handles open(IN, <, $infile) or die $!; open(OUT, >, $outfile) or die $!; ## sort input as numbers and print ## them into output file handle print OUT sort { $a <=> $b } <IN>;www.youtube.com/vkedco www.vkedco.blogspot.com
  8. 8. Sort as Strings with Filehandles my $infile = shift(); my $outfile = shift(); ## open input and output file handles open(IN, <, $infile) or die $!; open(OUT, >, $outfile) or die $!; ## sort input as numbers and print ## them into output file handle print OUT sort { $a cmp $b } <IN>;www.youtube.com/vkedco www.vkedco.blogspot.com
  9. 9. Pipeswww.youtube.com/vkedco www.vkedco.blogspot.com
  10. 10. Pipes ● A Pipe is a communication line b/w two running programs (processes) ● That communication line can be one- directional or bi-directional ● If you have two programs P1 and P2, then P1 | P2 means take the input of P1 and give it to P2 ● Pipes can be arbitrarily long: P1 | P2 | … | Pnwww.youtube.com/vkedco www.vkedco.blogspot.com
  11. 11. Exercise Write a program that sorts a file of integers and outputs every odd integer.www.youtube.com/vkedco www.vkedco.blogspot.com
  12. 12. Sample File: UNSORTED_NUMBERS.TXT 4 5 10 0 9 25 90 2 3 4www.youtube.com/vkedco www.vkedco.blogspot.com
  13. 13. Perl Solution: NMBR_FILE_SORT.PL ## this script sorts input as numbers use strict; use warnings; my $infile = shift(); open(IN, <, $infile) or die $!; my @lines = <IN>; print sort { $a <=> $b } @lines; source is herewww.youtube.com/vkedco www.vkedco.blogspot.com
  14. 14. Perl Solution ## 1. get infile path argument from command line my $infile = shift(); ## 2. open a file handle and pipe in ## the output of nmbr_sort.pl after nmbr_sort.pl ## is applied to $infile. open(FH, -|, ./nmbr_sort.pl . $infile); ## 3. pick out and print odd numbers while(<FH>) { if ( $_ % 2 != 0 ) { print $_; } } source is herewww.youtube.com/vkedco www.vkedco.blogspot.com
  15. 15. Unix/Linux Pipe Solution: PICK_ODDS.PL ## this script picks out and prints odd numbers use strict; use warnings; while (<>) { print $_ if ( $_ % 2 != 0 ); } source is herewww.youtube.com/vkedco www.vkedco.blogspot.com
  16. 16. Unix/Linux Pipe Solution: Pipe Chain >>> nmbr_sort.pl unsorted_input.txt | pick_odds.pl or >>> ./nmbr_sort.pl unsorted_input.txt | ./pick_odds.plwww.youtube.com/vkedco www.vkedco.blogspot.com
  17. 17. Exercise Write a program that sorts a file, picks every odd number, and outputs only those numbers that are less than 10.www.youtube.com/vkedco www.vkedco.blogspot.com
  18. 18. Perl Solution: PICK_LT_10.PL ## this script picks out numbers less than 10 use strict; use warnings; while (<>) { print $_ if ( $_ < 10 ); } source is herewww.youtube.com/vkedco www.vkedco.blogspot.com
  19. 19. Perl Solution ## 1. get infile path argument from command line my $infile = shift(); ## 2. open file handle and pipe in the output of nmbr_sort.pl after nmbr_sort.pl ## is applied to $infile. open(IN_FH, -|, ./nmbr_file_sort.pl . $infile); ## 3. open output pipe to pick_lt_10.pl open(OUT_FH, |-, ./pick_lt_10.pl) ## 3. pick out odds and pipe them out into OUT_FH while(<IN_FH>) { if ( $_ % 2 != 0 ) { print OUT_FH $_; } } ## close both file handles close(IN_FH); close(OUT_FH); source is herewww.youtube.com/vkedco www.vkedco.blogspot.com
  20. 20. Unix Pipe Solution: Pipe Chain >>> nmbr_file_sort.pl unsorted_numbers.txt | pick_odds.pl | pick_lt_10.pl or >>> ./nmbr_file_sort unsorted_numbers.txt | ./pick_odds.pl | ./pick_lt_10.plwww.youtube.com/vkedco www.vkedco.blogspot.com
  21. 21. Reading & References ● Unix Pipeline ● perldoc.perl.org ● www.perlmonks.comwww.youtube.com/vkedco www.vkedco.blogspot.com

×