The document discusses pipes and piping in Unix and other systems. It proposes a Perl module called Pipe.pm that implements piping using Moose and MooseX::App::Cmd. Commands like find, grep, sort are presented as classes that can be composed into a pipeline. The pipeline runs commands in sequence, passing output from one to the next. This allows piping commands together programmatically like in shell pipelines.
4. Unix philosophy
“This is the Unix philosophy: Write programs that do
one thing and do it well. Write programs to work
together. Write programs to handle text streams,
because that is a universal interface.”
– Doug McIlroy
http://www.faqs.org/docs/artu/ch01s06.html
9. ls
SYNOPSIS
ls [OPTION]... [FILE]...
OPTIONS
--format=WORD
across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable
human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN
do not list implied entries matching shell PATTERN
-r, --reverse
reverse order while sorting
--sort=WORD
none -U, extension -X, size -S, time -t, version -v
10. ls
SYNOPSIS
ls [OPTION]... [FILE]...
OPTIONS
--format=WORD
across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable
human readable sizes (e.g., 1K 234M 2G)
grep
--hide=PATTERN
do not list implied entries matching shell PATTERN
-r, --reverse
reverse order while sorting
--sort=WORD
none -U, extension -X, size -S, time -t, version -v
11. ls
SYNOPSIS
ls [OPTION]... [FILE]...
OPTIONS
--format=WORD
across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable
human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN
do not list implied entries matching shell PATTERN
-r, --reverse
reverse order while sorting
sort
--sort=WORD
none -U, extension -X, size -S, time -t, version -v
12. ls
SYNOPSIS
ls [OPTION]... [FILE]...
OPTIONS
map/format
--format=WORD
across -x, commas -m, horizontal -x, long -l, ...
-h, --human-readable
human readable sizes (e.g., 1K 234M 2G)
--hide=PATTERN
do not list implied entries matching shell PATTERN
-r, --reverse
reverse order while sorting
--sort=WORD
none -U, extension -X, size -S, time -t, version -v
14. ls -ltr | … ?
total 3272
-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe
-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl
-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README
drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib
-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list
-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
15. ls -ltr | … ?
using cut
total 3272
-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe
-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl
-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README
drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib
-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list
-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
16. ls -ltr | … ?
cut … --bytes? or --delimiter?
total 3272
-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe
-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl
-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README
drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib
-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list
-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
17. ls -ltr | … ?
total 3272
-rwxr-xr-x 1 hakim hakim˽˽˽˽˽˽˽˽˽˽˽52 2009-12-02 20:52 pipe
-rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽798 2009-12-03 00:49 pipe.pl
-rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽905 2009-12-03 01:13 README
drwxr-xr-x 3 hakim hakim˽˽˽˽˽˽˽4096 2009-12-03 01:39 lib
-rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽238 2009-12-04 23:05 images_list
-rw-r--r-- 1 hakim hakim˽˽3323129 2009-12-04 23:17 semantic.odp
18. ls -ltr | … ?
total 3272
-rwxr-xr-x 1 hakim hakim˽˽˽˽˽˽˽˽˽˽˽52 2009-12-02 20:52 pipe
-rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽798 2009-12-03 00:49 pipe.pl
-rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽905 2009-12-03 01:13 README
drwxr-xr-x 3 hakim hakim˽˽˽˽˽˽˽4096 2009-12-03 01:39 lib
-rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽238 2009-12-04 23:05 images_list
-rw-r--r-- 1 hakim hakim˽˽3323129 2009-12-04 23:17 semantic.odp
19. OK, find...
● rather baroque
● separate command
● “consistent interface” ?
29. Pipe::Command::find
use Moose;
extends 'Pipe::Command';
use MooseX::Types::Path::Class qw(File to_File);
use File::Next;
sub go {
my ($self, $session, $args) = @_;
my @files = @$args;
push @files, '.' unless @files;
my $it = File::Next::files( @files );
$session->iterator(
sub {
if (my $file = $it->()) {
return to_File($file);
} else {
return;
}
});
}
30. Pipe::Command::grep
has where => (
isa => 'Str',
is => 'rw',
documentation => 'string to match',
);
has where_sub => ( … );
sub filter {
my ($self, $session, $iterator) = @_;
my $where_sub = $self->where_sub;
my @queue;
return sub { # yucky code, go read HOP instead
return pop @queue if @queue;
{
my @values = $iterator->() or return;
@queue = grep {
my $result = eval { $where_sub->($_) };
$result;
} @values or redo;
}
return pop @queue;
};
}
31.
32. Pipe in action
$ ./pipe find lib
List of files found!
lib/Pipe/Command.pm
lib/Pipe/Command/find.pm
lib/Pipe/Command/grep.pm
lib/Pipe/Session.pm
lib/Pipe.pm
33. Pipe in action
$ ./pipe find lib |
./pipe grep '$_->stat->size > 1000'
List of files found!
lib/Pipe/Command.pm
34. Pipe::Command::find
sub pretty_print {
my ($self, $session) = @_;
say "List of files found!nn";
my $it = $session->iterator;
while (my @files = $it->()) {
say join "n", @files;
}
}
45. Database
● from mydb.foo |
grep '$_->name =~ /Bob/' |
select foo bar baz
46. Database
● from mydb.foo | # 1 billion rows?
grep '$_->name =~ /Bob/' |
select foo bar baz
47. “You see, wire
telegraph is a kind of
a very, very long cat.
You pull his tail in
New York and his
head is meowing in
Los Angeles. Do you
understand this?
And radio operates
exactly the same
way: you send
signals here, they
receive them there.
The only difference
is that there is no
cat.”
48. “You see, wire
telegraph is a kind of
a very, very long cat.
You pull his tail in
New York and his
head is meowing in
Los Angeles. Do you
understand this?
And radio operates
exactly the same
way: you send
signals here, they
receive them there.
The only difference
is that there is no
cat.”
54. Thank you
●
Details/links
●
Early version of this talk given at NorthWestEngland.pm, 1st July 2009
●
email: osfameron@cpan.org
●
http://github.com/osfameron/pipe/
●
http://github.com/rhaen/Bicycle-Workshop (nice MooseX::App::Cmd example)
●
Images
●
CC-licensed, via Flickr (thanks for making your photos free to use!!)
– http://www.flickr.com/photos/rxmflickr/4102530508/ Pipe Dream (title picture), by Rishi Menon
– http://www.flickr.com/photos/technodad/3827297755/ The Unix® License Plate takes a beach break, by technodad
– http://www.flickr.com/photos/darwinbell/2422933100/ Pipe Dreams, by Darwin Bell
– http://www.flickr.com/photos/heritagefutures/2106174676/ Urban Moose, by ausphoto!
– http://www.flickr.com/photos/james_michael_hill/88311128/ Pipes, by james_michael_hill
– http://www.flickr.com/photos/intherough/3244476512/ The Chain, by ...-Wink-...
– http://www.flickr.com/photos/carlos/3224149/ Pipe, by Nuevo Anden
●
Classic images, via Google
– Einstein with pipe
– I'm a PC, from the Mac ads
– Ceci n'est pas une pipe
– Einstein