Semantic Pipes (London Perl Workshop 2009)

P ip es
m an tic
Se

London
Perl Wor
k sh o p 5 D
e c 2009

osfam
eron

Unix philosophy

“This is the Unix philosophy: Write programs that do
one thing and do it well. Write programs to work
together. Write programs to handle text streams,
because that is a universal interface.”
– Doug McIlroy
http://www.faqs.org/docs/artu/ch01s06.html

Pipes
generate

Foo foo foo
Bar bar bar
Baz baz baz

map

foo
bar
baz

Pipes
generate

Foo foo foo
Bar bar bar
Baz baz baz

grep

Bar bar bar

Pipes
generate

Foo foo foo
Bar bar bar
Baz baz baz

sort

Bar bar bar
Baz baz baz
Foo foo foo

ls
SYNOPSIS
ls [OPTION]... [FILE]...

OPTIONS

--format=WORD
across -x, commas -m, horizontal -x, long -l, ...

-h, --human-readable
human readable sizes (e.g., 1K 234M 2G)

--hide=PATTERN
do not list implied entries matching shell PATTERN

-r, --reverse
reverse order while sorting

--sort=WORD
none -U, extension -X, size -S, time -t, version -v

ls
SYNOPSIS

OPTIONS

--format=WORD

grep
--hide=PATTERN

-r, --reverse

--sort=WORD

ls
SYNOPSIS

OPTIONS

--format=WORD


--hide=PATTERN

-r, --reverse
sort
--sort=WORD

ls
SYNOPSIS

OPTIONS
map/format
--format=WORD


--hide=PATTERN

-r, --reverse

--sort=WORD

ls | … ?

ls | grep

ls | wc -l

ls -ltr | … ?

total 3272
-rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe
-rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl
-rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README
drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib
-rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list
-rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp

ls -ltr | … ?

using cut
total 3272

ls -ltr | … ?

cut … --bytes? or --delimiter?
total 3272

ls -ltr | … ?

total 3272
-rwxr-xr-x 1 hakim hakim˽˽˽˽˽˽˽˽˽˽˽52 2009-12-02 20:52 pipe
-rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽798 2009-12-03 00:49 pipe.pl
-rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽905 2009-12-03 01:13 README
drwxr-xr-x 3 hakim hakim˽˽˽˽˽˽˽4096 2009-12-03 01:39 lib
-rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽238 2009-12-04 23:05 images_list
-rw-r--r-- 1 hakim hakim˽˽3323129 2009-12-04 23:17 semantic.odp

OK, find...
● rather baroque
● separate command
● “consistent interface” ?

So who does do piping properly?

Windows Powershell
● originally called Monad Shell
● (this is important)
● streams of .Net objects
● flexible and consistent filter/map/sort

Disclaimer
● I've never used Powershell...

Disclaimer
● I've never used Powershell...
● but the idea is genuinely exciting

Pipe.pm

package Pipe;
use Moose;
extends 'MooseX::App::Cmd';

1;

pipe

#!/usr/bin/perl
use lib 'lib';
use Pipe;
Pipe->run;

Pipe::Command::find
use Moose;
extends 'Pipe::Command';
use MooseX::Types::Path::Class qw(File to_File);
use File::Next;

sub go {
my ($self, $session, $args) = @_;

my @files = @$args;
push @files, '.' unless @files;

my $it = File::Next::files( @files );

$session->iterator(
sub {
if (my $file = $it->()) {
return to_File($file);
} else {
return;
}
});
}

Pipe::Command::grep
has where => (
isa => 'Str',
is => 'rw',
documentation => 'string to match',
);
has where_sub => ( … );

sub filter {
my ($self, $session, $iterator) = @_;

my $where_sub = $self->where_sub;

my @queue;
return sub { # yucky code, go read HOP instead
return pop @queue if @queue;
{
my @values = $iterator->() or return;
@queue = grep {
my $result = eval { $where_sub->($_) };
$result;
} @values or redo;
}
return pop @queue;
};
}

Pipe in action

$ ./pipe find lib

List of files found!

lib/Pipe/Command.pm
lib/Pipe/Command/find.pm
lib/Pipe/Command/grep.pm
lib/Pipe/Session.pm
lib/Pipe.pm

Pipe in action

$ ./pipe find lib |
./pipe grep '$_->stat->size > 1000'

List of files found!

lib/Pipe/Command.pm

Pipe::Command::find

sub pretty_print {
my ($self, $session) = @_;

say "List of files found!nn";
my $it = $session->iterator;
while (my @files = $it->()) {
say join "n", @files;
}
}

Pipe to an external tool
$ ./pipe find lib | cat
---
command_stack: !!perl/hash:Pipe::Command::find
app: !!perl/hash:Pipe
arg0: pipe
.
. <snip>
.
--- !!perl/hash:Path::Class::File
dir: !!perl/hash:Path::Class::Dir
dirs:
- lib
file_spec_class: ~
volume: ''
file: Pipe.pm
file_spec_class: ~

Pipe to an external tool

$ ./pipe find lib –-pretty | cat

lib/Pipe/Command.pm
lib/Pipe/Command/find.pm
lib/Pipe/Command/grep.pm
lib/Pipe/Session.pm
lib/Pipe.pm

generate
Pipes
app: !!Pipe
stack: generate
Foo foo foo
blah blah blah Bar bar bar
Baz baz baz

generate
Pipes
app: !!Pipe
stack: generate
Foo foo foo
blah blah blah Bar bar bar
I'm using YAML, Baz baz baz
but could just as well be
XML etc.

generate
Pipes
app: !!Pipe
stack: generate
blah blah blah

map

app: !!Pipe foo
stack: map
generate bar
blah blah blah baz

generate
Pipes
app: !!Pipe
stack: generate
blah blah blah

map

app: !!Pipe
stack: map
generate
blah blah blah

sort

app: !!Pipe bar
stack: sort
map baz
generate foo
blah blah blah

generate
Pipes
app: !!Pipe
stack: generate
blah blah blah

map

app: !!Pipe
stack: map
generate
blah blah blah
●nothing left to pipe to
(! -p)
sort ●or --pretty

app: !!Pipe bar
stack: sort
map baz
generate foo
blah blah blah

generate
Pipes
app: !!Pipe
stack: generate
blah blah blah
pretty_print
map

app: !!Pipe
stack: map
generate
blah blah blah

sort

app: !!Pipe
stack: sort
map
generate
blah blah blah

Database
● from mydb.foo |
grep '$_->name =~ /Bob/' |
select foo bar baz

Database
● from mydb.foo | # 1 billion rows?
grep '$_->name =~ /Bob/' |
select foo bar baz

“You see, wire
telegraph is a kind of
a very, very long cat.
You pull his tail in
New York and his
head is meowing in
Los Angeles. Do you
understand this?
And radio operates
exactly the same
way: you send
signals here, they
receive them there.
The only difference
is that there is no
cat.”

generate
Pipes
app: !!Pipe
stack: generate
blah blah blah

map

app: !!Pipe
stack: map
generate
blah blah blah

sort

app: !!Pipe
stack: sort
map
generate
blah blah blah

generate
Pipes Monads
app: !!Pipe
stack: generate

map

app: !!Pipe
stack: map
generate

sort

app: !!Pipe generate
stack: sort >> map
map >> sort
generate >> pretty_print
(first [sort, map, generate])

Thank you
●
Details/links
●
Early version of this talk given at NorthWestEngland.pm, 1st July 2009
●
email: osfameron@cpan.org
●
http://github.com/osfameron/pipe/
●
http://github.com/rhaen/Bicycle-Workshop (nice MooseX::App::Cmd example)
●
Images
●
CC-licensed, via Flickr (thanks for making your photos free to use!!)
– http://www.flickr.com/photos/rxmflickr/4102530508/ Pipe Dream (title picture), by Rishi Menon
– http://www.flickr.com/photos/technodad/3827297755/ The Unix® License Plate takes a beach break, by technodad
– http://www.flickr.com/photos/darwinbell/2422933100/ Pipe Dreams, by Darwin Bell
– http://www.flickr.com/photos/heritagefutures/2106174676/ Urban Moose, by ausphoto!
– http://www.flickr.com/photos/james_michael_hill/88311128/ Pipes, by james_michael_hill
– http://www.flickr.com/photos/intherough/3244476512/ The Chain, by ...-Wink-...
– http://www.flickr.com/photos/carlos/3224149/ Pipe, by Nuevo Anden
●
Classic images, via Google
– Einstein with pipe
– I'm a PC, from the Mac ads
– Ceci n'est pas une pipe
– Einstein

Semantic Pipes (London Perl Workshop 2009)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (12)

Similar to Semantic Pipes (London Perl Workshop 2009)

Similar to Semantic Pipes (London Perl Workshop 2009) (20)

More from osfameron

More from osfameron (15)

Recently uploaded

Recently uploaded (20)

Semantic Pipes (London Perl Workshop 2009)