Semantic Pipes (London Perl Workshop 2009)

P ip es m an tic Se London Perl Wor k sh o p 5 D e c 2009 osfam eron

Everyone likes pipes

Unix philosophy “This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” – Doug McIlroy http://www.faqs.org/docs/artu/ch01s06.html

Pipes generate Foo foo foo Bar bar bar Baz baz baz map foo bar baz

Pipes generate Foo foo foo Bar bar bar Baz baz baz grep Bar bar bar

Pipes generate Foo foo foo Bar bar bar Baz baz baz sort Bar bar bar Baz baz baz Foo foo foo

ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting --sort=WORD none -U, extension -X, size -S, time -t, version -v

ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) grep --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting --sort=WORD none -U, extension -X, size -S, time -t, version -v

ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting sort --sort=WORD none -U, extension -X, size -S, time -t, version -v

ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS map/format --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting --sort=WORD none -U, extension -X, size -S, time -t, version -v

ls | … ? ls | grep ls | wc -l

ls -ltr | … ? total 3272 -rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp

ls -ltr | … ? using cut total 3272 -rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp

ls -ltr | … ? cut … --bytes? or --delimiter? total 3272 -rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp

ls -ltr | … ? total 3272 -rwxr-xr-x 1 hakim hakim˽˽˽˽˽˽˽˽˽˽˽52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim˽˽˽˽˽˽˽4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim˽˽3323129 2009-12-04 23:17 semantic.odp

OK, find... ● rather baroque ● separate command ● “consistent interface” ?

So who does do piping properly?

Windows Powershell ● originally called Monad Shell ● (this is important) ● streams of .Net objects ● flexible and consistent filter/map/sort

Disclaimer ● I've never used Powershell...

Disclaimer ● I've never used Powershell... ● but the idea is genuinely exciting

Pipe.pm package Pipe; use Moose; extends 'MooseX::App::Cmd'; 1;

pipe #!/usr/bin/perl use lib 'lib'; use Pipe; Pipe->run;

Pipe.pm

Pipe::Command::find use Moose; extends 'Pipe::Command'; use MooseX::Types::Path::Class qw(File to_File); use File::Next; sub go { my ($self, $session, $args) = @_; my @files = @$args; push @files, '.' unless @files; my $it = File::Next::files( @files ); $session->iterator( sub { if (my $file = $it->()) { return to_File($file); } else { return; } }); }

Pipe::Command::grep has where => ( isa => 'Str', is => 'rw', documentation => 'string to match', ); has where_sub => ( … ); sub filter { my ($self, $session, $iterator) = @_; my $where_sub = $self->where_sub; my @queue; return sub { # yucky code, go read HOP instead return pop @queue if @queue; { my @values = $iterator->() or return; @queue = grep { my $result = eval { $where_sub->($_) }; $result; } @values or redo; } return pop @queue; }; }

Pipe in action $ ./pipe find lib List of files found! lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm

Pipe in action $ ./pipe find lib | ./pipe grep '$_->stat->size > 1000' List of files found! lib/Pipe/Command.pm

Pipe::Command::find sub pretty_print { my ($self, $session) = @_; say "List of files found!nn"; my $it = $session->iterator; while (my @files = $it->()) { say join "n", @files; } }

Pipe to an external tool $ ./pipe find lib | cat --- command_stack: !!perl/hash:Pipe::Command::find app: !!perl/hash:Pipe arg0: pipe . . <snip> . --- !!perl/hash:Path::Class::File dir: !!perl/hash:Path::Class::Dir dirs: - lib file_spec_class: ~ volume: '' file: Pipe.pm file_spec_class: ~

Pipe to an external tool $ ./pipe find lib –-pretty | cat lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm

generate Pipes app: !!Pipe stack: generate Foo foo foo blah blah blah Bar bar bar Baz baz baz

generate Pipes app: !!Pipe stack: generate Foo foo foo blah blah blah Bar bar bar I'm using YAML, Baz baz baz but could just as well be XML etc.

generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe foo stack: map generate bar blah blah blah baz

generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe bar stack: sort map baz generate foo blah blah blah

generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah ●nothing left to pipe to (! -p) sort ●or --pretty app: !!Pipe bar stack: sort map baz generate foo blah blah blah

generate Pipes app: !!Pipe stack: generate blah blah blah pretty_print map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe stack: sort map generate blah blah blah

Database ● from mydb.foo | grep '$_->name =~ /Bob/' | select foo bar baz

Database ● from mydb.foo | # 1 billion rows? grep '$_->name =~ /Bob/' | select foo bar baz

“You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”

generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe stack: sort map generate blah blah blah

generate Pipes Monads app: !!Pipe stack: generate map app: !!Pipe stack: map generate sort app: !!Pipe generate stack: sort >> map map >> sort generate >> pretty_print (first [sort, map, generate])

Thank you ● Details/links ● Early version of this talk given at NorthWestEngland.pm, 1st July 2009 ● email: osfameron@cpan.org ● http://github.com/osfameron/pipe/ ● http://github.com/rhaen/Bicycle-Workshop (nice MooseX::App::Cmd example) ● Images ● CC-licensed, via Flickr (thanks for making your photos free to use!!) – http://www.flickr.com/photos/rxmflickr/4102530508/ Pipe Dream (title picture), by Rishi Menon – http://www.flickr.com/photos/technodad/3827297755/ The Unix® License Plate takes a beach break, by technodad – http://www.flickr.com/photos/darwinbell/2422933100/ Pipe Dreams, by Darwin Bell – http://www.flickr.com/photos/heritagefutures/2106174676/ Urban Moose, by ausphoto! – http://www.flickr.com/photos/james_michael_hill/88311128/ Pipes, by james_michael_hill – http://www.flickr.com/photos/intherough/3244476512/ The Chain, by ...-Wink-... – http://www.flickr.com/photos/carlos/3224149/ Pipe, by Nuevo Anden ● Classic images, via Google – Einstein with pipe – I'm a PC, from the Mac ads – Ceci n'est pas une pipe – Einstein

Semantic Pipes (London Perl Workshop 2009)

by osfameron

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 5

Statistics

Semantic Pipes (London Perl Workshop 2009) Presentation Transcript