Semantic Pipes (London Perl Workshop 2009)
Upcoming SlideShare
Loading in...5
×
 

Semantic Pipes (London Perl Workshop 2009)

on

  • 2,475 views

Unix has always had a philosophy of composable tools, where one tool outputs to the next in a pipeline. ...

Unix has always had a philosophy of composable tools, where one tool outputs to the next in a pipeline.

But the technique of piping a *textual* stream of data, and having to extract data out of it is looking a bit long in the tooth. Microsoft (not historically an innovator in its shell environment :-) has stolen a march with its Powershell.

Can we do better in Perl? With composable streams of objects? Written in a modern OO framework (Moose)?

You bet we can!

This talk was given at the London Perl Workshop 2009, http://conferences.yapceurope.org/lpw2009/talk/2456

Statistics

Views

Total Views
2,475
Views on SlideShare
2,470
Embed Views
5

Actions

Likes
3
Downloads
22
Comments
0

1 Embed 5

http://www.slideshare.net 5

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Semantic Pipes (London Perl Workshop 2009) Semantic Pipes (London Perl Workshop 2009) Presentation Transcript

  • P ip es m an tic Se London Perl Wor k sh o p 5 D e c 2009 osfam eron
  • Everyone likes pipes
  • Unix philosophy “This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” – Doug McIlroy http://www.faqs.org/docs/artu/ch01s06.html
  • Pipes generate Foo foo foo Bar bar bar Baz baz baz map foo bar baz
  • Pipes generate Foo foo foo Bar bar bar Baz baz baz grep Bar bar bar
  • Pipes generate Foo foo foo Bar bar bar Baz baz baz sort Bar bar bar Baz baz baz Foo foo foo
  • ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting --sort=WORD none -U, extension -X, size -S, time -t, version -v
  • ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) grep --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting --sort=WORD none -U, extension -X, size -S, time -t, version -v
  • ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting sort --sort=WORD none -U, extension -X, size -S, time -t, version -v
  • ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS map/format --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting --sort=WORD none -U, extension -X, size -S, time -t, version -v
  • ls | … ? ls | grep ls | wc -l
  • ls -ltr | … ? total 3272 -rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
  • ls -ltr | … ? using cut total 3272 -rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
  • ls -ltr | … ? cut … --bytes? or --delimiter? total 3272 -rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
  • ls -ltr | … ? total 3272 -rwxr-xr-x 1 hakim hakim˽˽˽˽˽˽˽˽˽˽˽52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim˽˽˽˽˽˽˽4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim˽˽3323129 2009-12-04 23:17 semantic.odp
  • ls -ltr | … ? total 3272 -rwxr-xr-x 1 hakim hakim˽˽˽˽˽˽˽˽˽˽˽52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim˽˽˽˽˽˽˽4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim˽˽3323129 2009-12-04 23:17 semantic.odp
  • OK, find... ● rather baroque ● separate command ● “consistent interface” ?
  • So who does do piping properly?
  • So who does do piping properly?
  • Windows Powershell ● originally called Monad Shell ● (this is important) ● streams of .Net objects ● flexible and consistent filter/map/sort
  • Disclaimer ● I've never used Powershell...
  • Disclaimer ● I've never used Powershell... ● but the idea is genuinely exciting
  • Pipe.pm package Pipe; use Moose; extends 'MooseX::App::Cmd'; 1;
  • pipe #!/usr/bin/perl use lib 'lib'; use Pipe; Pipe->run;
  • Pipe.pm
  • Pipe::Command::find use Moose; extends 'Pipe::Command'; use MooseX::Types::Path::Class qw(File to_File); use File::Next; sub go { my ($self, $session, $args) = @_; my @files = @$args; push @files, '.' unless @files; my $it = File::Next::files( @files ); $session->iterator( sub { if (my $file = $it->()) { return to_File($file); } else { return; } }); }
  • Pipe::Command::grep has where => ( isa => 'Str', is => 'rw', documentation => 'string to match', ); has where_sub => ( … ); sub filter { my ($self, $session, $iterator) = @_; my $where_sub = $self->where_sub; my @queue; return sub { # yucky code, go read HOP instead return pop @queue if @queue; { my @values = $iterator->() or return; @queue = grep { my $result = eval { $where_sub->($_) }; $result; } @values or redo; } return pop @queue; }; }
  • Pipe in action $ ./pipe find lib List of files found! lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm
  • Pipe in action $ ./pipe find lib | ./pipe grep '$_->stat->size > 1000' List of files found! lib/Pipe/Command.pm
  • Pipe::Command::find sub pretty_print { my ($self, $session) = @_; say "List of files found!nn"; my $it = $session->iterator; while (my @files = $it->()) { say join "n", @files; } }
  • Pipe to an external tool $ ./pipe find lib | cat --- command_stack: !!perl/hash:Pipe::Command::find app: !!perl/hash:Pipe arg0: pipe . . <snip> . --- !!perl/hash:Path::Class::File dir: !!perl/hash:Path::Class::Dir dirs: - lib file_spec_class: ~ volume: '' file: Pipe.pm file_spec_class: ~
  • Pipe to an external tool $ ./pipe find lib –-pretty | cat lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm
  • generate Pipes app: !!Pipe stack: generate Foo foo foo blah blah blah Bar bar bar Baz baz baz
  • generate Pipes app: !!Pipe stack: generate Foo foo foo blah blah blah Bar bar bar I'm using YAML, Baz baz baz but could just as well be XML etc.
  • generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe foo stack: map generate bar blah blah blah baz
  • generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe bar stack: sort map baz generate foo blah blah blah
  • generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah ●nothing left to pipe to (! -p) sort ●or --pretty app: !!Pipe bar stack: sort map baz generate foo blah blah blah
  • generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah ●nothing left to pipe to (! -p) sort ●or --pretty app: !!Pipe bar stack: sort map baz generate foo blah blah blah
  • generate Pipes app: !!Pipe stack: generate blah blah blah pretty_print map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe stack: sort map generate blah blah blah
  • Database ● from mydb.foo | grep '$_->name =~ /Bob/' | select foo bar baz
  • Database ● from mydb.foo | # 1 billion rows? grep '$_->name =~ /Bob/' | select foo bar baz
  • “You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”
  • “You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”
  • generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe stack: sort map generate blah blah blah
  • generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe stack: sort map generate blah blah blah
  • generate Pipes Monads app: !!Pipe stack: generate map app: !!Pipe stack: map generate sort app: !!Pipe generate stack: sort >> map map >> sort generate >> pretty_print (first [sort, map, generate])
  • Thank you ● Details/links ● Early version of this talk given at NorthWestEngland.pm, 1st July 2009 ● email: osfameron@cpan.org ● http://github.com/osfameron/pipe/ ● http://github.com/rhaen/Bicycle-Workshop (nice MooseX::App::Cmd example) ● Images ● CC-licensed, via Flickr (thanks for making your photos free to use!!) – http://www.flickr.com/photos/rxmflickr/4102530508/ Pipe Dream (title picture), by Rishi Menon – http://www.flickr.com/photos/technodad/3827297755/ The Unix® License Plate takes a beach break, by technodad – http://www.flickr.com/photos/darwinbell/2422933100/ Pipe Dreams, by Darwin Bell – http://www.flickr.com/photos/heritagefutures/2106174676/ Urban Moose, by ausphoto! – http://www.flickr.com/photos/james_michael_hill/88311128/ Pipes, by james_michael_hill – http://www.flickr.com/photos/intherough/3244476512/ The Chain, by ...-Wink-... – http://www.flickr.com/photos/carlos/3224149/ Pipe, by Nuevo Anden ● Classic images, via Google – Einstein with pipe – I'm a PC, from the Mac ads – Ceci n'est pas une pipe – Einstein