Semantic Pipes (London Perl Workshop 2009)

  • 1,301 views
Uploaded on

Unix has always had a philosophy of composable tools, where one tool outputs to the next in a pipeline. …

Unix has always had a philosophy of composable tools, where one tool outputs to the next in a pipeline.

But the technique of piping a *textual* stream of data, and having to extract data out of it is looking a bit long in the tooth. Microsoft (not historically an innovator in its shell environment :-) has stolen a march with its Powershell.

Can we do better in Perl? With composable streams of objects? Written in a modern OO framework (Moose)?

You bet we can!

This talk was given at the London Perl Workshop 2009, http://conferences.yapceurope.org/lpw2009/talk/2456

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,301
On Slideshare
0
From Embeds
0
Number of Embeds
1

Actions

Shares
Downloads
23
Comments
0
Likes
3

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. P ip es m an tic Se London Perl Wor k sh o p 5 D e c 2009 osfam eron
  • 2. Everyone likes pipes
  • 3. Unix philosophy “This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” – Doug McIlroy http://www.faqs.org/docs/artu/ch01s06.html
  • 4. Pipes generate Foo foo foo Bar bar bar Baz baz baz map foo bar baz
  • 5. Pipes generate Foo foo foo Bar bar bar Baz baz baz grep Bar bar bar
  • 6. Pipes generate Foo foo foo Bar bar bar Baz baz baz sort Bar bar bar Baz baz baz Foo foo foo
  • 7. ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting --sort=WORD none -U, extension -X, size -S, time -t, version -v
  • 8. ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) grep --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting --sort=WORD none -U, extension -X, size -S, time -t, version -v
  • 9. ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting sort --sort=WORD none -U, extension -X, size -S, time -t, version -v
  • 10. ls SYNOPSIS ls [OPTION]... [FILE]... OPTIONS map/format --format=WORD across -x, commas -m, horizontal -x, long -l, ... -h, --human-readable human readable sizes (e.g., 1K 234M 2G) --hide=PATTERN do not list implied entries matching shell PATTERN -r, --reverse reverse order while sorting --sort=WORD none -U, extension -X, size -S, time -t, version -v
  • 11. ls | … ? ls | grep ls | wc -l
  • 12. ls -ltr | … ? total 3272 -rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
  • 13. ls -ltr | … ? using cut total 3272 -rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
  • 14. ls -ltr | … ? cut … --bytes? or --delimiter? total 3272 -rwxr-xr-x 1 hakim hakim 52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim 798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim 905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim 4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim 238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim 3323129 2009-12-04 23:17 semantic.odp
  • 15. ls -ltr | … ? total 3272 -rwxr-xr-x 1 hakim hakim˽˽˽˽˽˽˽˽˽˽˽52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim˽˽˽˽˽˽˽4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim˽˽3323129 2009-12-04 23:17 semantic.odp
  • 16. ls -ltr | … ? total 3272 -rwxr-xr-x 1 hakim hakim˽˽˽˽˽˽˽˽˽˽˽52 2009-12-02 20:52 pipe -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽798 2009-12-03 00:49 pipe.pl -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽905 2009-12-03 01:13 README drwxr-xr-x 3 hakim hakim˽˽˽˽˽˽˽4096 2009-12-03 01:39 lib -rw-r--r-- 1 hakim hakim˽˽˽˽˽˽˽˽˽238 2009-12-04 23:05 images_list -rw-r--r-- 1 hakim hakim˽˽3323129 2009-12-04 23:17 semantic.odp
  • 17. OK, find... ● rather baroque ● separate command ● “consistent interface” ?
  • 18. So who does do piping properly?
  • 19. So who does do piping properly?
  • 20. Windows Powershell ● originally called Monad Shell ● (this is important) ● streams of .Net objects ● flexible and consistent filter/map/sort
  • 21. Disclaimer ● I've never used Powershell...
  • 22. Disclaimer ● I've never used Powershell... ● but the idea is genuinely exciting
  • 23. Pipe.pm package Pipe; use Moose; extends 'MooseX::App::Cmd'; 1;
  • 24. pipe #!/usr/bin/perl use lib 'lib'; use Pipe; Pipe->run;
  • 25. Pipe.pm
  • 26. Pipe::Command::find use Moose; extends 'Pipe::Command'; use MooseX::Types::Path::Class qw(File to_File); use File::Next; sub go { my ($self, $session, $args) = @_; my @files = @$args; push @files, '.' unless @files; my $it = File::Next::files( @files ); $session->iterator( sub { if (my $file = $it->()) { return to_File($file); } else { return; } }); }
  • 27. Pipe::Command::grep has where => ( isa => 'Str', is => 'rw', documentation => 'string to match', ); has where_sub => ( … ); sub filter { my ($self, $session, $iterator) = @_; my $where_sub = $self->where_sub; my @queue; return sub { # yucky code, go read HOP instead return pop @queue if @queue; { my @values = $iterator->() or return; @queue = grep { my $result = eval { $where_sub->($_) }; $result; } @values or redo; } return pop @queue; }; }
  • 28. Pipe in action $ ./pipe find lib List of files found! lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm
  • 29. Pipe in action $ ./pipe find lib | ./pipe grep '$_->stat->size > 1000' List of files found! lib/Pipe/Command.pm
  • 30. Pipe::Command::find sub pretty_print { my ($self, $session) = @_; say "List of files found!nn"; my $it = $session->iterator; while (my @files = $it->()) { say join "n", @files; } }
  • 31. Pipe to an external tool $ ./pipe find lib | cat --- command_stack: !!perl/hash:Pipe::Command::find app: !!perl/hash:Pipe arg0: pipe . . <snip> . --- !!perl/hash:Path::Class::File dir: !!perl/hash:Path::Class::Dir dirs: - lib file_spec_class: ~ volume: '' file: Pipe.pm file_spec_class: ~
  • 32. Pipe to an external tool $ ./pipe find lib –-pretty | cat lib/Pipe/Command.pm lib/Pipe/Command/find.pm lib/Pipe/Command/grep.pm lib/Pipe/Session.pm lib/Pipe.pm
  • 33. generate Pipes app: !!Pipe stack: generate Foo foo foo blah blah blah Bar bar bar Baz baz baz
  • 34. generate Pipes app: !!Pipe stack: generate Foo foo foo blah blah blah Bar bar bar I'm using YAML, Baz baz baz but could just as well be XML etc.
  • 35. generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe foo stack: map generate bar blah blah blah baz
  • 36. generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe bar stack: sort map baz generate foo blah blah blah
  • 37. generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah ●nothing left to pipe to (! -p) sort ●or --pretty app: !!Pipe bar stack: sort map baz generate foo blah blah blah
  • 38. generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah ●nothing left to pipe to (! -p) sort ●or --pretty app: !!Pipe bar stack: sort map baz generate foo blah blah blah
  • 39. generate Pipes app: !!Pipe stack: generate blah blah blah pretty_print map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe stack: sort map generate blah blah blah
  • 40. Database ● from mydb.foo | grep '$_->name =~ /Bob/' | select foo bar baz
  • 41. Database ● from mydb.foo | # 1 billion rows? grep '$_->name =~ /Bob/' | select foo bar baz
  • 42. “You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”
  • 43. “You see, wire telegraph is a kind of a very, very long cat. You pull his tail in New York and his head is meowing in Los Angeles. Do you understand this? And radio operates exactly the same way: you send signals here, they receive them there. The only difference is that there is no cat.”
  • 44. generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe stack: sort map generate blah blah blah
  • 45. generate Pipes app: !!Pipe stack: generate blah blah blah map app: !!Pipe stack: map generate blah blah blah sort app: !!Pipe stack: sort map generate blah blah blah
  • 46. generate Pipes Monads app: !!Pipe stack: generate map app: !!Pipe stack: map generate sort app: !!Pipe generate stack: sort >> map map >> sort generate >> pretty_print (first [sort, map, generate])
  • 47. Thank you ● Details/links ● Early version of this talk given at NorthWestEngland.pm, 1st July 2009 ● email: osfameron@cpan.org ● http://github.com/osfameron/pipe/ ● http://github.com/rhaen/Bicycle-Workshop (nice MooseX::App::Cmd example) ● Images ● CC-licensed, via Flickr (thanks for making your photos free to use!!) – http://www.flickr.com/photos/rxmflickr/4102530508/ Pipe Dream (title picture), by Rishi Menon – http://www.flickr.com/photos/technodad/3827297755/ The Unix® License Plate takes a beach break, by technodad – http://www.flickr.com/photos/darwinbell/2422933100/ Pipe Dreams, by Darwin Bell – http://www.flickr.com/photos/heritagefutures/2106174676/ Urban Moose, by ausphoto! – http://www.flickr.com/photos/james_michael_hill/88311128/ Pipes, by james_michael_hill – http://www.flickr.com/photos/intherough/3244476512/ The Chain, by ...-Wink-... – http://www.flickr.com/photos/carlos/3224149/ Pipe, by Nuevo Anden ● Classic images, via Google – Einstein with pipe – I'm a PC, from the Mac ads – Ceci n'est pas une pipe – Einstein