Rubish: A Quixotic Shell
Howard Yeh

Quix.ot.ic
• \\kwik-’sä-tik\\
• foolishly impractical especially in the pursuit
of ideals ; especially : marked by rash lofty
romantic ideas or extravagantly chivalrous
action.

Why Do You Care?
• Closes the gap between script and shell
• Object-Oriented Meta-shell (frobbable)
• Extensible
– Namespace
– Local extensions (with singleton objects)
– Abstraction as codified memory
• Ruby you know and love

Symbiosis With Ruby
• Meta access
• Data type
• Plausibly concise syntax
• Object#instance_eval(&block)
• Rubish has no metasyntax
• Rubish uses no monkey patches
– Not strictly true, but…

Introduction to Rubish
abandon bash all ye who enters…

Overview
• Executable
– Command, Pipe
– Sed, Awk
– Batch
• Job Control
– Concurrency; Exception Mechanism
• Context
– Dynamically scoped IO; Workspace

Command
Rubish Bash
foo foo
foo :abc foo -abc
foo “a b c” foo a b c
foo.q “a b c” foo “a b c”
foo “a”, “b”, “c”
foo [\"a\",\"b\",\"c\"]
foo [[\"a\",\"b\"],\"c\",\"d\"]
Handling weird filenames
rsh> touch(\"a b\",\"c d\").q
rsh> wc(ls.map).q

Pipe
• Build pipe with factory
rsh> p { cmd1; cmd2; cmd3 }
• Build pipe from array of commands
rsh> @cmds = [cmd1,cmd2,cmd3]
rsh> p(@cmds)

IO Redirection
• Methods defined on the Executable class
– Executable#{i,o,err}
– Common API to all subclasses
– So far only supports stdin, stdout, stderr
• With File
• With Ruby IO object
• With a block that receives a pipe
– Good for building further abstractions

IO to File
rsh> cat.o(\"output\")
a
b
c
^D
rsh> wc(\"output\").first.to_i
3

IO with Block
rsh> @c = cat.i {|pipe| pipe.puts 1,2,3 }
rsh> @c
1
2
3
# ask the command to write to a pipe instead
rsh> @c.o { |pipe| ... }

IO Abstractions
• Enumerating methods
– Common API to all Executable subclasses
– Executable#{each,map}
– Executable#{head(n=1),tail(n=1),first,last}
• Implemented with the Rubish IO architecture.
– OOP for the win!

Executable#each!
def each!
self.o do |pipe|
pipe.each_line do |line|
line.chomp!
yield(line)
end
end
job = self.exec!
return job
end

Processing with Ruby
# array of listed file names
ls.map
# last file as string
ls.last # == ls.tail(1).first
# extension name of last 10 files
ls.tail(10) {|f| File.extname(f) }
# ditto with pipe
p { …}.map {|line| … }

Awk & Sed

Sed “One-Liner”
• Courtesy J. Zawinsky (Unix Hater Handbook)
# find *.el files that didn't have corresponding *.elc files
# only two processes per file.
sh> find . -name ’*.el’ -print \\
| sed ’s/^/FOO=/’|\\
sed ’s/$/; if * ! -f \\ ${FOO}c ]; then \\
echo \\ $FOO ; fi/’ | sh
• Rubish Sed
rsh> find(\". -name '*.el'\").sed { p if !File.exist?(line + \"c\") }
rsh> find(\". -name '*.el'\").sed { p if !File.exist?(line + \"c\") }.map {|f| … }

Awk & Sed
• Unix Powertools are for Powerfools
• Hard to predict complexity.
• Hard to get right.
• Hard to remember.
• Hard to extend.
• Hard to generalize.
– “weird chars” problem

Rubish Sed & Awk
• Doesn’t aim for full generality
– Since we are embedded in Ruby anyway
• Captures common usage patterns
– Common-Lisp loopesque helpers
• Be explicit
– Sed does not print by default
• Fits well
– Both are subclasses of Executable

grep with context (Sed)
# grep -A1 -B1
sed -n -e '/regexp/{=;x;1!p;g;$!N;p;D;}' -e h

grep with context (Rubish)
# rubish sed
sed {
if line =~ /regexp/
p \"====\"
# special case if the first line matches
p prev if respond_to?(:prev)
p
p peek(1)
end
hold(:prev,1,line) # dynamically creates a method “prev”
}

Streamer
• Awk and Sed are subclasses of Streamer
• Streamer#{peek(n=1),skip(n=1)}
– No more sedistic register shufflings
• Streamer#{max,min,count,collect,…}
– Common usage patterns
• Streamer#{quit,stop}
• Streamer#{each,map,head,tail,…}
– Inherited methods from Executable

grep with context (Rubish)
rsh> cat.i {|p| p.puts((1..11).to_a)}.sed { ... }
====
1
2
====
9
10
11
====
10
11

grep with context (Rubish)
• Easy to generalize
# simulates ‘grep regexp -Aa -Bb’
sed {
if line =~ regexp
p \"====\"
# special case if the first line matches
p prev if respond_to?(:prev)
p
p peek(a)
end
hold(:prev,b,line)
}

Aggregator
• Streamer#{max,min,count,collect,…}
– Capture common usage patterns
– Inspired by Common Lisp’s Loop Facility
– Method signature: helper(name,value,key)
• Aggregation partitioned by key
– nil is the special global key under which everything
is aggregated.

Aggregator
# longest and shortest filename lengths
rsh> ls.awk {
max(:mx,line)
min(:mn,line)
}.end {[mx,mn]}
[25, 3]

Aggregator
rsh> ls.awk { f=a[0]; collect(:fn,f,File.extname(f))}.end { fn}
{\"\"=> [\"a\",...,\"util\"],
\".gz\"=>[\"ruby-termios-0.9.5.tar.gz\"],
\".yml\"=>[\"VERSION.yml\"],
\".sh\"=>[\"test.sh\"],
\".output\"=>[\"awk.output\"],
\".5\"=>[\"ruby-termios-0.9.5\"],
\".textile\"=>[\"README.textile\"],
\".rb\"=>[\"address.rb\", \"foo.rb\", \"my.rb\"],
\".bar\"=>[\"foo.bar\"]
nil=>[\"a\", ...,\"VERSION.yml\"]}
# fn(“.rb”) contains the array of *.rb

Addressed Patterns
.sed(3) { … } # triggered for line 3
.sed(3,9) { … } # lines 3 to 9 inclusive
.sed(3,:eof) { … } # lines 3 to end of file
.sed(/a/) { … } # triggered for matching lines
.sed(/a/,/b/) # triggered for lines between
# ditto for awk
.awk(/a/,/b/)

Rubish Concurrency

Concurrency
• Coarse Grained
– For independent tasks
• No interaction between tasks
– No deadlock
– No shared memory (by abstinence)
– But is safety by policy (read: no safety)
• Ruby Green Thread
– You could always fork… I guess?

Background Jobs
• Executable#{exec!,each!,map!}
– #exec! returns an instance of Job
– #each! invokes the iterator in a background thread
– #map!(acc) << into an (hopefully thread safe)
accumulator.
• Job#{wait,stop}
– #wait returns the result of a job after it completes
– #stop signal a job to terminate, then wait
• For Command,Pipe,Sed,Awk, and more.

Background Jobs
# returns immediately
rsh> @job = slowcat(3).exec!
rsh> jobs # == [@job]
# slowcat(3) takes 3 seconds to complete
rsh> waitall # after 3 seconds => @job.wait
rsh> @acc = [] # should use a thread-safe acc
rsh> ls.map!(@acc)
rsh> ls.map!(@acc)

Exception Handling
• Job#wait would raise on abnormal completion
– exitstatus != 0
• Exception avoids error checking cruft
– It’s the 21st century!
• A little suprising
– grep \"notfound *\"
– wc a-directory

Rubish in Context

Context
• Encapsulates IOs
– Dynamically scoped
• Encapsulates Bindings
– Lexically scoped
– Namespace management
– Local extensibility
• A context defines the meaning of a closure
with free bindings.

Context-Sensitive Block
• Object#instance_eval(&block)
obj.instance_eval {
binding1
binding2
}
• Local extensibility
obj.extend(module).instance_eval { … }

Context IO
with {
cmd1.exec
cmd2.exec
with { cmd3 }.o(\"output3\").exec
}.o(\"output1-2\").exec

Context Extension
# ad hoc, local extensions
with(derive({def foo; ...; end})) {
... # outer foo
with(derive({def foo; ...; end})) {
... #inner foo
}
... # outer foo
}

Context Extension with Modules
# prebaked (modules from ‘load’)
with(derive(mod1,mod2)) {
…
}

Batch
• A batch job is a contextualized block executed
in a thread.
• Schematically :
@job = Thread.new { context.eval { … }}
@job.wait
• Similar to subshell, but in the same process
• A Batch is also an Executable!

Batch
# first extend context
# then carry out a block within a batch thread
batch(derive(...)) { ... }
# all Executable methods apply
@job = batch(derive(...)) { ... }.exec!
batch(derive(...)) { ... }.map
batch(derive(...)) { ... }.awk
…

Structured Concurrency
# Concurrent Jobs arranged in a tree:
batch {
exec! cmd1, cmd2
batch {
exec! cmd3, cmd4
batch { exec! cmd5 }
}.exec # we’ll wait till this batch completes
…
}.exec!

Conclusion

• OOP is great!
– Inheritance makes code confusing
– Polymorphism is powerful
– Excellent namespace management
• Design by Symbiosis
• Singleton objects for local extensibility
• Object#instance_eval(&block)
• Metaprogramming
– Make things frobbable
– I don’t miss lisp that much…

Thank You
• http://github.com/hayeah/rubish/tree/master
– Need lots more work.
– Generalize Rubish for remote scripting.
• Looking for interesting projects
– I like weird languages.
• hayeah@gmail.com