Thnad's Revenge

Welcome to “Thnad’s Revenge,” a programming language implementation tale in three acts.

http://en.wikipedia.org/wiki/Yars'_Revenge
...Yars’ Revenge, the awesome Atari video game from the ’80s.

Cucumber Recipes

Ian Dees
with Aslak Hellesøy
and Matt Wynne

pragprog/titles/JRUBY
discount code: JRubyIanDees

Before we get to the talk, let me make a couple of quick announcements. First, we’re
updating the JRuby book this summer with a JRuby 1.7-ready PDF. To celebrate that, we’re
offering a discount code on the book during the conference. Second, I’m working on a new
book with the Cucumber folks, which has some JRuby/JVM stuff in it—if you’d like to be a
tech reviewer, please ﬁnd me after this talk.

I. Meet Thnad
II. Enter the Frenemy
III. Thnad’s Revenge

(with apologies to Ira Glass) Act I, Meet Thnad, in which we encounter Thnad, a programming
language built with JRuby and designed not for programmer happiness, but for implementer
happiness. Act II, Enter the Frenemy, in which we meet a new Ruby runtime. Act III, Thnad's
Revenge, in which we port Thnad to run on the Rubinius runtime and encounter some
surprises along the way.

I. Meet Thnad

Thnad is a programming language I created last summer as an excuse to learn some fun
JRuby tools and see what it's like to write a compiler.

The name comes from a letter invented by Dr. Seuss in his book, “On Beyond Zebra.” Since
most of the real letters are already taken by programming languages, a ﬁctional one seems
appropriate.

A Fictional Programming
Language
Optimized for Implementer Happiness

Just as Ruby is optimized for programmer happiness, Thnad is optimized for implementer
happiness. It was designed to be implemented with a minimum of time and effort, and a
maximum amount of fun.

function factorial(n) {
if (eq(n, 1)) {
1
} else {
times(n, factorial(minus(n, 1)))
}
}

print(factorial(4))

Here’s a sample Thnad program demonstrating all the major features. Thnad has integers,
functions, conditionals, and... not much else. These minimal features were easy to add,
thanks to the great tools available in the JRuby ecosystem (and other ecosystems, as we’ll
see).

Thnad Features

1. Names and Numbers
2. Function Calls
3. Conditionals
4. Function Deﬁnitions

In the next few minutes, we’re going to trace through each of these four language features,
from parsing the source all the way to generating the ﬁnal binary. We won’t show every
single grammar rule, but we will hit the high points.

As Tom mentioned in his talk, there are a number of phases a piece of source code goes
through during compilation.

Stages of Parsing

tokenize
parse
transform
emit

These break down into four main stages in a typical language: ﬁnding the tokens or parts of
speech of the text, parsing the tokens into an in-memory tree, transforming the tree, and
generating the bytecode. We’re going to look at each of Thnad’s major features in the
context of these stages.


First, let’s look at the easiest language feature: numbers and function parameters.

{:number => '42'}

root

'42' :number

"42"

Our parser needs to transform this input text into some kind of Ruby data structure.

Parslet
kschiess.github.com/parslet

I used a library called Parslet for that. Parslet handles the ﬁrst two stages of compilation
(tokenizing and parsing) using a Parsing Expression Grammar, or PEG. PEGs are like regular
expressions attached to blocks of code. They sound like a hack, but there’s solid compiler
theory behind them.

{:number => '42'}

root

'42' :number

"42"

rule(:number) {
match('[0-9]').repeat(1).as(:number) >> space? }

The rule at the bottom of the page is Parslet’s notation for matching one or more numbers
followed by a optional space.

{:number => '42'} Thnad::Number.new(42)

root
root
Thnad::Number
:number
:value

"42" 42

rule(:number => simple(:value)) {
Number.new(value.to_i) }

Now for the third stage, transformation. We could generate the bytecode straight from the
original tree, using a bunch of hard-to-test case statements. But it would be nicer to have a
speciﬁc Ruby class for each Thnad language feature. The rule at the bottom of this slide tells
Parslet to transform a Hash with a key called :number into an instance of a Number class we
provide.

BiteScript
github/headius/bitescript

The ﬁnal stage, outputting bytecode, is handled by the BiteScript library, which is basically a
domain-speciﬁc language for emitting JVM opcodes.

main do
ldc 42
ldc 1
invokestatic :Example, :baz, [int, int, int]
returnvoid
end

Here's an example, just to get an idea of the ﬂavor. To call a method, you just push the
arguments onto the stack and then call a speciﬁc opcode, in this case invokestatic. The VM
you're writing for is aware of classes, interfaces, and so on—you don't have to implement
method lookup like you would with plain machine code.

“JVM Bytecode for Dummies”
Charles Nutter, Øredev 2010
slideshare/CharlesNutter/redev-2010-jvm-bytecode-for-dummies
When I ﬁrst saw the BiteScript, I thought it was something you'd only need if you were doing
deep JVM hacking. But when I read the slides from Charlie's presentation at Øredev, it
clicked. This library takes me way back to my college days, when we'd write assembler
programs for a really simple instruction set like MIPS. BiteScript evokes that same kind of
feeling. I'd always thought the JVM would have a huge, crufty instruction set—but it's actually
quite manageable to keep the most important parts of it in your head.

class Number < Struct.new :value
def eval(context, builder)
builder.ldc value
end
end

We can generate the bytecode any way we want. One simple way is to give each of our
classes an eval() method that takes a BiteScript generator and calls various methods on it to
generate JVM instructions.

class Name < Struct.new :name
param_names = context[:params] || []
position = param_names.index(name)
raise "Unknown parameter #{name}" unless position

builder.iload position
end
end

Dealing with passed-in parameters is nearly as easy as dealing with raw integers; we just
look up the parameter name by position, and then push the nth parameter onto the stack.

2. Function Calls

The next major feature is function calls. Once we have those, we will be able to run a trivial
Thnad program.

{:funcall =>
{:name => 'baz',
:args => [
{:arg => {:number => '42'}}]}}
{:arg => {:name => 'foo'}}]}}

root

'baz(42, foo)' :funcall

:name :args

"baz" :arg :arg

:number :name

"42" "foo"

We’re going to move a little faster here, to leave time for Rubinius. Here, we want to
transform this source code into this Ruby data structure representing a function call.

Thnad::Funcall.new 'foo',
[Thnad::Number.new(42)]

root

Thnad::Funcall

:name :args

"foo" Thnad::Number

:value

42

Now, we want to transform generic Ruby data structures into purpose-built ones that we can
attach bytecode-emitting behavior to.

class Funcall < Struct.new :name, :args
args.each { |a| a.eval(context, builder) }
types = [builder.int] * (args.length + 1)

builder.invokestatic
builder.class_builder, name, types
end
end

The bytecode for a function call is really simple in BiteScript. All functions in Thnad are static
methods on a single class.

3. Conditionals

The ﬁrst two features we’ve deﬁned are enough to write simple programs like print(42). The
next two features will let us add conditionals and custom functions.

{:cond =>
{:number => '0'},
:if_true =>
{:body => {:number => '42'}},
:if_false =>
{:body => {:number => '667'}}}
'if (0) {
42
root
} else {
667
:cond :if_true :if_false
}'
:number :body :body

"0" :number :number

"42" "667"

A conditional consists of the “if” keyword, followed by a body of code inside braces, then the
“else” keyword, followed by another body of code in braces.

Thnad::Conditional.new
Thnad::Number.new(0),
Thnad::Number.new(42),
Thnad::Number.new(667)

root

Thnad::Conditional

:cond :if_true :if_false

Thnad::Number Thnad::Number Thnad::Number

:value :value :value

0 42 667

Here’s the transformed tree representing a set of custom Ruby classes.

class Conditional < Struct.new :cond, :if_true, :if_false
cond.eval context, builder
builder.ifeq :else
if_true.eval context, builder
builder.goto :endif
builder.label :else
if_false.eval context, builder
builder.label :endif
end
end

The bytecode emitter for conditionals has a new twist. The Conditional struct points to three
other Thnad nodes. It needs to eval() them at the right time to emit their bytecode in
between all the zero checks and gotos.


On to the ﬁnal piece of Thnad: deﬁning new functions.

{:func =>
{:name => 'foo'},
:params =>
{:param =>
{:name => 'x'}},
:body =>
{:number => '5'}}

'function foo(x) {
root
5
}'
:func :params :body

:name :param :number

"foo" :name "5"

"x"

A function deﬁnition looks a lot like a function call, but with a body attached to it.

Thnad::Function.new
'foo',
[Thnad::Name.new('x')],
Thnad::Number.new(5)

root

Thnad::Function

:name :params :body

"foo" Thnad::Name Thnad::Number

:name :value

"x" 5

Here’s the transformation we want to perform for this language feature.

class Function < Struct.new :name, :params, :body
param_names = [params].flatten.map(&:name)
context[:params] = param_names
types = [builder.int] * (param_names.count + 1)

builder.public_static_method(self.name, [], *types) do
|method|

self.body.eval(context, method)
method.ireturn
end
end
end

Since all Thnad parameters and return types are integers, emitting a function deﬁnition is
really easy. We count the parameters so that we can give the JVM a correct signature. Then,
we just pass a block to the public_static_method helper, a feature of BiteScript that will
inspire the Rubinius work later on.

Compiler

We’ve seen how to generate individual chunks of bytecode; how do they all get stitched
together into a .class ﬁle?

builder = BiteScript::FileBuilder.build(@filename) do
public_class classname, object do |klass|
# ...

klass.public_static_method 'main', [], void, string[] do
|method|

context = Hash.new
exprs.each do |e|
e.eval(context, method)
end

method.returnvoid
end
end
end

Here’s the core of class generation. We output a standard Java main() function...

builder = BiteScript::FileBuilder.build(@filename) do
public_class classname, object do |klass|
# ...

klass.public_static_method 'main', [], void, string[] do
|method|

context = Hash.new
exprs.each do |e|
e.eval(context, method)
end

method.returnvoid
end
end
end

...inside which we eval() our Thnad expressions (not counting function deﬁnitions) one by
one.

Built-ins
plus, minus, times, eq, print

Thnad ships with a few basic arithmetic operations, plus a print() function. Let’s look at one
of those now.

public_static_method 'minus', [], int, int, int do
iload 0
iload 1
isub
ireturn
end

Here’s the deﬁnition of minus(). It just pushes its two arguments onto the stack and then
subtracts them. The rest of the built-ins are nearly identical to this one, so we won’t show
them here.

II. Enter the Frenemy

So that's a whirlwind tour of Thnad. Last year, I was telling someone about this project—it
was either Shane Becker or Brian Ford, I think—and he said,...

Rubinius

...“Hey, you should port this to Rubinius!” I thought, “Hmm, why not? Sounds fun.” Let’s
take a look at this other runtime that has sprung up as a rival for Thnad’s affections.

Ruby in Ruby

• As much as performance allows
• Initially 100%, now around half (?)
• Core in C++ / LLVM
• Tons in Ruby: primitives, parser, bytecode

The goal of Rubinius is to implement Ruby in Ruby as much as performance allows. Quite a
lot of functionality you’d think would need to be in C is actually in Ruby.

RubySpec, FFI
Brought to you by Rubinius
(Thank you!)

We have Rubinius to thank for the executable Ruby speciﬁcation that all Rubies are now
judged against, and for the excellent foreign-function interface that lets you call C code in a
way that’s compatible with at least four Rubies.

Looking Inside Your Code

Rubinius also has tons of mechanisms for looking inside your code, which was very helpful
when I needed to learn what bytecode I’d need to output to accomplish a particular task in
Thnad.

class Example
def add(a, b)
a + b
end
end

For example, with this class,...

AST
$ rbx compile -S example.rb
[:script,
[:class,
:Example,
nil,
[:scope,
[:block,
[:defn,
:add,
[:args, :a, :b],
[:scope,
[:block,
[:call,
[:lvar, :a], :+, [:arglist, [:lvar, :b]]]]]]]]]]

...you can get a Lisp-like representation of the syntax tree,...

Bytecode
$ rbx compile -B example.rb
...
================= :add =================
Arguments: 2 required, 2 total
Locals: 2: a, b
Stack size: 4
Lines to IP: 2: -1..-1, 3: 0..6

0000: push_local 0 # a
0002: push_local 1 # b
0004: meta_send_op_plus :+
0006: ret
----------------------------------------

...or a dump of the actual bytecode for the Rubinius VM.

“Ruby Platform Throwdown”
Moderated by Dr Nic, 2011
vimeo/26773441

For more on the similarities and differences between Rubinius and JRuby, see the throwdown
video moderated by Dr Nic.

III: Thnad’s Revenge

Now that we’ve gotten to know Rubinius a little...

Let’s port Thnad to Rubinius!

...let’s see what it would take to port Thnad to it.

photo: JSConf US

Our Guide Through the Wilderness
@brixen

Brian Ford was a huge help during this effort, answering tons of my “How do I...?” questions
in an awesome Socratic way (“Let’s take a look at the Generator class source code....”)

Same parser
Same AST transformation
Different bytecode
(But similar bytecode ideas)

Because the Thnad syntax is unchanged, we can reuse the parser and syntax transformation.
All we need to change is the bytecode output. And even that’s not drastically different.

Thnad’s Four Features,
Revisited

Let’s go back through Thnad’s four features in the context of Rubinius.


First, function parameters and integers.

JVM RBX

# Numbers: # Numbers:
ldc 42 push 42

# Names: # Names:
iload 0 push_local 0

See how similar the JVM and Rubinius bytecode is for these basic features?

class Number < Struct.new :value
builder.push value
end
end

All we had to change was the name of the opcode both for numbers...

class Name < Struct.new :name
param_names = context[:params] || []
position = param_names.index(name)
raise "Unknown parameter #{name}" unless position

builder.push_local position
end
end

...and for parameter names.

2. Function Calls

Function calls were similarly easy.

JVM RBX

push_const :Example
ldc 42 push 42
ldc 1 push 1
invokestatic #2; //Method send_stack #<CM>, 2
//add:(II)I

In Rubinius, there are no truly static methods. We are calling the method on a Ruby object—
namely, an entire Ruby class. So we have to push the name of that class onto the stack ﬁrst.
The other big difference is that in Rubinius, we don’t just push the method name onto the
stack—we push a reference to the compiled code itself. Fortunately, there’s a helper method
to make this look more Bitescript-like.

class Funcall < Struct.new :name, :args
builder.push_const :Thnad
args.each { |a| a.eval(context, builder) }
builder.allow_private
builder.send name.to_sym, args.length
end
end

Here’s how that difference affects the bytecode. Notice the allow_private() call? I’m not sure
exactly why we need this. It may be an “onion in the varnish,” a reference to a story by Primo
Levi in _The Periodic Table_.

ﬂickr/black-and-white-prints/1366095561
ﬂickr/ianfuller/76775606
In the story, the workers at a varnish factory wondered why the recipe called for an onion.
They couldn’t work out chemically why it would be needed, but it had always been one of the
ingredients. It turned out that it was just a crude old-school thermometer: when the onion
sizzled, the varnish was ready.

3. Conditionals

On to conditionals.

JVM RBX
0: iconst_0 37: push 0
1: ifeq 9 38: push 0
4: bipush 42 39: send :==
6: goto 12 41: goto_if_false 47
9: sipush 667 43: push 42
12: ... 45: goto 49
47: push 667
49: ...

Here, the JVM directly supports an “if equal to zero” opcode, whereas in Rubinius we have to
explicitly compare the item on the stack with zero.

class Conditional < Struct.new :cond, :if_true, :if_false
else_label = builder.new_label
endif_label = builder.new_label

cond.eval context, builder
builder.push 0
builder.send :==, 1

builder.goto_if_true else_label

if_true.eval context, builder
builder.goto endif_label

else_label.set!
if_false.eval context, builder
endif_label.set!
end
end
Labels are also a little different in Rubinius, too; here’s what the bytecode for conditionals
looks like now.


The trickiest part to implement was function calls.

JVM RBX
public int add(int, int); push_rubinius
iload_1 push :add
iload_2 push #<CM>
iadd push_scope
ireturn push_self
push_const :Thnad
send :attach_method, 4

Remember that in Ruby, there’s no compile-time representation of a class. So rather than
emitting a class deﬁnition, we emit code that creates a class at runtime.

class Function < Struct.new :name, :params, :body
param_names = [params].flatten.map(&:name)
context[:params] = param_names

# create a new Rubinius::Generator
builder.begin_method name.to_sym, params.count
self.body.eval(context, builder.current_method)
builder.current_method.ret
builder.end_method
end
end

The code to deﬁne a method in Rubinius requires spinning up a completely separate
bytecode generator. I stuck all this hairy logic in a set of helpers to make it more BiteScript-
like.

class Rubinius::Generator
def end_method
# ...

cm = @inner.package Rubinius::CompiledMethod

push_rubinius
push_literal inner.name
push_literal cm
push_scope
push_const :Thnad
send :attach_method, 4
pop
end
end

Here’s the most interesting part of those helpers. After the function deﬁnition is compiled,
we push it onto the stack and tell Rubinius to attach it to our class.

Compiler

How does the compiled code make its way into a .rbc ﬁle?

g = Rubinius::Generator.new

# ...

context = Hash.new
exprs.each do |e|
e.eval(context, g)
end

# ...

As with JRuby, we create a bytecode generation object, then evaluate all the Thnad
statements into it.

main = g.package Rubinius::CompiledMethod

Rubinius::CompiledFile.dump
main, @outname, Rubinius::Signature, 18

Finally, we tell Rubinius to marshal the compiled code to a .rbc ﬁle.

Runner (new!)

That means we now need a small script to unmarshal that compiled code and run it. This is
new; on the Java runtime, we already have a runner: the java binary.

#!/usr/bin/env rbx -rubygems

(puts("Usage: #{} BINARY"); exit) if ARGV.empty?

loader = Rubinius::CodeLoader.new(ARGV.first)
method = loader.load_compiled_file(
ARGV.first, Rubinius::Signature, 18)
result = Rubinius.run_script(method)

Here’s the entirety of the code to load and run a compiled Rubinius ﬁle.

Built-ins

As we’ve just seen, deﬁning a function in Rubinius takes a lot of steps, even with helper
functions to abstract away some of the hairiness.

g.begin_method :minus, 2
g.current_method.push_local 0
g.current_method.push_local 1
g.current_method.send :-, 1
g.current_method.ret
g.end_method

For example, here’s the built-in minus() function. I wanted to avoid writing a bunch of these.

function plus(a, b) {
minus(a, minus(0, b))
}

I realized that you could write plus() in Thnad instead, deﬁning it in terms of minus.

function times(a, b) {
if (eq(b, 0)) {
0
} else {
plus(a, times(a, minus(b, 1)))
}
}

If you don’t care about bounds checking, you can also do times()...

function eq(a, b) {
if (minus(a, b)) {
0
} else {
1
}
}

...and if()!

stdthnadlib?!?
We have a standard library!

That means we have a standard library! Doing the Rubinius implementation helped me
improve the JRuby version. I was able to go back and rip out most of the built-in functions
from that implementation.

Thnad Online
github/undees/thnad/tree/master
github/undees/thnad/tree/rbx

Here’s where you can download and play with either implementation.

This has been a fantastic conference. Thank you to our hosts...

Special Thanks

Kaspar Schiess for Parslet
Charles Nutter for BiteScript
Ryan Davis and Aja Hammerly for Graph
Brian Ford for guidance
Our tireless conference organizers!

...and to the makers of JRuby, Rubinius, Parslet, BiteScript, and everything else that made this
project possible. Cheers!

Thnad's Revenge

More Related Content

What's hot

Similar to Thnad's Revenge

More from Erin Dees

Recently uploaded

Thnad's Revenge

Editor's Notes