1. Building Interpreters
with PyPy

2. About me
• Computer science bachelor student at TU Berlin!
• Programming/Python since ~2008!
• Primarily involved with Pocoo projects (Sphinx,
Werkzeug, Flask, Babel, …)

3. PyPy Python Interpreter
• Fast Python implementation!
• Just-in-Time compilation!
• Proper garbage collection (no reference counting)!
• Written in Python

4. PyPy Translation Toolchain
• Capable of compiling (R)Python!
• Garbage collection!
• Tracing just-in-time compiler generator!
• Software transactional memory?

5. PyPy based interpreters
• Topaz (Ruby)!
• HippyVM (PHP)!
• Pyrolog (Prolog)!
• pycket (Racket)!
• Various other interpreters for (Scheme, Javascript,
io, Gameboy)

6. RPython
• Python subset!
• Statically typed!
• Garbage collected!
• Standard library almost entirely unavailable!
• Some missing builtins (print, open(), …)!
• rpython.rlib!
• exceptions are (sometimes) ignored!
• Not a really a language, rather a "state"

7. Hello RPython
# hello_rpython.py
import os
!
def entry_point(argv):
os.write(2, “Hello, World!n”)
return 0
!
def target(driver, argv):
return entry_point, None

8. $ rpython hello_rpython.py
…
$ ./hello_python-c
Hello, RPython!

9. Goal
• BASIC interpreter capable of running Hamurabi!
• Bytecode based!
• Garbage Collection!
• Just-In-Time Compilation

11. Live play session

12. Architecture
Parser
Compiler
Virtual Machine
AST
Bytecode
Source

13. 10 PRINT TAB(32);"HAMURABI"
20 PRINT TAB(15);"CREATIVE COMPUTING MORRISTOWN, NEW JERSEY"
30 PRINT:PRINT:PRINT
80 PRINT "TRY YOUR HAND AT GOVERNING ANCIENT SUMERIA"
90 PRINT "FOR A TEN-YEAR TERM OF OFFICE.":PRINT
95 D1=0: P1=0
100 Z=0: P=95:S=2800: H=3000: E=H-S
110 Y=3: A=H/Y: I=5: Q=1
210 D=0
215 PRINT:PRINT:PRINT "HAMURABI: I BEG TO REPORT TO YOU,": Z=Z+1
217 PRINT "IN YEAR";Z;",";D;"PEOPLE STARVED,";I;"CAME TO THE CITY,"
218 P=P+I
227 IF Q>0 THEN 230
228 P=INT(P/2)
229 PRINT "A HORRIBLE PLAGUE STRUCK! HALF THE PEOPLE DIED."
230 PRINT "POPULATION IS NOW";P
232 PRINT "THE CITY NOW OWNS ";A;"ACRES."
235 PRINT "YOU HARVESTED";Y;"BUSHELS PER ACRE."
250 PRINT "THE RATS ATE";E;"BUSHELS."
260 PRINT "YOU NOW HAVE ";S;"BUSHELS IN STORE.": PRINT
270 REM *** MORE CODE THAT DID NOT FIT INTO THE SLIDE FOLLOWS

14. Parser
Parser
Abstract Syntax Tree (AST)
Source

15. Parser
Parser
AST
Source
Lexer
Tokens
Source
Parser
AST

16. RPLY
• Based on PLY, which is based on Lex and Yacc!
• Lexer generator!
• LALR parser generator

17. Lexer
from rply import LexerGenerator
!
lg = LexerGenerator()
!
lg.add(“NUMBER”, “[0-9]+”)
# …
lg.ignore(“ +”) # whitespace
!
lexer = lg.build().lex

18. lg.add('NUMBER', r'[0-9]*.[0-9]+')
lg.add('PRINT', r'PRINT')
lg.add('IF', r'IF')
lg.add('THEN', r'THEN')
lg.add('GOSUB', r'GOSUB')
lg.add('GOTO', r'GOTO')
lg.add('INPUT', r'INPUT')
lg.add('REM', r'REM')
lg.add('RETURN', r'RETURN')
lg.add('END', r'END')
lg.add('FOR', r'FOR')
lg.add('TO', r'TO')
lg.add('NEXT', r'NEXT')
lg.add('NAME', r'[A-Z][A-Z0-9$]*')
lg.add('(', r'(')
lg.add(')', r')')
lg.add(';', r';')
lg.add('STRING', r'"[^"]*"')
lg.add(':', r'r?n')
lg.add(':', r':')
lg.add('=', r'=')
lg.add('<>', r'<>')
lg.add('-', r'-')
lg.add('/', r'/')
lg.add('+', r'+')
lg.add('>=', r'>=')
lg.add('>', r'>')
lg.add('***', r'***.*')
lg.add('*', r'*')
lg.add('<=', r'<=')
lg.add('<', r'<')

19. >>> from basic.lexer import lex
>>> source = open("hello.bas").read()
>>> for token in lex(source):
... print token
Token("NUMBER", "10")
Token("PRINT", "PRINT")
Token("STRING",'"HELLO BASIC!"')
Token(":", "n")

20. Grammar
• A set of formal rules that defines the syntax!
• terminals = tokens!
• nonterminals = rules defining a sequence of one or
more (non)terminals

21. 10 PRINT TAB(32);"HAMURABI"
20 PRINT TAB(15);"CREATIVE COMPUTING MORRISTOWN, NEW JERSEY"
30 PRINT:PRINT:PRINT
80 PRINT "TRY YOUR HAND AT GOVERNING ANCIENT SUMERIA"
90 PRINT "FOR A TEN-YEAR TERM OF OFFICE.":PRINT
95 D1=0: P1=0
100 Z=0: P=95:S=2800: H=3000: E=H-S
110 Y=3: A=H/Y: I=5: Q=1
210 D=0
215 PRINT:PRINT:PRINT "HAMURABI: I BEG TO REPORT TO YOU,": Z=Z+1
217 PRINT "IN YEAR";Z;",";D;"PEOPLE STARVED,";I;"CAME TO THE CITY,"
218 P=P+I
227 IF Q>0 THEN 230
228 P=INT(P/2)
229 PRINT "A HORRIBLE PLAGUE STRUCK! HALF THE PEOPLE DIED."
230 PRINT "POPULATION IS NOW";P
232 PRINT "THE CITY NOW OWNS ";A;"ACRES."
235 PRINT "YOU HARVESTED";Y;"BUSHELS PER ACRE."
250 PRINT "THE RATS ATE";E;"BUSHELS."
260 PRINT "YOU NOW HAVE ";S;"BUSHELS IN STORE.": PRINT
270 REM *** MORE CODE THAT DID NOT FIT INTO THE SLIDE FOLLOWS

22. program :
program : line
program : line program

23. line : NUMBER statements

24. statements : statement
statements : statement statements

25. statement : PRINT :
statement : PRINT expressions :
expressions : expression
expressions : expression ;
expressions : expression ; expressions

26. statement : NAME = expression :

27. statement : IF expression THEN number :

28. statement : INPUT name :

29. statement : GOTO NUMBER :
statement : GOSUB NUMBER :
statement : RETURN :

30. statement : REM *** :

31. statement : FOR NAME = NUMBER TO NUMBER :
statement : NEXT NAME :

32. statement : END :

33. expression : NUMBER
expression : NAME
expression : STRING
expression : operation
expression : ( expression )
expression : NAME ( expression )

34. operation : expression + expression
operation : expression - expression
operation : expression * expression
operation : expression / expression
operation : expression <= expression
operation : expression < expression
operation : expression = expression
operation : expression <> expression
operation : expression > expression
operation : expression >= expression

35. from rply.token import BaseBox
!
class Program(BaseBox):
def __init__(self, lines):
self.lines = lines
AST

36. class Line(BaseBox):
def __init__(self, lineno, statements):
self.lineno = lineno
self.statements = statements

37. class Statements(BaseBox):
def __init__(self, statements):
self.statements = statements

38. class Print(BaseBox):
def __init__(self, expressions, newline=True):
self.expressions = expressions
self.newline = newline

39. …

40. from rply import ParserGenerator
!
pg = ParserGenerator(["NUMBER", "PRINT", …])
Parser

41. @pg.production("program : ")
@pg.production("program : line")
@pg.production("program : line program")
def program(p):
if len(p) == 2:
return Program([p[0]] + p[1].get_lines())
return Program(p)

42. @pg.production("line : number statements")
def line(p):
return Line(p[0], p[1].get_statements())

43. @pg.production("op : expression + expression")
@pg.production("op : expression * expression")
def op(p):
if p[1].gettokentype() == "+":
return Add(p[0], p[2])
elif p[1].gettokentype() == "*":
return Mul(p[0], p[2])

44. pg = ParserGenerator([…], precedence=[
("left", ["+", "-"]),
("left", ["*", "/"])
])

45. parse = pg.build().parse

46. Compiler/Virtual Machine
Compiler
Virtual Machine
AST
Bytecode

47. class VM(object):
def __init__(self, program):
self.program = program

48. class VM(object):
def __init__(self, program):
self.program = program
self.pc = 0

49. class VM(object):
def __init__(self, program):
self.program = program
self.pc = 0
self.frames = []

50. class VM(object):
def __init__(self, program):
self.program = program
self.pc = 0
self.frames = []
self.iterators = []

51. class VM(object):
def __init__(self, program):
self.program = program
self.pc = 0
self.frames = []
self.iterators = []
self.stack = []

52. class VM(object):
def __init__(self, program):
self.program = program
self.pc = 0
self.frames = []
self.iterators = {}
self.stack = []
self.variables = {}

53. class VM(object):
…
def execute(self):
while self.pc < len(self.program.instructions):
self.execute_bytecode(self.program.instructions[self.pc])

54. class VM(object):
…
def execute_bytecode(self, code):
raise NotImplementedError(code)

55. class VM(object):
...
def execute_bytecode(self):
if isinstance(code, TYPE):
self.execute_TYPE(code)
...
else:
raise NotImplementedError(code)

56. class Program(object):
def __init__(self):
self.instructions = []
Bytecode

57. class Instruction(object):
pass

58. class Number(Instruction):
def __init__(self, value):
self.value = value
!
class String(Instructions):
def __init__(self, value):
self.value = value

59. class Print(Instruction):
def __init__(self, expressions, newline):
self.expressions = expressions
self.newline = newline

60. class Call(Instruction):
def __init__(self, function_name):
self.function_name = function_name

61. class Let(Instruction):
def __init__(self, name):
self.name = name

62. class Lookup(Instruction):
def __init__(self, name):
self.name = name

63. class Add(Instruction):
pass
!
class Sub(Instruction):
pass
!
class Mul(Instruction):
pass
!
class Equal(Instruction):
pass
!
...

64. class GotoIfTrue(Instruction):
def __init__(self, target):
self.target = target
!
class Goto(Instruction):
def __init__(self, target, with_frame=False):
self.target = target
self.with_frame = with_frame
!
class Return(Instruction):
pass

65. class Input(object):
def __init__(self, name):
self.name = name

66. class For(Instruction):
def __init__(self, variable):
self.variable = variable
!
class Next(Instruction):
def __init__(self, variable):
self.variable = variable

67. class Program(object):
def __init__(self):
self.instructions = []
self.lineno2instruction = {}
!
def __enter__(self):
return self
!
def __exit__(self, exc_type, exc_value, tb):
if exc_type is None:
for i, instruction in enumerate(self.instructions):
instruction.finalize(self, i)

68. def finalize(self, program, index):
self.target = program.lineno2instruction[self.target]

69. class Program(BaseBox):
…
def compile(self):
with bytecode.Program() as program:
for line in self.lines:
line.compile(program)
return program

70. class Line(BaseBox):
...
def compile(self, program):
program.lineno2instruction[self.lineno] = len(program.instructions)
for statement in self.statements:
statement.compile(program)

71. class Line(BaseBox):
...
def compile(self, program):
program.lineno2instruction[self.lineno] = len(program.instructions)
for statement in self.statements:
statement.compile(program)

72. class Print(Statement):
def compile(self, program):
for expression in self.expressions:
expression.compile(program)
program.instructions.append(
bytecode.Print(
len(self.expressions),
self.newline
)
)

73. class Print(Statement):
...
def compile(self, program):
for expression in self.expressions:
expression.compile(program)
program.instructions.append(
bytecode.Print(
len(self.expressions),
self.newline
)
)

74. class Let(Statement):
...
def compile(self, program):
self.value.compile(program)
program.instructions.append(
bytecode.Let(self.name)
)

75. class Input(Statement):
...
def compile(self, program):
program.instructions.append(
bytecode.Input(self.variable)
)

76. class Goto(Statement):
...
def compile(self, program):
program.instructions.append(
bytecode.Goto(self.target)
)
!
class Gosub(Statement):
...
def compile(self, program):
program.instructions.append(
bytecode.Goto(
self.target,
with_frame=True
)
)
!
class Return(Statement):
...
def compile(self, program):
program.instructions.append(
bytecode.Return()
)

77. class For(Statement):
...
def compile(self, program):
self.start.compile(program)
program.instructions.append(
bytecode.Let(self.variable)
)
self.end.compile(program)
program.instructions.append(
bytecode.For(self.variable)
)

78. class WrappedObject(object):
pass
!
class WrappedString(WrappedObject):
def __init__(self, value):
self.value = value
!
class WrappedFloat(WrappedObject):
def __init__(self, value):
self.value = value

79. class VM(object):
…
def execute_number(self, code):
self.stack.append(WrappedFloat(code.value))
self.pc += 1
!
def execute_string(self, code):
self.stack.append(WrappedString(code.value))
self.pc += 1

80. class VM(object):
…
def execute_call(self, code):
argument = self.stack.pop()
if code.function_name == "TAB":
self.stack.append(WrappedString(" " * int(argument)))
elif code.function_name == "RND":
self.stack.append(WrappedFloat(random.random()))
...
self.pc += 1

81. class VM(object):
…
def execute_let(self, code):
value = self.stack.pop()
self.variables[code.name] = value
self.pc += 1
!
def execute_lookup(self, code):
value = self.variables[code.name]
self.stack.append(value)
self.pc += 1

82. class VM(object):
…
def execute_add(self, code):
right = self.stack.pop()
left = self.stack.pop()
self.stack.append(WrappedFloat(left + right))
self.pc += 1

83. class VM(object):
…
def execute_goto_if_true(self, code):
condition = self.stack.pop()
if condition:
self.pc = code.target
else:
self.pc += 1

84. class VM(object):
…
def execute_goto(self, code):
if code.with_frame:
self.frames.append(self.pc + 1)
self.pc = code.target

85. class VM(object):
…
def execute_return(self, code):
self.pc = self.frames.pop()

86. class VM(object):
…
def execute_input(self, code):
value = WrappedFloat(float(raw_input() or “0.0”))
self.variables[code.name] = value
self.pc += 1

87. class VM(object):
…
def execute_for(code):
self.pc += 1
self.iterators[code.variable] = (
self.pc,
self.stack.pop()
)

88. class VM(object):
…
def execute_next(self, code):
loop_begin, end = self.iterators[code.variable]
current_value = self.variables[code.variable].value
next_value = current_value + 1.0
if next_value <= end:
self.variables[code.variable] =
WrappedFloat(next_value)
self.pc = loop_begin
else:
del self.iterators[code.variable]
self.pc += 1

89. def entry_point(argv):
try:
filename = argv[1]
except IndexError:
print(“You must supply a filename”)
return 1
content = read_file(filename)
tokens = lex(content)
ast = parse(tokens)
program = ast.compile()
vm = VM(program)
vm.execute()
return 0
Entry Point

90. JIT (in PyPy)
1. Identify “hot" loops!
2. Create trace inserting guards based on observed
values!
3. Optimize trace!
4. Compile trace!
5. Execute machine code instead of interpreter

91. from rpython.rlib.jit import JitDriver
!
jitdriver = JitDriver(
greens=[“pc”, “vm”, “program”, “frames”, “iterators”],
reds=[“stack”, “variables"]
)

92. class VM(object):
…
def execute(self):
while self.pc < len(self.program.instructions):
jitdriver.merge_point(
vm=self,
pc=self.pc,
…
)

93. Benchmark
10 N = 1
20 IF N <= 10000 THEN 40
30 END
40 GOSUB 100
50 IF R = 0 THEN 70
60 PRINT "PRIME"; N
70 N = N + 1: GOTO 20
100 REM *** ISPRIME N -> R
110 IF N <= 2 THEN 170
120 FOR I = 2 TO (N - 1)
130 A = N: B = I: GOSUB 200
140 IF R <> 0 THEN 160
150 R = 0: RETURN
160 NEXT I
170 R = 1: RETURN
200 REM *** MOD A -> B -> R
210 R = A - (B * INT(A / B))
220 RETURN

94. cbmbasic 58.22s
basic-c 5.06s
basic-c-jit 2.34s
Python implementation (CPython) 2.83s
Python implementation (PyPy) 0.11s
C implementation 0.03s

95. Questions?

96. These slides are licensed under a Creative Commons
Attribution-ShareAlike 4.0 International License