1.
Hugging Abstract Syntax Trees:
A Pythonic Love Story
<3
Tom Lee
Shine Technologies

2.
About Me <3
● Consultant for Shine Technologies
● Software consultancy with offices in Melbourne and Brisbane
● http://www.shinetech.com
● Python contributor
● try/except/finally
● compile() ASTs
● … and other bits and bobs.
● Would-be compiler geek.

3.
Overview <3
● What are we discussing?
● Python Compiler Architecture
● Abstract Syntax Trees (ASTs)
● Abstract Syntax Trees in CPython
● Writing Compilers for the Python VM
● Benefits
● Demo?

4.
What are we discussing? <3
● A bird's-eye view of bytecode compilers.
● Features introduced in Python 2.5 & 2.6.
● Abstract Syntax Trees and their use in Python.
● What they are.
● Why they're cool!
● Implement your own language in Python for the Python VM.

5.
Python Compiler Architecture <3

6.
Abstract Syntax Trees <3
● Data structure.
● Constructed by a parser.
● Consumed by a code generator.
● In-memory, logical representation of a program.
● Syntax and bytecode independent.
● Contrast to parse trees.

7.
Abstract Syntax Trees in CPython <3
● Python 2.5: Used internally by the Python compiler.
● API access from Python, but limited utility.
● Python 2.6: compile() ASTs to bytecode.
● Parse Python source to an AST.
● Analyze & manipulate ASTs in-memory.
– Optimisation
– Static analysis
● Generate and execute bytecode from an AST.
● What if you construct your own ASTs?

8.
Writing Compilers for the Python VM <3
● Write a scanner (in Python!), generate tokens.
● Write a parser (in Python!), construct an AST.
● Use Python's ast module.
● Use compile() to construct a code object from the AST.
● Use exec/eval() to run the code object on the Python VM.

9.
Benefits <3
● Write compilers in Python!
● ASTs more easily manipulated than generated source.
● Optimise code at the AST level.
● Use Python modules from within your language.
● OR Compile your language to .pyc files and call 'em from
Python.

10.
Proof of Concept: Viking <3
http://github.com/thomaslee/viking-poc

11.
About Viking <3
● Think of it as Python with braces...
● A front end for the Python bytecode compiler.
● Compiles source to an equivalent using Python's AST.
● Written over the course of a few hours.
● <400 LOPC (including spaces and comments).
● Quick, dirty, horrible, incomplete...
● … but surprisingly powerful as a demonstration.

12.
Demo? <3
http://www.funnyhub.com/pictures/pages/snake-bites-face.html

13.
Summary <3
● ASTs are cool!
● Python + ASTs + compile() = powerful language prototyping.
● Practical? Eh.

14.
Questions? <3
Tom Lee
http://tomlee.co
Shine Technologies
http://www.shinetech.com
thomas.lee@shinetech.com