Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Hugging Abstract Syntax Trees: A Pythonic Love Story (OSDC 2010)

3,025 views

Published on

OSDC 2010 presentation on cool stuff that can be done with Python Abstract Syntax Trees and the compile() function.

Demo Python compiler (incomplete and hacky!) can be found here: http://github.com/thomaslee/viking-poc

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Hugging Abstract Syntax Trees: A Pythonic Love Story (OSDC 2010)

  1. 1. Hugging Abstract Syntax Trees: A Pythonic Love Story <3 Tom Lee Shine Technologies
  2. 2. About Me <3 ● Consultant for Shine Technologies ● Software consultancy with offices in Melbourne and Brisbane ● http://www.shinetech.com ● Python contributor ● try/except/finally ● compile() ASTs ● … and other bits and bobs. ● Would-be compiler geek.
  3. 3. Overview <3 ● What are we discussing? ● Python Compiler Architecture ● Abstract Syntax Trees (ASTs) ● Abstract Syntax Trees in CPython ● Writing Compilers for the Python VM ● Benefits ● Demo?
  4. 4. What are we discussing? <3 ● A bird's-eye view of bytecode compilers. ● Features introduced in Python 2.5 & 2.6. ● Abstract Syntax Trees and their use in Python. ● What they are. ● Why they're cool! ● Implement your own language in Python for the Python VM.
  5. 5. Python Compiler Architecture <3
  6. 6. Abstract Syntax Trees <3 ● Data structure. ● Constructed by a parser. ● Consumed by a code generator. ● In-memory, logical representation of a program. ● Syntax and bytecode independent. ● Contrast to parse trees.
  7. 7. Abstract Syntax Trees in CPython <3 ● Python 2.5: Used internally by the Python compiler. ● API access from Python, but limited utility. ● Python 2.6: compile() ASTs to bytecode. ● Parse Python source to an AST. ● Analyze & manipulate ASTs in-memory. – Optimisation – Static analysis ● Generate and execute bytecode from an AST. ● What if you construct your own ASTs?
  8. 8. Writing Compilers for the Python VM <3 ● Write a scanner (in Python!), generate tokens. ● Write a parser (in Python!), construct an AST. ● Use Python's ast module. ● Use compile() to construct a code object from the AST. ● Use exec/eval() to run the code object on the Python VM.
  9. 9. Benefits <3 ● Write compilers in Python! ● ASTs more easily manipulated than generated source. ● Optimise code at the AST level. ● Use Python modules from within your language. ● OR Compile your language to .pyc files and call 'em from Python.
  10. 10. Proof of Concept: Viking <3 http://github.com/thomaslee/viking-poc
  11. 11. About Viking <3 ● Think of it as Python with braces... ● A front end for the Python bytecode compiler. ● Compiles source to an equivalent using Python's AST. ● Written over the course of a few hours. ● <400 LOPC (including spaces and comments). ● Quick, dirty, horrible, incomplete... ● … but surprisingly powerful as a demonstration.
  12. 12. Demo? <3 http://www.funnyhub.com/pictures/pages/snake-bites-face.html
  13. 13. Summary <3 ● ASTs are cool! ● Python + ASTs + compile() = powerful language prototyping. ● Practical? Eh.
  14. 14. Questions? <3 Tom Lee http://tomlee.co Shine Technologies http://www.shinetech.com thomas.lee@shinetech.com

×