Hugging Abstract Syntax Trees:
A Pythonic Love Story
<3
Tom Lee
Shine Technologies
About Me <3
● Consultant for Shine Technologies
● Software consultancy with offices in Melbourne and Brisbane
● http://www...
Overview <3
● What are we discussing?
● Python Compiler Architecture
● Abstract Syntax Trees (ASTs)
● Abstract Syntax Tree...
What are we discussing? <3
● A bird's-eye view of bytecode compilers.
● Features introduced in Python 2.5 & 2.6.
● Abstrac...
Python Compiler Architecture <3
Abstract Syntax Trees <3
● Data structure.
● Constructed by a parser.
● Consumed by a code generator.
● In-memory, logical...
Abstract Syntax Trees in CPython <3
● Python 2.5: Used internally by the Python compiler.
● API access from Python, but li...
Writing Compilers for the Python VM <3
● Write a scanner (in Python!), generate tokens.
● Write a parser (in Python!), con...
Benefits <3
● Write compilers in Python!
● ASTs more easily manipulated than generated source.
● Optimise code at the AST ...
Proof of Concept: Viking <3
http://github.com/thomaslee/viking-poc
About Viking <3
● Think of it as Python with braces...
● A front end for the Python bytecode compiler.
● Compiles source t...
Demo? <3
http://www.funnyhub.com/pictures/pages/snake-bites-face.html
Summary <3
● ASTs are cool!
● Python + ASTs + compile() = powerful language prototyping.
● Practical? Eh.
Questions? <3
Tom Lee
http://tomlee.co
Shine Technologies
http://www.shinetech.com
thomas.lee@shinetech.com
Upcoming SlideShare
Loading in …5
×

Hugging Abstract Syntax Trees: A Pythonic Love Story (OSDC 2010)

2,908 views

Published on

OSDC 2010 presentation on cool stuff that can be done with Python Abstract Syntax Trees and the compile() function.

Demo Python compiler (incomplete and hacky!) can be found here: http://github.com/thomaslee/viking-poc

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,908
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
31
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • TODO:Make the diagram clearer: show the flow of data from scanner to parser to bytecode generator.
  • Hugging Abstract Syntax Trees: A Pythonic Love Story (OSDC 2010)

    1. 1. Hugging Abstract Syntax Trees: A Pythonic Love Story <3 Tom Lee Shine Technologies
    2. 2. About Me <3 ● Consultant for Shine Technologies ● Software consultancy with offices in Melbourne and Brisbane ● http://www.shinetech.com ● Python contributor ● try/except/finally ● compile() ASTs ● … and other bits and bobs. ● Would-be compiler geek.
    3. 3. Overview <3 ● What are we discussing? ● Python Compiler Architecture ● Abstract Syntax Trees (ASTs) ● Abstract Syntax Trees in CPython ● Writing Compilers for the Python VM ● Benefits ● Demo?
    4. 4. What are we discussing? <3 ● A bird's-eye view of bytecode compilers. ● Features introduced in Python 2.5 & 2.6. ● Abstract Syntax Trees and their use in Python. ● What they are. ● Why they're cool! ● Implement your own language in Python for the Python VM.
    5. 5. Python Compiler Architecture <3
    6. 6. Abstract Syntax Trees <3 ● Data structure. ● Constructed by a parser. ● Consumed by a code generator. ● In-memory, logical representation of a program. ● Syntax and bytecode independent. ● Contrast to parse trees.
    7. 7. Abstract Syntax Trees in CPython <3 ● Python 2.5: Used internally by the Python compiler. ● API access from Python, but limited utility. ● Python 2.6: compile() ASTs to bytecode. ● Parse Python source to an AST. ● Analyze & manipulate ASTs in-memory. – Optimisation – Static analysis ● Generate and execute bytecode from an AST. ● What if you construct your own ASTs?
    8. 8. Writing Compilers for the Python VM <3 ● Write a scanner (in Python!), generate tokens. ● Write a parser (in Python!), construct an AST. ● Use Python's ast module. ● Use compile() to construct a code object from the AST. ● Use exec/eval() to run the code object on the Python VM.
    9. 9. Benefits <3 ● Write compilers in Python! ● ASTs more easily manipulated than generated source. ● Optimise code at the AST level. ● Use Python modules from within your language. ● OR Compile your language to .pyc files and call 'em from Python.
    10. 10. Proof of Concept: Viking <3 http://github.com/thomaslee/viking-poc
    11. 11. About Viking <3 ● Think of it as Python with braces... ● A front end for the Python bytecode compiler. ● Compiles source to an equivalent using Python's AST. ● Written over the course of a few hours. ● <400 LOPC (including spaces and comments). ● Quick, dirty, horrible, incomplete... ● … but surprisingly powerful as a demonstration.
    12. 12. Demo? <3 http://www.funnyhub.com/pictures/pages/snake-bites-face.html
    13. 13. Summary <3 ● ASTs are cool! ● Python + ASTs + compile() = powerful language prototyping. ● Practical? Eh.
    14. 14. Questions? <3 Tom Lee http://tomlee.co Shine Technologies http://www.shinetech.com thomas.lee@shinetech.com

    ×