Python Compiler Internals Presentation Slides
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

Python Compiler Internals Presentation Slides

  • 7,316 views
Uploaded on

My OSDC 2008 presentation on the internals of the Python programming language.

My OSDC 2008 presentation on the internals of the Python programming language.

More in: Technology
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • hey there,could you please mail this across to me,it will truly help me for my function.thank you very much.
    Sharika
    http://financeadded.com http://traveltreble.com
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
7,316
On Slideshare
7,278
From Embeds
38
Number of Embeds
7

Actions

Shares
Downloads
122
Comments
1
Likes
9

Embeds 38

http://www.slideshare.net 24
http://www.linkedin.com 5
https://www.linkedin.com 5
http://translate.googleusercontent.com 1
http://coderwall.com 1
http://www.slashdocs.com 1
http://www.docseek.net 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Python Compiler Internals Thomas Lee Shine Technologies, Melbourne
  • 2. My Contributions to Python
    • try/except/finally syntax for 2.5
    • AST to bytecode compilation from Python
    • “ optimization” at the AST level
  • 3. What will you get out of this? Compiler development != rocket science. No magic: it's just code.
  • 4. Compiler? Isn't Python interpreted? Well, yes and no.
  • 5. WTF It ... It's Java?
  • 6. Aaanyway Let's screw with the compiler.
  • 7. Erm ... Before we start ... A patch to fix a quirk in the build process. This has been reported on the bug tracker!
  • 8. An overview of the compiler Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 9. New construct: the “unless” statement unless may_give_you_up: print “never gonna give you up...”
  • 10. Semantics of “unless” Works just like “if not” ... if not would_consider_letting_you_down: print “never gonna let you down...”
  • 11. Implementing “unless”
    • Modify the Grammar
    • Change the AST definition
    • Generate bytecode for “unless”
  • 12. So what's first?
    • Modify the Grammar
    • Change the AST definition
    • Generate bytecode for “unless”
  • 13. What does this change affect? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 14. WTF is a “grammar”? Defines the syntactic structure of the language.
  • 15. Why a tokenizer? Makes parsing easier.
  • 16. So what does a tokenizer do, then? Generates a stream of events for the parser.
  • 17. What does the parser do? Organises tokens into the structure defined by the grammar. This is the “parse tree”.
  • 18. What's next?
    • Modify the Grammar
    • Change the AST definition
    • Generate bytecode for “unless”
  • 19. What's affected by this change? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 20. What's an AST? Similar to a parse tree, but not bound to the syntax “ Abstract Syntax Tree”
  • 21. Why does Python use an AST? Much easier to operate upon an AST than a parse tree.
  • 22. AST reuse New constructs can sometimes be expressed using existing AST nodes ...
  • 23. AST reuse ... so we could implement “unless” using “if” and “not” AST nodes. But then I wouldn't be able to show you the code generator ... Unless (test, body) = If ( Not (test), body)
  • 24. What's next?
    • Modify the Grammar
    • Change the AST definition
    • Generate bytecode for “unless”
  • 25. What's affected by this change? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 26. Generating bytecode for “unless”
    • Add a hook for our new AST node
    • Generate bytecode implementing “unless” logic
    • Use basicblocks as labels for jumps
  • 27. What about the rest of the compiler? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 28. “unless” in action Let's see if this stuff works ...
  • 29. Where to go from here?
    • Use the Source.
    • Don't try to grok it all at once.
    • Don't be afraid to be wrong.
    • Annoy the python-dev mailing list, like I do!
  • 30. Thanks!
  • 31. Questions?