SlideShare a Scribd company logo
1 of 31
Download to read offline
Python Compiler Internals Thomas Lee Shine Technologies, Melbourne
My Contributions to Python ,[object Object],[object Object],[object Object]
What will you get out of this? Compiler development != rocket science. No magic: it's just code.
Compiler? Isn't Python interpreted? Well, yes and no.
WTF It ... It's Java?
Aaanyway Let's screw with the compiler.
Erm ... Before we start ... A patch to fix a quirk in the build process. This has been reported on the bug tracker!
An overview of the compiler Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
New construct: the “unless” statement unless  may_give_you_up: print   “never gonna give you up...”
Semantics of “unless” Works just like “if not” ... if not  would_consider_letting_you_down: print   “never gonna let you down...”
Implementing “unless” ,[object Object],[object Object],[object Object]
So what's first? ,[object Object],[object Object],[object Object]
What does this change affect? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
WTF is a “grammar”? Defines the syntactic structure of the language.
Why a tokenizer? Makes parsing easier.
So what does a tokenizer do, then? Generates a stream of events for the parser.
What does the parser do? Organises tokens into the structure defined by the grammar. This is the “parse tree”.
What's next? ,[object Object],[object Object],[object Object]
What's affected by this change? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
What's an AST? Similar to a parse tree, but not bound to the syntax “ Abstract Syntax Tree”
Why does Python use an AST? Much easier to operate upon an AST than a parse tree.
AST reuse New constructs can sometimes be expressed using existing AST nodes ...
AST reuse ... so we  could  implement “unless” using “if” and “not” AST nodes. But then I wouldn't be able to show you the code generator ... Unless (test, body) =  If ( Not (test), body)
What's next? ,[object Object],[object Object],[object Object]
What's affected by this change? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
Generating bytecode for “unless” ,[object Object],[object Object],[object Object]
What about the rest of the compiler? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
“unless” in action Let's see if this stuff works ...
Where to go from here? ,[object Object],[object Object],[object Object],[object Object]
Thanks!
Questions?

More Related Content

What's hot

Introduction to Python programming
Introduction to Python programmingIntroduction to Python programming
Introduction to Python programmingDamian T. Gordon
 
Introduction to C Programming
Introduction to C ProgrammingIntroduction to C Programming
Introduction to C ProgrammingAniket Patne
 
Presentation on C Switch Case Statements
Presentation on C Switch Case StatementsPresentation on C Switch Case Statements
Presentation on C Switch Case StatementsDipesh Pandey
 
Object-oriented Programming-with C#
Object-oriented Programming-with C#Object-oriented Programming-with C#
Object-oriented Programming-with C#Doncho Minkov
 
IF Statement
IF StatementIF Statement
IF StatementYunis20
 
Looping statement in python
Looping statement in pythonLooping statement in python
Looping statement in pythonRaginiJain21
 
Time complexity of union find
Time complexity of union findTime complexity of union find
Time complexity of union findWei (Terence) Li
 
Clean Code - The Next Chapter
Clean Code - The Next ChapterClean Code - The Next Chapter
Clean Code - The Next ChapterVictor Rentea
 
Fundamentals of Programming Constructs.pptx
Fundamentals of  Programming Constructs.pptxFundamentals of  Programming Constructs.pptx
Fundamentals of Programming Constructs.pptxvijayapraba1
 
oops concept in java | object oriented programming in java
oops concept in java | object oriented programming in javaoops concept in java | object oriented programming in java
oops concept in java | object oriented programming in javaCPD INDIA
 
This keyword and final keyword
This keyword and final  keywordThis keyword and final  keyword
This keyword and final keywordkinjalbirare
 
Expression evaluation
Expression evaluationExpression evaluation
Expression evaluationJeeSa Sultana
 
Conditional statement c++
Conditional statement c++Conditional statement c++
Conditional statement c++amber chaudary
 
Loops and conditional statements
Loops and conditional statementsLoops and conditional statements
Loops and conditional statementsSaad Sheikh
 

What's hot (20)

Introduction to Python programming
Introduction to Python programmingIntroduction to Python programming
Introduction to Python programming
 
Introduction to C Programming
Introduction to C ProgrammingIntroduction to C Programming
Introduction to C Programming
 
Presentation on C Switch Case Statements
Presentation on C Switch Case StatementsPresentation on C Switch Case Statements
Presentation on C Switch Case Statements
 
Object-oriented Programming-with C#
Object-oriented Programming-with C#Object-oriented Programming-with C#
Object-oriented Programming-with C#
 
C# Loops
C# LoopsC# Loops
C# Loops
 
IF Statement
IF StatementIF Statement
IF Statement
 
Java Generics
Java GenericsJava Generics
Java Generics
 
Looping statement in python
Looping statement in pythonLooping statement in python
Looping statement in python
 
Time complexity of union find
Time complexity of union findTime complexity of union find
Time complexity of union find
 
C++ Language
C++ LanguageC++ Language
C++ Language
 
C# loops
C# loopsC# loops
C# loops
 
Clean Code - The Next Chapter
Clean Code - The Next ChapterClean Code - The Next Chapter
Clean Code - The Next Chapter
 
Fundamentals of Programming Constructs.pptx
Fundamentals of  Programming Constructs.pptxFundamentals of  Programming Constructs.pptx
Fundamentals of Programming Constructs.pptx
 
oops concept in java | object oriented programming in java
oops concept in java | object oriented programming in javaoops concept in java | object oriented programming in java
oops concept in java | object oriented programming in java
 
This keyword and final keyword
This keyword and final  keywordThis keyword and final  keyword
This keyword and final keyword
 
Java 8 features
Java 8 featuresJava 8 features
Java 8 features
 
Expression evaluation
Expression evaluationExpression evaluation
Expression evaluation
 
Conditional statement c++
Conditional statement c++Conditional statement c++
Conditional statement c++
 
Threading in C#
Threading in C#Threading in C#
Threading in C#
 
Loops and conditional statements
Loops and conditional statementsLoops and conditional statements
Loops and conditional statements
 

Similar to Python Compiler Internals Presentation Slides

AST for JavaScript developers
AST for JavaScript developersAST for JavaScript developers
AST for JavaScript developersBohdan Liashenko
 
Open Source Compiler Construction for the JVM
Open Source Compiler Construction for the JVMOpen Source Compiler Construction for the JVM
Open Source Compiler Construction for the JVMTom Lee
 
Creating your own Abstract Processor
Creating your own Abstract ProcessorCreating your own Abstract Processor
Creating your own Abstract ProcessorAodrulez
 
Open Source Compiler Construction for the JVM [LCA2011 Miniconf]
Open Source Compiler Construction for the JVM [LCA2011 Miniconf]Open Source Compiler Construction for the JVM [LCA2011 Miniconf]
Open Source Compiler Construction for the JVM [LCA2011 Miniconf]Tom Lee
 
Streams of information - Chicago crystal language monthly meetup
Streams of information - Chicago crystal language monthly meetupStreams of information - Chicago crystal language monthly meetup
Streams of information - Chicago crystal language monthly meetupBrian Cardiff
 
"the Bund" language. A PEG grammar.
"the Bund" language. A PEG grammar."the Bund" language. A PEG grammar.
"the Bund" language. A PEG grammar.Vladimir Ulogov
 
Python Training Topics
Python Training TopicsPython Training Topics
Python Training Topicsvibrantuser
 
The Holistic Programmer
The Holistic ProgrammerThe Holistic Programmer
The Holistic ProgrammerAdam Keys
 
Leo Tolstoy and static code analysis
Leo Tolstoy and static code analysisLeo Tolstoy and static code analysis
Leo Tolstoy and static code analysisPVS-Studio
 
Os Keysholistic
Os KeysholisticOs Keysholistic
Os Keysholisticoscon2007
 
Inside Python [OSCON 2012]
Inside Python [OSCON 2012]Inside Python [OSCON 2012]
Inside Python [OSCON 2012]Tom Lee
 
An unusual bug in Lucene.Net
An unusual bug in Lucene.NetAn unusual bug in Lucene.Net
An unusual bug in Lucene.NetPVS-Studio
 
The Little Wonders of C# 6
The Little Wonders of C# 6The Little Wonders of C# 6
The Little Wonders of C# 6BlackRabbitCoder
 
Dive into CPython Bytecode
Dive into CPython BytecodeDive into CPython Bytecode
Dive into CPython BytecodeAlex Gaynor
 
React Component Library Design @WalmartLabs
React Component Library Design @WalmartLabsReact Component Library Design @WalmartLabs
React Component Library Design @WalmartLabschaseadamsio
 
The Great Semicolon Debate
The Great Semicolon DebateThe Great Semicolon Debate
The Great Semicolon DebateMike Wilcox
 
Os Worthington
Os WorthingtonOs Worthington
Os Worthingtonoscon2007
 
Python - Getting to the Essence - Points.com - Dave Park
Python - Getting to the Essence - Points.com - Dave ParkPython - Getting to the Essence - Points.com - Dave Park
Python - Getting to the Essence - Points.com - Dave Parkpointstechgeeks
 
Web Application Hacking
Web Application HackingWeb Application Hacking
Web Application HackingSensePost
 

Similar to Python Compiler Internals Presentation Slides (20)

AST for JavaScript developers
AST for JavaScript developersAST for JavaScript developers
AST for JavaScript developers
 
Open Source Compiler Construction for the JVM
Open Source Compiler Construction for the JVMOpen Source Compiler Construction for the JVM
Open Source Compiler Construction for the JVM
 
Creating your own Abstract Processor
Creating your own Abstract ProcessorCreating your own Abstract Processor
Creating your own Abstract Processor
 
Open Source Compiler Construction for the JVM [LCA2011 Miniconf]
Open Source Compiler Construction for the JVM [LCA2011 Miniconf]Open Source Compiler Construction for the JVM [LCA2011 Miniconf]
Open Source Compiler Construction for the JVM [LCA2011 Miniconf]
 
Streams of information - Chicago crystal language monthly meetup
Streams of information - Chicago crystal language monthly meetupStreams of information - Chicago crystal language monthly meetup
Streams of information - Chicago crystal language monthly meetup
 
"the Bund" language. A PEG grammar.
"the Bund" language. A PEG grammar."the Bund" language. A PEG grammar.
"the Bund" language. A PEG grammar.
 
Python Training Topics
Python Training TopicsPython Training Topics
Python Training Topics
 
The Holistic Programmer
The Holistic ProgrammerThe Holistic Programmer
The Holistic Programmer
 
Leo Tolstoy and static code analysis
Leo Tolstoy and static code analysisLeo Tolstoy and static code analysis
Leo Tolstoy and static code analysis
 
Os Keysholistic
Os KeysholisticOs Keysholistic
Os Keysholistic
 
Inside Python
Inside PythonInside Python
Inside Python
 
Inside Python [OSCON 2012]
Inside Python [OSCON 2012]Inside Python [OSCON 2012]
Inside Python [OSCON 2012]
 
An unusual bug in Lucene.Net
An unusual bug in Lucene.NetAn unusual bug in Lucene.Net
An unusual bug in Lucene.Net
 
The Little Wonders of C# 6
The Little Wonders of C# 6The Little Wonders of C# 6
The Little Wonders of C# 6
 
Dive into CPython Bytecode
Dive into CPython BytecodeDive into CPython Bytecode
Dive into CPython Bytecode
 
React Component Library Design @WalmartLabs
React Component Library Design @WalmartLabsReact Component Library Design @WalmartLabs
React Component Library Design @WalmartLabs
 
The Great Semicolon Debate
The Great Semicolon DebateThe Great Semicolon Debate
The Great Semicolon Debate
 
Os Worthington
Os WorthingtonOs Worthington
Os Worthington
 
Python - Getting to the Essence - Points.com - Dave Park
Python - Getting to the Essence - Points.com - Dave ParkPython - Getting to the Essence - Points.com - Dave Park
Python - Getting to the Essence - Points.com - Dave Park
 
Web Application Hacking
Web Application HackingWeb Application Hacking
Web Application Hacking
 

Recently uploaded

Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Nikki Chapple
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Alkin Tezuysal
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024TopCSSGallery
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Memoori
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...Wes McKinney
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFMichael Gough
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...Karmanjay Verma
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialJoão Esperancinha
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceOpsTree solutions
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 

Recently uploaded (20)

Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
Microsoft 365 Copilot: How to boost your productivity with AI – Part one: Ado...
 
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
Unleashing Real-time Insights with ClickHouse_ Navigating the Landscape in 20...
 
Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024Top 10 Hubspot Development Companies in 2024
Top 10 Hubspot Development Companies in 2024
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!
 
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
The Future Roadmap for the Composable Data Stack - Wes McKinney - Data Counci...
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
All These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDFAll These Sophisticated Attacks, Can We Really Detect Them - PDF
All These Sophisticated Attacks, Can We Really Detect Them - PDF
 
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...React JS; all concepts. Contains React Features, JSX, functional & Class comp...
React JS; all concepts. Contains React Features, JSX, functional & Class comp...
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Kuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorialKuma Meshes Part I - The basics - A tutorial
Kuma Meshes Part I - The basics - A tutorial
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
Deliver Latency Free Customer Experience
Deliver Latency Free Customer ExperienceDeliver Latency Free Customer Experience
Deliver Latency Free Customer Experience
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 

Python Compiler Internals Presentation Slides

  • 1. Python Compiler Internals Thomas Lee Shine Technologies, Melbourne
  • 2.
  • 3. What will you get out of this? Compiler development != rocket science. No magic: it's just code.
  • 4. Compiler? Isn't Python interpreted? Well, yes and no.
  • 5. WTF It ... It's Java?
  • 6. Aaanyway Let's screw with the compiler.
  • 7. Erm ... Before we start ... A patch to fix a quirk in the build process. This has been reported on the bug tracker!
  • 8. An overview of the compiler Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 9. New construct: the “unless” statement unless may_give_you_up: print “never gonna give you up...”
  • 10. Semantics of “unless” Works just like “if not” ... if not would_consider_letting_you_down: print “never gonna let you down...”
  • 11.
  • 12.
  • 13. What does this change affect? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 14. WTF is a “grammar”? Defines the syntactic structure of the language.
  • 15. Why a tokenizer? Makes parsing easier.
  • 16. So what does a tokenizer do, then? Generates a stream of events for the parser.
  • 17. What does the parser do? Organises tokens into the structure defined by the grammar. This is the “parse tree”.
  • 18.
  • 19. What's affected by this change? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 20. What's an AST? Similar to a parse tree, but not bound to the syntax “ Abstract Syntax Tree”
  • 21. Why does Python use an AST? Much easier to operate upon an AST than a parse tree.
  • 22. AST reuse New constructs can sometimes be expressed using existing AST nodes ...
  • 23. AST reuse ... so we could implement “unless” using “if” and “not” AST nodes. But then I wouldn't be able to show you the code generator ... Unless (test, body) = If ( Not (test), body)
  • 24.
  • 25. What's affected by this change? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 26.
  • 27. What about the rest of the compiler? Tokens Parse Tree AST Bytecode + Data AST + Symtable Tokenizer Parser AST Translation Bytecode Generation Bytecode Optimization Execution Symtable Construction
  • 28. “unless” in action Let's see if this stuff works ...
  • 29.