- Understanding the CPython implementation - Hacking Python Grammar - Parsing an LL(1) Grammar - Hacking the disassembler - Hacking the AST - Building a Flake8 plugin - Understanding CPython internals - Pycon Germany, Karlsruhe, 2017 - Compiler theory

1. Hacking Python ASTs
@jargnar
suhas.org

2. Abstract Syntax Trees
Intermediate Representation (IR)

3. Abstract Syntax Trees
Intermediate Representation (IR)
A data structure to represent source code

4. Abstract Syntax Trees
Intermediate Representation (IR)
A data structure to represent source code
So, code as a data structure?

5. Metaprogramming

6. Metaprogramming
Data for traditional programs: Files, network packets, …

7. Metaprogramming
Data for traditional programs: Files, network packets, …
Data for meta programs: Source code

8. It’s just useful to think in data structures

9. First, the obvious use-cases

10. Can you guess what this code does?
def F(n):
A = [0] * (n+1)
for m in range(n+1):
A[m] = Fraction(1, m+1)
for j in range(m, 0, -1):
A[j-1] = j*(A[j-1] - A[j])
return A[0]

11. The Nth
Bernoulli Number
def F(n):
A = [0] * (n+1)
for m in range(n+1):
A[m] = Fraction(1, m+1)
for j in range(m, 0, -1):
A[j-1] = j*(A[j-1] - A[j])
return A[0]
# 1, ½, ⅙, …

12. But what’s wrong with it?
def F(n):
A = [0] * (n+1)
for m in range(n+1):
A[m] = Fraction(1, m+1)
for j in range(m, 0, -1):
A[j-1] = j*(A[j-1] - A[j])
return A[0]
# 1, ½, ⅙, …

13. But what’s wrong with it?
Hint: It’s not the logic
def F(n):
A = [0] * (n+1)
for m in range(n+1):
A[m] = Fraction(1, m+1)
for j in range(m, 0, -1):
A[j-1] = j*(A[j-1] - A[j])
return A[0]
# 1, ½, ⅙, …

14. But what’s wrong with it?
def F(n):
A = [0] * (n+1)
for m in range(n+1):
A[m] = Fraction(1, m+1)
for j in range(m, 0, -1):
A[j-1] = j*(A[j-1] - A[j])
return A[0]
# 1, ½, ⅙, …
The function name
feels a bit
disrespectful,
doesn’t it?

15. $ pip install
flake8-disrespectful-code

16. The bigger picture

17. A brief primer on compilers
Front end Middle end Back end
Scala,
Python
IR IR
ARM, x86

18. Book recommendation

19. CPython
Let’s take a quick look at the source

20. The Design of the CPython Compiler
Python/
ast.c
Parser/
pgen.c
Python/
compile.c
Python/
compile.c
Parse
Tree
AST CFG
Bytecode
example.py
Grammar
Python/
ceval.c
x86, ARM

21. The Design of the CPython Compiler
Python/
ast.c
Parser/
pgen.c
Python/
compile.c
Python/
compile.c
Parse
Tree
AST CFG
Bytecode
example.py
Grammar
Python/
ceval.c
x86, ARM

22. Let’s parse a grammar
Now, we parse
S
AB S -> AB
aAB A -> aA
aaAB A -> aA
aaaAB A -> aA
aaaεB A -> ε
aaab B -> b
First, we formally define a grammar
S -> AB
A -> aA | ε
B -> b | bB
Does aaab belong to this grammar?

23. LL(1) Grammar

24. Grammar/Grammar

25. The Design of the CPython Compiler
Python/
ast.c
Parser/
pgen.c
Python/
compile.c
Python/
compile.c
Parse
Tree
AST CFG
Bytecode
example.py
Grammar
Python/
ceval.c
x86, ARM

26. dis

27. The Design of the CPython Compiler
Python/
ast.c
Parser/
pgen.c
Python/
compile.c
Python/
compile.c
Parse
Tree
AST CFG
Bytecode
example.py
Grammar
Python/
ceval.c
x86, ARM

28. ast

29. ast.NodeVisitor

30. ast.NodeTransformer

31. Flake8

32. $ pip install
flake8-disrespectful-code
https://github.com/jargnar/flake8-disrespectful-code

33. twitter.com/jargnar
github.com/jargnar
suhas.org