An Insight into CPython compiler design

2,496 views

Published on

Describes the construction of the Python compiler by taking apart the Unladen Swallow codebase

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,496
On SlideShare
0
From Embeds
0
Number of Embeds
147
Actions
Shares
0
Downloads
57
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

An Insight into CPython compiler design

  1. 1. An Insight into CPython Compiler Design Ramkumar Ramachandra FOSS.IN/2009 03 December 2009 Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 1 / 12
  2. 2. Short Discussion of the CPython Compiler 1 Short Discussion of the CPython Compiler 2 A Gentle Introduction to LLVM 3 Enter: Unladen Swallow 4 Conclusion Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 2 / 12
  3. 3. Short Discussion of the CPython Compiler How Python is compiled Do the boring grammar parsing Compile the parse tree to bytecode Apply optimizations Interpret the bytecode Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 2 / 12
  4. 4. Short Discussion of the CPython Compiler The various stages of compilation PyAST FromNode() in Python/ast.c | Parse tree → AST PyAST Compile() in compile.c | AST → CFG → Bytecode PyAST Compile() calls PySymtable Build() and compiler mod() | AST → CFG assemble() | Post-order DFS | CFG → Bytecode Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 3 / 12
  5. 5. Short Discussion of the CPython Compiler What the final bytecode looks like a, b = 1, 0 if a or b : print " Hello " , a 1 0 LOAD_CONST 4 ((1 , 0) ) 3 UNPACK_SEQUEN CE 2 6 STORE_NAME 0 (a) 9 STORE_NAME 1 (b) 2 12 LOAD_NAME 0 (a) 15 JUMP_IF_TRUE 7 ( to 25) 18 POP_TOP 19 LOAD_NAME 1 (b) 22 JUMP_IF_FALSE 13 ( to 38) >> 25 POP_TOP 3 26 LOAD_CONST 2 ( ' Hello ' ) 29 PRINT_ITEM 30 LOAD_NAME 0 (a) 33 PRINT_ITEM 34 PRINT_NEWLINE 35 JUMP_FORWARD 1 ( to 39) >> 38 POP_TOP >> 39 LOAD_CONST 3 ( None ) 42 RETURN_VALUE Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 4 / 12
  6. 6. Short Discussion of the CPython Compiler Execute the bytecode 1 PyObject * Py E va l_ E va l Fr a m e E x ( PyFrameObject *f , int throwflag ) { 2 PyObject * result ; 3 result = PyEval_EvalFr a m e ( f ) ; 4 return result ; 5 } 1 PyObject * PyEval_EvalFra m e ( PyFrameObject * f ) 2 { 3 register PyObject ** stack_pointer ; /* Next free slot */ 4 register unsigned char * next_instr ; 5 register int opcode ; /* Current opcode */ 6 register int oparg ; /* Current opcode argument , if any */ 7 PyObject * retval = NULL ; /* Return value */ 8 PyCodeObject * co ; /* Code object */ 9 } Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 5 / 12
  7. 7. A Gentle Introduction to LLVM What is LLVM and why is it relevant? Compiler infrastructure Invents a new IR Replaces lower levels of GCC Provides static GCC-like compilation and JIT Python frontend possible Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 6 / 12
  8. 8. Enter: Unladen Swallow How Unladen Swallow started Objective: To speed up CPython Experiment with Psyco Temporarily use VMgen for eval loop Remove rarely used opcodes Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 7 / 12
  9. 9. Enter: Unladen Swallow Compile Python bytecode to LLVM IR 1 extern " C " _LlvmFunction * 2 _ P y C o d e _ T o L l v m I r ( PyCodeObject * code ) 3 { 4 _LlvmFunction * wrapper = new _LlvmFunction () ; 5 /* fbuilder functions in llvm_fbuilder . cc */ 6 wrapper - > lf_function = fbuilder . function () ; 7 return wrapper ; 8 } Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 8 / 12
  10. 10. Enter: Unladen Swallow Changes to the eval loop 1 static int 2 m a r k _ c a l l e d _ a n d _ m a y b e _ c o m p i l e ( PyCodeObject * co , PyFrameObject * f ) 3 { 4 co - > co_hotness += 10; 5 if ( co - > co_hotness > P Y _ H O T N E S S _ T H R E S H O L D ) { 6 if ( co - > co_llvm_func t i o n == NULL ) { 7 int t a r g e t _ o p t im i z a t i o n = 8 std :: max ( Py_DEFAULT_JIT_OPT_LEVEL , 9 Py_Optim iz eF l ag ) ; 10 if ( co - > co_optimiz at io n < t a r g e t _ o p t i m i z a t i o n ) { 11 // If the LLVM version of the function wasn ' t 12 // created yet , setting the optimization level 13 // will create it . 14 r = _ P y C o d e _ T o O p t i m i z e d L l v m I r ( co , t a r g e t _ o p t i m i z a t i o n ) ; 15 } 16 } 17 if ( co - > co _ na t iv e_ f u n c t i o n == NULL ) { 18 // Now try to JIT the IR function to machine code . 19 co - > c o _n a ti ve _ fu n c t i o n = 20 _ L lvm Fun cti on_J i t ( co - > c o_ l l v m _ f u n c t i o n ) ; 21 } 22 } 23 return 0; 24 } Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 9 / 12
  11. 11. Enter: Unladen Swallow Implement feedback-directed optimization Optimize native code, not bytecode Speed up builtin lookups/ inline simple builtins Don’t compile cold branches Inline simple operators using type feedback Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 10 / 12
  12. 12. Conclusion References [1] Abelson, H., Sussman, G. J., and Sussman, J. Structure and Interpretation of Computer Programs (SICP). The MIT Press, 1984. [2] Aho, A. V., Sethi, R., and Ullman, J. D. Compilers: Principles, Techniques, and Tools. Pearson Education, Inc, 2006. [3] Aycock, J. Compiling Little Languages in Python. [4] Cannon, B. Design of the CPython Compiler, 2005. [5] Ertl, M. A., Gregg, D., Krall, A., and Paysan, B. vmgen — a generator of efficient virtual machine interpreters. Software—Practice and Experience 32, 3 (2002), 265–294. [6] Montanaro, S. In A Peephole Optimizer for Python (1998). [7] Wang, D. C., Appel, A. W., Korn, J. L., and Serra, C. S. In The Zephyr Abstract Syntax Description Language (1997). Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 11 / 12
  13. 13. Conclusion Contact information Ramkumar Ramachandra artagnon@gmail.com http://artagnon.com Indian Institute of Technology, Kharagpur Presentation source available on http://github.com/artagnon/foss.in Ramkumar Ramachandra (FOSS.IN/2009) An Insight into CPython Compiler Design 03 December 2009 12 / 12

×