Cython - Making Python as Fast as C

3,206 views

Published on

It introduces how to make Python as fast as C by Cython at Taipei.py [1], COSCUP 2014 [2], Tainan.py [3], Taipei.py 2015 [4], Kaohsiung Python Meetup [5].

[1]: http://www.meetup.com/Taipei-py/events/169077442/
[2]: http://coscup.org/2014
[3]: http://www.meetup.com/Tainan-py-Python-Tainan-User-Group/events/219957183/
[4]: http://www.meetup.com/Taipei-py/events/222038313/
[5]: http://www.meetup.com/Kaohsiung-Python-Meetup/events/229978032/

Published in: Software

Cython - Making Python as Fast as C

  1. 1. Cython — 
 Making Python as Fast as C Mosky
  2. 2. Mosky ➤ Python Charmer at Pinkoi ➤ has spoken at 10+ conferences ➤ TEDxNTUST 2015 ➤ PyCons in TW/JP/SG/HK ➤ etc. ➤ has taught Python for 100+ hours ➤ has serval Python packages ➤ MoSQL, Clime, etc. ➤ http://mosky.tw/ 2
  3. 3. Outline 1. Introduction 2. Setup 3. Foundation 4. Practicing 5. Tips 6. Uncovered Topics 7. Is Cython the best solution? 3
  4. 4. Introduction
  5. 5. Cython ➤ Cython is a source-to-source compiler (aka. transcompiler). ➤ Cython is a superset of Python. ➤ Provides optional static type declarations. ➤ Makes writing C extensions for Python easier. ➤ Makes Python program faster by pre-compiling and 
 static type. ➤ Sometimes faster by orders of magnitude 5
  6. 6. .pyx .c Cython C compiler import.so .py Python
  7. 7. Setup
  8. 8. Install C Compiler ➤ Mac: ➤ xcode-select --install ➤ Ubuntu / Debian: ➤ sudo apt-get install build-essential ➤ Other: ➤ http://docs.cython.org/src/quickstart/install.html 8
  9. 9. Install Cython ➤ Recommend to use PIP: ➤ sudo pip install cython ➤ Other: ➤ http://docs.cython.org/src/quickstart/install.html 9
  10. 10. The setup.py from distutils.core import setup from Cython.Build import cythonize setup(     name = 'cython-lab',     ext_modules = cythonize('*.pyx'), ) 10
  11. 11. The hello_cython.pyx print 'Hello Cython!' 11
  12. 12. Build ➤ Into package folder for development: ➤ python setup.py build_ext --inplace ➤ Into system for production: ➤ python setup.py install ➤ If clang: error: unknown argument: '-mno-fused-madd', ➤ export CFLAGS=-Qunused-arguments 12
  13. 13. Foundation
  14. 14. Define Static Types cdef int i, j, k cdef float f, g[42], *h 14
  15. 15. cdef struct Grail: int age float volume cdef union Food: char* spam float* eggs 
 cdef enum CheeseType: cheddar, edam, camembert cdef enum CheeseState: hard = 1 soft = 2 runny = 3 15
  16. 16. ctypedef unsigned long ULong ctypedef int* IntPtr 16
  17. 17. cdef struct Point: int x int y # either `struct` or `ctypedef` is not need cdef Point p 17
  18. 18. cdef: struct Point: int x int y Point p 18
  19. 19. Define Function def say_hello(name='World'): return 'Hello, %s!' % name cdef say_hello(name='World'): return 'Hello, %s!' % name 19
  20. 20. cdef say_hello(object name='World'): return 'Hello, %s!' % name cdef say_hello(char* name='World'): return 'Hello, %s!' % name 20
  21. 21. cdef int add(int a, int b): return a+b cpdef say_hello(char* name='World'): return 'Hello, %s!' % name 21
  22. 22. .pxd Exposes cdef Func # mylib.pxd cdef say_hello(char* name=?) # another.pyx from mylib cimport say_hello 22
  23. 23. Using C Lib from libc.math cimport sin # or cdef extern from "math.h": double sin(double x) 23
  24. 24. Function Visibility 24 Same 
 File Other .pyx .py Func in .h/.c Visible
 directly Visible
 via cdef extern Invisible cdef Visible
 via .pxd & cimport cpdef Visible
 via import def
  25. 25. Binding During 25 Same 
 File Other .pyx .py Func in .h/.c compile-time compile-time x cdef cpdef run-time def
  26. 26. Type Conversions 26 C From PY To PY [unsigned] 
 char/short int int/long int long unsigned int/long long [unsigned] long long
  27. 27. C From PY To PY float/double int/long/float float long double char* str/bytes struct dict
  28. 28. Practicing
  29. 29. Suggestions ➤ LIB_NAME.pyx ➤ has an execute_self_tests function ➤ test_LIB_NAME.py ➤ call the execute_self_tests function 29
  30. 30. Overflow ➤ Static types may also overflow in Cython silently. ➤ Try to make an overflow! ➤ Hint: ➤ http://j.mp/test_overflow_in_c_c ➤ Ans: ➤ http://j.mp/overflow_in_pyx_pyx 30
  31. 31. Functions ➤ Write three functions defined in 
 def, cdef, and cpdef. ➤ Try to call them in ➤ the same file, ➤ another .pyx file, ➤ and a .py file. ➤ Hints: ➤ Refer to the table, “Function Visibility”. ➤ http://j.mp/ lib_in_pyx_pyx ➤ Ans: ➤ http://j.mp/ use_lib_in_pyx_pyx ➤ http://j.mp/ test_lib_in_pyx_py 31
  32. 32. Using C Function ➤ Try to use the functions in C. ➤ Playing with fork, the system call, may be fun. ➤ Hint: ➤ http://j.mp/test_fork_c ➤ Ans: ➤ http://j.mp/fork_in_pyx_pyx 32
  33. 33. Tips
  34. 34. cython -a ➤ cython -a NAME.pyx ➤ open NAME.html ➤ Lines are colored according to the level of “typedness” – white lines translates to pure C without any Python API calls. 34
  35. 35. PYXIMPORT import pyximport; pyximport.install() import my_pyx_lib # compile .pyx into .so # or pyximport.install(pyimport=True) import my_py_lib # compile .py into .so 35
  36. 36. Uncovered Topics
  37. 37. Uncovered Topics ➤ Differences between C and Cython expressions ➤ http://docs.cython.org/src/userguide/ language_basics.html#differences-between-c-and-cython- expressions ➤ Propagating Exceptions in cdef ➤ http://docs.cython.org/src/userguide/ language_basics.html#error-return-values 37
  38. 38. ➤ Extension Type — cdef class ➤ http://docs.cython.org/src/userguide/ extension_types.html ➤ Generic programming using Cython's Template ➤ http://docs.cython.org/src/userguide/fusedtypes.html ➤ Conditional Compilation ➤ http://docs.cython.org/src/userguide/ language_basics.html#conditional-compilation 38
  39. 39. ➤ Profiling ➤ http://docs.cython.org/src/tutorial/profiling_tutorial.html ➤ Parallelism (No GIL + OpenMP) ➤ http://docs.cython.org/src/userguide/parallelism.html ➤ Using C++ in Cython ➤ http://docs.cython.org/src/userguide/ wrapping_CPlusPlus.html 39
  40. 40. Is Cython 
 the best solution?
  41. 41. PYTHON 
 COMMUNITY 
 IS 
 DORAEMON!
  42. 42. Other Solutions ➤ Boost.Python — exposes C++ to Python ➤ Numba — compiles annotated code into LLVM by JIT compiler ➤ PyPy — speeds up existent code by JIT compiler ➤ NumPy or Blaze — provides efficient array ➤ SciPy — provides fast scientific computing 42
  43. 43. Cool Down ➤ Algorithm still does matter in any case. ➤ Profile your program. ➤ Consider the portability — you are writing C program! ➤ Consider the improvement is enough or not. ➤ Then pick the most suitable tools. 43
  44. 44. Ending
  45. 45. Ending ➤ cdef ➤ static types ➤ functions ➤ extern for C functions ➤ .pxd exposes cdef functions ➤ Tips ➤ mosky.tw ➤ Any question? 45

×