Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Cluj.py Meetup: Extending Python in C

893 views

Published on

Slides for the Cluj.py meetup where we explored the inner workings of CPython, the reference implementation of Python. Includes examples of writing a C extension to Python, and introduces Cython - ultimately the sanest way of writing C extensions.

Also check out the code samples on GitHub: https://github.com/trustyou/meetups/tree/master/python-c

Published in: Technology
  • Be the first to comment

Cluj.py Meetup: Extending Python in C

  1. 1. Extending Python in C Cluj.py meetup, Nov 19th Steffen Wenz, CTO TrustYou
  2. 2. Goals of today’s talk ● Look behind the scenes of the CPython interpreter - gain insights into “how Python works” ● Explore the CPython C API ● Build a Python extension in C ● Introduction to Cython
  3. 3. Who are we? ● For each hotel on the planet, provide a summary of all reviews ● Expertise: ○ NLP ○ Machine Learning ○ Big Data ● Clients: …
  4. 4. TrustYou Tech Stack Batch Layer ● Hadoop (HDP 2.1) ● Python ● Pig ● Luigi Service Layer ● PostgreSQL ● MongoDB ● Redis ● Cassandra Data Data Queries Hadoop cluster (100 nodes) Application machines
  5. 5. Let’s dive in! Assigning an integer a = 4 PyObject* a = PyInt_FromLong(4); // what's the difference to int a = 4? Documentation: PyInt_FromLong
  6. 6. List item access x = xs[i] PyObject* x = PyList_GetItem(xs, i); Documentation: PyList_GetItem
  7. 7. Returning None … return None Py_INCREF(Py_None); return Py_None; Documentation: Py_INCREF
  8. 8. Calling a function foo(1337, "bar") // argument list PyObject *args = Py_BuildValue ("is", 1337, "bar"); // make call PyObject_CallObject(foo, args); // release arguments Py_DECREF(args); Documentation: Py_BuildValue, PyObject_CallObject
  9. 9. What’s the CPython C API? ● API to manipulate Python objects, and interact with Python code, from C/C++ ● Purpose: Extend Python with new modules/types ● Why?
  10. 10. CPython internals def slangify(s): return s + ", yo!" C API Compiler Interpreter >>> slangify("hey") 'hey, yo!' |x00x00dx01x00x17S Not true for Jython, IronPython, PyPy, Stackless …
  11. 11. Why is Python slow? a = 1 a = a + 1 int a = 1; a++;
  12. 12. Why is Python slow? class Point: def __init__(self, x, y): self.x = x; self.y = y p = Point(1, 2) print p.x typedef struct { int x, y; } point; int main() { point p = {1, 2}; printf("%i", p.x); }
  13. 13. Why is Python slow? The GIL
  14. 14. Writing in C ● No OOP : typedef struct { /* ... */ } complexType; void fun(complexType* obj, int x, char* y) { // ... } ● Macros for code generation: #define SWAP(x,y) {int tmp = x; x = y; y = tmp;} SWAP(a, b);
  15. 15. Writing in C ● Manual memory management: ○ C: static, stack, malloc/free ○ Python C API: Reference counting ● No exceptions ○ Error handling via returning values ○ CPython: return null; signals an error
  16. 16. Reference Counting void Py_INCREF(PyObject *o) Increment the reference count for object o. The object must not be NULL; if you aren’t sure that it isn’t NULL, use Py_XINCREF(). = I want to hold on to this object and use it again after a while* *) any interaction with Python interpreter that may invalidate my reference void Py_DECREF(PyObject *o) Decrement the reference count for object o. The object must not be NULL; if you aren’t sure that it isn’t NULL, use Py_XDECREF(). If the reference count reaches zero, the object’s type’s deallocation function (which must not beNULL) is invoked. = I’m done, and don’t care if the object is discarded at this call See documentation
  17. 17. Anatomy of a refcount bug void buggy(PyObject *list) { PyObject *item = PyList_GetItem(list, 0); // borrowed ref. PyList_SetItem(list, 1, PyInt_FromLong(0L)); // calls destructor of previous element PyObject_Print(item, stdout, 0); // BUG! }
  18. 18. Our First Extension Module
  19. 19. Adding integers in C >>> import arithmetic >>> arithmetic.add(1, 1337) 1338
  20. 20. #include <Python.h> static PyObject* arithmetic_add(PyObject* self, PyObject* args) { int i, j; PyArg_ParseTuple(args, "ii", &i, &j); PyObject* sum = PyInt_FromLong(i + j); return sum; }
  21. 21. static PyObject* arithmetic_add(PyObject* self, PyObject* args) { int i, j; PyObject* sum = NULL; if (!PyArg_ParseTuple(args, "ii", &i, &j)) goto error; sum = PyInt_FromLong(i + j); if (sum == NULL) goto error; return sum; error: Py_XDECREF(sum); return NULL; }
  22. 22. Boilerplate² static PyMethodDef ArithmeticMethods[] = { {"add", arithmetic_add, METH_VARARGS, "Add two integers."}, {NULL, NULL, 0, NULL} // sentinel }; PyMODINIT_FUNC initarithmetic(void) { (void) Py_InitModule("arithmetic", ArithmeticMethods); }
  23. 23. … and build your module from distutils.core import setup, Extension module = Extension("arithmetic", sources=["arithmeticmodule.c"]) setup( name="Arithmetic", version="1.0", ext_modules=[module] )
  24. 24. $ sudo python setup.py install # build with gcc, any compiler errors & warnings are shown here $ python >>> import arithmetic >>> arithmetic <module 'arithmetic' from '/usr/local/lib/python2.7/dist-packages/ arithmetic.so'> >>> arithmetic.add <built-in function add> >>> arithmetic.add(1, "1337") Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: an integer is required
  25. 25. Why on earth would I do that?
  26. 26. Why go through all this trouble? ● Performance ○ C extensions & Cython optimize CPU-bound code (vs. memory-bound, IO-bound) ○ Pareto principle: 20% of the code responsible for 80% of the runtime ● Also: Interfacing with existing C/C++ code
  27. 27. Is my Python code performance-critical? import cProfile, pstats, sys pr = cProfile.Profile() pr.enable() setup() # run code you want to profile pr.disable() stats = pstats.Stats(pr, stream=sys.stdout).sort_stats("time") stats.print_stats()
  28. 28. 55705589 function calls (55688041 primitive calls) in 69.216 seconds Ordered by: internal time ncalls tottime percall cumtime percall filename:lineno(function) 45413 21.856 0.000 21.856 0.000 {method 'get' of 'pytc.HDB' objects} 32275 9.490 0.000 9.656 0.000 /usr/local/lib/python2.7/dist-packages/simplejson/decoder.py:376(raw_decode) 18760 6.403 0.000 12.797 0.001 /home/steffen/apps/group/lib/util/timeseries.py:29(reindex_pad) 56992 2.586 0.000 2.624 0.000 {sorted} 1383832 2.244 0.000 2.244 0.000 /home/steffen/apps/group/lib/hotel/index.py:231(<lambda>) 2708692 1.845 0.000 5.657 0.000 /home/steffen/apps/group/lib/hotel/index.py:21(<genexpr>) 497989 1.718 0.000 2.456 0.000 {_heapq.heapreplace} 4734466 1.624 0.000 2.491 0.000 /home/steffen/apps/group/lib/util/timeseries.py:43(<genexpr>) 346738 1.475 0.000 1.475 0.000 /usr/lib/python2.7/json/decoder.py:371(raw_decode) 510726 1.354 0.000 10.432 0.000 /usr/lib/python2.7/heapq.py:357(merge) 2691966 1.310 0.000 1.310 0.000 /home/steffen/apps/group/lib/util/timeseries.py:21(float_parse) 357260 1.160 0.000 5.122 0.000 /home/steffen/apps/group/lib/hotel/index.py:471(<genexpr>) 5348564 0.912 0.000 0.912 0.000 /home/steffen/apps/group/lib/util/timeseries.py:90(<genexpr>) 758026 0.882 0.000 0.882 0.000 {method 'match' of '_sre.SRE_Pattern' objects} 9470443 0.868 0.000 0.868 0.000 {method 'append' of 'list' objects} 4715746 0.867 0.000 0.867 0.000 /home/steffen/apps/group/lib/util/timeseries.py:31(bound) 1 0.857 0.857 69.220 69.220 /home/steffen/apps/group/lib/pages/table_page.py:37(calculate) 644766 0.839 0.000 1.752 0.000 {sum}
  29. 29. You can’t observe without changing … import timeit def setup(): pass def stmt(): pass print timeit.timeit(stmt=stmt, setup=setup, number=100)
  30. 30. Example: QuickSort
  31. 31. Pythonic QuickSort def quicksort(xs): if len(xs) <= 1: return xs middle = len(xs) / 2 pivot = xs[middle] del xs[middle] left, right = [], [] for x in xs: append_to = left if x < pivot else right append_to.append(x) return quicksort(left) + [pivot] + quicksort(right)
  32. 32. Results: Python vs. C extension Pythonic QuickSort: 2.0s C extension module: 0.092s
  33. 33. Cython
  34. 34. Adding integers in Cython # add.pyx def add(i, j): return i + j # main.py import pyximport; pyximport.install() import add if __name__ == "__main__": print add.add(1, 1337)
  35. 35. What is Cython? ● Compiles Python to C code ● “Superset” of Python: Accepts type annotations to compile more efficient code (optional!) cdef int i = 2 ● No reference counting, error handling, boilerplate … plus nicer compiling workflows
  36. 36. Results: Pythonic QuickSort: 2.0s C extension module: 0.092s Cython QuickSort (unchanged): 0.82s
  37. 37. cdef partition(xs, int left, int right, int pivot_index): cdef int pivot = xs[pivot_index] cdef int el xs[pivot_index], xs[right] = xs[right], xs[pivot_index] pivot_index = left for i in xrange(left, right): el = xs[i] if el <= pivot: xs[i], xs[pivot_index] = xs[pivot_index], xs[i] pivot_index += 1 xs[pivot_index], xs[right] = xs[right], xs[pivot_index] return pivot_index def quicksort(xs, left=0, right=None): if right is None: right = len(xs) - 1 if left < right: middle = (left + right) / 2 pivot_index = partition(xs, left, right, middle) quicksort(xs, left, pivot_index - 1) quicksort(xs, pivot_index + 1, right)
  38. 38. Results: Pythonic QuickSort: 2.0s C extension module: 0.092s Cython QuickSort (unchanged): 0.82s Cython QuickSort (C-like): 0.37s ● Unscientific result. Cython can be faster than hand-written C extensions!
  39. 39. Further Reading on Cython See code samples on ● O’Reilly Book TrustYou GitHub account: https://github. com/trustyou/meetups/tre e/master/python-c
  40. 40. TrustYou wants you! We offer positions in Cluj & Munich: ● Data engineer ● Application developer ● Crawling engineer Write me at swenz@trustyou.net, check out our website, or see you at the next meetup!
  41. 41. Thank you!
  42. 42. Python Bytecode >>> def slangify(s): ... return s + ", yo!" ... >>> slangify.func_code.co_code '|x00x00dx01x00x17S' >>> import dis >>> dis.dis(slangify) 2 0 LOAD_FAST 0 (s) 3 LOAD_CONST 1 (', yo!') 6 BINARY_ADD 7 RETURN_VALUE
  43. 43. Anatomy of a memory leak void buggier() { PyObject *lst = PyList_New(10); return Py_BuildValue("Oi", lst, 10); // increments refcount } // read the doc carefully before using *any* C API function

×