This document discusses using Pybind11 to create C extensions for Python. Pybind11 helps extend Python with C/C++ and provides functionality similar to CFFI. Examples are given of integrating NumPy and the Eigen linear algebra library. Testing C code with Google Test (GTEST) is recommended. GTEST example code is provided. Memory leaks should be checked with Valgrind. A case study of using Azure DevOps for continuous integration is also presented. In summary, Pybind11 allows mixing C/C++ with Python for efficiency reasons while leveraging Python libraries, and testing C code is important.

1. Development C-extension with using PyBind11
PyConJP 2022
Takayuki Suzuki
Shiseido, co, ltd,.

2. Table of contents
1. Introduction
2. What is Pybind11?
3. Pybind11 usage and integration with peripheral libraries
4. Test & Memory Check
5. DevOps Case Studies
6. Summary
1

3. Intro

4. Self Introduction
• Python Experience：about 10 years
• Working at Shiseido Co.
• Research and development of evaluation methods for skin images
2

5. Awareness of the problem
Organizations have cultures, and programming languages accompany cultures. Especially when
efficiency is required, there are historical assets, or for reasons of member skills. There are
cases where development in C/C++ is necessary.
On the other hand, we want to use Python because of its properties as a glue language and the
abundance of libraries.
What to do?
In this hour, I will introduce the libraries I have used to link Python and C and give examples of
how to test them!
• Source code is available on Github
• https://github.com/SzTk/pyconjp2022_dev_with_pybind11
3

6. How Python and the C language work together
How to call C from Python?
The official document says...
• How to include and build Python.h on the C language side (C extension)
• How to import ctypes on the Python side and call the library
• CFFI
Currently, CFFI seems to be the best choice(?).
PyBind11 basically provides functionality similar to CFFI → shared this time
4

7. What is Pybind11?

8. What is Pybind11?
Pybind11 is a library that helps Python extend C
Apparently derived from Boost.Python.
The machine learning library dlib also
uses it.
5

9. Quick Start
Examples of usage
C language side
https://pybind11.readthedocs.io/en/stable/basics.html
setup.py
6

10. Quick Start
Examples of usage
C language, so different data types will cause an error.
7

11. Pybind11 usage and integration
with peripheral libraries

12. Numby data type
Modules can also be created that use Numpy as input/output.
C language side
• Function that takes two np.arrays and
returns an element-wise summed np.array
• The returned numpy type data cannot be
created in standard C (of course), so it is
created by calling Python’s API.
PyBind11 provides a wrapper in
pybind11/numpy.h.
• There are multiple ways to access numpy
data. Here, we use the method of
accessing without range checks, etc.,
giving priority to efficiency.
8

13. Numby data type
Examples of usage
Naturally, if a type other than the specified type is received, an error (or unexpected behavior)
will occur.
9

14. Eigen
Eigen is a Header Only linear algebra library available in C++
Fast enough to be comparable to GOTO Blas
0
2000
4000
6000
8000
10000
12000
14000
16000
18000
20000
10 100 1000
MFLOPS
matrix size
matrix matrix product
’eigen3’
’GOTO2’
’INTEL_MKL’
’eigen2’
’ATLAS’
’gmm’
’ublas’
http://download.tuxfamily.org/eigen/btl-results-110323/axpy.pdf
It seems that the matrix-matrix multiplication is no different from GOTO2/INTEL MKL
10

15. Eigen is also available
C language side
setup.py
• Function that performs multiplication of
two matrices using Eigen and returns the
result
• In setup.py, specify the folder where you
downloaded eigen for include dirs
• The returned data is allocated memory
space on the C language side (unlike the
numpy example).
• The method of memory management
(whether to manage in C or to pass
management to Python) can be specified
by specifying return value policy.
11

16. Eigen Execution Example
12

17. Eigen Notes
• Efficient execution of matrix operations (not only
Eigen) requires attention to the order in which data
is stored in memory.
• Numpy arrays are C CONTIGUOUSS (=row-major)
by default
• Eigen’s default is column-major
• We need to either explicitly specify that Eigen also
receives RowMajor (which is the method employed
in the example above) or specify F CONTIGUOUSS
in numpy
13

18. Eigen Notes
In the above example, an error will occur if the array is passed after setting F CONTIGUOUSS
(because Raw Major = C CONTIGUOUSS in the argument type)
14

19. Linking to libraries
If the C language side depends on an external library (.dll for Windows, .so for Linux, etc.), the
dependent library should be listed in setup.py.
This example is linked with opencv in /usr/local/lib
If you distribute packages, you need to figure out how to distribute libraries as well
15

20. Test & Memory Check

21. Test
About the Test
When developing extensions in C or C++, Python’s GC does not manage memory areas
allocated in C modules.
It is necessary to check for memory leaks in the C language part.
I will share my method of checking.
16

22. GTEST
GTEST can be used for testing at the C language level.
https://github.com/google/googletest
GTEST is a library for testing the C++ language developed by Google
Note that testing at the Python level is done using nosetests.
17

23. GTEST Usage Example
My practice of GTEST
• Separate IF and C language parts with Python
• Test the C language part with GTEST
• Since it is C language, it is compiled (using CMake, etc.) to generate an executable binary
for testing.
• Python part tested with nosetests
18

24. GTEST Usage Example
Consider testing the matmul function used in eigen
Cut out only the matmul function to a separate file for testing
This has an intentional memory leak in it.
19

25. GTEST Usage Example
Write test code for GTEST
20

26. GTEST Usage Example 例
CMakeFile.txt
21

27. GTEST Usage Example
Build and Run
Execution Result
22

28. GTEST Usage Exampl
The Python IF part can be done as follows
• Write matmul is defined externally
• Add the file in which matmul is defined to setup.py
23

29. memory check
In addition to testing, memory leaks must also be checked.
• Valgrind is an OSS development tool for memory debugging, memory leak detection,
profiling, etc.
• This time, the test is done by passing the GTEST run command to valgrind.
24

30. memory check
• For memory leaks, just look at
LEAK SUMMARY.
• The leak summary points out
that 40 bytes of memory have
been forgotten to be deleted.
• should be reduced to zero,
including POSSIBLY
25

31. DevOps Case Studies

32. DevOps Pipeline
In our case, the skin analysis program is available for internal use.
To automate the process up to this point
In my case, I am using Azure DevOps to create a pipeline
After access management using Azure Active Directory It is useful to be able to publish
packages for internal use
26

33. Summary

34. Summary
Summary
• Depending on organizational culture and efficiency requirements, a mixture of C and
Python may be used for development
• Introduced how to use Pybind11 to link Python and C languages
• Practical methods of testing the C language portion of the program were introduced.
• Introduced a case study of DevOps use in our company.
Even today, there are still cultural areas where the C language is still alive.
I believe that working with other languages is also a cross-cultural exchange.
Why don’t you all try to touch the C extension of Python and have a cross-cultural exchange.
27

35. Questions?
27