Pysmbc Python C Modules are Easy


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Hi everybody, I'm Roberto Polli from Babel and I'm going to tell you a python story. A story about how easy is to contribute to a python project even if you're a py-noob like me, if you just use all the tools that the FLOSS world gives you. Before starting I'd like to thank Babel – the proud sponsor of this talk. - for letting me to play with this project even after I've finished the job. Babel delivers large mail infrastructures for ISP, PA and GOV using and contributing to open source software. This story is just a drop in the sea of the various contribution of Babel to FLOSS.
  • People loves FLOSS – mostly because it's free as in beer. Many companies think that contributing is costly and useless. Or simply because they don't have the required knowledge to mess with the code. But today there are many technologies that should make them re-think about that, because the hardest part is to start. With social coding platform like GitHub, and a clean testing process it's easy to write working code and contribute it back. You don't have to be a guru to contribute!
  • People loves FLOSS – mostly because it's free as in beer. Many companies think that contributing is costly and useless. Or simply because they don't have the required knowledge to mess with the code. But today there are many technologies that should make them re-think about that, because the hardest part is to start. With social coding platform like GitHub, and a clean testing process it's easy to write working code and contribute it back. You don't have to be a guru to contribute!
  • So, what's GitHub? Is there somebody here that doesn't know GitHub? Raise your hand! The standard FLOSS development patter was based on check-out from main repository and send patches to mailing list. GitHub one is based on fork. [read the slide] Did you say fork???
  • Fork is – historically – the nightmare of floss maintainer. It fragments the user base, and parcellize development efforts. The 1 st image shows various forks. Merging back the patches is hard, and even applying fixes from the master branch to the forks. So fork childs are sparse writes, merge difficulties, lost changes. GitHub acts like the Linux IO Scheduler, helping us to merge writes! It's done tracking forks, and allowing to review, push and merge changes from forks to master. When I commit a possibily small change on my repo – eg a fix – I can push it to the master branch. The maintainer can validate it – eventually adding some more test – and the apply. At this point I just have to merge my branch with the master one – which contains now my patch!
  • Here are some GitHub features: [read slide]
  • GitHub was essential for our contribution. But first we had to understand how pysmbc work. Linux smbclient library implements almost all functionalities. Is continuously maintained by the Samba team. Writing a pure python implementation of SMB client protocol was redundant! Python has a nice feature: C Extension. It's a way to wrap existing C functions in a python module. Main advantages are mentioned in the slide, while a drawback is that the given module is platform-dependant. ipython is a python console supporting gnu-readline and many nice features. To use C Extension that wraps the just set the PYTHONPATH and # import smbc And enjoy all the features of smbc
  • Here is a code example of wrapping a simple factorial() C function. The C Extension is a C file named eg. wrapperfy.c The wrapper function is my_factorial(), that parses the arguments received from python, calls the wrapped factorial() function, and return a Py_Object – in this case is a Python_Long one. To associate the my_factorial() function to a python method we need a map – the MethodDef variable contained in wrapperfy.c. Once built wrapperfy.c into with gcc, we can play again with ipython: import _wrapper _wrapper.factorial(4)
  • Writing C Extension we should be careful to avod memory leaks and segmentation faults. All the memory management is up to you! A good starting point is to validate memory areas and variable content, adding new exceptions to our module and throwing them when issues could raise. This is done: properly setting the python error and stacktrace variables with PyErr_ functions – that are similar to the errno C variable; Returning NULL in the function. In our example, we created an Exception in the initialization function of the extention. Then we raise it using PyErr_SetString() in case of an invalid parameter in factorial()
  • Main sections of our wrapper modules: - wrapper functions - exceptions - function/method map - initialization function All functions should be static – statically mapped into the .so files and not usable outside that file. And – as I told above - you have to track memory usage!
  • We can even enjoy python code wrapping the extension in a module. The convention is to prepend an underscore “_” to the C extension library. becomes In this way we can create a module named “wrapper”. The slide shows the module directory structure. It's initialized by which includes: all the _wrapper methods; methods contained in further .py files. This could be a future asset of pysmbc, letting us to add further python methods to manage eg. xattr strings.
  • Now that we understood how C extensions and modules works, let's see how to get our job to be accepted by maintainers. Once you clone (checkout) your project, you have to setup your environment – in our case a Samba Server – in a way that all tests are successful. So check the credentials, the workgroup and all the environment! If you care about tests, the maintainer will care about your work! Running tests on pysmbc is easy. Just run #nosetests against the test directory. Nose will browse the directory and discover all your test cases. When all tests are ok, you can start developing!
  • Before coding, branch your code tree and name it with a meaningful target of your changes. I used ioggstream_setxattr, so that the maintainer could easily find who and what changes were involved. Branching let us preserve the original tree and merge the “master” with the new changes made on the maintainer repository. Writing tests before code is even simpler with Nose Test!
  • Does everybody knows something about unit testing? The standard python library for unit testing is python-unittest. It lets you write Test classes, with setup/teardown methods. Nose works in a similar way. With the nosetests script you can save some bytes and avoid writing main(). And you can even run unittest programs.
  • Nose simplifies test writing. In many cases you don't even need to include nose. Just write your test methods, beginning with test_. Moreover it supports basic fixtures like @SkipTest and @with_setup, to customize setup and teardown tests.
  • Nosetests script is quite flexible. You can run all tests, select a single file, class or even a single test. Everything works via command-line and is scriptable. By default nosetests silences tests writings to stdout. You can de-silence it with -s. And enable verbose output with -v.
  • Nose ensures that we're not going to break anything. The next step is to to write some test cases for the new functionalities. Writing tests let us focus on the expected input and output, and require us to read the wrapped library documentation. We started with the simpler one: get_xattr. T1 - check the existence of all the new constants T2 – test get_xattr with invalid parameters T3 – test get_xattr with valid parameters
  • setxattr is not user friendly. You have to create ACL strings by hand. The slide shows a simple ACL with a file accessible from two different groups. Python is really good at handling strings, so I started to work on a python class to manage ACL. To include it in pysmbc I had to reorganize pysmbc structure on my branch. Test classes are still the same, living in the same path, so I'm sure I'm not going to break anything. I'm currently discussing with the maintainer the new asset of the module.
  • That's all. Your questions are welcome! I leave just few links, if you're interested in contributing to PySmbC or just want to find some useful posts on Linux and Coding on Babel company blog vaunaspada.
  • Pysmbc Python C Modules are Easy

    1. 1. PySmbC:C Modules are EasyEuroPython 2012, 6th July - FirenzeBabel Srl P.zza S. Benedetto da Norcia, 33 0040, Pomezia (RM) – Roberto Polli -
    2. 2. What? Who? Why?A story about how easy is to contribute to a Python project usingGitHub and Nose testing framework.Roberto Polli - Community Manager @ Loves writing in C,Java and Python. Red Hat Certified Engineer.Fabio Isgrò – System Engineer @ Linux and Samba expert.Babel – Proud sponsor of this talk ;) Delivers large mail infrastructurebased on Open Source software for Italian ISP and PA. Contributesto various FLOSS. Roberto Polli -
    3. 3. Agenda - 1 Contributing to a Python C Extension is easier than thought+ GitHub allows a fast way to fork and merge patches to opensource projects+ Write and run Unit and Functional tests is easy with NoseTest = Dont have to be a Guru to support FLOSS! Roberto Polli -
    4. 4. Agenda - 2SMB is the protocol used by Windows for sharing folders. Samba is itsFLOSS implementation.smbclient library supports almost all SMB features. PySmbC wrapssome functions provided by this library.To support extended permissions (ACL) we wrapped two morefunctions: get_xattr and set_xattr. You dont need to know SMB to extend PySmbC Roberto Polli -
    5. 5. GitHub – social coding GitHub is a social coding platform.● Clone my repository and fork the project● Patch and commit to your repository● Push your changes to my repo● Discuss for approval● Merge my changes Did you say...fork? Roberto Polli -
    6. 6. GitHub – social codingForking is the nightmare of everyfloss maintainer.Sparse writes, increasing mergeefforts, lost changes.GitHub, like the I/O Scheduler,helps to merge writes! – Tracks forks; – Push changes to master. Roberto Polli -
    7. 7. GitHub – social codingPatch and commit to your repository● dont lose your changes;● track your own history.Push your changes to my repo● no more for .patch;● I get the change history.Discuss for approval● Github portal supports code annotations.Merge my changes● use git to merge from a remote repository! Roberto Polli -
    8. 8. Enter PySmbC - ExtensionsPython wrapper around libsmbclient: run C code from python – enjoy a stable C library; – just manage input, output and errors: – usually faster than a pure python implementation. rpolli$ ipython ln [1]: import smbc ln [2]: print smbc.XATTR_ACL system.nt_sec_desc.acl ln [3]: Roberto Polli -
    9. 9. Example - Wrapping factorial() - 1The wrapping function my_factorial(): // Python C Extension // uses factorial from fact.c● Parses and validates the input; #include <fact.h> wrapperfy.c● Calls the wrapped function(); // returns a python object! PyObject *my_factorial(...) { ...● Returns a python object. ret = factorial(n); ... return PyLong_asLong(ret); }A given structure maps python methods toC functions. // Maps _wrapper.factorial # python script // to my_factorial from _wrapper import factorialNow we can invoke a PyMethodDef BabelMethods[] = { {"factorial", my_factorial, ... }, print _wrapper.factorial(4)wrapped function! }; {NULL, NULL, 0, NULL} /*Sentinel*/ Roberto Polli -
    10. 10. Example - Wrapping factorial() - 2Parsing and validating Input and Output // returns a python object! PyObject *my_factorial(..., *args) {is fundamental. We dont want python   // NULL indicates an error   if (!PyArg_ParseTuple(args, "i", &n))to SEGFAULT!     return NULL;   // n! needs more than 8byte   if (n>21) {     ...     PyErr_SetString(FactError, Create new exceptions in the       “Bad value”);   }initialization function.   ...    return PyLong_asLong(ret); } PyObject *FactError;Throw exceptions in the function: // in extension initialization... ...● setting PyErr; init_wrapper(void) {   ...   // define an exception● returning NULL.   FactError =      PyErr_NewException("_wrapper.error",        NULL, NULL);   ... } Roberto Polli -
    11. 11. Example - Wrapping factorial() - 3 // Python C ExtensionC Extension components: #include <Python.h>● wrapping functions; // exceptions PyObject *FactError; PyObject *FiboError;● method/function map; // functions● exceptions; PyObject *my_factorial(...); PyObject *my_fibonacci(...);● initialization function. // Function/Method Maps  PyMethodDef BabelMethods[] = {   {"factorial",my_factorial,... },Functions and Exception should be static   {"fibonacci",my_fibonacci,... },   {NULL, NULL, 0, NULL} /*Sentinel*/ }; PyMODINIT_FUNC You have to track memory usage! init_wrapper(void) { PyObject *m; m = Py_InitModule("_wrapper", BabelMethods); // … Allocate Exceptions FactError = PyErr_NewException(...) FiboError = PyErr_NewException(...) } Roberto Polli -
    12. 12. Enters PySmbC - ModulesPython C extensionsmay enjoy both C andPython code.Wrap the C extension ina Python module.Extend the module withpython classes.$PYTHONPATH/ wrapper/ In [1]: import wrapper In [2]: assert wrapper.helpers In [3]: wrapper.helpers.is_integer(10) Roberto Polli -
    13. 13. Nose – lets contribute - 1Before adding features to PySmbC we checked the project status # git clone . # vim tests/ # set samba credential # nosetests test/NoseTest - a python script that auto-discovers and run test cases.Wraps python-unittest.Add new features only after successful tests. Verify your environment(eg. Samba credentials, directory acls ) Roberto Polli -
    14. 14. Nose – lets contribute - 2On successful tests, we can start developingFollow the Git way: create a separate branch. Well merge it onsuccess # git checkout -b ioggstream_setxattrWrite the tests before writing the code. Youll be more focused onyour targets With nosetest its simpler than ever! Roberto Polli -
    15. 15. Nose – is like UnitTestUnitTest Nosefrom unittest import TestCase, main import noseclass MyTest(UnitTest): class MyTest: def setUp(self): def setup(self): print”setup every” print ”setup” def tearDown(self): def teardown(self): print “teardown every” print “teardown” def test_do(self): def test_do(self): print “do 1” print “do 1”if __name__== “__main__”: # nose script will auto-discover main() # this script named Roberto Polli -
    16. 16. Nose – is simpler than UnitTestNose: simple test Nose: annotations# dont need to import nose from nose import SkipTest,with_setup# or define a class def pre(): print “setup”def setup(): def post(): print “teardown” print”setup once for all tests”def teardown(): @with_setup(pre,post) print “teardown once for all test” def test_do(): print “do 1”def test_do(): print “do 1” @SkipTestdef test_fail(): def test_dont(): assert False Print “not done yet” Roberto Polli -
    17. 17. Nose – InvocationYou can run your all tests in a given directory # nosetests ./path/Or just one file # nosetests ./path/test_sample.pyOr even a single test method # nosetests ./path/ suite, eventually setting the working directory ex1# nosetests ./path/ ex2# nosetests -w ./path test_class:TestOneFor a verbose output just use: #nosetests -sv [args] Roberto Polli -
    18. 18. PySmbC – add getxattrNose ensures that were not going to break # from def test_xattr_constants():anything. reuse variable defined in smbclient.h assert smbc.XATTR_ACL assert smbc.XATTR_OWNERStart writing tests, not coding functionalities. assert smbc.XATTR_GROUPYou can @SkipTest until new functions are def test_xattr_get():ready. test xattr with all possible values . . . for xa in valid_xatts:Play with the wrapped functions. assert ctx.getxattr(url, xa) def test_xattr_get_error():Start with the simpler one: getxattr() xattr_get should recognize bad values● embed C constants into python; . . . for xa in invalid_xatts:● test good values; try: ctx.getxattr(url, xa)● check behavior with bad values. assert False except RuntimeError as e: . . . #get errnoCode until tests are successful. assert errno == EINVAL Roberto Polli -
    19. 19. PySmbC – add setxattr and futuresHelper methods for parsing and creating # from test_context.pyACL def test_xattr_set():attrs_new = uREVISION:1 . . . ctx.setxattr(url, a_name, + ,OWNER:RPOLLIbabel attrs_new, REPLACE) + ,GROUP:Unix Groupbabel attrs_1 = ctx.getxattr(url, a_name) + ,ACL:RPOLLIbabel:0/0/0x001e01ff assert attrs_1 == attrs_new + ,ACL:Unix Groupbabel:0/0/0x00120089 + ,ACL:Unix Groupgames:0/0/0x001e01ff def test_xattr_set_error(): setxattr should + ,ACL:Everyone:0/0/0x00120089 recognize bad values . . . for xa in invalid_xatts:Shift from to smbc module: try: ctx.setxattr(url, a_name,● smbc/ xa, REPLACE) assert False● smbc/ except RuntimeError as e: . . . #get errno● smbc/ assert errno == EINVAL except TypeError pass Roberto Polli -
    20. 20. PySmbC Thank You! Roberto Polli -