Testing Apache Modules with Python
and ctypes
ApacheCon US 2009

Markus Litz   - 06.11.2009


                            ...
Agenda for today

  Why?


  Introduction to ctypes


  Preparing the apache


  Creating tests


  Demo

                ...
DLR
German Aerospace Center




   Research Institution
   Research Areas
      Aeronautics
      Space
      Transport
  ...
Locations and employees

6200 employees across                 Hamburg 
29 research institutes and                       ...
Background

 DataFinder – a application for scientific data management
     Storing and managing huge amounts of data
    ...
Catacomb – A WebDAV Server Module for
Apache




                                                           Slide 6
      ...
Catacomb – The Difference to mod_dav_fs

    Saving the resources
        mod_dav_fs save content and properties in files ...
Catacomb – A WebDAV Server Module for
Apache
  WebDAV repository module for mod_dav

  Catacomb uses relational databases ...
Catacomb – History and Current State

   Initial development at the University of California under the
  chair of Jim Whit...
Why testing your code?

  Development is faster and easier


  Code is more robust


  Code is more maintainable


  Code ...
Why testing with Python and ctypes?

  Writing tests is easy


  No need to start an apache instance every time


  Tests ...
What is ctypes

  ctypes is a wrapper for C-librarys for python


  ctypes allows to call functions in dlls/shared librari...
How to use ctypes

  from ctypes import *


  Loading dynamic link libraries
     libc = cdll.msvcr
     libc = CDLL("libc...
Fundamental data types

  Good support for many primitive C compatible data
 types:


      C                 Python
     ...
Fundamental data types - usage

All these types can be created by calling them with an optional initializer
of the correct...
Using pointers

  byref() passes parameters by reference
     libc.sscanf("1 3.14 Hello", "%d %f
               %s", byref...
Return types

  Default return type: int

  strcat = libc.strcat
  strcat("abc", "def"))       # „8059983“


  strcat.rest...
Arrays
  Create an array-type
     TenIntsArrayType = c_int * 10


  Create an array-instance
     array1 = TenIntegers()
...
Structures and unions


  class POINT(Structure):
     _fields_ = [("x", c_int),
                 ("y", c_int)]

  point =...
UnitTesting Apache Modules

  The problem
      (Most) functions of a module could only be tested with
     a running apac...
Calling module functions directly

  Causes a exception stops execution
     On runtime, ctypes tries to resolve all dynam...
Building-kernel apache as a share core

  Building the apache kernel as shared module
     On apache 1.x
         --enable...
Compiling Apache

  Compiling apache

    make clean
    CFLAGS='-D SHARED_CORE -fPIC '
   ./configure
    make




      ...
Linking the Shared Core

  After compiling, the make command links apache
      libtool ... -mode=link gcc ... -o httpd
  ...
Modifications of the Module

  Module must be linked against the shared core

     LDFLAGS = -lhttpd -L </…/libhttpd.so>

...
Apache Data Structures in Python

class apr_allocator_t(Structure):

class apr_memnode_t(Structure):

class apr_pool_t(Str...
Setting Up Data Structures – apt_pool_t
class apr_pool_t(Structure):
    _fields_ = [("cleanups",POINTER(cleanup_t)),
    ...
Setting Up Data Structures – GCC

  Ctypes code generator – modified version of GCC

  Looks for declarations in C header ...
Unit Test Framwork (nose)

  Simple structure, one class for each testing object

     Setup_class()
     Test1()
     …
 ...
Setting up the Test Environment
def setup (self) :
    self.catacomb = CDLL("/apachecon/libmod_dav_repos.so")
    self.htt...
Writing the Test

def testSomething(self):
   assert self.catacomb.function_to_test(arg1,
                     byref(arg2)...
Shutting down the Test Environment


def teardown(self):
    self.apr.apr_pool_destroy(self.pool)
    self.apr.apr_allocat...
Summary of Steps

  Compile Apache as a shared core


  Link own module against shared core


  Define the data structures...
Conclusion

  Powerful possibility to create tests with no need of a
 running Apache.
   Tests could be made in an easy la...
Demonstration

  Before the demo:


     Thanks to Steven Mohr




                                                       ...
Upcoming SlideShare
Loading in …5
×

Testing Apache Modules with Python and Ctypes

3,982 views

Published on

Writing tests for your Apache module is often a developer's least favourite task, especially if you don't like using the Perl Apache Testing Framework! One of the main reasons for this is that it's difficult to test your C code on-the-fly without a running Apache server. Using ctypes, you can test your modules without a running httpd, just by writing and using simple Python scripts! In this talk we will look at how to compile a specific version of the Apache webserver for testing, and how to use a common Python unit testing framework to write and set up tests. We will also cover examples of writing simple test cases for the Catacomb WebDAV Server module.

Talk from Apachecon US 2009

Published in: Technology
  • Be the first to comment

Testing Apache Modules with Python and Ctypes

  1. 1. Testing Apache Modules with Python and ctypes ApacheCon US 2009 Markus Litz - 06.11.2009 Slide 1 Standard presentation deck > Sep. 2009
  2. 2. Agenda for today Why? Introduction to ctypes Preparing the apache Creating tests Demo Slide 2 Standard presentation deck > Sep. 2009
  3. 3. DLR German Aerospace Center Research Institution Research Areas Aeronautics Space Transport Energy Space Agency Slide 3 Standard presentation deck > Sep. 2009
  4. 4. Locations and employees 6200 employees across Hamburg  29 research institutes and  Neustrelitz Bremen   Trauen facilities at Berlin   13 sites. Braunschweig   Dortmund  Goettingen Offices in Brussels,  Koeln Paris and Washington.  Bonn Lampoldshausen  Stuttgart   Oberpfaffenhofen Weilheim  Slide 4 Standard presentation deck > Sep. 2009
  5. 5. Background DataFinder – a application for scientific data management Storing and managing huge amounts of data Search through the resource content and metadata Various ways to store data, for example ftp, network share, offline stores Metadata management with the WebDAV protocol Two supported WebDAV Server: Tamino XML Server & Catacomb Slide 5 Standard presentation deck > Sep. 2009
  6. 6. Catacomb – A WebDAV Server Module for Apache Slide 6 Standard presentation deck > Sep. 2009
  7. 7. Catacomb – The Difference to mod_dav_fs Saving the resources mod_dav_fs save content and properties in files on the filesystem mod_dav_fs creates for every resource, and also for every collection, their own property file Consequence: A single query of server side searching needs to open many files Implementation of complex queries is difficult Full text search is expensive Slide 7 Standard presentation deck > Sep. 2009
  8. 8. Catacomb – A WebDAV Server Module for Apache WebDAV repository module for mod_dav Catacomb uses relational databases to store the metadata Strong search performance through SQL statements Catacomb is: Good for Content management Good for Collaborated web authoring Support locks, avoid the “lost update” problem Capable of searching (DASL) and versioning (DeltaV) resources Slide 8 Standard presentation deck > Sep. 2009
  9. 9. Catacomb – History and Current State Initial development at the University of California under the chair of Jim Whitehead Open Source project since 2002 DeltaV and DASL implementation Since 2006 contribution of the DLR ACP support Database abstraction using mod_dbd License changed to ASL2.0 Slide 9 Standard presentation deck > Sep. 2009
  10. 10. Why testing your code? Development is faster and easier Code is more robust Code is more maintainable Code is more reliable Slide 10 Standard presentation deck > Sep. 2009
  11. 11. Why testing with Python and ctypes? Writing tests is easy No need to start an apache instance every time Tests could be automatically done with various Apache versions Slide 11 Standard presentation deck > Sep. 2009
  12. 12. What is ctypes ctypes is a wrapper for C-librarys for python ctypes allows to call functions in dlls/shared libraries from python code It is possible to implement C callback function Since Python 2.5.x, ctypes is in the standard library Slide 12 Standard presentation deck > Sep. 2009
  13. 13. How to use ctypes from ctypes import * Loading dynamic link libraries libc = cdll.msvcr libc = CDLL("libc.so.6") Calling functions print libc.time(None) Slide 13 Standard presentation deck > Sep. 2009
  14. 14. Fundamental data types Good support for many primitive C compatible data types: C Python char  c_char int  c_int long  c_long void*  c_void_p Slide 14 Standard presentation deck > Sep. 2009
  15. 15. Fundamental data types - usage All these types can be created by calling them with an optional initializer of the correct type and value: i = c_int(42) print i.value # „42“ i.value = -1 print i.value # „-1“ num = c_double(3.14) libc.printf("Number: %fn“, num) # „Numner: 3.14“ Slide 15 Standard presentation deck > Sep. 2009
  16. 16. Using pointers byref() passes parameters by reference libc.sscanf("1 3.14 Hello", "%d %f %s", byref(i), byref(f), s) Creating a pointer i = c_int(42) pi = pointer(i) Slide 16 Standard presentation deck > Sep. 2009
  17. 17. Return types Default return type: int strcat = libc.strcat strcat("abc", "def")) # „8059983“ strcat.restype = c_char_p strcat("abc", "def")) # „abcdef“ Slide 17 Standard presentation deck > Sep. 2009
  18. 18. Arrays Create an array-type TenIntsArrayType = c_int * 10 Create an array-instance array1 = TenIntegers() array2 = TenIntegers(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) Using arrays Array1[3]  “0” Array2[3]  “4” Slide 18 Standard presentation deck > Sep. 2009
  19. 19. Structures and unions class POINT(Structure): _fields_ = [("x", c_int), ("y", c_int)] point = POINT(10, 20) print point.x, point.y  „10 20“ Slide 19 Standard presentation deck > Sep. 2009
  20. 20. UnitTesting Apache Modules The problem (Most) functions of a module could only be tested with a running apache Module-functions could not be called directly The solutions Starting and stopping an apache on each test Test functions from the module directly using ctypes Slide 20 Standard presentation deck > Sep. 2009
  21. 21. Calling module functions directly Causes a exception stops execution On runtime, ctypes tries to resolve all dynamic symbols All apache specific methods and data structures are not available Solution: Building Apache as a shared core Slide 21 Standard presentation deck > Sep. 2009
  22. 22. Building-kernel apache as a share core Building the apache kernel as shared module On apache 1.x --enable-rule=SHARED_CORE On apache 2.x build infrastructure doesn't seem to know this anymore Slide 22 Standard presentation deck > Sep. 2009
  23. 23. Compiling Apache Compiling apache make clean CFLAGS='-D SHARED_CORE -fPIC ' ./configure make Slide 23 Standard presentation deck > Sep. 2009
  24. 24. Linking the Shared Core After compiling, the make command links apache libtool ... -mode=link gcc ... -o httpd .. Linking command for a shared core libtool ... -mode=link gcc ... -shared -o libhttpd.so ..server/exports.o Slide 24 Standard presentation deck > Sep. 2009
  25. 25. Modifications of the Module Module must be linked against the shared core LDFLAGS = -lhttpd -L </…/libhttpd.so> Could be an extra make-target Slide 25 Standard presentation deck > Sep. 2009
  26. 26. Apache Data Structures in Python class apr_allocator_t(Structure): class apr_memnode_t(Structure): class apr_pool_t(Structure): class cleanup_t(Structure): Slide 26 Standard presentation deck > Sep. 2009
  27. 27. Setting Up Data Structures – apt_pool_t class apr_pool_t(Structure): _fields_ = [("cleanups",POINTER(cleanup_t)), ("free_cleanups",POINTER(cleanup_t)), ("allocator",POINTER(apr_allocator_t)), ("subprocesses",POINTER(process_chain)), ("abort_fn",c_void_p), ("user_data",c_void_p), ("tag",c_char_p), ("active",POINTER(apr_memnode_t)), ("self",POINTER(apr_memnode_t)), ("self_first_avail",c_char_p), ("parent",POINTER(apr_pool_t)), ("child",POINTER(apr_pool_t)), ("sibling",POINTER(apr_pool_t)), ("ref",POINTER(POINTER(apr_pool_t)))] Slide 27 Standard presentation deck > Sep. 2009
  28. 28. Setting Up Data Structures – GCC Ctypes code generator – modified version of GCC Looks for declarations in C header files. Generates python codes for: enums, structs, unions, function declarations, com interfaces, and preprocessor definitions Very early stage Slide 28 Standard presentation deck > Sep. 2009
  29. 29. Unit Test Framwork (nose) Simple structure, one class for each testing object Setup_class() Test1() … TestX() TearDown_class() Slide 29 Standard presentation deck > Sep. 2009
  30. 30. Setting up the Test Environment def setup (self) : self.catacomb = CDLL("/apachecon/libmod_dav_repos.so") self.httpd = CDLL("/apachecon/libhttpd.so") self.apr = CDLL("/apachecon/lib/libapr-1.so") self.pool = c_void_p() self.allocator = c_void_p() self.apr.apr_initialize() self.apr.apr_allocator_create(byref(self.allocator)) self.apr.apr_pool_create_ex(byref(self.pool), None, None, self.allocator) Slide 30 Standard presentation deck > Sep. 2009
  31. 31. Writing the Test def testSomething(self): assert self.catacomb.function_to_test(arg1, byref(arg2)) == “true” Slide 31 Standard presentation deck > Sep. 2009
  32. 32. Shutting down the Test Environment def teardown(self): self.apr.apr_pool_destroy(self.pool) self.apr.apr_allocator_destroy(self.allocator) self.apr.apr_terminate() Slide 32 Standard presentation deck > Sep. 2009
  33. 33. Summary of Steps Compile Apache as a shared core Link own module against shared core Define the data structures you need Write the tests Run the test Slide 33 Standard presentation deck > Sep. 2009
  34. 34. Conclusion Powerful possibility to create tests with no need of a running Apache. Tests could be made in an easy language with possibility to easily make moc-objects. Writing a test is in most cases less than writing 10 lines of code. Tests are easily portable to other systems/apache- versions. Slide 34 Standard presentation deck > Sep. 2009
  35. 35. Demonstration Before the demo: Thanks to Steven Mohr Slide 35 Standard presentation deck > Sep. 2009

×