Testing Apache Modules with Python and Ctypes
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Testing Apache Modules with Python and Ctypes

on

  • 4,511 views

Writing tests for your Apache module is often a developer's least favourite task, especially if you don't like using the Perl Apache Testing Framework! One of the main reasons for this is that it's ...

Writing tests for your Apache module is often a developer's least favourite task, especially if you don't like using the Perl Apache Testing Framework! One of the main reasons for this is that it's difficult to test your C code on-the-fly without a running Apache server. Using ctypes, you can test your modules without a running httpd, just by writing and using simple Python scripts! In this talk we will look at how to compile a specific version of the Apache webserver for testing, and how to use a common Python unit testing framework to write and set up tests. We will also cover examples of writing simple test cases for the Catacomb WebDAV Server module.

Talk from Apachecon US 2009

Statistics

Views

Total Views
4,511
Views on SlideShare
4,454
Embed Views
57

Actions

Likes
3
Downloads
54
Comments
0

7 Embeds 57

http://www.techgig.com 25
http://noatta.blogspot.com 10
http://www.slideshare.net 9
http://noatta.blogspot.tw 9
http://noatta.blogspot.co.uk 2
http://noatta.blogspot.com.es 1
http://115.112.206.131 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Testing Apache Modules with Python and Ctypes Presentation Transcript

  • 1. Testing Apache Modules with Python and ctypes ApacheCon US 2009 Markus Litz - 06.11.2009 Slide 1 Standard presentation deck > Sep. 2009
  • 2. Agenda for today Why? Introduction to ctypes Preparing the apache Creating tests Demo Slide 2 Standard presentation deck > Sep. 2009
  • 3. DLR German Aerospace Center Research Institution Research Areas Aeronautics Space Transport Energy Space Agency Slide 3 Standard presentation deck > Sep. 2009
  • 4. Locations and employees 6200 employees across Hamburg  29 research institutes and  Neustrelitz Bremen   Trauen facilities at Berlin   13 sites. Braunschweig   Dortmund  Goettingen Offices in Brussels,  Koeln Paris and Washington.  Bonn Lampoldshausen  Stuttgart   Oberpfaffenhofen Weilheim  Slide 4 Standard presentation deck > Sep. 2009
  • 5. Background DataFinder – a application for scientific data management Storing and managing huge amounts of data Search through the resource content and metadata Various ways to store data, for example ftp, network share, offline stores Metadata management with the WebDAV protocol Two supported WebDAV Server: Tamino XML Server & Catacomb Slide 5 Standard presentation deck > Sep. 2009
  • 6. Catacomb – A WebDAV Server Module for Apache Slide 6 Standard presentation deck > Sep. 2009
  • 7. Catacomb – The Difference to mod_dav_fs Saving the resources mod_dav_fs save content and properties in files on the filesystem mod_dav_fs creates for every resource, and also for every collection, their own property file Consequence: A single query of server side searching needs to open many files Implementation of complex queries is difficult Full text search is expensive Slide 7 Standard presentation deck > Sep. 2009
  • 8. Catacomb – A WebDAV Server Module for Apache WebDAV repository module for mod_dav Catacomb uses relational databases to store the metadata Strong search performance through SQL statements Catacomb is: Good for Content management Good for Collaborated web authoring Support locks, avoid the “lost update” problem Capable of searching (DASL) and versioning (DeltaV) resources Slide 8 Standard presentation deck > Sep. 2009
  • 9. Catacomb – History and Current State Initial development at the University of California under the chair of Jim Whitehead Open Source project since 2002 DeltaV and DASL implementation Since 2006 contribution of the DLR ACP support Database abstraction using mod_dbd License changed to ASL2.0 Slide 9 Standard presentation deck > Sep. 2009
  • 10. Why testing your code? Development is faster and easier Code is more robust Code is more maintainable Code is more reliable Slide 10 Standard presentation deck > Sep. 2009
  • 11. Why testing with Python and ctypes? Writing tests is easy No need to start an apache instance every time Tests could be automatically done with various Apache versions Slide 11 Standard presentation deck > Sep. 2009
  • 12. What is ctypes ctypes is a wrapper for C-librarys for python ctypes allows to call functions in dlls/shared libraries from python code It is possible to implement C callback function Since Python 2.5.x, ctypes is in the standard library Slide 12 Standard presentation deck > Sep. 2009
  • 13. How to use ctypes from ctypes import * Loading dynamic link libraries libc = cdll.msvcr libc = CDLL("libc.so.6") Calling functions print libc.time(None) Slide 13 Standard presentation deck > Sep. 2009
  • 14. Fundamental data types Good support for many primitive C compatible data types: C Python char  c_char int  c_int long  c_long void*  c_void_p Slide 14 Standard presentation deck > Sep. 2009
  • 15. Fundamental data types - usage All these types can be created by calling them with an optional initializer of the correct type and value: i = c_int(42) print i.value # „42“ i.value = -1 print i.value # „-1“ num = c_double(3.14) libc.printf("Number: %fn“, num) # „Numner: 3.14“ Slide 15 Standard presentation deck > Sep. 2009
  • 16. Using pointers byref() passes parameters by reference libc.sscanf("1 3.14 Hello", "%d %f %s", byref(i), byref(f), s) Creating a pointer i = c_int(42) pi = pointer(i) Slide 16 Standard presentation deck > Sep. 2009
  • 17. Return types Default return type: int strcat = libc.strcat strcat("abc", "def")) # „8059983“ strcat.restype = c_char_p strcat("abc", "def")) # „abcdef“ Slide 17 Standard presentation deck > Sep. 2009
  • 18. Arrays Create an array-type TenIntsArrayType = c_int * 10 Create an array-instance array1 = TenIntegers() array2 = TenIntegers(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) Using arrays Array1[3]  “0” Array2[3]  “4” Slide 18 Standard presentation deck > Sep. 2009
  • 19. Structures and unions class POINT(Structure): _fields_ = [("x", c_int), ("y", c_int)] point = POINT(10, 20) print point.x, point.y  „10 20“ Slide 19 Standard presentation deck > Sep. 2009
  • 20. UnitTesting Apache Modules The problem (Most) functions of a module could only be tested with a running apache Module-functions could not be called directly The solutions Starting and stopping an apache on each test Test functions from the module directly using ctypes Slide 20 Standard presentation deck > Sep. 2009
  • 21. Calling module functions directly Causes a exception stops execution On runtime, ctypes tries to resolve all dynamic symbols All apache specific methods and data structures are not available Solution: Building Apache as a shared core Slide 21 Standard presentation deck > Sep. 2009
  • 22. Building-kernel apache as a share core Building the apache kernel as shared module On apache 1.x --enable-rule=SHARED_CORE On apache 2.x build infrastructure doesn't seem to know this anymore Slide 22 Standard presentation deck > Sep. 2009
  • 23. Compiling Apache Compiling apache make clean CFLAGS='-D SHARED_CORE -fPIC ' ./configure make Slide 23 Standard presentation deck > Sep. 2009
  • 24. Linking the Shared Core After compiling, the make command links apache libtool ... -mode=link gcc ... -o httpd .. Linking command for a shared core libtool ... -mode=link gcc ... -shared -o libhttpd.so ..server/exports.o Slide 24 Standard presentation deck > Sep. 2009
  • 25. Modifications of the Module Module must be linked against the shared core LDFLAGS = -lhttpd -L </…/libhttpd.so> Could be an extra make-target Slide 25 Standard presentation deck > Sep. 2009
  • 26. Apache Data Structures in Python class apr_allocator_t(Structure): class apr_memnode_t(Structure): class apr_pool_t(Structure): class cleanup_t(Structure): Slide 26 Standard presentation deck > Sep. 2009
  • 27. Setting Up Data Structures – apt_pool_t class apr_pool_t(Structure): _fields_ = [("cleanups",POINTER(cleanup_t)), ("free_cleanups",POINTER(cleanup_t)), ("allocator",POINTER(apr_allocator_t)), ("subprocesses",POINTER(process_chain)), ("abort_fn",c_void_p), ("user_data",c_void_p), ("tag",c_char_p), ("active",POINTER(apr_memnode_t)), ("self",POINTER(apr_memnode_t)), ("self_first_avail",c_char_p), ("parent",POINTER(apr_pool_t)), ("child",POINTER(apr_pool_t)), ("sibling",POINTER(apr_pool_t)), ("ref",POINTER(POINTER(apr_pool_t)))] Slide 27 Standard presentation deck > Sep. 2009
  • 28. Setting Up Data Structures – GCC Ctypes code generator – modified version of GCC Looks for declarations in C header files. Generates python codes for: enums, structs, unions, function declarations, com interfaces, and preprocessor definitions Very early stage Slide 28 Standard presentation deck > Sep. 2009
  • 29. Unit Test Framwork (nose) Simple structure, one class for each testing object Setup_class() Test1() … TestX() TearDown_class() Slide 29 Standard presentation deck > Sep. 2009
  • 30. Setting up the Test Environment def setup (self) : self.catacomb = CDLL("/apachecon/libmod_dav_repos.so") self.httpd = CDLL("/apachecon/libhttpd.so") self.apr = CDLL("/apachecon/lib/libapr-1.so") self.pool = c_void_p() self.allocator = c_void_p() self.apr.apr_initialize() self.apr.apr_allocator_create(byref(self.allocator)) self.apr.apr_pool_create_ex(byref(self.pool), None, None, self.allocator) Slide 30 Standard presentation deck > Sep. 2009
  • 31. Writing the Test def testSomething(self): assert self.catacomb.function_to_test(arg1, byref(arg2)) == “true” Slide 31 Standard presentation deck > Sep. 2009
  • 32. Shutting down the Test Environment def teardown(self): self.apr.apr_pool_destroy(self.pool) self.apr.apr_allocator_destroy(self.allocator) self.apr.apr_terminate() Slide 32 Standard presentation deck > Sep. 2009
  • 33. Summary of Steps Compile Apache as a shared core Link own module against shared core Define the data structures you need Write the tests Run the test Slide 33 Standard presentation deck > Sep. 2009
  • 34. Conclusion Powerful possibility to create tests with no need of a running Apache. Tests could be made in an easy language with possibility to easily make moc-objects. Writing a test is in most cases less than writing 10 lines of code. Tests are easily portable to other systems/apache- versions. Slide 34 Standard presentation deck > Sep. 2009
  • 35. Demonstration Before the demo: Thanks to Steven Mohr Slide 35 Standard presentation deck > Sep. 2009