SlideShare a Scribd company logo
Testing Apache Modules with Python
and ctypes
ApacheCon US 2009

Markus Litz   - 06.11.2009


                                                           Slide 1
                              Standard presentation deck > Sep. 2009
Agenda for today

  Why?


  Introduction to ctypes


  Preparing the apache


  Creating tests


  Demo

                                                        Slide 2
                           Standard presentation deck > Sep. 2009
DLR
German Aerospace Center




   Research Institution
   Research Areas
      Aeronautics
      Space
      Transport
      Energy
   Space Agency



                                                       Slide 3
                          Standard presentation deck > Sep. 2009
Locations and employees

6200 employees across                 Hamburg 
29 research institutes and                                          Neustrelitz
                                  Bremen        Trauen
facilities at
                                                           Berlin 
         13 sites.                 Braunschweig 

                                    Dortmund  Goettingen
Offices in Brussels,
                                  Koeln
Paris and Washington.             Bonn


                             Lampoldshausen 

                                  Stuttgart 

                                                      Oberpfaffenhofen
                                           Weilheim 




                                                                                        Slide 4
                                                           Standard presentation deck > Sep. 2009
Background

 DataFinder – a application for scientific data management
     Storing and managing huge amounts of data
     Search through the resource content and metadata
     Various ways to store data, for example
          ftp, network share, offline stores
     Metadata management with the WebDAV protocol
          Two supported WebDAV Server:
                Tamino XML Server & Catacomb




                                                                             Slide 5
                                                Standard presentation deck > Sep. 2009
Catacomb – A WebDAV Server Module for
Apache




                                                           Slide 6
                              Standard presentation deck > Sep. 2009
Catacomb – The Difference to mod_dav_fs

    Saving the resources
        mod_dav_fs save content and properties in files on
       the filesystem
        mod_dav_fs creates for every resource, and also for
       every collection, their own property file

    Consequence:
       A single query of server side searching needs to open
       many files
       Implementation of complex queries is difficult
       Full text search is expensive

                                                                               Slide 7
                                                  Standard presentation deck > Sep. 2009
Catacomb – A WebDAV Server Module for
Apache
  WebDAV repository module for mod_dav

  Catacomb uses relational databases to store the metadata
      Strong search performance through SQL statements

  Catacomb is:
      Good for Content management
      Good for Collaborated web authoring
          Support locks, avoid the “lost update” problem
      Capable of searching (DASL) and versioning (DeltaV)
     resources
                                                                               Slide 8
                                                  Standard presentation deck > Sep. 2009
Catacomb – History and Current State

   Initial development at the University of California under the
  chair of Jim Whitehead

   Open Source project since 2002

   DeltaV and DASL implementation

   Since 2006 contribution of the DLR
       ACP support
       Database abstraction using mod_dbd
       License changed to ASL2.0
                                                                                 Slide 9
                                                    Standard presentation deck > Sep. 2009
Why testing your code?

  Development is faster and easier


  Code is more robust


  Code is more maintainable


  Code is more reliable




                                                                Slide 10
                                     Standard presentation deck > Sep. 2009
Why testing with Python and ctypes?

  Writing tests is easy


  No need to start an apache instance every time


  Tests could be automatically done with various
 Apache versions




                                                                        Slide 11
                                             Standard presentation deck > Sep. 2009
What is ctypes

  ctypes is a wrapper for C-librarys for python


  ctypes allows to call functions in dlls/shared libraries
 from python code


  It is possible to implement C callback function


  Since Python 2.5.x, ctypes is in the standard library



                                                                             Slide 12
                                                  Standard presentation deck > Sep. 2009
How to use ctypes

  from ctypes import *


  Loading dynamic link libraries
     libc = cdll.msvcr
     libc = CDLL("libc.so.6")


  Calling functions
     print libc.time(None)


                                                              Slide 13
                                   Standard presentation deck > Sep. 2009
Fundamental data types

  Good support for many primitive C compatible data
 types:


      C                 Python
     char              c_char
     int               c_int
     long              c_long
     void*             c_void_p



                                                                       Slide 14
                                            Standard presentation deck > Sep. 2009
Fundamental data types - usage

All these types can be created by calling them with an optional initializer
of the correct type and value:

    i = c_int(42)
    print i.value             # „42“
    i.value = -1
    print i.value             # „-1“


    num = c_double(3.14)
    libc.printf("Number: %fn“, num)
                                               # „Numner: 3.14“

                                                                                      Slide 15
                                                           Standard presentation deck > Sep. 2009
Using pointers

  byref() passes parameters by reference
     libc.sscanf("1 3.14 Hello", "%d %f
               %s", byref(i), byref(f), s)


  Creating a pointer
   i = c_int(42)
   pi = pointer(i)




                                                             Slide 16
                                  Standard presentation deck > Sep. 2009
Return types

  Default return type: int

  strcat = libc.strcat
  strcat("abc", "def"))       # „8059983“


  strcat.restype = c_char_p
  strcat("abc", "def"))       # „abcdef“




                                                              Slide 17
                                   Standard presentation deck > Sep. 2009
Arrays
  Create an array-type
     TenIntsArrayType = c_int * 10


  Create an array-instance
     array1 = TenIntegers()
     array2 = TenIntegers(1, 2, 3, 4, 5, 6,
    7, 8, 9, 10)


  Using arrays
     Array1[3]           “0”
     Array2[3]           “4”

                                                                Slide 18
                                     Standard presentation deck > Sep. 2009
Structures and unions


  class POINT(Structure):
     _fields_ = [("x", c_int),
                 ("y", c_int)]

  point = POINT(10, 20)
  print point.x, point.y            „10 20“




                                                                Slide 19
                                     Standard presentation deck > Sep. 2009
UnitTesting Apache Modules

  The problem
      (Most) functions of a module could only be tested with
     a running apache
      Module-functions could not be called directly

  The solutions
      Starting and stopping an apache on each test
      Test functions from the module directly using ctypes




                                                                            Slide 20
                                                 Standard presentation deck > Sep. 2009
Calling module functions directly

  Causes a exception stops execution
     On runtime, ctypes tries to resolve all dynamic
    symbols
      All apache specific methods and data structures
     are not available


  Solution:
      Building Apache as a shared core



                                                                        Slide 21
                                             Standard presentation deck > Sep. 2009
Building-kernel apache as a share core

  Building the apache kernel as shared module
     On apache 1.x
         --enable-rule=SHARED_CORE


    On apache 2.x build infrastructure doesn't seem to
   know this anymore




                                                                       Slide 22
                                            Standard presentation deck > Sep. 2009
Compiling Apache

  Compiling apache

    make clean
    CFLAGS='-D SHARED_CORE -fPIC '
   ./configure
    make




                                                            Slide 23
                                 Standard presentation deck > Sep. 2009
Linking the Shared Core

  After compiling, the make command links apache
      libtool ... -mode=link gcc ... -o httpd
    ..


  Linking command for a shared core
      libtool ... -mode=link gcc ...
    -shared -o libhttpd.so ..server/exports.o




                                                                   Slide 24
                                        Standard presentation deck > Sep. 2009
Modifications of the Module

  Module must be linked against the shared core

     LDFLAGS = -lhttpd -L </…/libhttpd.so>


     Could be an extra make-target




                                                                      Slide 25
                                           Standard presentation deck > Sep. 2009
Apache Data Structures in Python

class apr_allocator_t(Structure):

class apr_memnode_t(Structure):

class apr_pool_t(Structure):

class cleanup_t(Structure):




                                                               Slide 26
                                    Standard presentation deck > Sep. 2009
Setting Up Data Structures – apt_pool_t
class apr_pool_t(Structure):
    _fields_ = [("cleanups",POINTER(cleanup_t)),
                ("free_cleanups",POINTER(cleanup_t)),
                ("allocator",POINTER(apr_allocator_t)),
                ("subprocesses",POINTER(process_chain)),
                ("abort_fn",c_void_p),
                ("user_data",c_void_p),
                ("tag",c_char_p),
                ("active",POINTER(apr_memnode_t)),
                ("self",POINTER(apr_memnode_t)),
                ("self_first_avail",c_char_p),
                ("parent",POINTER(apr_pool_t)),
                ("child",POINTER(apr_pool_t)),
                ("sibling",POINTER(apr_pool_t)),
                ("ref",POINTER(POINTER(apr_pool_t)))]
                                                                          Slide 27
                                               Standard presentation deck > Sep. 2009
Setting Up Data Structures – GCC

  Ctypes code generator – modified version of GCC

  Looks for declarations in C header files. Generates python
 codes for:
      enums, structs, unions, function declarations, com
    interfaces, and preprocessor definitions

  Very early stage




                                                                            Slide 28
                                                 Standard presentation deck > Sep. 2009
Unit Test Framwork (nose)

  Simple structure, one class for each testing object

     Setup_class()
     Test1()
     …
     TestX()
     TearDown_class()




                                                                              Slide 29
                                                   Standard presentation deck > Sep. 2009
Setting up the Test Environment
def setup (self) :
    self.catacomb = CDLL("/apachecon/libmod_dav_repos.so")
    self.httpd = CDLL("/apachecon/libhttpd.so")
    self.apr = CDLL("/apachecon/lib/libapr-1.so")


    self.pool = c_void_p()
    self.allocator = c_void_p()


    self.apr.apr_initialize()
    self.apr.apr_allocator_create(byref(self.allocator))
    self.apr.apr_pool_create_ex(byref(self.pool), None,
                                   None, self.allocator)




                                                                              Slide 30
                                                   Standard presentation deck > Sep. 2009
Writing the Test

def testSomething(self):
   assert self.catacomb.function_to_test(arg1,
                     byref(arg2)) == “true”




                                                                       Slide 31
                                            Standard presentation deck > Sep. 2009
Shutting down the Test Environment


def teardown(self):
    self.apr.apr_pool_destroy(self.pool)
    self.apr.apr_allocator_destroy(self.allocator)
    self.apr.apr_terminate()




                                                                   Slide 32
                                        Standard presentation deck > Sep. 2009
Summary of Steps

  Compile Apache as a shared core


  Link own module against shared core


  Define the data structures you need


  Write the tests


  Run the test

                                                                   Slide 33
                                        Standard presentation deck > Sep. 2009
Conclusion

  Powerful possibility to create tests with no need of a
 running Apache.
   Tests could be made in an easy language with
 possibility to easily make moc-objects.
   Writing a test is in most cases less than writing 10
 lines of code.
  Tests are easily portable to other systems/apache-
 versions.




                                                                           Slide 34
                                                Standard presentation deck > Sep. 2009
Demonstration

  Before the demo:


     Thanks to Steven Mohr




                                                        Slide 35
                             Standard presentation deck > Sep. 2009

More Related Content

Similar to Testing Apache Modules with Python and Ctypes

02 c++g3 d
02 c++g3 d02 c++g3 d
02 c++g3 d
mahago
 
T2
T2T2
T2
lksoo
 
Framework engineering JCO 2011
Framework engineering JCO 2011Framework engineering JCO 2011
Framework engineering JCO 2011
YoungSu Son
 
Bounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise EnvironmentBounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise Environment
AdaCore
 
C# tutorial
C# tutorialC# tutorial
C# tutorial
sarangowtham_gunnam
 
9.oo languages
9.oo languages9.oo languages
9.oo languages
APU
 
9. oo languages
9. oo languages9. oo languages
9. oo languages
APU
 
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Animesh Singh
 
Intro to c# (vs. objective c and java)
Intro to c# (vs. objective c and java)Intro to c# (vs. objective c and java)
Intro to c# (vs. objective c and java)
Wes Yanaga
 
Intro to c# (vs. objective c and java)
Intro to c# (vs. objective c and java)Intro to c# (vs. objective c and java)
Intro to c# (vs. objective c and java)
John Smith
 
Legacy of Void*
Legacy of Void*Legacy of Void*
Legacy of Void*
Adam Crain
 
Srgoc dotnet_new
Srgoc dotnet_newSrgoc dotnet_new
Srgoc dotnet_new
Gaurav Singh
 
tybsc it asp.net full unit 1,2,3,4,5,6 notes
tybsc it asp.net full unit 1,2,3,4,5,6 notestybsc it asp.net full unit 1,2,3,4,5,6 notes
tybsc it asp.net full unit 1,2,3,4,5,6 notes
WE-IT TUTORIALS
 
Lotusphere 2007 AD505 DevBlast 30 LotusScript Tips
Lotusphere 2007 AD505 DevBlast 30 LotusScript TipsLotusphere 2007 AD505 DevBlast 30 LotusScript Tips
Lotusphere 2007 AD505 DevBlast 30 LotusScript Tips
Bill Buchan
 
High Quality Symfony Bundles tutorial - Dutch PHP Conference 2014
High Quality Symfony Bundles tutorial - Dutch PHP Conference 2014High Quality Symfony Bundles tutorial - Dutch PHP Conference 2014
High Quality Symfony Bundles tutorial - Dutch PHP Conference 2014
Matthias Noback
 
Introduction-to-C-Part-1.pdf
Introduction-to-C-Part-1.pdfIntroduction-to-C-Part-1.pdf
Introduction-to-C-Part-1.pdf
AnassElHousni
 
Visual studio.net
Visual studio.netVisual studio.net
Visual studio.net
Satish Verma
 
Advance Android Application Development
Advance Android Application DevelopmentAdvance Android Application Development
Advance Android Application Development
Ramesh Prasad
 
Mac ruby deployment
Mac ruby deploymentMac ruby deployment
Mac ruby deployment
Thilo Utke
 
Writing a Gem with native extensions
Writing a Gem with native extensionsWriting a Gem with native extensions
Writing a Gem with native extensions
Tristan Penman
 

Similar to Testing Apache Modules with Python and Ctypes (20)

02 c++g3 d
02 c++g3 d02 c++g3 d
02 c++g3 d
 
T2
T2T2
T2
 
Framework engineering JCO 2011
Framework engineering JCO 2011Framework engineering JCO 2011
Framework engineering JCO 2011
 
Bounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise EnvironmentBounded Model Checking for C Programs in an Enterprise Environment
Bounded Model Checking for C Programs in an Enterprise Environment
 
C# tutorial
C# tutorialC# tutorial
C# tutorial
 
9.oo languages
9.oo languages9.oo languages
9.oo languages
 
9. oo languages
9. oo languages9. oo languages
9. oo languages
 
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
Hybrid Cloud, Kubeflow and Tensorflow Extended [TFX]
 
Intro to c# (vs. objective c and java)
Intro to c# (vs. objective c and java)Intro to c# (vs. objective c and java)
Intro to c# (vs. objective c and java)
 
Intro to c# (vs. objective c and java)
Intro to c# (vs. objective c and java)Intro to c# (vs. objective c and java)
Intro to c# (vs. objective c and java)
 
Legacy of Void*
Legacy of Void*Legacy of Void*
Legacy of Void*
 
Srgoc dotnet_new
Srgoc dotnet_newSrgoc dotnet_new
Srgoc dotnet_new
 
tybsc it asp.net full unit 1,2,3,4,5,6 notes
tybsc it asp.net full unit 1,2,3,4,5,6 notestybsc it asp.net full unit 1,2,3,4,5,6 notes
tybsc it asp.net full unit 1,2,3,4,5,6 notes
 
Lotusphere 2007 AD505 DevBlast 30 LotusScript Tips
Lotusphere 2007 AD505 DevBlast 30 LotusScript TipsLotusphere 2007 AD505 DevBlast 30 LotusScript Tips
Lotusphere 2007 AD505 DevBlast 30 LotusScript Tips
 
High Quality Symfony Bundles tutorial - Dutch PHP Conference 2014
High Quality Symfony Bundles tutorial - Dutch PHP Conference 2014High Quality Symfony Bundles tutorial - Dutch PHP Conference 2014
High Quality Symfony Bundles tutorial - Dutch PHP Conference 2014
 
Introduction-to-C-Part-1.pdf
Introduction-to-C-Part-1.pdfIntroduction-to-C-Part-1.pdf
Introduction-to-C-Part-1.pdf
 
Visual studio.net
Visual studio.netVisual studio.net
Visual studio.net
 
Advance Android Application Development
Advance Android Application DevelopmentAdvance Android Application Development
Advance Android Application Development
 
Mac ruby deployment
Mac ruby deploymentMac ruby deployment
Mac ruby deployment
 
Writing a Gem with native extensions
Writing a Gem with native extensionsWriting a Gem with native extensions
Writing a Gem with native extensions
 

Recently uploaded

Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
Jason Yip
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
DanBrown980551
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Pitangent Analytics & Technology Solutions Pvt. Ltd
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
DanBrown980551
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
saastr
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
Edge AI and Vision Alliance
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
Ajin Abraham
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
c5vrf27qcz
 

Recently uploaded (20)

Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
[OReilly Superstream] Occupy the Space: A grassroots guide to engineering (an...
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
Crafting Excellence: A Comprehensive Guide to iOS Mobile App Development Serv...
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
9 CEO's who hit $100m ARR Share Their Top Growth Tactics Nathan Latka, Founde...
 
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
“Temporal Event Neural Networks: A More Efficient Alternative to the Transfor...
 
AppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSFAppSec PNW: Android and iOS Application Security with MobSF
AppSec PNW: Android and iOS Application Security with MobSF
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance PanelsNorthern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
Northern Engraving | Modern Metal Trim, Nameplates and Appliance Panels
 
Y-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PPY-Combinator seed pitch deck template PP
Y-Combinator seed pitch deck template PP
 

Testing Apache Modules with Python and Ctypes

  • 1. Testing Apache Modules with Python and ctypes ApacheCon US 2009 Markus Litz - 06.11.2009 Slide 1 Standard presentation deck > Sep. 2009
  • 2. Agenda for today Why? Introduction to ctypes Preparing the apache Creating tests Demo Slide 2 Standard presentation deck > Sep. 2009
  • 3. DLR German Aerospace Center Research Institution Research Areas Aeronautics Space Transport Energy Space Agency Slide 3 Standard presentation deck > Sep. 2009
  • 4. Locations and employees 6200 employees across Hamburg  29 research institutes and  Neustrelitz Bremen   Trauen facilities at Berlin   13 sites. Braunschweig   Dortmund  Goettingen Offices in Brussels,  Koeln Paris and Washington.  Bonn Lampoldshausen  Stuttgart   Oberpfaffenhofen Weilheim  Slide 4 Standard presentation deck > Sep. 2009
  • 5. Background DataFinder – a application for scientific data management Storing and managing huge amounts of data Search through the resource content and metadata Various ways to store data, for example ftp, network share, offline stores Metadata management with the WebDAV protocol Two supported WebDAV Server: Tamino XML Server & Catacomb Slide 5 Standard presentation deck > Sep. 2009
  • 6. Catacomb – A WebDAV Server Module for Apache Slide 6 Standard presentation deck > Sep. 2009
  • 7. Catacomb – The Difference to mod_dav_fs Saving the resources mod_dav_fs save content and properties in files on the filesystem mod_dav_fs creates for every resource, and also for every collection, their own property file Consequence: A single query of server side searching needs to open many files Implementation of complex queries is difficult Full text search is expensive Slide 7 Standard presentation deck > Sep. 2009
  • 8. Catacomb – A WebDAV Server Module for Apache WebDAV repository module for mod_dav Catacomb uses relational databases to store the metadata Strong search performance through SQL statements Catacomb is: Good for Content management Good for Collaborated web authoring Support locks, avoid the “lost update” problem Capable of searching (DASL) and versioning (DeltaV) resources Slide 8 Standard presentation deck > Sep. 2009
  • 9. Catacomb – History and Current State Initial development at the University of California under the chair of Jim Whitehead Open Source project since 2002 DeltaV and DASL implementation Since 2006 contribution of the DLR ACP support Database abstraction using mod_dbd License changed to ASL2.0 Slide 9 Standard presentation deck > Sep. 2009
  • 10. Why testing your code? Development is faster and easier Code is more robust Code is more maintainable Code is more reliable Slide 10 Standard presentation deck > Sep. 2009
  • 11. Why testing with Python and ctypes? Writing tests is easy No need to start an apache instance every time Tests could be automatically done with various Apache versions Slide 11 Standard presentation deck > Sep. 2009
  • 12. What is ctypes ctypes is a wrapper for C-librarys for python ctypes allows to call functions in dlls/shared libraries from python code It is possible to implement C callback function Since Python 2.5.x, ctypes is in the standard library Slide 12 Standard presentation deck > Sep. 2009
  • 13. How to use ctypes from ctypes import * Loading dynamic link libraries libc = cdll.msvcr libc = CDLL("libc.so.6") Calling functions print libc.time(None) Slide 13 Standard presentation deck > Sep. 2009
  • 14. Fundamental data types Good support for many primitive C compatible data types: C Python char  c_char int  c_int long  c_long void*  c_void_p Slide 14 Standard presentation deck > Sep. 2009
  • 15. Fundamental data types - usage All these types can be created by calling them with an optional initializer of the correct type and value: i = c_int(42) print i.value # „42“ i.value = -1 print i.value # „-1“ num = c_double(3.14) libc.printf("Number: %fn“, num) # „Numner: 3.14“ Slide 15 Standard presentation deck > Sep. 2009
  • 16. Using pointers byref() passes parameters by reference libc.sscanf("1 3.14 Hello", "%d %f %s", byref(i), byref(f), s) Creating a pointer i = c_int(42) pi = pointer(i) Slide 16 Standard presentation deck > Sep. 2009
  • 17. Return types Default return type: int strcat = libc.strcat strcat("abc", "def")) # „8059983“ strcat.restype = c_char_p strcat("abc", "def")) # „abcdef“ Slide 17 Standard presentation deck > Sep. 2009
  • 18. Arrays Create an array-type TenIntsArrayType = c_int * 10 Create an array-instance array1 = TenIntegers() array2 = TenIntegers(1, 2, 3, 4, 5, 6, 7, 8, 9, 10) Using arrays Array1[3]  “0” Array2[3]  “4” Slide 18 Standard presentation deck > Sep. 2009
  • 19. Structures and unions class POINT(Structure): _fields_ = [("x", c_int), ("y", c_int)] point = POINT(10, 20) print point.x, point.y  „10 20“ Slide 19 Standard presentation deck > Sep. 2009
  • 20. UnitTesting Apache Modules The problem (Most) functions of a module could only be tested with a running apache Module-functions could not be called directly The solutions Starting and stopping an apache on each test Test functions from the module directly using ctypes Slide 20 Standard presentation deck > Sep. 2009
  • 21. Calling module functions directly Causes a exception stops execution On runtime, ctypes tries to resolve all dynamic symbols All apache specific methods and data structures are not available Solution: Building Apache as a shared core Slide 21 Standard presentation deck > Sep. 2009
  • 22. Building-kernel apache as a share core Building the apache kernel as shared module On apache 1.x --enable-rule=SHARED_CORE On apache 2.x build infrastructure doesn't seem to know this anymore Slide 22 Standard presentation deck > Sep. 2009
  • 23. Compiling Apache Compiling apache make clean CFLAGS='-D SHARED_CORE -fPIC ' ./configure make Slide 23 Standard presentation deck > Sep. 2009
  • 24. Linking the Shared Core After compiling, the make command links apache libtool ... -mode=link gcc ... -o httpd .. Linking command for a shared core libtool ... -mode=link gcc ... -shared -o libhttpd.so ..server/exports.o Slide 24 Standard presentation deck > Sep. 2009
  • 25. Modifications of the Module Module must be linked against the shared core LDFLAGS = -lhttpd -L </…/libhttpd.so> Could be an extra make-target Slide 25 Standard presentation deck > Sep. 2009
  • 26. Apache Data Structures in Python class apr_allocator_t(Structure): class apr_memnode_t(Structure): class apr_pool_t(Structure): class cleanup_t(Structure): Slide 26 Standard presentation deck > Sep. 2009
  • 27. Setting Up Data Structures – apt_pool_t class apr_pool_t(Structure): _fields_ = [("cleanups",POINTER(cleanup_t)), ("free_cleanups",POINTER(cleanup_t)), ("allocator",POINTER(apr_allocator_t)), ("subprocesses",POINTER(process_chain)), ("abort_fn",c_void_p), ("user_data",c_void_p), ("tag",c_char_p), ("active",POINTER(apr_memnode_t)), ("self",POINTER(apr_memnode_t)), ("self_first_avail",c_char_p), ("parent",POINTER(apr_pool_t)), ("child",POINTER(apr_pool_t)), ("sibling",POINTER(apr_pool_t)), ("ref",POINTER(POINTER(apr_pool_t)))] Slide 27 Standard presentation deck > Sep. 2009
  • 28. Setting Up Data Structures – GCC Ctypes code generator – modified version of GCC Looks for declarations in C header files. Generates python codes for: enums, structs, unions, function declarations, com interfaces, and preprocessor definitions Very early stage Slide 28 Standard presentation deck > Sep. 2009
  • 29. Unit Test Framwork (nose) Simple structure, one class for each testing object Setup_class() Test1() … TestX() TearDown_class() Slide 29 Standard presentation deck > Sep. 2009
  • 30. Setting up the Test Environment def setup (self) : self.catacomb = CDLL("/apachecon/libmod_dav_repos.so") self.httpd = CDLL("/apachecon/libhttpd.so") self.apr = CDLL("/apachecon/lib/libapr-1.so") self.pool = c_void_p() self.allocator = c_void_p() self.apr.apr_initialize() self.apr.apr_allocator_create(byref(self.allocator)) self.apr.apr_pool_create_ex(byref(self.pool), None, None, self.allocator) Slide 30 Standard presentation deck > Sep. 2009
  • 31. Writing the Test def testSomething(self): assert self.catacomb.function_to_test(arg1, byref(arg2)) == “true” Slide 31 Standard presentation deck > Sep. 2009
  • 32. Shutting down the Test Environment def teardown(self): self.apr.apr_pool_destroy(self.pool) self.apr.apr_allocator_destroy(self.allocator) self.apr.apr_terminate() Slide 32 Standard presentation deck > Sep. 2009
  • 33. Summary of Steps Compile Apache as a shared core Link own module against shared core Define the data structures you need Write the tests Run the test Slide 33 Standard presentation deck > Sep. 2009
  • 34. Conclusion Powerful possibility to create tests with no need of a running Apache. Tests could be made in an easy language with possibility to easily make moc-objects. Writing a test is in most cases less than writing 10 lines of code. Tests are easily portable to other systems/apache- versions. Slide 34 Standard presentation deck > Sep. 2009
  • 35. Demonstration Before the demo: Thanks to Steven Mohr Slide 35 Standard presentation deck > Sep. 2009