Autotools adaptation for integrating autotmatic unit tests and covering for KPI-1

1. Autotools adaptation for:
- Automatic unitary tests (Google test/mock)
- Generation of KPI report using lcov
Thierry GAYET (EUROGICIEL)
Revolution-S Rennes - 2011

2. The main goal of the following slides is to describe how to:
1. Automate the launch of the unit tests
2. Generate KPI from the unit test coverage
2
Objective
06/17/10

3. 3
Hudson workflow
06/17/10
Hudson
Server
Code covering for KPI project needs using lcov by
analyzing the gcov data and generate an html report
Latest doxygen documentation of the
public API in html format
Launch the unit tests with Google{test/gmock}
(generate junit xml files as a report)
Set the environment
Autotools targets
available on Hudson 
mkdir tmp_srcdir
./autogen.sh
./configure --prefix=/usr --srcdir=`pwd`/tmp_srcdir …
make coverity
make check
Make the latest stable archive, extract it in a specific
directory, then launch the configure script before
building the component
make dist
tar zxvf composant-version.tar.gz -C <BUILD_PATH>
cd <BUILD_PATH>/composant-version/
./autogen.sh
./configure --prefix=/usr –enable-covering –enable-test
make –j7
make DESTDIR=<STAGINGDIR> install
make html
Deliver to default CI stream
+ Baseline
+ Recommended baseline

4. Overview of the autotool process
Makefile.am
configure.ac aclocal.m4
configure
xxx.pc.in
config.h.in
For each package:
• provided by the developers
• stored in repository
Makefile.in
config.cache
config.log
config.h
Makefile
xxx.pc
autogen.sh

automake
autoheader
aclocal
autoconf
USER VIEW
DEV. VIEW

5. Overview
./autogen.sh
./configure
--prefix=/usr
--enable-test
--enable-covering make covering
--enable-test :
- Enable the unitary tests
- If the Google tests are used the libgtest is checked
- Same for the libgmock
--enable-covering :
- Add the gcov flags to gcc
Step #1 Step #2 Step #3
t
make check
Will compile and launch the unit test
[ with gcov data at the runtime ]
make
It will generate:
- Makefile.in
- configure script
- config.h.in if used
It will generate:
- Makefile
- config.h header
- .pc file
It will generate the
expected ELF objects
(binaries, libraries, … )
including gcov support. Analyze the
data
generated
by gcov and
make html
output
It will generate an html
report that includes a
synthesis of the gcov data
generated at the runtime.
Thus, there is a
correlation between the
unit tests and the lcov
results !!
Step #4

6. /
Root of the
component
/src/ : implementation’s source code
/inc/ : private headers
Example of component pattern
<name>/ : public headers
autogen.sh : mandatory file for the autotools
configure.ac : main configure template
Makefile.am : main Makefile template
<name>.pc.in : pkgconfig’s metadata used by
libraries.
/test/
src/ : implementation of the Google test/mock
inc/ : headers used by the Google test/mock
 This template is used by individuals components (binary or library). In case of a binary, no
unit tests are provided.

7. Makefile dependencies
make covering
make DESTDIR=<PATH> install
./autogen.sh
./configure –prefix …
make dist
make or make –j7
make check
make html
 This target generates html output from the lcov
analysis and provides human readable data about the
source code covering by the unitary tests.
--enable-covering should be set to the configure script
 This target launches the unitary tests using the Google
test/mock and provides junit.xml report per unit test.
If gcov has been set, it will generate some information
that will be analysed by lcov.
--enable-test should be set to the configure script
make fullhtml
 This target generates a documentation
of the implementation.
 This target generates a
documentation from the latest
public API (headers).
This target generates the
(cross)compilation
and the link of the component.
This target generates a tarball..
This target intalls the files generated during
the build process.
Generates the final files (Makefile, config.h, … )

8. Adaptations needed
 configure.ac :
 Add the --enable-test support that exports HAVE_TEST to
the Makefiles.
Add the --enable-covering support for adding the gcov flags
to the build process.
 Makefile.am :
Add the make check support to make the unit tests automatic
Add the make covering support to make the lcov analysis
automatic.

9. Unit tests 1/5
 Name : --enable-test
 Mandatory : no
 If this flags is not set, the unit test binaries are not built. Thus, the “make check” command
doesn’t do anything at all.

10. Unit tests 2/5
dnl --------------------------------------------
dnl Brief : enable or disable the unitary test
dnl Mandatory : no
dnl Values : none (just enable the tests if set)
dnl Default's value : disable
dnl --------------------------------------------
AC_ARG_ENABLE(test,
AS_HELP_STRING([--enable-test], [Enable the test unitary support [default=no]]),
[case "${enableval}" in
yes) have_test=true ;;
no) have_test=false ;;
*) AC_MSG_ERROR(bad value ${enableval} for --enable-test) ;;
esac],
[have_test=false])
dnl --------------------------------------------
dnl Export the conditional HAVE_TEST variable to the Makefiles
dnl --------------------------------------------
AM_CONDITIONAL(HAVE_TEST, $have_test)
dnl --------------------------------------------
dnl Test the have_test variable and if equal to true
dnl --------------------------------------------
AC_MSG_CHECKING(Checking the test support)
if test "$have_test" != "false" ; then
dnl --------------------------------------------
dnl Display the result of the test ... yes
dnl --------------------------------------------
AC_MSG_RESULT([yes])
LIBGTEST_REQUIRED_VERSION=1.5.0
PKG_CHECK_MODULES([GTEST],[gtest >= $LIBGTEST_REQUIRED_VERSION],[have_libgtest=yes],[have_libgtest=no])
if test "$have_libgtest" = no ; then
AC_MSG_ERROR([Missing libgtest library (http://code.google.com/p/googletest/) !!])
fi
LIBGMOCK_REQUIRED_VERSION=1.5.0
PKG_CHECK_MODULES([GMOCK],[gmock >= $LIBGMOCK_REQUIRED_VERSION],[have_libgmock=yes],[have_libgmock=no])
if test "$have_libgmock" = no ; then
AC_MSG_ERROR([Missing libgmock library !!])
fi
else
dnl --------------------------------------------
dnl Display the result of the test ... no
dnl --------------------------------------------
AC_MSG_RESULT([no])
fi
If you are not using the Google test/mock you can avoid this step. If you need to check both the gtest and gmock
dependencies.
configure.ac

11. dnl --------------------------------------------
dnl Test if the libgtest is well available in the stagingdir
dnl If so get the cflags and ldflag from the .pc file
dnl --------------------------------------------
LIBGTEST_REQUIRED_VERSION=1.5.0
PKG_CHECK_MODULES([GTEST],[gtest >= $LIBGTEST_REQUIRED_VERSION],[have_libgtest=yes],[have_libgtest=no])
if test "$have_libgtest" = no ; then
AC_MSG_ERROR([Missing libgtest library (http://code.google.com/p/googletest/) !!])
fi
Unit tests 3/5
dnl --------------------------------------------
dnl Test if the libgmock is well available in the stagingdir
dnl If so get the cflags and ldflag from the .pc file
dnl --------------------------------------------
LIBGMOCK_REQUIRED_VERSION=1.5.0
PKG_CHECK_MODULES([GMOCK],[gmock >= $LIBGMOCK_REQUIRED_VERSION],[have_libgmock=yes],[have_libgmock=no])
if test "$have_libgmock" = no ; then
AC_MSG_ERROR([Missing libgmock (http://code.google.com/p/googlemock/) library !!])
fi
 Check the Google test library :
 Check the Google mock library :
configure.ac
configure.ac

12. Unit tests : direct call 4/5
# ----------------------------------------------------------------
# Unit Tests
# ----------------------------------------------------------------
# Listing of the program to launch
# Those programs should not be installed
# ----------------------------------------------------------------
if HAVE_TEST
TESTS = toe_gtest_assert
noinst_PROGRAMS = $(TESTS)
toe_gtest_assert_SOURCES = $(top_srcdir)/test/src/toe_gtest_assert.cpp
toe_gtest_assert_CPPFLAGS = -I$(HEADER_DIR)
$(GTEST_CFLAGS)
$(LIBCONFIG_CFLAGS)
$(LIBLOG_CFLAGS)
-DLOG_NAME=TOE_ASSERT
-DLOG_MASK_PRIO=255
toe_gtest_assert_LDFLAGS = $(top_builddir)/.libs/libtoe.a
-lpthread
-lrt
toe_gtest_assert_LDADD = $(GTEST_LIBS)
$(LIBCONFIG_LIBS)
$(LIBLOG_LIBS)
$(EFENCE_LIBS)
else
TESTS =
endif
Makefile.am
 The Makefile will call the toe_gtest_assert binary directly.
make check

13. Unit tests : call over a script 5/5
# ----------------------------------------------------------------
# Unit Tests for par_si
# ----------------------------------------------------------------
# - Listing of the program to launch
# - Those programs should not be installed
# ----------------------------------------------------------------
if HAVE_TEST
check_SCRIPTS = unitary_test.sh
TESTS = $(check_SCRIPTS)
unitary_test.sh:
@rm -f test/pasi_gtest
@cp ./pasi_gtest test/pasi_gtest
@echo "cd test" >> ./unitary_test.sh
@echo "./pasi_gtest " >> ./unitary_test.sh
@chmod +x ./unitary_test.sh
noinst_PROGRAMS = pasi_gtest
pasi_gtest_SOURCES = test/pasi_test.cpp
test/pasi_handle.cpp
test/pasi_atsc_handle.cpp
test/pasi_mpeg_parsing.cpp
test/pasi_dvb_parsing.cpp
test/pasi_atsc_parsing.cpp
test/pasi_string_converter.cpp
pasi_gtest_CPPFLAGS = -I$(HEADER_DIR)
-g
-Wall
-Wextra
pasi_gtest_LDFLAGS = -lrt
-lstdc++
pasi_gtest_LDADD = $(GTEST_LIBS)
$(EFENCE_LIBS)
else
TESTS =
endif
 This Makefile first generates a shell script called unitary_test.sh, then the binary, then launch it.
Makefile.am
make check

14. 14
Code covering 1/2
06/17/10
 Name : --enable-covering
 Mandatory flag : no
 If this flag is set, the ELF objects will be compiled and linked with the following options:
CFLAGS/CXXFLAGS : --fprofile-arcs -ftest-coverage.
LDFLAGS : --fprofile-arcs
 If this flag is not set, it won't add all the covering flags needed.
 This creates an instrumented executable which contains additional instructions that record the
number of times each line of the program is executed. The option -ftest-coverage adds instructions
for counting the number of times individual lines are executed, while -fprofile-arcs incorporates
instrumentation code for each branch of the program. Branch instrumentation records how
frequently different paths are taken through ‘if’ statements and other conditionals. The executable
must then be run to create the coverage data.
 Integration in Hudson through the Cobertura plugin :
http://meekrosoft.wordpress.com/2010/06/02/continuous-code-coverage-with-gcc-googletest-and-
hudson/

15. Code covering 2/2
dnl --------------------------------------------
dnl Brief : enable or disable the buildin covering mode
dnl Mandatory : no
dnl Values : none (just enable the covering mode if set)
dnl --------------------------------------------
AC_ARG_ENABLE(covering,
AS_HELP_STRING([--enable-covering], [Enable the building covering (gnu/gcov) support [default=no]]),
[case "${enableval}" in
yes) have_covering=true ;;
no) have_covering=false ;;
*) AC_MSG_ERROR(bad value ${enableval} for --enable-covering) ;;
esac],
[have_covering=false])
dnl --------------------------------------------
dnl Test the have_covering variable and if equal to true
dnl --------------------------------------------
AC_MSG_CHECKING([Checking the covering support])
if test "$have_covering" = "true" ; then
dnl --------------------------------------------
dnl Display the result of the test ... yes
dnl --------------------------------------------
AC_MSG_RESULT([yes])
dnl --------------------------------------------
dnl Update the CFLAGS and CPPFLAGS with gcov options for gcc
dnl --------------------------------------------
COVERING_CFLAGS=" -fprofile-arcs -ftest-coverage "
COVERING_CPPFLAGS=" -fprofile-arcs -ftest-coverage "
COVERING_LDFLAGS=" -fprofile-arcs "
else
dnl --------------------------------------------
dnl Display the result of the test ... no
dnl --------------------------------------------
AC_MSG_RESULT([no])
fi
configure.ac

16. make covering : launch the kpi
 Analyse the data generated by the unit tests compiled with the gcov support. It makes an html
report in the output directory.
 The lcov host tool (lcov and genhtml) must be available in the server.
 Example of result available in the output directory:
# ----------------------------------------------------------------
# Automatic LCOV analyzing
# ----------------------------------------------------------------
covering: all check
@echo -n "Removing the previous html generation : "
@rm -fR ./output
@echo "OK"
@echo -n "Analyzing the covering : "
@lcov -c -d . -b . -o toe.log
@echo "OK"
@echo -n "Generating the html report : "
@genhtml -o output toe.log
@echo "OK"
@echo "--> You can now browse the html generated covering using the output/index.html file."
@echo " firefox output/index.html & "
@rm -f *.gcno *.gcda
Makefile.am

17. TIPS
 The more the unitary tests (based on Google test/mock) will test
functions, the more the KPI will give good data.
 The main goal is to target 100% but because some tests can
be difficult to check (exception, interraction with the system, … )
 The tests should be runable on Hudson.
 Actually each component provides the following data:
o Number of lines (wc –l)
o % of lines tested
o % of functions tested
o % of branches tested