An overview of how to add CTest support to your existing CMake build and upload the test results to a web based dashboard with CDash.

1. Richard Thomson
Senior Software Engineer, NVIDIA
@LegalizeAdulthd
http://LegalizeAdulthood.wordpress.com
legalize@xmission.com

2. Outline
 Adding test support in CMakeLists.txt
 Running Tests with CTest
 Viewing Test Trends in CDash

3. Tests in CMake
 Tests are commands
 Tests pass when:
 Process return code is zero
 Process output matches success regex
 Process output does not match fail regex

4. CMakeLists.txt Requirements
 Call enable_testing() after project()
 Call add_test() to add each test command

5. add_test Syntax
add_test(
NAME <name>
COMMAND <command> [<arg>...]
[CONFIGURATIONS <config>...]
[WORKING_DIRECTORY <dir>]
[COMMAND_EXPAND_LISTS]
)

6. add_test Arguments
 <name> names the test
 Multiple tests can run the same command
 <command> can be a CMake target
 <arg> can use generator expressions
 Use absolute paths for working directory
 Use COMMAND_EXPAND_LISTS if your
arguments can expand to empty strings

7. Setting Test Properties
Tests are configured through properties
set_property(
TEST <test>
PROPERTY <name> [<value>...] )
set_tests_properties(
<test> [<test> ...]
PROPERTIES <prop> <value>
[<prop> <value>...] )
Use ; separated value lists with set_tests_properties

8. Common Test Properties
 Running tests
 ENVIRONMENT
 WORKING_DIRECTORY
 DEPENDS
 FIXTURES_SETUP
 FIXTURES_REQUIRED
 FIXTURES_CLEANUP
 DISABLED
 PROCESSORS
 RESOURCE_GROUPS
 RESOURCE_LOCK
 RUN_SERIAL
 TIMEOUT
 TIMEOUT_AFTER_MATCH
 Success determination:
 PASS_REGULAR_EXPRESSION
 FAIL_REGULAR_EXPRESSION
 SKIP_REGULAR_EXPRESSION
 SKIP_RETURN_CODE
 WILL_FAIL
 Test organization:
 LABELS

9. Configuring Test Result
 WILL_FAIL
 Inverts the success/fail sense
 SKIP_RETURN_CODE
 If command returns this code, test is skipped
 PASS_REGULAR_EXPRESSION
 FAIL_REGULAR_EXPRESSION
 SKIP_REGULAR_EXPRESSION
 Evaluated against test output to determine test
result

10.  LABELS property contains a list of label
names
 Use with –L <regex> argument to ctest to
select tests to run by regex match
Organizing Tests

11. Test Dependencies
 Does one test depend on another test to be
run first?
 Use the DEPENDS property to specify the
relationship
 Only controls order of test execution;
dependent test is run regardless of success
of the dependency

12. Test Fixtures
 Use a fixture to orchestrate set up and tear
down for a test case
 A fixture is just a name
 Test fixture set up is just a test with
FIXTURES_SETUP property
 Test fixture tear down is just a test with
FIXTURES_CLEANUP property
 Test using a fixture has
FIXTURES_REQUIRED property

13. Test Fixture Example
set_tests_properties(startServer
PROPERTIES FIXTURES_SETUP server)
set_tests_properties(stopServer
PROPERTIES FIXTURES_CLEANUP server)
set_tests_properties(databaseUp
PROPERTIES FIXTURES_SETUP database)
set_tests_properties(clientNoDB
PROPERTIES FIXTURES_REQUIRED server)
set_tests_properties(client
PROPERTIES
FIXTURES_REQUIRED "server;database")

14. How CTest Works
 enable_testing establishes the location of a
cmake script in the build directory
 add_test writes commands to the cmake
script
 ctest processes the script to run tests
 Run ctest from the location of the cmake
script (usually top-level build directory)

15. Running CTest
 ctest [args...]
 -C <config> select config to run
 needed for multi-config generators
 -R <regex>, -E <regex>
 Specify tests to run/exclude based on test name
 --timeout <seconds>
 --stop-time <time-of-day>
 -j <n> for parallel execution
 --resource-spec-file <path>
 -L <regex> to select tests by label

16. Test Resource Constraints
 Tests might need lots of CPU
 Tests might need lots of RAM
 Tests might need lots of GPU
 etc.
 Running tests in parallel can cause false
failures due of resource exhaustion
 RUN_SERIAL property can force a test to run
alone

17. RESOURCE_LOCK Property
 A resource is just a name
 Tests requiring the resource have the name
in its RESOURCE_LOCK property
 Only one test per named resource will run
at a time

18. RESOURCE_GROUPS
Property
 Specify quantity of resources in each group
needed by a test
 Configure the resources available in a
JSON file
 Check the documentation for details of
specifying and configuring resources

19. GoogleTest Support 3.9
include(GoogleTest)
gtest_add_tests(
TARGET <target>
[SOURCES file1...]
[EXTRA_ARGS arg1...]
[WORKING_DIRECTORY dir]
[TEST_PREFIX prefix]
[TEST_SUFFIX suffix]
[SKIP_DEPENDENCY]
[TEST_LIST outVar]
)

20. GoogleTest Support 3.9
 Scans source files to identify tests
 Adds one CTest test case per Google Test
test case
 Variable containing test names can be used
to further customize test cases
 Misses parameterized tests
 Misses tests defined through custom
macros

21. GoogleTest Support 3.10
include(GoogleTest)
gtest_discover_tests(
TARGET <target>
[EXTRA_ARGS arg1...]
[WORKING_DIRECTORY dir]
[TEST_PREFIX prefix]
[TEST_SUFFIX suffix]
[NO_PRETTY_TYPES]
[NO_PRETTY_VALUES]
[PROPERTIES name1 value1...]
[TEST_LIST outVar]
[DISCOVERY_TIMEOUT seconds]
)

22. GoogleTest Support 3.10
 Runs test executable to get test names
 Tests names aren't available until CTest is
run
 Can't easily further customize tests via
properties at CMake configuration time
 Customization can be done by including
additional hand-written files into the
generated CTest script

23. CDash
 CDash is a web based dashborad for test
results and trends (FOSS)
 CTest can prepare results for submission to
CDash
 CDash organizes data from pipeline steps
according to models that are displayed in
tracks in the dashboard

24. CDash Tracks and Step
Display
 Track
 Step

25. CDash Steps
 Start
 Update
 Configure
 Build
 Test
 Coverage
 Always displays results in the Coverage track
 MemCheck
 Always displays results in the Dynamic Analysis
track
 Submit

26. CDash Models
 Every pipeline is associated with a model
 Model defines default steps and error behavior
 Nightly
 Excludes MemCheck step, continues if Update fails
 Continuous
 Excludes MemCheck step, stops if Update fails
 Experimental
 Excludes Update and MemCheck steps

27. Executing Steps and Pipelines
 ctest -M <model> -T <step> --track <track>
 At least -M or -T must be specified
 Needs a CDash configuration in place
 ex: ctest -M Nightly --track "Nightly Master"

28. CDash Configuration
 Mostly handled by CTest module
 include(CTest)
 After project()
 CTest module defines BUILD_TESTING
 Allows you to add custom test-only build code
 CTest module calls enable_testing() for you
 Requires CTestConfig.cmake at top-level

29. Sample CTestConfig.cmake
# Name used by CDash to refer to the project
set(CTEST_PROJECT_NAME "Foo")
# Start of day to organize results by day
set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")
# CDash submission details
set(CTEST_DROP_METHOD "http")
set(CTEST_DROP_SITE "my.cdash.org")
set(CTEST_DROP_LOCATION
"/submit.php?project=${CTEST_PROJECT_NAME}")
set(CTEST_DROP_SITE_CDASH YES)
# Show command lines in logs
set(CTEST_USE_LAUNCHERS YES)

30. Simpler CTestConfig.cmake
(3.14+)
# Name used by CDash to refer to the project
set(CTEST_PROJECT_NAME "Foo")
# Start of day to organize results by day
set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")
# CDash submission details
set(CTEST_SUBMIT_URL
"http://my.cdash.org/submit.php?project=${CTEST_PROJECT_NAME}")
# Show command lines in logs
set(CTEST_USE_LAUNCHERS YES)

31. Custom Pipeline Execution
 Write a custom CMake script that calls
ctest_xxx commands
 Invoke with ctest -S <script>
 Allows arbitrary payloads to be uploaded to
CDash

32. Custom Test Results
 ATTACHED_FILES,
ATTACHED_FILES_ON_FAIL properties
specify a list of files to be attached to the
test results in the Upload step
 MEASUREMENT property allows a test to
specify a single value that can be tracked
for that test in CDash
 CTest snoops test output for XML fragments
to define measurements; see docs for
details.

33. Going Further
 Consult the CMake, CTest and CDash
documentation for more details
 More details and examples provided in
"Professional CMake: A Practical Guide" by
Craig Scott, http://crascit.com