SlideShare a Scribd company logo
Richard Thomson
Senior Software Engineer, NVIDIA
@LegalizeAdulthd
http://LegalizeAdulthood.wordpress.com
legalize@xmission.com
Outline
 Adding test support in CMakeLists.txt
 Running Tests with CTest
 Viewing Test Trends in CDash
Tests in CMake
 Tests are commands
 Tests pass when:
 Process return code is zero
 Process output matches success regex
 Process output does not match fail regex
CMakeLists.txt Requirements
 Call enable_testing() after project()
 Call add_test() to add each test command
add_test Syntax
add_test(
NAME <name>
COMMAND <command> [<arg>...]
[CONFIGURATIONS <config>...]
[WORKING_DIRECTORY <dir>]
[COMMAND_EXPAND_LISTS]
)
add_test Arguments
 <name> names the test
 Multiple tests can run the same command
 <command> can be a CMake target
 <arg> can use generator expressions
 Use absolute paths for working directory
 Use COMMAND_EXPAND_LISTS if your
arguments can expand to empty strings
Setting Test Properties
Tests are configured through properties
set_property(
TEST <test>
PROPERTY <name> [<value>...] )
set_tests_properties(
<test> [<test> ...]
PROPERTIES <prop> <value>
[<prop> <value>...] )
Use ; separated value lists with set_tests_properties
Common Test Properties
 Running tests
 ENVIRONMENT
 WORKING_DIRECTORY
 DEPENDS
 FIXTURES_SETUP
 FIXTURES_REQUIRED
 FIXTURES_CLEANUP
 DISABLED
 PROCESSORS
 RESOURCE_GROUPS
 RESOURCE_LOCK
 RUN_SERIAL
 TIMEOUT
 TIMEOUT_AFTER_MATCH
 Success determination:
 PASS_REGULAR_EXPRESSION
 FAIL_REGULAR_EXPRESSION
 SKIP_REGULAR_EXPRESSION
 SKIP_RETURN_CODE
 WILL_FAIL
 Test organization:
 LABELS
Configuring Test Result
 WILL_FAIL
 Inverts the success/fail sense
 SKIP_RETURN_CODE
 If command returns this code, test is skipped
 PASS_REGULAR_EXPRESSION
 FAIL_REGULAR_EXPRESSION
 SKIP_REGULAR_EXPRESSION
 Evaluated against test output to determine test
result
 LABELS property contains a list of label
names
 Use with –L <regex> argument to ctest to
select tests to run by regex match
Organizing Tests
Test Dependencies
 Does one test depend on another test to be
run first?
 Use the DEPENDS property to specify the
relationship
 Only controls order of test execution;
dependent test is run regardless of success
of the dependency
Test Fixtures
 Use a fixture to orchestrate set up and tear
down for a test case
 A fixture is just a name
 Test fixture set up is just a test with
FIXTURES_SETUP property
 Test fixture tear down is just a test with
FIXTURES_CLEANUP property
 Test using a fixture has
FIXTURES_REQUIRED property
Test Fixture Example
set_tests_properties(startServer
PROPERTIES FIXTURES_SETUP server)
set_tests_properties(stopServer
PROPERTIES FIXTURES_CLEANUP server)
set_tests_properties(databaseUp
PROPERTIES FIXTURES_SETUP database)
set_tests_properties(clientNoDB
PROPERTIES FIXTURES_REQUIRED server)
set_tests_properties(client
PROPERTIES
FIXTURES_REQUIRED "server;database")
How CTest Works
 enable_testing establishes the location of a
cmake script in the build directory
 add_test writes commands to the cmake
script
 ctest processes the script to run tests
 Run ctest from the location of the cmake
script (usually top-level build directory)
Running CTest
 ctest [args...]
 -C <config> select config to run
 needed for multi-config generators
 -R <regex>, -E <regex>
 Specify tests to run/exclude based on test name
 --timeout <seconds>
 --stop-time <time-of-day>
 -j <n> for parallel execution
 --resource-spec-file <path>
 -L <regex> to select tests by label
Test Resource Constraints
 Tests might need lots of CPU
 Tests might need lots of RAM
 Tests might need lots of GPU
 etc.
 Running tests in parallel can cause false
failures due of resource exhaustion
 RUN_SERIAL property can force a test to run
alone
RESOURCE_LOCK Property
 A resource is just a name
 Tests requiring the resource have the name
in its RESOURCE_LOCK property
 Only one test per named resource will run
at a time
RESOURCE_GROUPS
Property
 Specify quantity of resources in each group
needed by a test
 Configure the resources available in a
JSON file
 Check the documentation for details of
specifying and configuring resources
GoogleTest Support 3.9
include(GoogleTest)
gtest_add_tests(
TARGET <target>
[SOURCES file1...]
[EXTRA_ARGS arg1...]
[WORKING_DIRECTORY dir]
[TEST_PREFIX prefix]
[TEST_SUFFIX suffix]
[SKIP_DEPENDENCY]
[TEST_LIST outVar]
)
GoogleTest Support 3.9
 Scans source files to identify tests
 Adds one CTest test case per Google Test
test case
 Variable containing test names can be used
to further customize test cases
 Misses parameterized tests
 Misses tests defined through custom
macros
GoogleTest Support 3.10
include(GoogleTest)
gtest_discover_tests(
TARGET <target>
[EXTRA_ARGS arg1...]
[WORKING_DIRECTORY dir]
[TEST_PREFIX prefix]
[TEST_SUFFIX suffix]
[NO_PRETTY_TYPES]
[NO_PRETTY_VALUES]
[PROPERTIES name1 value1...]
[TEST_LIST outVar]
[DISCOVERY_TIMEOUT seconds]
)
GoogleTest Support 3.10
 Runs test executable to get test names
 Tests names aren't available until CTest is
run
 Can't easily further customize tests via
properties at CMake configuration time
 Customization can be done by including
additional hand-written files into the
generated CTest script
CDash
 CDash is a web based dashborad for test
results and trends (FOSS)
 CTest can prepare results for submission to
CDash
 CDash organizes data from pipeline steps
according to models that are displayed in
tracks in the dashboard
CDash Tracks and Step
Display
 Track
 Step
CDash Steps
 Start
 Update
 Configure
 Build
 Test
 Coverage
 Always displays results in the Coverage track
 MemCheck
 Always displays results in the Dynamic Analysis
track
 Submit
CDash Models
 Every pipeline is associated with a model
 Model defines default steps and error behavior
 Nightly
 Excludes MemCheck step, continues if Update fails
 Continuous
 Excludes MemCheck step, stops if Update fails
 Experimental
 Excludes Update and MemCheck steps
Executing Steps and Pipelines
 ctest -M <model> -T <step> --track <track>
 At least -M or -T must be specified
 Needs a CDash configuration in place
 ex: ctest -M Nightly --track "Nightly Master"
CDash Configuration
 Mostly handled by CTest module
 include(CTest)
 After project()
 CTest module defines BUILD_TESTING
 Allows you to add custom test-only build code
 CTest module calls enable_testing() for you
 Requires CTestConfig.cmake at top-level
Sample CTestConfig.cmake
# Name used by CDash to refer to the project
set(CTEST_PROJECT_NAME "Foo")
# Start of day to organize results by day
set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")
# CDash submission details
set(CTEST_DROP_METHOD "http")
set(CTEST_DROP_SITE "my.cdash.org")
set(CTEST_DROP_LOCATION
"/submit.php?project=${CTEST_PROJECT_NAME}")
set(CTEST_DROP_SITE_CDASH YES)
# Show command lines in logs
set(CTEST_USE_LAUNCHERS YES)
Simpler CTestConfig.cmake
(3.14+)
# Name used by CDash to refer to the project
set(CTEST_PROJECT_NAME "Foo")
# Start of day to organize results by day
set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC")
# CDash submission details
set(CTEST_SUBMIT_URL
"http://my.cdash.org/submit.php?project=${CTEST_PROJECT_NAME}")
# Show command lines in logs
set(CTEST_USE_LAUNCHERS YES)
Custom Pipeline Execution
 Write a custom CMake script that calls
ctest_xxx commands
 Invoke with ctest -S <script>
 Allows arbitrary payloads to be uploaded to
CDash
Custom Test Results
 ATTACHED_FILES,
ATTACHED_FILES_ON_FAIL properties
specify a list of files to be attached to the
test results in the Upload step
 MEASUREMENT property allows a test to
specify a single value that can be tracked
for that test in CDash
 CTest snoops test output for XML fragments
to define measurements; see docs for
details.
Going Further
 Consult the CMake, CTest and CDash
documentation for more details
 More details and examples provided in
"Professional CMake: A Practical Guide" by
Craig Scott, http://crascit.com

More Related Content

What's hot

Oracle AFD
Oracle AFDOracle AFD
Git best practices workshop
Git best practices workshopGit best practices workshop
Git best practices workshop
Otto Kekäläinen
 
Introduction to Maven
Introduction to MavenIntroduction to Maven
Introduction to Maven
Mindfire Solutions
 
Git and github 101
Git and github 101Git and github 101
Git and github 101
Senthilkumar Gopal
 
Git and Github slides.pdf
Git and Github slides.pdfGit and Github slides.pdf
Git and Github slides.pdf
Tilton2
 
An Introduction to CMake
An Introduction to CMakeAn Introduction to CMake
An Introduction to CMake
ICS
 
Stream1 change sets delivery to stream2 in RTC
Stream1 change sets delivery to stream2 in RTCStream1 change sets delivery to stream2 in RTC
Stream1 change sets delivery to stream2 in RTC
Ankit Vashistha
 
remote-method-guesser - BHUSA2021 Arsenal
remote-method-guesser - BHUSA2021 Arsenal remote-method-guesser - BHUSA2021 Arsenal
remote-method-guesser - BHUSA2021 Arsenal
Tobias Neitzel
 
What's Coming In CloudStack 4.18
What's Coming In CloudStack 4.18What's Coming In CloudStack 4.18
What's Coming In CloudStack 4.18
ShapeBlue
 
Effective CMake
Effective CMakeEffective CMake
Effective CMake
Daniel Pfeifer
 
Learning git
Learning gitLearning git
Learning git
Sid Anand
 
Git in 10 minutes
Git in 10 minutesGit in 10 minutes
Git in 10 minutes
Safique Ahmed Faruque
 
Enterprise Manager: Write powerful scripts with EMCLI
Enterprise Manager: Write powerful scripts with EMCLIEnterprise Manager: Write powerful scripts with EMCLI
Enterprise Manager: Write powerful scripts with EMCLI
Gokhan Atil
 
Container Security Vulnerability Scanning with Trivy
Container Security Vulnerability Scanning with TrivyContainer Security Vulnerability Scanning with Trivy
Container Security Vulnerability Scanning with Trivy
Faheem Memon
 
Performance Tuning Oracle Weblogic Server 12c
Performance Tuning Oracle Weblogic Server 12cPerformance Tuning Oracle Weblogic Server 12c
Performance Tuning Oracle Weblogic Server 12c
Ajith Narayanan
 
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
ScaleGrid.io
 
Windows Operating System Archaeology
Windows Operating System ArchaeologyWindows Operating System Archaeology
Windows Operating System Archaeology
enigma0x3
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
Kyle Hailey
 
Getting the most out of your Oracle 12.2 Optimizer (i.e. The Brain)
Getting the most out of your Oracle 12.2 Optimizer (i.e. The Brain)Getting the most out of your Oracle 12.2 Optimizer (i.e. The Brain)
Getting the most out of your Oracle 12.2 Optimizer (i.e. The Brain)
SolarWinds
 
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
Jo Hoon
 

What's hot (20)

Oracle AFD
Oracle AFDOracle AFD
Oracle AFD
 
Git best practices workshop
Git best practices workshopGit best practices workshop
Git best practices workshop
 
Introduction to Maven
Introduction to MavenIntroduction to Maven
Introduction to Maven
 
Git and github 101
Git and github 101Git and github 101
Git and github 101
 
Git and Github slides.pdf
Git and Github slides.pdfGit and Github slides.pdf
Git and Github slides.pdf
 
An Introduction to CMake
An Introduction to CMakeAn Introduction to CMake
An Introduction to CMake
 
Stream1 change sets delivery to stream2 in RTC
Stream1 change sets delivery to stream2 in RTCStream1 change sets delivery to stream2 in RTC
Stream1 change sets delivery to stream2 in RTC
 
remote-method-guesser - BHUSA2021 Arsenal
remote-method-guesser - BHUSA2021 Arsenal remote-method-guesser - BHUSA2021 Arsenal
remote-method-guesser - BHUSA2021 Arsenal
 
What's Coming In CloudStack 4.18
What's Coming In CloudStack 4.18What's Coming In CloudStack 4.18
What's Coming In CloudStack 4.18
 
Effective CMake
Effective CMakeEffective CMake
Effective CMake
 
Learning git
Learning gitLearning git
Learning git
 
Git in 10 minutes
Git in 10 minutesGit in 10 minutes
Git in 10 minutes
 
Enterprise Manager: Write powerful scripts with EMCLI
Enterprise Manager: Write powerful scripts with EMCLIEnterprise Manager: Write powerful scripts with EMCLI
Enterprise Manager: Write powerful scripts with EMCLI
 
Container Security Vulnerability Scanning with Trivy
Container Security Vulnerability Scanning with TrivyContainer Security Vulnerability Scanning with Trivy
Container Security Vulnerability Scanning with Trivy
 
Performance Tuning Oracle Weblogic Server 12c
Performance Tuning Oracle Weblogic Server 12cPerformance Tuning Oracle Weblogic Server 12c
Performance Tuning Oracle Weblogic Server 12c
 
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...
 
Windows Operating System Archaeology
Windows Operating System ArchaeologyWindows Operating System Archaeology
Windows Operating System Archaeology
 
UKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction LocksUKOUG, Oracle Transaction Locks
UKOUG, Oracle Transaction Locks
 
Getting the most out of your Oracle 12.2 Optimizer (i.e. The Brain)
Getting the most out of your Oracle 12.2 Optimizer (i.e. The Brain)Getting the most out of your Oracle 12.2 Optimizer (i.e. The Brain)
Getting the most out of your Oracle 12.2 Optimizer (i.e. The Brain)
 
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
왜 쿠버네티스는 systemd로 cgroup을 관리하려고 할까요
 

Similar to Automated Testing with CMake, CTest and CDash

Mxunit
MxunitMxunit
Mxunit
VIkas Patel
 
Pragmatic unittestingwithj unit
Pragmatic unittestingwithj unitPragmatic unittestingwithj unit
Pragmatic unittestingwithj unit
liminescence
 
Intro to Testing in Zope, Plone
Intro to Testing in Zope, PloneIntro to Testing in Zope, Plone
Intro to Testing in Zope, Plone
Quintagroup
 
Apex Testing and Best Practices
Apex Testing and Best PracticesApex Testing and Best Practices
Apex Testing and Best Practices
Jitendra Zaa
 
Adding unit tests with tSQLt to the database deployment pipeline
Adding unit tests with tSQLt to the database deployment pipelineAdding unit tests with tSQLt to the database deployment pipeline
Adding unit tests with tSQLt to the database deployment pipeline
Eduardo Piairo
 
Nativescript angular
Nativescript angularNativescript angular
Nativescript angular
Christoffer Noring
 
Aspects of 10 Tuning
Aspects of 10 TuningAspects of 10 Tuning
Aspects of 10 Tuning
Sage Computing Services
 
Scala test
Scala testScala test
Scala test
Meetu Maltiar
 
Scala test
Scala testScala test
Building frameworks over Selenium
Building frameworks over SeleniumBuilding frameworks over Selenium
Building frameworks over Selenium
Cristian COȚOI
 
Example of TAF with batch execution of test cases
 Example of TAF with batch execution of test cases  Example of TAF with batch execution of test cases
Example of TAF with batch execution of test cases
COMAQA.BY
 
Cassandra Day NY 2014: Getting Started with the DataStax C# Driver
Cassandra Day NY 2014: Getting Started with the DataStax C# DriverCassandra Day NY 2014: Getting Started with the DataStax C# Driver
Cassandra Day NY 2014: Getting Started with the DataStax C# Driver
DataStax Academy
 
Adding unit tests to the database deployment pipeline
Adding unit tests to the database deployment pipelineAdding unit tests to the database deployment pipeline
Adding unit tests to the database deployment pipeline
Eduardo Piairo
 
Practical Glusto Example
Practical Glusto ExamplePractical Glusto Example
Practical Glusto Example
Gluster.org
 
Adding unit tests with tSQLt to the database deployment pipeline
 Adding unit tests with tSQLt to the database deployment pipeline Adding unit tests with tSQLt to the database deployment pipeline
Adding unit tests with tSQLt to the database deployment pipeline
Eduardo Piairo
 
Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)
Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)
Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)
Matt Fuller
 
Google App Engine Developer - Day3
Google App Engine Developer - Day3Google App Engine Developer - Day3
Google App Engine Developer - Day3
Simon Su
 
Infrastructure-as-Code (IaC) using Terraform
Infrastructure-as-Code (IaC) using TerraformInfrastructure-as-Code (IaC) using Terraform
Infrastructure-as-Code (IaC) using Terraform
Adin Ermie
 
Protractor framework – how to make stable e2e tests for Angular applications
Protractor framework – how to make stable e2e tests for Angular applicationsProtractor framework – how to make stable e2e tests for Angular applications
Protractor framework – how to make stable e2e tests for Angular applications
Ludmila Nesvitiy
 
Intro to tsql
Intro to tsqlIntro to tsql
Intro to tsql
Syed Asrarali
 

Similar to Automated Testing with CMake, CTest and CDash (20)

Mxunit
MxunitMxunit
Mxunit
 
Pragmatic unittestingwithj unit
Pragmatic unittestingwithj unitPragmatic unittestingwithj unit
Pragmatic unittestingwithj unit
 
Intro to Testing in Zope, Plone
Intro to Testing in Zope, PloneIntro to Testing in Zope, Plone
Intro to Testing in Zope, Plone
 
Apex Testing and Best Practices
Apex Testing and Best PracticesApex Testing and Best Practices
Apex Testing and Best Practices
 
Adding unit tests with tSQLt to the database deployment pipeline
Adding unit tests with tSQLt to the database deployment pipelineAdding unit tests with tSQLt to the database deployment pipeline
Adding unit tests with tSQLt to the database deployment pipeline
 
Nativescript angular
Nativescript angularNativescript angular
Nativescript angular
 
Aspects of 10 Tuning
Aspects of 10 TuningAspects of 10 Tuning
Aspects of 10 Tuning
 
Scala test
Scala testScala test
Scala test
 
Scala test
Scala testScala test
Scala test
 
Building frameworks over Selenium
Building frameworks over SeleniumBuilding frameworks over Selenium
Building frameworks over Selenium
 
Example of TAF with batch execution of test cases
 Example of TAF with batch execution of test cases  Example of TAF with batch execution of test cases
Example of TAF with batch execution of test cases
 
Cassandra Day NY 2014: Getting Started with the DataStax C# Driver
Cassandra Day NY 2014: Getting Started with the DataStax C# DriverCassandra Day NY 2014: Getting Started with the DataStax C# Driver
Cassandra Day NY 2014: Getting Started with the DataStax C# Driver
 
Adding unit tests to the database deployment pipeline
Adding unit tests to the database deployment pipelineAdding unit tests to the database deployment pipeline
Adding unit tests to the database deployment pipeline
 
Practical Glusto Example
Practical Glusto ExamplePractical Glusto Example
Practical Glusto Example
 
Adding unit tests with tSQLt to the database deployment pipeline
 Adding unit tests with tSQLt to the database deployment pipeline Adding unit tests with tSQLt to the database deployment pipeline
Adding unit tests with tSQLt to the database deployment pipeline
 
Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)
Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)
Presto Testing Tools: Benchto & Tempto (Presto Boston Meetup 10062015)
 
Google App Engine Developer - Day3
Google App Engine Developer - Day3Google App Engine Developer - Day3
Google App Engine Developer - Day3
 
Infrastructure-as-Code (IaC) using Terraform
Infrastructure-as-Code (IaC) using TerraformInfrastructure-as-Code (IaC) using Terraform
Infrastructure-as-Code (IaC) using Terraform
 
Protractor framework – how to make stable e2e tests for Angular applications
Protractor framework – how to make stable e2e tests for Angular applicationsProtractor framework – how to make stable e2e tests for Angular applications
Protractor framework – how to make stable e2e tests for Angular applications
 
Intro to tsql
Intro to tsqlIntro to tsql
Intro to tsql
 

More from Richard Thomson

Vintage Computing Festival Midwest 18 2023-09-09 What's In A Terminal.pdf
Vintage Computing Festival Midwest 18 2023-09-09 What's In A Terminal.pdfVintage Computing Festival Midwest 18 2023-09-09 What's In A Terminal.pdf
Vintage Computing Festival Midwest 18 2023-09-09 What's In A Terminal.pdf
Richard Thomson
 
Feature and platform testing with CMake
Feature and platform testing with CMakeFeature and platform testing with CMake
Feature and platform testing with CMake
Richard Thomson
 
BEFLIX
BEFLIXBEFLIX
SIMD Processing Using Compiler Intrinsics
SIMD Processing Using Compiler IntrinsicsSIMD Processing Using Compiler Intrinsics
SIMD Processing Using Compiler Intrinsics
Richard Thomson
 
Modern C++
Modern C++Modern C++
Modern C++
Richard Thomson
 
Cross Platform Mobile Development with Visual Studio 2015 and C++
Cross Platform Mobile Development with Visual Studio 2015 and C++Cross Platform Mobile Development with Visual Studio 2015 and C++
Cross Platform Mobile Development with Visual Studio 2015 and C++
Richard Thomson
 
Consuming and Creating Libraries in C++
Consuming and Creating Libraries in C++Consuming and Creating Libraries in C++
Consuming and Creating Libraries in C++
Richard Thomson
 
Web mashups with NodeJS
Web mashups with NodeJSWeb mashups with NodeJS
Web mashups with NodeJS
Richard Thomson
 
C traps and pitfalls for C++ programmers
C traps and pitfalls for C++ programmersC traps and pitfalls for C++ programmers
C traps and pitfalls for C++ programmers
Richard Thomson
 

More from Richard Thomson (9)

Vintage Computing Festival Midwest 18 2023-09-09 What's In A Terminal.pdf
Vintage Computing Festival Midwest 18 2023-09-09 What's In A Terminal.pdfVintage Computing Festival Midwest 18 2023-09-09 What's In A Terminal.pdf
Vintage Computing Festival Midwest 18 2023-09-09 What's In A Terminal.pdf
 
Feature and platform testing with CMake
Feature and platform testing with CMakeFeature and platform testing with CMake
Feature and platform testing with CMake
 
BEFLIX
BEFLIXBEFLIX
BEFLIX
 
SIMD Processing Using Compiler Intrinsics
SIMD Processing Using Compiler IntrinsicsSIMD Processing Using Compiler Intrinsics
SIMD Processing Using Compiler Intrinsics
 
Modern C++
Modern C++Modern C++
Modern C++
 
Cross Platform Mobile Development with Visual Studio 2015 and C++
Cross Platform Mobile Development with Visual Studio 2015 and C++Cross Platform Mobile Development with Visual Studio 2015 and C++
Cross Platform Mobile Development with Visual Studio 2015 and C++
 
Consuming and Creating Libraries in C++
Consuming and Creating Libraries in C++Consuming and Creating Libraries in C++
Consuming and Creating Libraries in C++
 
Web mashups with NodeJS
Web mashups with NodeJSWeb mashups with NodeJS
Web mashups with NodeJS
 
C traps and pitfalls for C++ programmers
C traps and pitfalls for C++ programmersC traps and pitfalls for C++ programmers
C traps and pitfalls for C++ programmers
 

Recently uploaded

Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
Upturn India Technologies
 
Going AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applicationsGoing AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applications
Alina Yurenko
 
TMU毕业证书精仿办理
TMU毕业证书精仿办理TMU毕业证书精仿办理
TMU毕业证书精仿办理
aeeva
 
42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert
vaishalijagtap12
 
The Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdfThe Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdf
mohitd6
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
Patrick Weigel
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
dakas1
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
Drona Infotech
 
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
kalichargn70th171
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
kgyxske
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
KrishnaveniMohan1
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
ervikas4
 
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
dakas1
 
Stork Product Overview: An AI-Powered Autonomous Delivery Fleet
Stork Product Overview: An AI-Powered Autonomous Delivery FleetStork Product Overview: An AI-Powered Autonomous Delivery Fleet
Stork Product Overview: An AI-Powered Autonomous Delivery Fleet
Vince Scalabrino
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
Yara Milbes
 
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and MoreManyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
narinav14
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
campbellclarkson
 
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISDECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
Tier1 app
 
Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...
Paul Brebner
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
sandeepmenon62
 

Recently uploaded (20)

Upturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in NashikUpturn India Technologies - Web development company in Nashik
Upturn India Technologies - Web development company in Nashik
 
Going AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applicationsGoing AOT: Everything you need to know about GraalVM for Java applications
Going AOT: Everything you need to know about GraalVM for Java applications
 
TMU毕业证书精仿办理
TMU毕业证书精仿办理TMU毕业证书精仿办理
TMU毕业证书精仿办理
 
42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert42 Ways to Generate Real Estate Leads - Sellxpert
42 Ways to Generate Real Estate Leads - Sellxpert
 
The Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdfThe Role of DevOps in Digital Transformation.pdf
The Role of DevOps in Digital Transformation.pdf
 
WWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders AustinWWDC 2024 Keynote Review: For CocoaCoders Austin
WWDC 2024 Keynote Review: For CocoaCoders Austin
 
一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理一比一原版(USF毕业证)旧金山大学毕业证如何办理
一比一原版(USF毕业证)旧金山大学毕业证如何办理
 
Mobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona InfotechMobile App Development Company In Noida | Drona Infotech
Mobile App Development Company In Noida | Drona Infotech
 
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdfThe Comprehensive Guide to Validating Audio-Visual Performances.pdf
The Comprehensive Guide to Validating Audio-Visual Performances.pdf
 
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
一比一原版(sdsu毕业证书)圣地亚哥州立大学毕业证如何办理
 
Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.Penify - Let AI do the Documentation, you write the Code.
Penify - Let AI do the Documentation, you write the Code.
 
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptxMigration From CH 1.0 to CH 2.0 and  Mule 4.6 & Java 17 Upgrade.pptx
Migration From CH 1.0 to CH 2.0 and Mule 4.6 & Java 17 Upgrade.pptx
 
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
一比一原版(UMN毕业证)明尼苏达大学毕业证如何办理
 
Stork Product Overview: An AI-Powered Autonomous Delivery Fleet
Stork Product Overview: An AI-Powered Autonomous Delivery FleetStork Product Overview: An AI-Powered Autonomous Delivery Fleet
Stork Product Overview: An AI-Powered Autonomous Delivery Fleet
 
The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024The Rising Future of CPaaS in the Middle East 2024
The Rising Future of CPaaS in the Middle East 2024
 
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and MoreManyata Tech Park Bangalore_ Infrastructure, Facilities and More
Manyata Tech Park Bangalore_ Infrastructure, Facilities and More
 
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
🏎️Tech Transformation: DevOps Insights from the Experts 👩‍💻
 
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSISDECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
DECODING JAVA THREAD DUMPS: MASTER THE ART OF ANALYSIS
 
Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...Superpower Your Apache Kafka Applications Development with Complementary Open...
Superpower Your Apache Kafka Applications Development with Complementary Open...
 
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptxOperational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
Operational ease MuleSoft and Salesforce Service Cloud Solution v1.0.pptx
 

Automated Testing with CMake, CTest and CDash

  • 1. Richard Thomson Senior Software Engineer, NVIDIA @LegalizeAdulthd http://LegalizeAdulthood.wordpress.com legalize@xmission.com
  • 2. Outline  Adding test support in CMakeLists.txt  Running Tests with CTest  Viewing Test Trends in CDash
  • 3. Tests in CMake  Tests are commands  Tests pass when:  Process return code is zero  Process output matches success regex  Process output does not match fail regex
  • 4. CMakeLists.txt Requirements  Call enable_testing() after project()  Call add_test() to add each test command
  • 5. add_test Syntax add_test( NAME <name> COMMAND <command> [<arg>...] [CONFIGURATIONS <config>...] [WORKING_DIRECTORY <dir>] [COMMAND_EXPAND_LISTS] )
  • 6. add_test Arguments  <name> names the test  Multiple tests can run the same command  <command> can be a CMake target  <arg> can use generator expressions  Use absolute paths for working directory  Use COMMAND_EXPAND_LISTS if your arguments can expand to empty strings
  • 7. Setting Test Properties Tests are configured through properties set_property( TEST <test> PROPERTY <name> [<value>...] ) set_tests_properties( <test> [<test> ...] PROPERTIES <prop> <value> [<prop> <value>...] ) Use ; separated value lists with set_tests_properties
  • 8. Common Test Properties  Running tests  ENVIRONMENT  WORKING_DIRECTORY  DEPENDS  FIXTURES_SETUP  FIXTURES_REQUIRED  FIXTURES_CLEANUP  DISABLED  PROCESSORS  RESOURCE_GROUPS  RESOURCE_LOCK  RUN_SERIAL  TIMEOUT  TIMEOUT_AFTER_MATCH  Success determination:  PASS_REGULAR_EXPRESSION  FAIL_REGULAR_EXPRESSION  SKIP_REGULAR_EXPRESSION  SKIP_RETURN_CODE  WILL_FAIL  Test organization:  LABELS
  • 9. Configuring Test Result  WILL_FAIL  Inverts the success/fail sense  SKIP_RETURN_CODE  If command returns this code, test is skipped  PASS_REGULAR_EXPRESSION  FAIL_REGULAR_EXPRESSION  SKIP_REGULAR_EXPRESSION  Evaluated against test output to determine test result
  • 10.  LABELS property contains a list of label names  Use with –L <regex> argument to ctest to select tests to run by regex match Organizing Tests
  • 11. Test Dependencies  Does one test depend on another test to be run first?  Use the DEPENDS property to specify the relationship  Only controls order of test execution; dependent test is run regardless of success of the dependency
  • 12. Test Fixtures  Use a fixture to orchestrate set up and tear down for a test case  A fixture is just a name  Test fixture set up is just a test with FIXTURES_SETUP property  Test fixture tear down is just a test with FIXTURES_CLEANUP property  Test using a fixture has FIXTURES_REQUIRED property
  • 13. Test Fixture Example set_tests_properties(startServer PROPERTIES FIXTURES_SETUP server) set_tests_properties(stopServer PROPERTIES FIXTURES_CLEANUP server) set_tests_properties(databaseUp PROPERTIES FIXTURES_SETUP database) set_tests_properties(clientNoDB PROPERTIES FIXTURES_REQUIRED server) set_tests_properties(client PROPERTIES FIXTURES_REQUIRED "server;database")
  • 14. How CTest Works  enable_testing establishes the location of a cmake script in the build directory  add_test writes commands to the cmake script  ctest processes the script to run tests  Run ctest from the location of the cmake script (usually top-level build directory)
  • 15. Running CTest  ctest [args...]  -C <config> select config to run  needed for multi-config generators  -R <regex>, -E <regex>  Specify tests to run/exclude based on test name  --timeout <seconds>  --stop-time <time-of-day>  -j <n> for parallel execution  --resource-spec-file <path>  -L <regex> to select tests by label
  • 16. Test Resource Constraints  Tests might need lots of CPU  Tests might need lots of RAM  Tests might need lots of GPU  etc.  Running tests in parallel can cause false failures due of resource exhaustion  RUN_SERIAL property can force a test to run alone
  • 17. RESOURCE_LOCK Property  A resource is just a name  Tests requiring the resource have the name in its RESOURCE_LOCK property  Only one test per named resource will run at a time
  • 18. RESOURCE_GROUPS Property  Specify quantity of resources in each group needed by a test  Configure the resources available in a JSON file  Check the documentation for details of specifying and configuring resources
  • 19. GoogleTest Support 3.9 include(GoogleTest) gtest_add_tests( TARGET <target> [SOURCES file1...] [EXTRA_ARGS arg1...] [WORKING_DIRECTORY dir] [TEST_PREFIX prefix] [TEST_SUFFIX suffix] [SKIP_DEPENDENCY] [TEST_LIST outVar] )
  • 20. GoogleTest Support 3.9  Scans source files to identify tests  Adds one CTest test case per Google Test test case  Variable containing test names can be used to further customize test cases  Misses parameterized tests  Misses tests defined through custom macros
  • 21. GoogleTest Support 3.10 include(GoogleTest) gtest_discover_tests( TARGET <target> [EXTRA_ARGS arg1...] [WORKING_DIRECTORY dir] [TEST_PREFIX prefix] [TEST_SUFFIX suffix] [NO_PRETTY_TYPES] [NO_PRETTY_VALUES] [PROPERTIES name1 value1...] [TEST_LIST outVar] [DISCOVERY_TIMEOUT seconds] )
  • 22. GoogleTest Support 3.10  Runs test executable to get test names  Tests names aren't available until CTest is run  Can't easily further customize tests via properties at CMake configuration time  Customization can be done by including additional hand-written files into the generated CTest script
  • 23. CDash  CDash is a web based dashborad for test results and trends (FOSS)  CTest can prepare results for submission to CDash  CDash organizes data from pipeline steps according to models that are displayed in tracks in the dashboard
  • 24. CDash Tracks and Step Display  Track  Step
  • 25. CDash Steps  Start  Update  Configure  Build  Test  Coverage  Always displays results in the Coverage track  MemCheck  Always displays results in the Dynamic Analysis track  Submit
  • 26. CDash Models  Every pipeline is associated with a model  Model defines default steps and error behavior  Nightly  Excludes MemCheck step, continues if Update fails  Continuous  Excludes MemCheck step, stops if Update fails  Experimental  Excludes Update and MemCheck steps
  • 27. Executing Steps and Pipelines  ctest -M <model> -T <step> --track <track>  At least -M or -T must be specified  Needs a CDash configuration in place  ex: ctest -M Nightly --track "Nightly Master"
  • 28. CDash Configuration  Mostly handled by CTest module  include(CTest)  After project()  CTest module defines BUILD_TESTING  Allows you to add custom test-only build code  CTest module calls enable_testing() for you  Requires CTestConfig.cmake at top-level
  • 29. Sample CTestConfig.cmake # Name used by CDash to refer to the project set(CTEST_PROJECT_NAME "Foo") # Start of day to organize results by day set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC") # CDash submission details set(CTEST_DROP_METHOD "http") set(CTEST_DROP_SITE "my.cdash.org") set(CTEST_DROP_LOCATION "/submit.php?project=${CTEST_PROJECT_NAME}") set(CTEST_DROP_SITE_CDASH YES) # Show command lines in logs set(CTEST_USE_LAUNCHERS YES)
  • 30. Simpler CTestConfig.cmake (3.14+) # Name used by CDash to refer to the project set(CTEST_PROJECT_NAME "Foo") # Start of day to organize results by day set(CTEST_NIGHTLY_START_TIME "01:00:00 UTC") # CDash submission details set(CTEST_SUBMIT_URL "http://my.cdash.org/submit.php?project=${CTEST_PROJECT_NAME}") # Show command lines in logs set(CTEST_USE_LAUNCHERS YES)
  • 31. Custom Pipeline Execution  Write a custom CMake script that calls ctest_xxx commands  Invoke with ctest -S <script>  Allows arbitrary payloads to be uploaded to CDash
  • 32. Custom Test Results  ATTACHED_FILES, ATTACHED_FILES_ON_FAIL properties specify a list of files to be attached to the test results in the Upload step  MEASUREMENT property allows a test to specify a single value that can be tracked for that test in CDash  CTest snoops test output for XML fragments to define measurements; see docs for details.
  • 33. Going Further  Consult the CMake, CTest and CDash documentation for more details  More details and examples provided in "Professional CMake: A Practical Guide" by Craig Scott, http://crascit.com