Links from the talk are available at: https://github.com/claremacrae/talks/blob/master/Cpp_Testing_Techniques_Tips_and_Tricks.md#top Presented at C++ London Meetup, on Thursday, November 21, 2019. Abstract: An assortment of practical techniques to make it easier to write effective automated tests of C++ code, both old and new. I share some approaches for easier handling of commonly troublesome testing scenarios. This is a brand new talk, that is independent of test frameworks, and even covers a little for those creating Qt desktop applications.

1. 1
C++ Testing Techniques, Tips and Tricks
Clare Macrae (She/her)
clare@claremacrae.co.uk
21 November 2019
C++ London

2. 2
About Me
• Scientific C++ and Qt developer since 1999
• My mission: Sustainable and efficient testing and refactoring of legacy code
– Co-author of “Approval Tests for C++”
• Consulting & training via “Clare Macrae Consulting Ltd”
– https://claremacrae.co.uk
• All links from this talk via:
– github.com/claremacrae/talks

3. 3

4. 4
Why?
• Share things I wish I knew earlier
– Some well known
– Some less so…
• High-level overview, with links
– Avoiding detail
• Anything unclear?
– Please ask!

5. 5
Getting Started Testing C++

6. 6
Tip: Pick a Test Framework
• Lots to choose from, including:
• Get to know it well!

7. 7
Tip: Practice!

8. 8
Practice!

9. 9
Scenario: Writing your first test
• Getting from 0 to 1 tests is always the hardest – that’s normal
• Getting from 1 to 2, and then 2 to 3, always much easier
• Step 1: Make the first test as easy as possible – it’s going to be hard!
• Step 2: Write at least 3 tests before deciding if it’s a good idea!

10. 10
Techniques for Testing

11. 11
Traditional unit tests
// Let Catch provide main():
#define CATCH_CONFIG_MAIN
#include <catch2/catch.hpp>
int Factorial( int number ) {
return number <= 1 ? number : Factorial( number - 1 ) * number; // fail
// return number <= 1 ? 1 : Factorial( number - 1 ) * number; // pass
}
TEST_CASE( "Factorial of 0 is 1 (fail)", "[single-file]" ) {
REQUIRE( Factorial(0) == 1 );
}
TEST_CASE( "Factorials of 1 and higher are computed (pass)", "[single-file]" ) {
REQUIRE( Factorial(1) == 1 );
REQUIRE( Factorial(2) == 2 );
REQUIRE( Factorial(3) == 6 );
REQUIRE( Factorial(10) == 3628800 );
}

12. 12
Property Based Testing
• State the Invariants – what is required to be true

13. 13
Kevlin Henney: FizzBuzz Trek HD

14. 14
C++ Property Based Testing
• Can roll your own, in helper functions
• RapidCheck:
– github.com/emil-e/rapidcheck

15. 15
Summary: Techniques for Testing
• Traditional Unit Tests
• Property Based Testing

16. 16
Inheriting Legacy/Existing Code

17. 17
Typical Scenario
• I've inherited some
legacy code
• It's valuable
• I need to add feature
• Or fix bug
• How can I ever break
out of this loop?
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests

18. 18
Typical Scenario
•
•
•
•
•
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests
Topics of
this part

19. 19
How good are your existing tests?

20. 20
Measure Code Coverage
• Caution!
• Useful to see unexecuted code
• Use a code coverage tool
– What to measure?

21. 21
Line Coverage: CLion 2019.3

22. 22
Branch Coverage

23. 23
Reference: Set-up on my Mac
• Slightly hard-won:
• g++-8
• CodeCoverage.cmake in github.com/bilke/cmake-modules
• lcov
• ~/.lcovrc
genhtml_branch_coverage = 1
lcov_branch_coverage = 1
lcov_excl_br_line = LCOV_EXCL_BR_LINE|CHECK|REQUIRE

24. 24
Mutation testing: Sabotage the code!
• Test the tests
• Small changes
• Re-run tests
• Fail: 
• Pass: 

26. 26
Golden Master Test Setup
Input
Data
Existing
Code
Save
“Golden
Master”

27. 27
Golden Master Tests In Use
Input
Data
Updated
Code
Pass
Fail
Output
same?
Yes
No

28. 28
Approval Tests
TEST_CASE( "Factorials up to 10" )
{
// Create a container with values 0 ... 10 inclusive
std::vector<int> inputs(11);
std::iota(inputs.begin(), inputs.end(), 0);
// Act on all values in inputs container:
Approvals::verifyAll("Factorial", inputs,
[](auto i, auto& os)
{
os << i << "! => " << Factorial(i);
});
}

29. 29
Quickly Testing Legacy C++ Code
with Approval Tests

30. 30
Scenario: Golden Master is a log file
• Dates and times?
• Object addresses?
2019-01-29 15:27

31. 31
Options for unstable output
• Introduce date-time abstraction?
• Customised comparison function?
• Or: strip dates from the log file

32. 32
Tip: Rewrite output file, before testing it
2019-01-29 15:27 [date-time-stamp]

33. 33
Summary: Legacy Code
• Validating existing tests
– Consider Code coverage to validate initial tests
• Prefer Branch coverage to Line coverage
– Consider Mutation testing to break tests
• Consider Fuzz testing to find extra test cases
• Approval Tests
• Be creative to make your tests work for you!

34. 34
Maintaining Tests

35. 35
Scenario: Tests too slow for CI
• “Builds take 2 to 3 days!” => No hope of Continuous Integration
• Most of that was running tests
• No prospect of speeding up some very slow tests

36. 36
Solution: Don’t run slow tests during day
• Mark some tests as Slow
– Filtering built in to Google Test Framework
• Teach CI system to run sub-set of tests
– Only run the Slow tests at night
– Everything else run on every commit
• Thanks to Michael Platings for making this happen!

37. 37
Speeding up tests
• Incredible value of powerful tools for seeing where time goes

38. 38
Tip: Never tolerate Flickering tests
• Tests that fail randomly
• Rapidly devalue other tests
• Fix them, or delete them!
• Especially don’t mix performance tests with unit tests

39. 39
Summary: Maintaining Tests
• Fast feedback from Continuous Integration
• Stomp on flickering tests

40. 40
Testing Qt User Interfaces

41. 41

42. 42
Qt’s GUI powers make
Automated Testing Harder

43. 43
Introducing ApprovalTests.cpp.Qt
•Goals
• Rapid start testing Qt code
– Useful even if you don’t use Approval Tests!
• Approval Tests support for Qt types
– Easy saving of state in Golden Master files
• https://github.com/approvals/ApprovalTests.cpp.Qt
• https://github.com/approvals/ApprovalTests.cpp.Qt.StarterProject
• v.0.0.1

44. 44
Example: Checking Table Contents
• Inherited complex code to set up a table
• Want to add at least a first test – of the text in the cells

45. 45
Verifying a QTableWidget
TEST_CASE("It approves a QTableWidget")
{
// A note on naming: QTableWidget is a concrete class that implements
// the more general QTableView. Here we create a QTableWidget,
// for convenience.
QTableWidget tableWidget;
populateTable(tableWidget);
ApprovalTestsQt::verifyQTableView(tableWidget);
}

46. 46
Approval file: .tsv

47. 47
More Info…
• Key point: Use Fixtures to write Expressive, Maintainable Tests
– slideshare.net/ClareMacrae/quickly-testing-qt-desktop-applications

48. 48
Summary
• It’s never too late to start testing!
– Pick a framework and practice it
– Explore different types of testing
• You can test legacy code too!
• Keep maintaining your tests
• Even GUIs can be tested

49. 49
C++ Testing Techniques, Tips and Tricks
• All links from this talk, and more, via:
– github.com/claremacrae/talks
• Sustainable and efficient testing and refactoring of legacy code
• Consulting & training via “Clare Macrae Consulting Ltd”
– https://claremacrae.co.uk
– clare@claremacrae.co.uk
• Any Questions?
• Any More Tips?