This document provides tips and techniques for testing C++ code, including: - Use a test framework and get to know it well - Prioritize testable code over perfect code - Be creative in testing legacy code, such as using golden master tests or approval tests - Maintain tests by filtering slow tests from continuous integration and fixing flickering tests

1. 1
C++ Testing: Techniques, Tips and Tricks
Clare Macrae (She/her)
clare@claremacrae.co.uk
25 February 2020
Cpp Europe

2. 2
About Me
• C++ and Qt developer since 1999
• My mission: Sustainable and efficient testing and refactoring of legacy code
– Co-author of “Approval Tests for C++”
• Consulting & training
– https://claremacrae.co.uk
• All links from this talk via:
– github.com/claremacrae/talks

3. 3
Why?
• Share things I wish I knew earlier
– Some well known
– Some less so…
• High-level overview, with links
– Avoiding detail
• Anything unclear?
– Please ask!

4. 4
Assumptions
• Value of testing – safety net!
• No worries about types of tests
– (unit, integration, regression)

5. 5
Foundations

6. 6
Tip: Pick a Test Framework
• Lots to choose from, including:
• Get to know it well!

7. 7
Catch2 basic example
// Let Catch provide main():
#define CATCH_CONFIG_MAIN
#include <catch2/catch.hpp>
int Factorial( int number ) {
return number <= 1 ? 1 : Factorial( number - 1 ) * number;
}
TEST_CASE( "Factorial of 0 is 1" ) {
REQUIRE( Factorial(0) == 1 );
}
TEST_CASE( "Factorials of 1 and higher are computed" ) {
REQUIRE( Factorial(1) == 1 );
REQUIRE( Factorial(2) == 2 );
REQUIRE( Factorial(3) == 6 );
REQUIRE( Factorial(10) == 3628800 );
}

8. 8
But what about
Real World Code?

9. 9
Rarely that simple!
• Long methods
• Interdependent code
• Tests hard to set up
• Not designed to be testable

10. 10
Tip: Take any reasonable measure!
• … to get code tested!
• Don’t get hung up on perfect code…
• What does that mean?

11. 11
Tip: Take any reasonable measure!
• When you are stuck, write the test you want
• Then safely do whatever you need to your implementation – temporarily!
– Make method static to call from outside?
– Add getters – and maybe even setters?
– Public data?
• Hold your nose!
• Then refactor

12. 12
Prioritise Testable code
Over Perfect code

13. 13
How do I test private code?
Don’t!

14. 14
Tip: Don’t test private code
• Automated tests give freedom to safely change implementation
• Testing private code breaks that
• Only test observable – public – behaviours

15. 15
What could possibly go wrong?
• It’s easy to think your test is passing…
• I’ve been bitten by all of these:
– Forget to add test source to build
– IDE calls a different test
– Test is doing the wrong thing
– Test is accidentally disabled

16. 16
Never trust a test until
you have seen it fail!

17. 17
Book recommendations

18. 18
Summary: Foundations
• Use a test framework – get to know it well
• Take any reasonable measure to add tests
– Then improve it
• Test public behaviour – not private implementation
• Testable code over perfect code
• Never trust a test until you have seen it fail

19. 19
Private code revisited

20. 20
How do I test private code?
Don’t!

21. 21
But what if…
• Tiny real world example
• “GoToLineTool” widget
• User can type in to the spinner

22. 22
Testing goal
• Test that widget communicates changes correctly when user types in to widget
• lineNumberChanged()
• How to test that?

23. 23
Options – none are satisfactory
• Ideas
– Search the widget hierarchy for widgets by name
– Make the spinner widget data member public
– Provide a public accessor to the data member
– Make the test a friend of the class GoToLineTool class
• All if these require the test to be reworked if widget changes

24. 24
Goal: Make tests super easy to write
• And super easy to maintain…
• Test code accessing widgets directly makes that hard

25. 25
Technique: Separate concerns with Fixture
class GoToLineTool : public
QWidget
{
...
public/protected/private:
QSpinBox* spinBox();
QToolButton* goButton();
Implementation Tests – lots of them!Test Fixture
class GoToLineToolFixture
{
private:
GoToLineTool mGoToLineWidget;
public:
void typeCharacterIntoSpinner(
QChar character);
...
TEST_CASE_METHOD(
GoToLineToolFixture,
"GoToLineTool …")
{
// Ask Fixture to act
// on GoToLineTool
typeCharacterIntoSpinner('1');
}

26. 26
Summary: Fixtures
• Test Fixtures can make tests of complex code expressive
• And Maintainable

27. 27
Inheriting Legacy/Existing Code

28. 28
Typical Scenario
• I've inherited some
legacy code
• It's valuable
• I need to add feature
• Or fix bug
• How can I ever break
out of this loop?
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests

29. 29
Typical Scenario
•
•
•
•
•
Need to
change
the code
No tests
Not
designed
for testing
Needs
refactoring
to add
tests
Can’t
refactor
without
tests
Topics of
this part

30. 30
Golden Master Test Setup
Input
Data
Existing
Code
Save
“Golden
Master”

31. 31
Golden Master Tests In Use
Input
Data
Updated
Code
Pass
Fail
Output
same?
Yes
No

32. 32
Approval Tests
TEST_CASE( "Factorials up to 10" )
{
// Create a container with values 0 ... 10 inclusive
std::vector<int> inputs(11);
std::iota(inputs.begin(), inputs.end(), 0);
// Act on all values in inputs container:
Approvals::verifyAll("Factorial", inputs,
[](auto i, auto& os)
{
os << i << "! => " << Factorial(i);
});
}

33. 33
More Info: Quickly Testing Legacy C++ Code
with Approval Tests

34. 34
Scenario: Golden Master is a log file
• Dates and times?
• Object addresses?
2019-01-29 15:27

35. 35
Options for unstable output
• Introduce date-time abstraction?
• Customised comparison function?
• Or: strip dates from the log file
– Credit: Llewellyn Falco

36. 36
Tip: Rewrite output file, before testing it
2019-01-29 15:27 [date-time-stamp]

37. 37
Summary: Legacy Code
• Approval Tests
• Be creative to make your tests work for you!
• After adding tests, start refactoring for maintainability

38. 38
Testing Qt User Interfaces

39. 39

40. 40
Introducing ApprovalTests.cpp.Qt
•Goals
• Quickly start testing Qt code
– Useful even if you don’t use Approval Tests!
• Approval Tests support for Qt types
– Easy saving of state in Golden Master files
• https://github.com/approvals/ApprovalTests.cpp.Qt
• https://github.com/approvals/ApprovalTests.cpp.Qt.StarterProject
• v.0.0.1

41. 41
Example: Checking Table Contents
• Inherited complex code to set up a table
• Want to add at least a first test – of the text in the cells

42. 42
Verifying a QTableWidget
TEST_CASE("It approves a QTableWidget")
{
// A note on naming: QTableWidget is a concrete class that implements
// the more general QTableView. Here we create a QTableWidget,
// for convenience.
QTableWidget tableWidget;
populateTable(tableWidget);
ApprovalTestsQt::verifyQTableView(tableWidget);
}

43. 43
Approval file: .tsv

44. 44
Summary: ApprovalTests.cpp.Qt
• More on this and on using Fixtures to write Expressive, Maintainable Tests
– slideshare.net/ClareMacrae/quickly-testing-qt-desktop-applications

45. 45
Maintaining Tests

46. 46
Scenario: Tests too slow for CI
• “Builds take 2 to 3 days!” => No hope of Continuous Integration
• Most of that was running tests
• No prospect of speeding up some very slow tests

47. 47
Solution: Don’t run slow tests during day
• Mark some tests as Slow
– Filtering built in to Google Test Framework
• Teach CI system to run sub-set of tests
– Only run the Slow tests at night
– Everything else run on every commit
• Thanks to Michael Platings for making this happen!

48. 48
Tip: Never tolerate Flickering tests
• Tests that fail randomly
• Rapidly devalue other tests
• Fix them, or delete them!
• Especially don’t mix performance tests with unit tests

49. 49
Summary: Maintaining Tests
• Fast feedback from Continuous Integration
• Stomp on flickering tests

50. 50
Summary
• It’s never too late to start testing!
– Pick a framework and practice it
– Explore different types of testing
• You can test legacy code too!
• Keep maintaining your tests
• Even GUIs can be tested

51. 51
C++ Testing: Techniques, Tips and Tricks
• All links from this talk, and more, via:
– bit.ly/CppTestingTips
– github.com/claremacrae/talks
• Sustainable and efficient testing and refactoring of legacy code
• Consulting & Training
– https://claremacrae.co.uk
– clare@claremacrae.co.uk
• Any Questions?
• Any More Tips?