Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

C++ Testing Techniques Tips and Tricks - C++ London

118 views

Published on

Links from the talk are available at:
https://github.com/claremacrae/talks/blob/master/Cpp_Testing_Techniques_Tips_and_Tricks.md#top

Presented at C++ London Meetup, on Thursday, November 21, 2019.

Abstract:

An assortment of practical techniques to make it easier to write effective automated tests of C++ code, both old and new.

I share some approaches for easier handling of commonly troublesome testing scenarios. This is a brand new talk, that is independent of test frameworks, and even covers a little for those creating Qt desktop applications.

Published in: Software
  • Be the first to comment

C++ Testing Techniques Tips and Tricks - C++ London

  1. 1. 1 C++ Testing Techniques, Tips and Tricks Clare Macrae (She/her) clare@claremacrae.co.uk 21 November 2019 C++ London
  2. 2. 2 About Me • Scientific C++ and Qt developer since 1999 • My mission: Sustainable and efficient testing and refactoring of legacy code – Co-author of “Approval Tests for C++” • Consulting & training via “Clare Macrae Consulting Ltd” – https://claremacrae.co.uk • All links from this talk via: – github.com/claremacrae/talks
  3. 3. 3
  4. 4. 4 Why? • Share things I wish I knew earlier – Some well known – Some less so… • High-level overview, with links – Avoiding detail • Anything unclear? – Please ask!
  5. 5. 5 Getting Started Testing C++
  6. 6. 6 Tip: Pick a Test Framework • Lots to choose from, including: • Get to know it well!
  7. 7. 7 Tip: Practice!
  8. 8. 8 Practice!
  9. 9. 9 Scenario: Writing your first test • Getting from 0 to 1 tests is always the hardest – that’s normal • Getting from 1 to 2, and then 2 to 3, always much easier • Step 1: Make the first test as easy as possible – it’s going to be hard! • Step 2: Write at least 3 tests before deciding if it’s a good idea!
  10. 10. 10 Techniques for Testing
  11. 11. 11 Traditional unit tests // Let Catch provide main(): #define CATCH_CONFIG_MAIN #include <catch2/catch.hpp> int Factorial( int number ) { return number <= 1 ? number : Factorial( number - 1 ) * number; // fail // return number <= 1 ? 1 : Factorial( number - 1 ) * number; // pass } TEST_CASE( "Factorial of 0 is 1 (fail)", "[single-file]" ) { REQUIRE( Factorial(0) == 1 ); } TEST_CASE( "Factorials of 1 and higher are computed (pass)", "[single-file]" ) { REQUIRE( Factorial(1) == 1 ); REQUIRE( Factorial(2) == 2 ); REQUIRE( Factorial(3) == 6 ); REQUIRE( Factorial(10) == 3628800 ); }
  12. 12. 12 Property Based Testing • State the Invariants – what is required to be true
  13. 13. 13 Kevlin Henney: FizzBuzz Trek HD
  14. 14. 14 C++ Property Based Testing • Can roll your own, in helper functions • RapidCheck: – github.com/emil-e/rapidcheck
  15. 15. 15 Summary: Techniques for Testing • Traditional Unit Tests • Property Based Testing
  16. 16. 16 Inheriting Legacy/Existing Code
  17. 17. 17 Typical Scenario • I've inherited some legacy code • It's valuable • I need to add feature • Or fix bug • How can I ever break out of this loop? Need to change the code No tests Not designed for testing Needs refactoring to add tests Can’t refactor without tests
  18. 18. 18 Typical Scenario • • • • • Need to change the code No tests Not designed for testing Needs refactoring to add tests Can’t refactor without tests Topics of this part
  19. 19. 19 How good are your existing tests?
  20. 20. 20 Measure Code Coverage • Caution! • Useful to see unexecuted code • Use a code coverage tool – What to measure?
  21. 21. 21 Line Coverage: CLion 2019.3
  22. 22. 22 Branch Coverage
  23. 23. 23 Reference: Set-up on my Mac • Slightly hard-won: • g++-8 • CodeCoverage.cmake in github.com/bilke/cmake-modules • lcov • ~/.lcovrc genhtml_branch_coverage = 1 lcov_branch_coverage = 1 lcov_excl_br_line = LCOV_EXCL_BR_LINE|CHECK|REQUIRE
  24. 24. 24 Mutation testing: Sabotage the code! • Test the tests • Small changes • Re-run tests • Fail:  • Pass: 
  25. 25. 26 Golden Master Test Setup Input Data Existing Code Save “Golden Master”
  26. 26. 27 Golden Master Tests In Use Input Data Updated Code Pass Fail Output same? Yes No
  27. 27. 28 Approval Tests TEST_CASE( "Factorials up to 10" ) { // Create a container with values 0 ... 10 inclusive std::vector<int> inputs(11); std::iota(inputs.begin(), inputs.end(), 0); // Act on all values in inputs container: Approvals::verifyAll("Factorial", inputs, [](auto i, auto& os) { os << i << "! => " << Factorial(i); }); }
  28. 28. 29 Quickly Testing Legacy C++ Code with Approval Tests
  29. 29. 30 Scenario: Golden Master is a log file • Dates and times? • Object addresses? 2019-01-29 15:27
  30. 30. 31 Options for unstable output • Introduce date-time abstraction? • Customised comparison function? • Or: strip dates from the log file
  31. 31. 32 Tip: Rewrite output file, before testing it 2019-01-29 15:27 [date-time-stamp]
  32. 32. 33 Summary: Legacy Code • Validating existing tests – Consider Code coverage to validate initial tests • Prefer Branch coverage to Line coverage – Consider Mutation testing to break tests • Consider Fuzz testing to find extra test cases • Approval Tests • Be creative to make your tests work for you!
  33. 33. 34 Maintaining Tests
  34. 34. 35 Scenario: Tests too slow for CI • “Builds take 2 to 3 days!” => No hope of Continuous Integration • Most of that was running tests • No prospect of speeding up some very slow tests
  35. 35. 36 Solution: Don’t run slow tests during day • Mark some tests as Slow – Filtering built in to Google Test Framework • Teach CI system to run sub-set of tests – Only run the Slow tests at night – Everything else run on every commit • Thanks to Michael Platings for making this happen!
  36. 36. 37 Speeding up tests • Incredible value of powerful tools for seeing where time goes
  37. 37. 38 Tip: Never tolerate Flickering tests • Tests that fail randomly • Rapidly devalue other tests • Fix them, or delete them! • Especially don’t mix performance tests with unit tests
  38. 38. 39 Summary: Maintaining Tests • Fast feedback from Continuous Integration • Stomp on flickering tests
  39. 39. 40 Testing Qt User Interfaces
  40. 40. 41
  41. 41. 42 Qt’s GUI powers make Automated Testing Harder
  42. 42. 43 Introducing ApprovalTests.cpp.Qt •Goals • Rapid start testing Qt code – Useful even if you don’t use Approval Tests! • Approval Tests support for Qt types – Easy saving of state in Golden Master files • https://github.com/approvals/ApprovalTests.cpp.Qt • https://github.com/approvals/ApprovalTests.cpp.Qt.StarterProject • v.0.0.1
  43. 43. 44 Example: Checking Table Contents • Inherited complex code to set up a table • Want to add at least a first test – of the text in the cells
  44. 44. 45 Verifying a QTableWidget TEST_CASE("It approves a QTableWidget") { // A note on naming: QTableWidget is a concrete class that implements // the more general QTableView. Here we create a QTableWidget, // for convenience. QTableWidget tableWidget; populateTable(tableWidget); ApprovalTestsQt::verifyQTableView(tableWidget); }
  45. 45. 46 Approval file: .tsv
  46. 46. 47 More Info… • Key point: Use Fixtures to write Expressive, Maintainable Tests – slideshare.net/ClareMacrae/quickly-testing-qt-desktop-applications
  47. 47. 48 Summary • It’s never too late to start testing! – Pick a framework and practice it – Explore different types of testing • You can test legacy code too! • Keep maintaining your tests • Even GUIs can be tested
  48. 48. 49 C++ Testing Techniques, Tips and Tricks • All links from this talk, and more, via: – github.com/claremacrae/talks • Sustainable and efficient testing and refactoring of legacy code • Consulting & training via “Clare Macrae Consulting Ltd” – https://claremacrae.co.uk – clare@claremacrae.co.uk • Any Questions? • Any More Tips?

×