Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Quickly Testing Legacy Code - ACCU York April 2019

71 views

Published on

Presented at ACCU York, on 25th April 2019.

You've inherited some legacy code: it's valuable, but it doesn't have tests, and it wasn't designed to be testable, so you need to start refactoring! But you can't refactor safely until the code has tests, and you can't add tests without refactoring. How can you ever break out of this loop?

I present a new C++ library for applying Llewellyn Falco's "Approval Tests" approach to testing cross-platform C++ code - for both legacy and green-field systems, and a range of testing frameworks.

I describe its use in some real-world situations, including how to quickly lock down the behaviour of legacy code. I show how to quickly achieve good test coverage, even for very large sets of inputs. Finally, I also describe some general techniques I learned along the way.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Quickly Testing Legacy Code - ACCU York April 2019

  1. 1. 1 @ClareMacraeUK Quickly Testing Legacy Code Clare Macrae 25 April 2019
  2. 2. 2 @ClareMacraeUK @ClareMacraeUK
  3. 3. 3 @ClareMacraeUK Contents • Introduction • Legacy Code • Golden Master • Approval Tests • Example • Resources • Summary
  4. 4. 4 @ClareMacraeUK Contents • Introduction • Legacy Code • Golden Master • Approval Tests • Example • Resources • Summary
  5. 5. 5 @ClareMacraeUK
  6. 6. 6 @ClareMacraeUK
  7. 7. 7 @ClareMacraeUK A year of remote pairing later…
  8. 8. 8 @ClareMacraeUK Goal: Share techniques for easier testing in challenging scenarios
  9. 9. 9 @ClareMacraeUK Goal: Share Cross-language techniques for easier testing in challenging scenarios
  10. 10. 10 @ClareMacraeUK Contents • Introduction • Legacy Code • Golden Master • Approval Tests • Example • Resources • Summary
  11. 11. 11 @ClareMacraeUK Quickly Testing Legacy Code
  12. 12. 12 @ClareMacraeUK
  13. 13. 13 @ClareMacraeUK What does Legacy Code really mean? • Michael Feathers –“code without unit tests” • J.B. Rainsberger –“profitable code that we feel afraid to change.” • Kate Gregory –“any code that you want to change, but are afraid to”
  14. 14. 14 @ClareMacraeUK Typical Scenario • I've inherited some legacy code • It's valuable • I need to add feature • Or fix bug • How can I ever break out of this loop? Need to change the code No tests Not designed for testing Needs refactoring to add tests Can’t refactor without tests
  15. 15. 15 @ClareMacraeUK Typical Scenario • • • • • Need to change the code No tests Not designed for testing Needs refactoring to add tests Can’t refactor without tests Topics of this talk
  16. 16. 16 @ClareMacraeUK Assumptions • Value of testing • No worries about types of tests – (unit, integration, regression)
  17. 17. 17 @ClareMacraeUK Any existing tests?
  18. 18. 18 @ClareMacraeUK What, no tests? • If absolutely no tests… • Stop now! • Set one up! • Existing framework
  19. 19. 19 @ClareMacraeUK Popular C++ Test Frameworks Google Test • Google's C++ test framework • https://github.com/google/googletest Catch • Phil Nash’s test framework • https://github.com/catchorg/Catch2
  20. 20. 20 @ClareMacraeUK
  21. 21. 21 @ClareMacraeUK How good are your existing tests?
  22. 22. 22 @ClareMacraeUK First evaluate your tests • If you do have tests…. • Test the tests! • In area you are changing • Reliable?
  23. 23. 23 @ClareMacraeUK Code Coverage • Caution! • Unexecuted code • Techniques • Debugger • Code coverage tool – What to measure?
  24. 24. 24 @ClareMacraeUK 100% Test Coverage Or is it? OpenCppCoverage does not show branch/condition coverage.
  25. 25. 25 @ClareMacraeUK BullseyeCoverage: Unrolls If conditions (line 10) 73% of conditions covered
  26. 26. 26 @ClareMacraeUK Mutation testing: Sabotage the code! • Test the tests • Small changes • Re-run tests • Fail:  • Pass: 
  27. 27. 27 @ClareMacraeUK Mutation testing approaches • By hand, – e.g. + to - • By tool – e.g. Mutate++ - https://github.com/nlohmann/mutate_cpp
  28. 28. 28 @ClareMacraeUK If tests need improving… And unit tests not viable…
  29. 29. 29 @ClareMacraeUK Contents • Introduction • Legacy Code • Golden Master • Approval Tests • Example • Resources • Summary
  30. 30. 30 @ClareMacraeUK Quickly Testing Legacy Code
  31. 31. 31 @ClareMacraeUK Key Phrase : “Locking Down Current Behaviour”
  32. 32. 32 @ClareMacraeUK Writing Unit Tests for Legacy Code • Time-consuming • What’s intended behaviour? • Is there another way?
  33. 33. 33 @ClareMacraeUK Alternative: Golden Master Testing
  34. 34. 34 @ClareMacraeUK Golden Master Test Setup Input Data Existing Code Save “Golden Master”
  35. 35. 35 @ClareMacraeUK Golden Master Tests In Use Input Data Updated Code Pass Fail Output same? Yes No
  36. 36. 36 @ClareMacraeUK Thoughts on Golden Master Tests • Good to start testing legacy systems • Poor Person’s Integration Tests • Depends on ease of – Capturing output – Getting stable output – Reviewing any differences – Avoiding overwriting Master by mistake! • Doesn’t matter that it’s not a Unit Test
  37. 37. 37 @ClareMacraeUK Contents • Introduction • Legacy Code • Golden Master • Approval Tests • Example • Resources • Summary
  38. 38. 38 @ClareMacraeUK Quickly Testing Legacy Code
  39. 39. 39 @ClareMacraeUK One approach: “ApprovalTests” • Llewellyn Falco’s convenient, powerful, flexible Golden Master implementation • Supports many languages And now C++! GO Java Lua .NET .Net.ASP NodeJS Objective-C PHP Perl Python Swift TCL
  40. 40. 40 @ClareMacraeUK ApprovalTests.cpp • New C++ library for applying Llewellyn Falco’s “Approval Tests” approach • For testing cross-platform C++ code (Windows, Linux, Mac) • For legacy and green-field systems • It’s on github
  41. 41. 41 @ClareMacraeUK ApprovalTests.cpp • Header-only • Open source - Apache 2.0 licence
  42. 42. 42 @ClareMacraeUK ApprovalTests.cpp • Works with a range of testing frameworks • Currently supports Catch1, Catch2, Google Test and Okra
  43. 43. 43 @ClareMacraeUK Getting Started with Approvals in C++
  44. 44. 44 @ClareMacraeUK StarterProject with Catch2 • https://github.com/approvals/ApprovalTests.cpp.StarterProject • Download ZIP
  45. 45. 45 @ClareMacraeUK How to use it: Catch2 Boilerplate • Your main.cpp #define APPROVALS_CATCH #include "ApprovalTests.hpp"
  46. 46. 46 @ClareMacraeUK Pure Catch2 Test • A test file #include "Catch.hpp" // Catch-only test TEST_CASE( "Sums are calculated" ) { REQUIRE( 1 + 1 == 2 ); REQUIRE( 1 + 2 == 3 ); }
  47. 47. 47 @ClareMacraeUK Approvals Catch2 Test • A test file (Test02.cpp) #include "ApprovalTests.hpp" #include "Catch.hpp" // Approvals test - test static value, for demo purposes TEST_CASE("TestFixedInput") { Approvals::verify("SomenMulti-linenoutput"); }
  48. 48. 48 @ClareMacraeUK First run • 1. • 2. Differencing tool pops up (Araxis Merge, in this case)
  49. 49. 49 @ClareMacraeUK Actual/Received Expected/Approved
  50. 50. 50 @ClareMacraeUK Actual/Received Expected/Approved
  51. 51. 51 @ClareMacraeUK Actual/Received Expected/Approved
  52. 52. 52 @ClareMacraeUK Second run • I approved the output • Then we commit the test, and the approved file(s) to version control • The diffing tool only shows up if: – there is not (yet) an Approved file or – the Received differs from the Approved
  53. 53. 53 @ClareMacraeUK What’s going on here? • It’s a very convenient form of Golden Master testing • Objects being tested are stringified – that’s a requirement • Captures snapshot of current behaviour • Useful for locking down behaviour of existing code – Not intended to replace Unit Tests
  54. 54. 54 @ClareMacraeUK Example test directory
  55. 55. 55 @ClareMacraeUK How to use it: Catch2 Boilerplate • Your main.cpp #define APPROVALS_CATCH #include "ApprovalTests.hpp“ // Put all approved and received files in "approval_tests/" auto dir = Approvals::useApprovalsSubdirectory("approval_tests");
  56. 56. 56 @ClareMacraeUK How to use it: GoogleTest Boilerplate • Your main.cpp #define APPROVALS_GOOGLETEST #include "ApprovalTests.hpp"
  57. 57. 57 @ClareMacraeUK Pure Google Test • A test file #include <gtest/gtest.h> // Google-only test TEST( Test01, SumsAreCalculated ) { EXPECT_EQ( 1 + 1, 2 ); EXPECT_EQ( 1 + 2, 3 ); }
  58. 58. 58 @ClareMacraeUK Approvals Google Test • A test file #include "ApprovalTests.hpp" #include <gtest/gtest.h> // googletest - test static value, for demo purposes TEST( Test02, TestFixedInput ) { Approvals::verify("SomenMulti-linenoutput"); }
  59. 59. 59 @ClareMacraeUK Reference: Evolving User Guide
  60. 60. 62 @ClareMacraeUK A note on Tools • I’ll be using a variety of tools in the talk • I’ll mention them as I go • Slides of references at the end
  61. 61. 63 @ClareMacraeUK How does this help with legacy code?
  62. 62. 64 @ClareMacraeUK Applying this to legacy code TEST_CASE("New test of legacy feature") { // Standard pattern: // Wrap your legacy feature to test in a function call // that returns some state that can be written to a text file // for verification: const LegacyThing result = doLegacyOperation(); Approvals::verify(result); }
  63. 63. 65 @ClareMacraeUK Implementation class LegacyThing; std::ostream &operator<<(std::ostream &os, const LegacyThing &result) { // write interesting info from result here: os << result…; return os; } LegacyThing doLegacyOperation() { // your implementation here.. return LegacyThing(); }
  64. 64. 66 @ClareMacraeUK Feature: Consistency over machines • Naming of output files • Approved files version controlled
  65. 65. 67 @ClareMacraeUK Feature: Consistency over OSs • Line-endings
  66. 66. 68 @ClareMacraeUK Feature: Consistency over Languages • Consistent concepts and nomenclature in all the implementations
  67. 67. 69 @ClareMacraeUK Feature: Quick to write tests TEST_CASE("verifyAllWithHeaderBeginEndAndLambda") { std::list<int> numbers{ 0, 1, 2, 3}; // Multiple convenience overloads of Approvals::verifyAll() Approvals::verifyAll( "Test Squares", numbers.begin(), numbers.end(), [](int v, std::ostream& s) { s << v << " => " << v*v << 'n' ; }); }
  68. 68. 70 @ClareMacraeUK Feature: Quick to get good coverage TEST_CASE("verifyAllCombinationsWithLambda") { std::vector<std::string> strings{"hello", "world"}; std::vector<int> numbers{1, 2, 3}; CombinationApprovals::verifyAllCombinations< std::vector<std::string>, // The type of element in our first input container std::vector<int>, // The type of element in our second input container std::string>( // The return type from testing one combination of inputs // Lambda that acts on one combination of inputs, and returns the result to be approved: [](std::string s, int i) { return s + " " + std::to_string(i); }, strings, // The first input container numbers); // The second input container }
  69. 69. 71 @ClareMacraeUK All values in single output file: (hello, 1) => hello 1 (hello, 2) => hello 2 (hello, 3) => hello 3 (world, 1) => world 1 (world, 2) => world 2 (world, 3) => world 3
  70. 70. 72 @ClareMacraeUK Customisability: Reporters • Very simple but powerful abstraction • Gives complete user control over how to inspect and act on a failure • If you adopt Approvals, do review the supplied reporters for ideas
  71. 71. 73 @ClareMacraeUK Feature: Change the Reporter in one test TEST_CASE("Demo custom reporter") { std::vector<std::string> v{"hello", "world"}; Approvals::verifyAll(v, MyCustomReporter()); }
  72. 72. 74 @ClareMacraeUK Feature: Change the default Reporter // main.cpp - or in individual tests: #include <memory> auto disposer = Approvals::useAsDefaultReporter( std::make_shared<Windows::BeyondCompare4Reporter>() ); auto disposer = Approvals::useAsDefaultReporter( std::make_shared<GenericDiffReporter>( "C:/Program Files/Araxis/Araxis Merge/Compare.exe") );
  73. 73. 75 @ClareMacraeUK Feature: Don’t run GUIs on build servers // main.cpp auto disposer = Approvals::useAsFrontLoadedReporter( BlockingReporter::onMachineNamed("MyCIMachineName") );
  74. 74. 76 @ClareMacraeUK Challenge: Golden Master is a log file • Dates and times? • Object addresses? 2019-01-29 15:27
  75. 75. 77 @ClareMacraeUK Options for unstable output • Introduce date-time abstraction? • Customised comparison function? • Or: strip dates from the log file
  76. 76. 78 @ClareMacraeUK Tip: Rewrite output file 2019-01-29 15:27 [date-time-stamp]
  77. 77. 79 @ClareMacraeUK Customisability: ApprovalWriter interface class ApprovalWriter { public: virtual std::string getFileExtensionWithDot() = 0; virtual void write(std::string path) = 0; virtual void cleanUpReceived(std::string receivedPath) = 0; };
  78. 78. 80 @ClareMacraeUK Feature: Convention over Configuration • Fewer decisions for developers • Users only specify unusual behaviours
  79. 79. 81 @ClareMacraeUK Contents • Introduction • Legacy Code • Golden Master • Approval Tests • Example • Resources • Summary
  80. 80. 82 @ClareMacraeUK What I can do:
  81. 81. 83 @ClareMacraeUK What I need to do:
  82. 82. 84 @ClareMacraeUK Sugar - sucrose
  83. 83. 85 @ClareMacraeUK Sugar - sucrose
  84. 84. 86 @ClareMacraeUK Sugar - sucrose
  85. 85. 87 @ClareMacraeUK Sugar - sucrose
  86. 86. 88 @ClareMacraeUK Sugar - sucrose
  87. 87. 89 @ClareMacraeUK
  88. 88. 90 @ClareMacraeUK Approving Images
  89. 89. 91 @ClareMacraeUK What I’m aiming for TEST(PolyhedralGraphicsStyleApprovals, CUVXAT) { // CUVXAT identifies a crystal structure in the database const QImage crystal_picture = loadEntry("CUVXAT"); verifyQImage(crystal_picture, *currentReporter()); }
  90. 90. 92 @ClareMacraeUK Foundation of Approval tests void FileApprover::verify( ApprovalNamer& namer, ApprovalWriter& writer, const Reporter& reporter);
  91. 91. 93 @ClareMacraeUK Idiomatic verification of Qt image objects void verifyQImage( QImage image, const Reporter& reporter = DiffReporter()) { ApprovalTestNamer namer; QImageWriter writer(image); FileApprover::verify(namer, writer, reporter); }
  92. 92. 94 @ClareMacraeUK Customisability: ApprovalWriter interface class ApprovalWriter { public: virtual std::string getFileExtensionWithDot() = 0; virtual void write(std::string path) = 0; virtual void cleanUpReceived(std::string receivedPath) = 0; };
  93. 93. 95 @ClareMacraeUK QImageWriter declaration class QImageWriter : public ApprovalWriter { public: explicit QImageWriter(QImage image, std::string fileExtensionWithDot = ".png"); std::string getFileExtensionWithDot() override; void write(std::string path) override; void cleanUpReceived(std::string receivedPath) override; private: QImage image_; std::string ext; };
  94. 94. 96 @ClareMacraeUK QImageWriter::write() void QImageWriter::write(std::string path) { // Have to convert std::string to QString image_.save( QString::fromStdString(path) ); }
  95. 95. 97 @ClareMacraeUK Useful feedback BeyondCompare 4
  96. 96. 98 @ClareMacraeUK Back In work… • Previously, working from home via Remote Desktop • The tests started failing when I ran them in work
  97. 97. 99 @ClareMacraeUK What? Re-running now gives random failures Araxis Merge
  98. 98. 100 @ClareMacraeUK BeyondCompare 4
  99. 99. 101 @ClareMacraeUK
  100. 100. 102 @ClareMacraeUK
  101. 101. 103 @ClareMacraeUK
  102. 102. 104 @ClareMacraeUK
  103. 103. 105 @ClareMacraeUK
  104. 104. 106 @ClareMacraeUK Customisability: ApprovalComparator class ApprovalComparator { public: virtual ~ApprovalComparator() = default; virtual bool contentsAreEquivalent(std::string receivedPath, std::string approvedPath) const = 0; };
  105. 105. 107 @ClareMacraeUK Create custom QImage comparison class // Allow differences in up to 1/255 of RGB values at each pixel, as saved in 32-bit images // sometimes have a few slightly different pixels, which are not visible to the human eye. class QImageApprovalComparator : public ApprovalComparator { public: bool contentsAreEquivalent(std::string receivedPath, std::string approvedPath) const override { const QImage receivedImage(QString::fromStdString(receivedPath)); const QImage approvedImage(QString::fromStdString(approvedPath)); return compareQImageIgnoringTinyDiffs(receivedImage, approvedImage); } };
  106. 106. 108 @ClareMacraeUK Register the custom comparison class // Somewhere in main… FileApprover::registerComparator( ".png", std::make_shared<QImageApprovalComparator>());
  107. 107. 109 @ClareMacraeUK Does the difference matter? • Legacy code is often brittle • Testing makes changes visible • Then decide if change matters • Fast feedback cycle for efficient development
  108. 108. 110 @ClareMacraeUK Looking back • User perspective • Learned about own code • Enabled unit tests for graphics problems • Approvals useful even for temporary tests
  109. 109. 111 @ClareMacraeUK ApprovalTests Customisation Points • Reporter • ApprovalWriter • ApprovalComparator • ApprovalNamer
  110. 110. 112 @ClareMacraeUK Contents • Introduction • Legacy Code • Golden Master • Approval Tests • Example • Resources • Summary
  111. 111. 113 @ClareMacraeUK References: Tools Used • Diffing – Araxis Merge: https://www.araxis.com/merge/ – Beyond Compare: https://www.scootersoftware.com/ • Code Coverage – OpenCppCoverage and Visual Studio Plugin: https://github.com/OpenCppCoverage – BullseyeCoverage: https://www.bullseye.com/
  112. 112. 114 @ClareMacraeUK Beautiful C++: Updating Legacy Code • https://app.pluralsight.com/library/courses/cpp-updating-legacy-code/table-of-contents
  113. 113. 115 @ClareMacraeUK The Legacy Code Programmer’s Toolbox • https://leanpub.com/legacycode • The Legacy Code Programmer's Toolbox: Practical skills for software professionals working with legacy code, by Jonathan Boccara
  114. 114. 116 @ClareMacraeUK Video https://youtu.be/aWiwDdx_rdo
  115. 115. 117 @ClareMacraeUK Adam Tornhill’s Books • Mining actionable information from historical code bases (by Adam Tornhill) – Your Code as a Crime Scene – Software Design X-Rays: Fix Technical Debt with Behavioural Code Analysis
  116. 116. 118 @ClareMacraeUK
  117. 117. 119 @ClareMacraeUK
  118. 118. 120 @ClareMacraeUK Our original goals • Encourage under-represented people – To speak – To apply for jobs – To stay in this industry • Get conferences to have a code of conduct – Hadn’t even thought about enforcement • Get employers to value diversity somewhat • Provide some resources to conferences and employers 120
  119. 119. 121 @ClareMacraeUK What we did, April 2018 – April 2019 • Grew the discord server to over 1900 people – 30+ public channels – Plus more for “established” users • Sent real people to conferences – Admission provided by conference – Fundraising for travel expenses • Booth / table / stand at every major C++ conference – Welcoming people and happy space – Inviting people to Discord, providing answers, etc • Got shirts, stickers, badges etc out into the world – Speakers wear them on stage – They’ve been spotted at the ISO C++ meetings • People changing how they speak and write – Examples in demos – “guys” • And yes, got some conferences to add a CoC – And real enforcement
  120. 120. 122 @ClareMacraeUK Contents • Introduction • Legacy Code • Golden Master • Approval Tests • Example • Resources • Summary
  121. 121. 123 @ClareMacraeUK Adopting legacy code • You can do it! • Evaluate current tests • Quickly improve coverage with Golden Master • ApprovalTests.cpp makes that really easy • Even for non-text types
  122. 122. 124 @ClareMacraeUK ApprovalTests • Powerful Golden Master tool • verify(), verifyAll(), verifyAllCombinations() • Adjust output files, after writing, to simplify comparison
  123. 123. 125 @ClareMacraeUK ApprovalTests.cpp • Not (always) a replacement for Unit Tests! Golden Master Refactor code Add unit tests Archive Approval Tests?
  124. 124. 126 @ClareMacraeUK Thank You: Any Questions? • Example Code: https://github.com/claremacrae/cpponsea2019 • Slides: https://www.slideshare.net/ClareMacrae – (later this week) • ApprovalTests.cpp – https://github.com/approvals/ApprovalTests.cpp – https://github.com/approvals/ApprovalTests.cpp.StarterProject

×