Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Quickly and Effectively Testing Legacy C++ Code with Approval Tests

Presented at C++ on Sea 2020, 15 July 2020

Links from this talk are at: https://github.com/claremacrae/talks/blob/main/Quickly_and_Effectively_Testing_Legacy_C++_Code_with_Approval_Tests.md#top

Abstract:
You've inherited some legacy code: it's valuable, but it doesn't have tests, and it wasn't designed to be testable, so you need to start refactoring. But you can't refactor safely until the code has tests, and you can't add tests without refactoring! How can you ever break out of this loop?

Whether Legacy code for you means "old code", "code without tests", or "code you wish to redesign for new features or unit-tests", this talk will enable you to become productive and work safely, quickly.

The simplicity, convenience, ease-of-use, power and flexibility of Llewellyn Falco's "Approval Tests" approach has long been proven in a dozen programming languages. And now all this is now available to C++ developers too!

Clare will present a small but surprisingly effective C++11 library for applying "Approval Tests" to cross-platform C++ code - for both legacy and green-field systems, and with a range of testing frameworks.

She will demonstrate its use in some real-world situations, including how to quickly lock down the behaviour of legacy code. She will show how to quickly achieve good test coverage, even for very large sets of inputs.

She will also show recent work applying Approval Tests to testing Graphical User Interfaces

Finally, she will describe some powerful techniques used in the implementation, that are useful beyond the field

Attendees will discover some quick, practical techniques to use for common challenges, such as testing GUI systems, and outputs containing dates and times, that can be applied very easily using Approval Tests.

  • Be the first to comment

Quickly and Effectively Testing Legacy C++ Code with Approval Tests

  1. 1. 1 Quickly and Effectively Testing Legacy C++ Code with Approval Tests Clare Macrae (She/her) clare@claremacrae.co.uk 15 July 2020 C++ on Sea
  2. 2. 2 About Me • C++ and Qt developer since 1999 • My mission: Sustainable and efficient testing and refactoring of legacy code – Co-author of “Approval Tests for C++” • Consulting & training – https://claremacrae.co.uk • All links from this talk via: – github.com/claremacrae/talks
  3. 3. 3 Goal Share the Power of Approval Tests
  4. 4. 4 Contents •Intro • Getting Started • Legacy Example • Non-deterministic Output • Customisability • Visualising Differences • Testing GUIs • Summary
  5. 5. 5
  6. 6. 6
  7. 7. 7 “Approval Tests allow you to verify a chunk of output (such as a file) in one operation as opposed to writing test assertions for each element”
  8. 8. 8 Questions at the end…
  9. 9. 9 Contents • Intro •Getting Started • Legacy Example • Non-deterministic Output • Customisability • Visualising Differences • Testing GUIs • Summary
  10. 10. 10 Demo 1: Hello Approvals
  11. 11. 11 Main Methods • Approvals::verify(std::string, Options) – 2 other overloads take template types • Approvals::verifyAll() – For use with containers – Writes one element at a time • All are in namespace ApprovalTests
  12. 12. 12 object verify() writes "received" reads "approved" Same? Pass Fail Reporter Stages of Approvals::verify()
  13. 13. 13 Easy to: • … Add tests quickly • … Visualise differences • … Update output • Convenience – Separates test data from test code – Ignore end-of-line differences
  14. 14. 14 Easy setup • Single header – https://github.com/approvals/ApprovalTests.cpp/releases • Finds diff tool automatically • Sensible filenames automatically – source_directory/test_filename.test_name.approved.txt • “It just works”
  15. 15. 15 Supported Test Frameworks • Lots to choose from: • Get to know chosen one well!
  16. 16. 16 Contents • Intro • Getting Started •Legacy Example • Non-deterministic Output • Customisability • Visualising Differences • Testing GUIs • Summary
  17. 17. 17 Demo 2: Legacy Code: Gilded Rose
  18. 18. 18 Approvals and Legacy Code • Before you start refactoring… • Achieve good coverage • verifyAllCombinations() – Not just for legacy code
  19. 19. 19 Lots of String Conversion Options • String Conversions – Pass in a std::string – Use Approvals::verify( object, lambda ) – Write custom operator<<( std::ostream, … ) – Specialize ApprovalTests::StringMaker::toString( … ) – Use TApprovals<YourStringConvertingClass> • How to Use the Fmt Library To Print Objects – Use {fmt} library via FmtApprovals::verify()
  20. 20. 20 String Design Guidelines • Objects print their relevant data • The data is consistent between runs – (no times, pointers, random) • The data is easy to read, at a glance: • Advice: Tips for Designing Strings [0] = [x: 4 y: 50 width: 100 height: 61] [1] = [x: 50 y: 5200 width: 400 height: 62] [2] = [x: 60 y: 3 width: 7 height: 63] (x,y,width,height) = (4,50,100,61) (x,y,width,height) = (50,5200,400,62) (x,y,width,height) = (60,3,7,63)
  21. 21. 21 Contents • Intro • Getting Started • Legacy Example •Non-deterministic Output • Customisability • Visualising Differences • Testing GUIs • Summary
  22. 22. 22 Demo 3: Log Files
  23. 23. 23 Key Points • Approvals::verifyExistingFile() – For when the file is already written • Scrubbers for non-deterministic output – More scrubber options coming soon…
  24. 24. 24 Contents • Intro • Getting Started • Legacy Example • Non-deterministic Output •Customisability • Visualising Differences • Testing GUIs • Summary
  25. 25. 25 Customisation Points Overview
  26. 26. 26 Scrubber Namer Writer Comparator Pass Fail Reporter Stages of Approvals::verify()
  27. 27. 27 Everything is customisable • We take great pride in the docs • All examples generated from compiled, tested code github.com/approvals/ApprovalTests.cpp/blob/master/doc/ approvaltestscpp.readthedocs.io/en/latest/
  28. 28. 28 Contents • Intro • Getting Started • Legacy Example • Non-deterministic Output • Customisability •Visualising Differences • Testing GUIs • Summary
  29. 29. 29 Demo 4: SVG Files
  30. 30. 30 Key Points • Custom Reporters can help understand failures • Make it easy to visualise differences • No need to version control .png files
  31. 31. 31 Contents • Intro • Getting Started • Legacy Example • Non-deterministic Output • Customisability • Visualising Differences •Testing GUIs • Summary
  32. 32. 32 Demo 5: Testing GUIs
  33. 33. 33 Key Points • ApprovalTests.Cpp.Qt – Integrates Catch2 test framework with Qt event loop – Very early days! • Multiple output files per test – Make filenames unique • SeparateApprovedAndReceivedDirectoriesNamer – Worth being aware of • AutoApproveReporter –Caution! Only if approved files already in version control
  34. 34. 34 Annoyance
  35. 35. 35 • C# “Empty Files” project by Simon Cropp – github.com/SimonCropp/EmptyFiles Excellent Idea
  36. 36. 36 GUI-testing • What to test – Try to test smaller units of code, such as individual widgets – Test behaviors, not appearance (mainly) – Keep application logic separate from GUI code • ApprovalTests.Cpp.Qt feedback welcome! – github.com/approvals/ApprovalTests.cpp.Qt
  37. 37. 37 See Also ImageApprovals • github.com/p-podsiadly/ImageApprovals • Built on ApprovalTests.cpp • Not yet released • Recommended over ApprovalTests.cpp.Qt
  38. 38. 38 Contents • Intro • Getting Started • Legacy Example • Non-deterministic Output • Customisability • Visualising Differences • Testing GUIs •Summary
  39. 39. 39 Summary
  40. 40. 40 “Approval Tests allow you to verify a chunk of output (such as a file) in one operation as opposed to writing test assertions for each element”
  41. 41. 41 Approval Tests for C++ • Easy to use • Convenient • Powerful • Flexible • It just works!
  42. 42. 42
  43. 43. 43 Quickly & Effectively Test Legacy C++ Code with Approval Tests
  44. 44. 44 References • All links from this talk, and more, via: – github.com/claremacrae/talks • Sustainable and efficient testing and refactoring of legacy code • Consulting & Training – claremacrae.co.uk – clare@claremacrae.co.uk Any Questions?
  45. 45. 45 Contents • Intro • Getting Started • Legacy Example • Non-deterministic Output • Customisability • Visualising Differences • Testing GUIs • Summary •Appendix: Challenges
  46. 46. 46 Appendix: Common Challenges
  47. 47. 47 External Resources • I/O, Networking, System API calls… • Be creative! • Can you intercept and log calls? – And save them as “approved” results? – Then intercept again during later runs, and compare results? • You can even use this data as inputs to later tests! • Excellent example from Angie Jones: – https://angiejones.tech/verifying-entire-api-responses/
  48. 48. 48 Embedded Systems • Separate as much logic as possible • Test the logic on desktop machines and CI • Then if you get a failure on device, you’ve got less to test • It may become possible to run Approval Tests from cross-built code
  49. 49. 49 Output is OS-specific • Include the OS in the file-names • So verify() compares against output from same platform • For example – TestQtDialog.loginScreen.onMacOSX.approved.png – TestQtDialog.loginScreen.onWindows.approved.png – TestQtDialog.loginScreen.onLinux.approved.png • See Multiple output files per test
  50. 50. 50 Approving is fiddly or takes too long • Approving images? • Approving hundreds of files? • Try one of: – AutoApproveIfMissingReporter – AutoApproveReporter (and compare the diffs in version control!)

×