Regression Testing


Published on

So you're about to commit a change to the pkgsrc infrastructure. You've tested it -- it fixes the bug you're interested in and you can still build a few of your favourite packages. But you know that if you reintroduce a bug that caused havoc with threaded python last year, you're in for a right bricking. That's why you need pkg_regress. This presentation explains what it is, how it works and all the other reasons why it's for you.

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Regression Testing

  1. 1. Regression Testing Gavan Fantom pkgsrcCon 2005
  2. 2. Introduction Have you ever committed anything to mk? ● Did you break stuff? ● Has anybody else broken your code? ● What testing do you do before you commit? ●
  3. 3. Regression Testing Framework Automated tests of pkgsrc infrastructure ● Designed at pkgsrcCon 2004 ● Will solve all the world's problems ● (except those solved by pkgviews) But not a substitute for other forms of testing ●
  4. 4. How to run regression tests Install pkgtools/pkg_regress ● Run pkg_regress ● pkg_regress -v shows more details – Tests live in regress/ ●
  5. 5. Why should you run regression tests? Make sure stuff is broken before you commit ● Notice breakage more quickly ● You should run regression tests more often if ● you use non-standard settings or an esoteric Operating System.
  6. 6. How to write a regression test Test a specific feature of the infrastructure ● A test contains: ● spec file – Makefile (typically) – Any other files required – A test is only a test if it contains a spec file ● Other directories are ignored, so a test can consist of – more than one package if necessary.
  7. 7. Regression test example regress/pkgfail ● Makefile – spec – Tests that PKG_FAIL_REASON does what it ● says on the tin
  8. 8. Makefile DISTNAME= regress-pkgfail-0.0 CATEGORIES= regress MAINTAINER= COMMENT= Test PKG_FAIL_REASON PKG_FAIL_REASON= quot;This package should never buildquot; .include quot;../../mk/bsd.pkg.mkquot;
  9. 9. spec MAKEARGS_TEST=install check_result() { exit_status 1 output_require quot;This package should never buildquot; }
  10. 10. Things you can do in the spec file Override: ● do_setup, do_cleanup, do_test – check_result – Define: ● MAKEARGS_TEST – MAKEARGS_CLEAN – Use: ● exit_status status – output_require quot;Good Regular Expressionquot; – output_prohibit quot;Bad Regular Expressionquot; –
  11. 11. What makes a good test? Simplicity ● Platform-independence ● Environment-independence ● Consistency ●
  12. 12. What makes a bad test? Hard to understand ● Random or variable results ● Only works correctly on certain platforms ● Succeeds if infrastructure is broken ●
  13. 13. Why should you write tests? Stop people from breaking things you care ● about Formally specify desired pkgsrc behaviour ● Because you can ● Enter the competition for the most ● complicated regression test. Currently, jlam is winning.
  14. 14. Room for improvement Better reporting ● Locale support ● Support sub-tests ● Write more tests ●
  15. 15. Questions?
  16. 16. Regression Testing Gavan Fantom pkgsrcCon 2005