Your SlideShare is downloading. ×
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Regression Testing
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Regression Testing

1,444

Published on

So you're about to commit a change to the pkgsrc infrastructure. You've tested it -- it fixes the bug you're interested in and you can still build a few of your favourite packages. But you know that …

So you're about to commit a change to the pkgsrc infrastructure. You've tested it -- it fixes the bug you're interested in and you can still build a few of your favourite packages. But you know that if you reintroduce a bug that caused havoc with threaded python last year, you're in for a right bricking. That's why you need pkg_regress. This presentation explains what it is, how it works and all the other reasons why it's for you.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,444
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
70
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Regression Testing Gavan Fantom gavan@NetBSD.org pkgsrcCon 2005
  • 2. Introduction Have you ever committed anything to mk? ● Did you break stuff? ● Has anybody else broken your code? ● What testing do you do before you commit? ●
  • 3. Regression Testing Framework Automated tests of pkgsrc infrastructure ● Designed at pkgsrcCon 2004 ● Will solve all the world's problems ● (except those solved by pkgviews) But not a substitute for other forms of testing ●
  • 4. How to run regression tests Install pkgtools/pkg_regress ● Run pkg_regress ● pkg_regress -v shows more details – Tests live in regress/ ●
  • 5. Why should you run regression tests? Make sure stuff is broken before you commit ● Notice breakage more quickly ● You should run regression tests more often if ● you use non-standard settings or an esoteric Operating System.
  • 6. How to write a regression test Test a specific feature of the infrastructure ● A test contains: ● spec file – Makefile (typically) – Any other files required – A test is only a test if it contains a spec file ● Other directories are ignored, so a test can consist of – more than one package if necessary.
  • 7. Regression test example regress/pkgfail ● Makefile – spec – Tests that PKG_FAIL_REASON does what it ● says on the tin
  • 8. Makefile DISTNAME= regress-pkgfail-0.0 CATEGORIES= regress MAINTAINER= gavan@NetBSD.org COMMENT= Test PKG_FAIL_REASON PKG_FAIL_REASON= quot;This package should never buildquot; .include quot;../../mk/bsd.pkg.mkquot;
  • 9. spec MAKEARGS_TEST=install check_result() { exit_status 1 output_require quot;This package should never buildquot; }
  • 10. Things you can do in the spec file Override: ● do_setup, do_cleanup, do_test – check_result – Define: ● MAKEARGS_TEST – MAKEARGS_CLEAN – Use: ● exit_status status – output_require quot;Good Regular Expressionquot; – output_prohibit quot;Bad Regular Expressionquot; –
  • 11. What makes a good test? Simplicity ● Platform-independence ● Environment-independence ● Consistency ●
  • 12. What makes a bad test? Hard to understand ● Random or variable results ● Only works correctly on certain platforms ● Succeeds if infrastructure is broken ●
  • 13. Why should you write tests? Stop people from breaking things you care ● about Formally specify desired pkgsrc behaviour ● Because you can ● Enter the competition for the most ● complicated regression test. Currently, jlam is winning.
  • 14. Room for improvement Better reporting ● Locale support ● Support sub-tests ● Write more tests ●
  • 15. Questions?
  • 16. Regression Testing Gavan Fantom gavan@NetBSD.org pkgsrcCon 2005

×