0
Regression Testing



    Gavan Fantom

  gavan@NetBSD.org

   pkgsrcCon 2005
Introduction
    Have you ever committed anything to mk?
●




    Did you break stuff?
●




    Has anybody else broken ...
Regression Testing Framework
    Automated tests of pkgsrc infrastructure
●




    Designed at pkgsrcCon 2004
●




    W...
How to run regression tests
    Install pkgtools/pkg_regress
●




    Run pkg_regress
●


        pkg_regress -v shows mo...
Why should you run regression
               tests?
    Make sure stuff is broken before you commit
●




    Notice break...
How to write a regression test
    Test a specific feature of the infrastructure
●


    A test contains:
●


        spec...
Regression test example
    regress/pkgfail
●




        Makefile
    –


        spec
    –


    Tests that PKG_FAIL_RE...
Makefile
DISTNAME=   regress-pkgfail-0.0
CATEGORIES=   regress
MAINTAINER=   gavan@NetBSD.org
COMMENT= Test PKG_FAIL_REASO...
spec
MAKEARGS_TEST=install


check_result()
{
    exit_status 1
    output_require quot;This package should
     never bui...
Things you can do in the spec file
    Override:
●


        do_setup, do_cleanup, do_test
    –

        check_result
   ...
What makes a good test?
    Simplicity
●




    Platform-independence
●




    Environment-independence
●




    Consis...
What makes a bad test?
    Hard to understand
●




    Random or variable results
●




    Only works correctly on certa...
Why should you write tests?
    Stop people from breaking things you care
●

    about

    Formally specify desired pkgsr...
Room for improvement
    Better reporting
●




    Locale support
●




    Support sub-tests
●




    Write more tests
●
Questions?
Regression Testing



    Gavan Fantom

  gavan@NetBSD.org

   pkgsrcCon 2005
Upcoming SlideShare
Loading in...5
×

Regression Testing

1,478

Published on

So you're about to commit a change to the pkgsrc infrastructure. You've tested it -- it fixes the bug you're interested in and you can still build a few of your favourite packages. But you know that if you reintroduce a bug that caused havoc with threaded python last year, you're in for a right bricking. That's why you need pkg_regress. This presentation explains what it is, how it works and all the other reasons why it's for you.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,478
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
70
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Regression Testing"

  1. 1. Regression Testing Gavan Fantom gavan@NetBSD.org pkgsrcCon 2005
  2. 2. Introduction Have you ever committed anything to mk? ● Did you break stuff? ● Has anybody else broken your code? ● What testing do you do before you commit? ●
  3. 3. Regression Testing Framework Automated tests of pkgsrc infrastructure ● Designed at pkgsrcCon 2004 ● Will solve all the world's problems ● (except those solved by pkgviews) But not a substitute for other forms of testing ●
  4. 4. How to run regression tests Install pkgtools/pkg_regress ● Run pkg_regress ● pkg_regress -v shows more details – Tests live in regress/ ●
  5. 5. Why should you run regression tests? Make sure stuff is broken before you commit ● Notice breakage more quickly ● You should run regression tests more often if ● you use non-standard settings or an esoteric Operating System.
  6. 6. How to write a regression test Test a specific feature of the infrastructure ● A test contains: ● spec file – Makefile (typically) – Any other files required – A test is only a test if it contains a spec file ● Other directories are ignored, so a test can consist of – more than one package if necessary.
  7. 7. Regression test example regress/pkgfail ● Makefile – spec – Tests that PKG_FAIL_REASON does what it ● says on the tin
  8. 8. Makefile DISTNAME= regress-pkgfail-0.0 CATEGORIES= regress MAINTAINER= gavan@NetBSD.org COMMENT= Test PKG_FAIL_REASON PKG_FAIL_REASON= quot;This package should never buildquot; .include quot;../../mk/bsd.pkg.mkquot;
  9. 9. spec MAKEARGS_TEST=install check_result() { exit_status 1 output_require quot;This package should never buildquot; }
  10. 10. Things you can do in the spec file Override: ● do_setup, do_cleanup, do_test – check_result – Define: ● MAKEARGS_TEST – MAKEARGS_CLEAN – Use: ● exit_status status – output_require quot;Good Regular Expressionquot; – output_prohibit quot;Bad Regular Expressionquot; –
  11. 11. What makes a good test? Simplicity ● Platform-independence ● Environment-independence ● Consistency ●
  12. 12. What makes a bad test? Hard to understand ● Random or variable results ● Only works correctly on certain platforms ● Succeeds if infrastructure is broken ●
  13. 13. Why should you write tests? Stop people from breaking things you care ● about Formally specify desired pkgsrc behaviour ● Because you can ● Enter the competition for the most ● complicated regression test. Currently, jlam is winning.
  14. 14. Room for improvement Better reporting ● Locale support ● Support sub-tests ● Write more tests ●
  15. 15. Questions?
  16. 16. Regression Testing Gavan Fantom gavan@NetBSD.org pkgsrcCon 2005
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×