Test tutorial
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Test tutorial

on

  • 2,334 views

 

Statistics

Views

Total Views
2,334
Views on SlideShare
2,334
Embed Views
0

Actions

Likes
0
Downloads
3
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Test tutorial Presentation Transcript

  • 1. Test::Tutorial chromatic and Michael G Schwern
  • 2. Testing? Why do I care?
  • 3. What Is Testing? q Check that your code does what it's supposed to do. q At varying levels of granularity. q In varying environments. q At various points in its development.
  • 4. What Is Automated Testing? q Programs to check other programs. q As with any rote task, you let the computer do it. x Humans will forget rote tests, computers will not q Press a button and walk away. x No human required (very important) q Manually running tests is a waste of your time. q Tests should run as close to instantaneous as possible x so you won't have an excuse not to run them x so you'll run them as often as possible x Instant feedback
  • 5. Testing Promotes Automation q Testable code is decoupled q Testable code is scriptable
  • 6. Why Test? q no missing functionality q no accidental functionality q when your tests pass, you're done
  • 7. More informative bug reports q Better to get the diagnostic output of your tests than "It doesn't work" q Easily generated by end users x "Run the tests and send me the output" q Helps IMMENSELY with porting (this is why my code works on VMS) x You can often port code without ever using the machine you're porting to
  • 8. More More Reasons q Most of the time spent on a project is debugging and bug fixing. x Worse, it often comes at the end (hidden cost) x "Oh, I'm 99% done, I just need to do some testing" q Testing as you go will increase your development time, but reduce debugging time. x It will let you estimate more realistically x Increased project visibility x Reduced debug time once you get used to it
  • 9. The Real Reason For Writing Tests q Confidence. x No fear of change x No fear of lurking bugs x No fear of breaking old things x No fear that new things don't work Ë Knowing when things don't work. q So you can play and experiment without worry. q Enable refactoring
  • 10. Testing Is Laziness q Take an O(n) amount of work and make it O(1) x Instead of walking through the code by hand at each change x Teach the computer to do that.
  • 11. What to test
  • 12. Textbook Testing q Traditional testing philosophy says things like Test all subroutines Test all branches of all conditions Test the boundary conditions of all inputs ... q This is just big and scary and too much. q We're lazy, and we think we can still be effective with much less work.
  • 13. XP Testing q XP says to write your tests before you write your code. x It's hard enough to get people to write tests at all. x Changing their coding philosophy at the same time is worse. q If you can do Test First, excellent. q If you're not already testing, this is a chance to start some new habits...
  • 14. On Test-First Programming q Think of it as coding to teeny, tiny, mini-iterations. q Break each task into boolean expressions. q Ask "What feature do I need next?" x Test the smallest and most immediate element of the overall task. x Take small steps!
  • 15. The two test-first questions q "How can I prove that this feature works?" x Write the simplest test that will fail unless the feature works. x The test must fail. q "What is the least amount of code I can write to pass the test?" x The simpler the test, the simpler the code you need. x The test must now pass. q This produces known good code and a comprehensive test suite. q Be sure to run the entire test suite after you implement a task. q Don't be afraid of baby steps. That's the point.
  • 16. Test Bugs q Another good philosophy is to test bugs and new features. q Every time you find a bug, write a test for it. q Every time you add a new feature, write a test for it. q In the process, you might test a few other related things. q This is the simplest way to retrofit tests onto existing code.
  • 17. Effects Of Testing Bugs q This has pleasant effects: x Slowly grows your test suite x Focuses tests on the parts of the code which have the most bugs q You're allowed to make mistakes, but ONLY ONCE. Never twice. q A disproportionate amount of bugs use the same logic. x One test can catch lots of bugs
  • 18. Knowing You're Done t/Your-Code.t......ok All tests successful. q For once, your computer is telling you something good. q Instant, positive feedback
  • 19. There Is No Magic q You may have seen this from h2xs. ######## We start with some black magic to print on failure. # Change 1..1 below to 1..last_test_to_print . # (It may become useful if the test is moved to ./t subdirec BEGIN { $| = 1; print "1..1n"; } END {print "not ok 1n" unless $loaded;} use Foo; $loaded = 1; print "ok 1n"; ######## End of black magic. q Testing really isn't this frightening.
  • 20. And Now For Something Completely Different
  • 21. The Most Basic Perl Test Program #!/usr/bin/perl -w print "1..1n"; print 1 + 1 == 2 ? "ok 1n" : "not ok 1n"; q Since 1 + 1 is 2, this prints: 1..1 ok 1 x "1..1" I'm going to run one test. x "ok 1" The first test passed.
  • 22. Perl's Testing Protocol q There are two parts to running a test in Perl. x Your test x Test::Harness q The output of your test is piped to Test::Harness. q Test::Harness interprets your output and reports. $ perl -MTest::Harness -wle 'runtests @ARGV' contrived.t contrived....ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.02 cusr + 0.02 csys
  • 23. There's TMTOWTDI and there's this... q Here's some of the many ways people write their tests: x t/op/sysio.t print 'not ' unless (syswrite(O, $a, 2) == 2); print "ok 20n"; x ext/Cwd/t/cwd.t print +($getcwd eq $start ? "" : "not "), "ok 4n"; x t/pod/plainer.t unless( $returned eq $expected ) { print map { s/^/#/mg; $_; } map {+$_} # to avoid readonly values "EXPECTED:n", $expected, "GOT:n", $returned; print "not "; } printf "ok %dn", ++$test; q Maintenance nightmare.
  • 24. I'm ok, you're ok #!/usr/bin/perl -w use Test::Simple tests => 1; ok( 1 + 1 == 2 ); q "ok" is the backbone of Perl testing. x If the expression is true, the test pass. x False, it fails. q Every conceivable test can be performed just using ok().
  • 25. YOU FAILED!!! #!/usr/bin/perl -w use Test::Simple tests => 2; ok( 1 + 1 == 2 ); ok( 2 + 2 == 5 ); q from that comes: 1..2 ok 1 not ok 2 # Failed test (contrived.t at line 5) # Looks like you failed 1 tests of 2. x "1..2" I'm going to run two tests. x "ok 1" The first test passed. x "not ok 2" The second test failed. x Some helpful commentary from Test::Simple
  • 26. Date::ICal q We're going to write some tests for Date::ICal. x It's real code. x It's sufficiently complex. x Everyone understands dates. Ë Some people even have them.
  • 27. Where To Start? q This is the hardest part. q Retrofitting a test suite onto old code sucks. x Marching through the testing swamps. q Write tests from the start and your life will be easier. q In any event, begin at the beginning.
  • 28. new() q Since Date::ICal is OO, the beginning is when you make an object. x (white-lie: the beginning is when you load the module) #!/usr/bin/perl -w use Test::Simple tests => 2; use Date::ICal; my $ical = Date::ICal->new; # make an object ok( defined $ical ); # check we got something ok( $ical->isa('Date::ICal') ); # and it's the right class q This produces: 1..2 ok 1 ok 2 q This is your first useful test.
  • 29. Names q "ok 2" isn't terribly descriptive. x what if you have 102 tests, what did #64 do? q Each test can be given a little description. ok( defined $ical, 'new() returned something' ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) q This outputs 1..2 ok 1 - new() returned something ok 2 - and it's the right class
  • 30. What's In A Name q Two views on names. x A name is a descriptive tag so you can track the test output back to the code which produced it. (the original purpose) x A name is a short description of what was tested. q There's a subtle difference. q Don't pull your hair out over it. x More importantly, don't pull other people's hair out over it.
  • 31. Test The Manual q Simplest way to build up a test suite is to just test what the manual says it does. x Also a good way to find mistakes/omissions in the docs. x You can take this five steps further and put the tests IN the manual. Test::Inline, later. q If the docs are well written, they should cover usage of your code. x You do have docs, right?
  • 32. SYNOPSIS q A good place to start. x A broad overview of the whole system q Here's a piece of Date::ICal's SYNOPSIS. SYNOPSIS use Date::ICal; $ical = Date::ICal->new( year => 1964, month => 10, day = hour => 16, min => 12, sec => 47, tz => '0530' ); $hour = $ical->hour; $year = $ical->year; q Oddly enough, there is a bug in this.
  • 33. SYNOPSIS test use Test::Simple tests => 8; use Date::ICal; $ical = Date::ICal->new( year => 1964, month => 10, day => 16, hour => 16, min => 12, sec => 47, tz => '0530' ); ok( defined $ical, 'new() returned something' ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) ok( $ical->sec == 47, ' sec()' ); ok( $ical->min == 42, ' min()' ); ok( $ical->hour == 10, ' hour()' ); ok( $ical->day == 16, ' day()' ); ok( $ical->month == 10, ' month()' ); ok( $ical->year == 1964, ' year()' );
  • 34. SYNOPSIS results 1..8 ok 1 - new() returned something ok 2 - and it's the right class ok 3 - sec() not ok 4 - min() # Failed test (ical.t at line 14) not ok 5 - hour() # Failed test (ical.t at line 15) ok 6 - day() ok 7 - month() ok 8 - year() # Looks like you failed 2 tests of 8. q Whoops, failures! q We know what and where it failed, but not much else. q How do you find out more? x Throw in print statements x Run in the debugger. q That sounds like work.
  • 35. Test::More q Test::Simple is deliberately limited to one function. q Test::More does everything Test::Simple does. x You can literally s/use Test::Simple/use Test::More/ q It provides more informative ways to say "ok".
  • 36. is() you is() or is() you isnt() my $baby; q Test::More's is() function: x declares that something is supposed to be something else x "Is this, that?" is( $this, $that ); # From ok( $ical->day == 16, ' day()' ); # To is( $ical->day, 16, ' day()' );
  • 37. ok() to is() q Here's the test with ok() replaced with is() appropriately. use Test::More tests => 8; use Date::ICal; $ical = Date::ICal->new( year => 1964, month => 10, day => 16, hour => 16, min => 12, sec => 47, tz => '+0530' ); ok( defined $ical, 'new() returned something' ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) is( $ical->sec, 47, ' sec()' ); is( $ical->min, 42, ' min()' ); is( $ical->hour, 10, ' hour()' ); is( $ical->day, 16, ' day()' ); is( $ical->month, 10, ' month()' ); is( $ical->year, 1964, ' year()' ); q "Is $ical->sec, 47?" q "Is $ical->min, 12?"
  • 38. Diagnostic Output 1..8 ok 1 - new() returned something ok 2 - and it's the right class ok 3 - sec() not ok 4 - min() # Failed test (- at line 13) # got: '12' # expected: '42' not ok 5 - hour() # Failed test (- at line 14) # got: '21' # expected: '10' ok 6 - day() ok 7 - month() ok 8 - year() # Looks like you failed 2 tests of 8. q $ical->min returned 12 instead of 42. q $ical->hour returned 21 instead of 10.
  • 39. Interpreting The Results q Turns out, there is no 'tz' argument to new()! x And it didn't warn us about bad arguments q The real argument is 'offset' x So the synopsis is wrong. x This is a real bug I found while writing this q Damn those tests.
  • 40. When to use is() q Use instead of ok() when you're testing "this equals that". x Yes, there is an isnt() and isn't(). q is() does a string comparison which 99.99% of the time comes out right. x cmp_ok() exists to test with specific comparison operators
  • 41. Tests Are Sometimes Wrong q The previous example was supposed to be highly contrived to illustrate that tests are sometimes wrong. q When investigating a test failure, look at both the code and the test. q There's a fine line of trusting your testing code. x Too much trust, and you'll be chasing phantoms. x Too little trust, and you'll be changing your tests to cover up bugs.
  • 42. How Can I Be Sure The Test Is Right? q Write the test q Run it and make sure the new test fails q Add the new feature / fix the bug q Run the test and make sure the new test passes. q Some development systems, such as Aegis, can enforce this process. q It's difficult to do this when writing tests for existing code. x Another reason to test as you go
  • 43. Version Control and Testing q VC & testing work well. x Run the tests, make sure they pass x Make sure everything is checked in. x Write tests for the bug / feature. Ë Make sure they fail. x Fix your bug / write your feature x Run the tests. Ë If they pass, commit. You're done. Ë If they fail, look at the diff. The problem is revealed by that change. q The smaller the change, the better this works. q You are using version control, right?
  • 44. Testing vs Brooks's Law q Tests catch damage done by a new programmer immediately q Easier for other developers to help you x They can pre-test their patches. x Even if you write perfect code, the rest of us don't.
  • 45. Testing Lots Of Values q Date handling code is notorious for magic dates that cause problems x 1970, 2038, 1904, 10,000. Leap years. Daylight savings. q So we want to repeat sets of tests with different values.
  • 46. It's Just Programming use Test::More tests => 32; use Date::ICal; my %ICal_Dates = ( '19971024T120000' => # from the docs. [ 1997, 10, 24, 12, 0, 0 ], '20390123T232832' => # after the Unix epoch [ 2039, 1, 23, 23, 28, 32 ], '19671225T000000' => # before the Unix epoch [ 1967, 12, 25, 0, 0, 0 ], '18990505T232323' => # before the MacOS epoch [ 1899, 5, 5, 23, 23, 23 ], ); while( my($ical_str, $expect) = each %ICal_Dates ) { my $ical = Date::ICal->new( ical => $ical_str, offset => 0 ) ok( defined $ical, "new(ical => '$ical_str')" ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) is( $ical->year, $expect->[0], ' year()' ); is( $ical->month, $expect->[1], ' month()' ); is( $ical->day, $expect->[2], ' day()' ); is( $ical->hour, $expect->[3], ' hour()' ); is( $ical->min, $expect->[4], ' min()' ); is( $ical->sec, $expect->[5], ' sec()' ); }
  • 47. The Good News q If you can write good code, you can learn to write good tests. q Just a while loop. q Easy to throw in more dates.
  • 48. The Bad News q You have to keep adjusting the # of tests when you add a date. x use Test::More tests => ##; q There are some tricks: # For each date, we run 8 tests. use Test::More tests => keys %ICal_Dates * 8; q There's also 'no_plan': use Test::More 'no_plan';
  • 49. Plan? There Ain't No Plan! q The plan exists for protection against: x The test dying x Accidentally not printing tests to STDOUT x Exiting early q The first two have other protections, and the third will shortly. x So the plan isn't as useful as it used to be q Newer versions of Test::Harness allow the plan to be at the end: ok 1 ok 2 ok 3 1..3 q This allows Test::More to count your tests for you. x You have to upgrade Test::Harness for this to work.
  • 50. Boundary tests q Almost bad input q Bad input q No input q Lots of input q Input that revealed a bug
  • 51. Bad Input Can Do Bad Things q Garbage in / Error out x graceful exceptions, not perl errors x helpful warnings, not uninitialized value warnings q Make sure bad input causes predictable, graceful failure.
  • 52. Basic bad input example use Test::More tests => 2; local $!; ok( !open(FILE, "I_dont_exist"), 'non-existent file' ); isnt( $!, 0, ' $! set' ); q Note, the exact value of $! is unpredictable.
  • 53. Tests with warnings q Test::More used to have a problem testing undefined values use Test::More tests => 1; is( undef, undef, 'undef is undef' ); q The test will pass, but there would be warnings. x The user will see them, but the test will not. q There's a whole bunch of these in Test-Simple/t/undef.t
  • 54. Catching Warnings q Use $SIG{__WARN__}. my $warnings = ''; local $SIG{__WARN__} = sub { $warnings . join '', @_ }; use Test::More tests => 2; is( undef, undef, 'undef is undef' ); is( $warnings, '', ' no warnings' ); q Use the same technique to check for expected warnings.
  • 55. Dealing With Death q Use eval BLOCK. local $@; eval { croak "Wibble"; }; like( $@, qr/^Wibble/ ); q Use the same technique to check that things didn't die. x Useful for past bugs where certain inputs would cause a fatal error.
  • 56. Acceptance, Regression, Unit, Functional... q Same thing, just a matter of timing. q Unit: Detailed tests of individual parts x Unit tests are easy(er) x So we think of the rest in terms of unit tests q Functional: Tests of your API x Blackbox unit tests q Integration: Testing that the pieces work together x Just unit testing bigger units q Acceptance: Tests defining your requirements x Customer driven unit tests q Regression: Tests for backwards compatibility x Old tests never die, they just become regression tests q All can be done with the same techniques
  • 57. Blackbox vs Glassbox q No, they're not window managers. q Blackbox tests use only the public, documented API. x No cheating x You have to forget how the code is implemented x More closely approximates real world usage x Immune from internal changes x Often forces you to make the public API more flexible q Glassbox tests can use whatever you want. x Cheat, steal, lie, violate encapsulation x Often necessary to test certain 'untestable' parts x May be broken by internal changes, undermines the test suite. q Blackbox is preferred where possible, glassbox is sometimes necessary. x Sometimes you can just peek inside the box.
  • 58. Test::More toys q Test::More has 13 ways to say ok. x It also has a wonderful man page. q Here's some of the most common.
  • 59. like() q Next to is() and ok(), you'll be using like() the most. like( $this, qr/that/ ); q This is the same as: ok( $this =~ /that/ ); q It has nicer diagnostics: not ok 1 # Failed test (contrived.t at line 2) # 'wibble' # doesn't match '(?-xism:woof)' q Because qr// was added in 5.005, it understands a string that looks like a regex for older perls. like( $this, '/that/' ); q There is an unlike() which is the !~ version.
  • 60. isa_ok() q We've been doing this a lot. ok( defined $ical, "new(ical => '$ical_str')" ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) q You do this so much in OO code, there's a special function. isa_ok( $ical, 'Date::ICal' ); q It works on references, too. isa_ok( $foo, 'ARRAY' ); # is $foo an array ref? q It also has nice diagnostics. not ok 1 - The object isa Date::ICal # Failed test (- at line 2) # The object isn't a 'Date::ICal' it's a 'ARRAY'
  • 61. can_ok() q A test for $obj->can($some_method) ok( $obj->can('foo'), 'foo() method inherited' ); q Simple but useful test can be like: # Does the Foo class have these methods? can_ok( 'Foo', qw(this that whatever wibble) ); x Might seem silly, but can catch stupid mistakes like forgetting a "=cut" q Takes an object or a class. q Also useful for checking your functions are exported use Text::Soundex; can_ok(__PACKAGE__, 'soundex');
  • 62. use_ok() q The real first thing you test is if the module loaded. use Test::More tests => 1; BEGIN { use_ok( 'Date::ICal' ); } q Has to be inside a BEGIN block to act like a real 'use'. q Remember the black magic? That's what it was doing.
  • 63. is_deeply() q For comparing complex data structures x Hashes, lists, hash of lists of hashes of lists of scalar references... my %expect = ( this => 42, that => [qw(1 2 3)] ); my %got = some_function(); is_deeply( %got, %expect ); q Will show you where the two structures start to diverge not ok 1 # Failed test (- at line 2) # Structures begin differing at: # $got->{that}[2] = '3' # $expected->{that}[2] = Does not exist q In CS this is really a "shallow comparison" and is() is "deep". x So the name is wrong because Schwern failed CS. q A stopgap measure x Currently doesn't handle circular structures (patches welcome) q Waiting for someone to step up to the plate and write Test::Set
  • 64. diag() q Test::More's functions are pretty good about providing diagnostics. q Sometimes you need more... q diag() lets you display whatever diagnostic information you want. x Guaranteed not to interfere with Test::Harness x Not a test function x Will not display inside a TODO block q Useful for giving suggestions about tricky failures
  • 65. Odd User Reactions q Sometimes users react rather oddly to tests. x won't report failures x will react to failures as if the test caused the bug! x will report "the tests failed" and leave off all the diagnostics x won't run the tests at all
  • 66. Getting People to RUN Your Tests q Once you've gotten people writing tests... q ...your next problem is getting them to RUN them
  • 67. Make It Simple q Preferably ONE command. x no user interaction (or smart defaults) x 'make test' x 'quicktest' CVS integration.
  • 68. Test On Commit q Make running the tests part of your commit policy x Automate with CVS commit actions (CVSROOT/modules) x Use a system such as Aegis
  • 69. Daily Smoke Test q Run the whole battery of tests against the latest code every day, automatically x CPAN::Smoke is one example
  • 70. Test Before Release q Automatically run tests as part of your release process. x 'make disttest' x your release process is automated, right?
  • 71. Testing Is Eating Your Own Dog Food q It forces you to use your own API. q Code that's hard to test may be hard to use. q This often makes your API more flexible. x Tends to get rid of constants and assumptions
  • 72. 'make test' schwern@blackrider:~/src/devel/File-chdir$ make test PERL_DL_NONLAZY=1 /usr/local/bin/perl5.6.1 -Iblib/arch -Iblib/lib -I/usr/local/perl5.6.1/lib/5.6.1/ppc-linux-64int -I/usr/local/perl5.6.1/lib/5.6.1 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t t/array.............ok t/chdir.............ok t/var...............ok All tests successful. Files=3, Tests=48, 2 wallclock secs ( 1.71 cusr + 0.38 csys = 2.09 CPU) q When you run 'make test' on a CPAN module, you're using: ExtUtils::MakeMaker Test::Harness your test
  • 73. What in the hell is all that mess? PERL_DL_NONLAZY=1 q magic to force XS code to strictly check shared libraries -Iblib/lib -Iblib/lib q Changes @INC to use the module you're about to install -I/usr/local/perl5.6.1/lib/5.6.1/ppc-linux-64int ... q Mistake. Code specific for testing Perl itself that leaked out. q Causes problems with core modules on CPAN. q Fixed in latest versions of MakeMaker.
  • 74. The mess continued... -e 'use Test::Harness qw(&runtests $verbose); q import runtests and $verbose $verbose=0 q This is really $verbose=$(TEST_VERBOSE) runtests @ARGV;' t/*.t q Pass in all your tests to Test::Harness::runtests()
  • 75. Still more mess... t/array.............ok t/chdir.............ok t/var...............ok q Your tests are all ok All tests successful. q It's Miller Time. Files=3, Tests=48, 2 wallclock secs ( 1.71 cusr + 0.38 csys = 2.09 CPU) q Benchmark of how long your tests took to run. May go away.
  • 76. New MakeMaker Is A Little Different $ make test PERL_DL_NONLAZY=1 /usr/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/array....ok t/chdir....ok t/var......ok All tests successful. Files=3, Tests=48, 3 wallclock secs ( 2.27 cusr + 0.48 csys = 2.75 CPU) q The -I$(PERL_LIB) -I$(PERL_ARCH) mistake is gone q The hanging Test::Harness wires have been put away q Mostly done for non-Unix platforms.
  • 77. test.pl caveat q Some modules put tests in test.pl. q Do not do that. q 'make test' does not parse the output which means... x 'make test' won't exit with non-zero on failure. x Things like the CPAN shell won't know there was a failure. x Historical accident, MakeMaker predates Test::Harness.
  • 78. Testing and Perl versions q Test::Simple/More will be in 5.8.0. q Test.pm was put in 5.4.5. q Test::Harness has been around so long nobody remembers who wrote it. x pre-5.6.1 will not support TODO tests or no_plan. q They're all available from CPAN. q They all work back to 5.4.0. q They all work on every platform.
  • 79. Testing, CPAN Modules, PREREQ_PM q Some people worry about having too many prereqs on their CPAN modules x Don't want to add prereqs on testing modules q A prereq of Test::More in turn prereqs & upgrades Test::Harness. q Even though Test::More isn't yet in core, it's already widely installed. Acme::ComeFrom, Acme::Magpie, Acme::Time::Asparagus, Acme::USIG, Acme::Your, Alzabo, Apache::ConfigParser, Apache::DefaultCharset, Apache::GuessCharset, Apache::RSS, Apache::Session::CacheAny, Apache::Session::Generate::ModUniqueId, Apache::Session::Generate::ModUsertrack, Apache::Session::PHP, Apache::Session::SQLite, Apache::Singleton, Apache::StickyQuery, App::Info, Archive::Any, Astro::Funtools::Parse, Attribute::Profiles, Attribute::Protected, Attribute::Unimplemented, CPAN, Business::Tax::Vat, Cache::Mmap, Carp::Assert, CDDB::File, CGI::Application, CGI::FormMagick, CGI::Untaint, CGI::Untaint::creditcard, CGI::Untaint::email, CGI::Untaint::uk_postcode, Class::DBI,
  • 80. More modules with Test::Simple/Test::More prerequisites Class::DBI::FromCGI, Class::DBI::Join, Class::DBI::mysql, Class::DBI::SQLite, Class::Factory, Class::Observable, Class::PseudoHash, Class::Trigger, CompBio, File::Random, Crypt::CAST5_PP, Crypt::OOEnigma, Data::BFDump, Data::BT:PhoneBill, Date::Chinese, Date::DayOfWeek, Date::Discordian, Date::Easter, Date::Passover, Date::ICal, Date::ISO, Date::Japanese, Date::Leapyear, Date::Range, Date::Range::Birth, Date::Roman, Date::Set, Date::SundayLetter, Devel::Caller, Devel::LexAlias, Devel::Profiler, Devel::Tinderbox::Reporter, DNS::Singleton, Email::Find, Email::Valid::Loose, Encode::Punycode, Getopt::ArgvFile, GraphViz::Data::Structure, Hash::Merge, HTML::Calendar::Simple, HTML::DWT, HTML::ERuby, HTML::FromANSI, HTML::LBI, HTML::Lint, HTML::TableParser, HTML::Template::JIT, HTML::TextToHTML, I18N::Charset, IDNA::Punycode, Ima::DBI, Image::DS9, Inline::TT, IO::File::Log, Lingua::Pangram, Lingua::SoundChange, Lingua::Zompist::Barakhinei, Lingua::Zompist::Cadhinor, Lingua::Zompist::Kebreni, Lingua::Zombist::Verdurian, Locale::Maketext::Lexicon, Log::Dispatch::Config, Log::Dispatch::DBI, Mail::Address::MobileJp,
  • 81. Everybody's Depending on Us! Mail::Address::Tagged, Mail::ListDetector, Mail::ListDetector::Detector::Fml, MARC::Record, Math::Currency, Module::CoreList, Module::InstalledVersion, SPOPS, Net::DNS, Net::DNS::Zonefile, Net::ICal, Net::IDN::Nameprep, Net::IP::Match, Net::Services, Net::Starnet::DataAccounting, Net::Telnet::Cisco, OutNet::BBS, PerlPoint::Package, PHP::Session, Pod::Coverage, Test::Inline, POE::Component::IKC::ReallySimple, POE::Component::RSS, POE::Component::SubWrapper, POE::Session::Cascading, Proc::InvokeEditor, Regexp::English, Regexp::Network, Spreadsheet::ParseExcel::Simple, Storable, Sub::Context, Sub::Parameters, Term::Cap, Term::TtyRec, Test::Class, Test::Exception, Test::Mail, CGI::Application, Text::Quote, Text::WikiFormat, Tie::Array::Iterable, Tie::Hash::Approx, uny2k, WWW::Automate, WWW::Baseball::NPB, WWW::Page::Author, WWW::Page::Host, WWW::Page::Modified, WWW::Search, XML::Filter::BufferText, XML::SAX::Writer, XML::XPath::Simpl XML::XSLT, XTM, XTM::slides q So the prerequisite will likely already be resolved. q Brought to you by Schwern Of Borg.
  • 82. t/lib trick q If you still don't want to have prerequisites on testing modules x Copy Test/Builder.pm & Test/More.pm into t/lib/ x Slap a "use lib 't/lib'" on your tests x distribute the whole thing q Who does this? x CGI, CPANPLUS, MakeMaker, parrot, Test::Harness q Caveats x You'll be adding to Test::More's takeover of search.cpan.org x Adds 18K to your tarball. x Can't use TODO or no_plan.
  • 83. Make the GUI layer thin q GUIs, CGI programs, etc... are hard to test. q Make the problem as small as possible. x Separate the form from the functionality. x Put as much code into format agnostic libraries as possible x Large, stand-alone programs (especially CGIs) ring alarm bells. q You might wind up with a small amount that still needs to be tested by hand. x At least you don't have to test the whole thing by hand.
  • 84. Testing Web Stuff q WWW::Automate is your friend. x LWP with lots of help. x Easily deals with forms x "Click" on buttons x Follow links x Has a "back" button q Makes simulating a real web site user easier.
  • 85. Domain Specific Test Libraries q WWW::Automate x Technically not a test library, but sooooo useful q Test::Exception q Test::Differences x Testing large blocks of text and complicated structures q Test::Unit x Straight XUnit port to Perl x Great for those used to JUnit & PyUnit q Test::Class x XUnit, but adapted to Perl x Inherited tests q Test::MockObject q Test::Inline x Embed tests in your documentation q Test::Mail
  • 86. Test::Builder q Usually you want Test::More's general functions + domain specific ones. x Unfortunately, sometimes test libraries don't play well together x Who owns the test counter? x Who prints the plan? q Test::Builder is a single backend to solve that problem. x Singleton object to handle the plan and the counter x Test::More-like methods you can write wrappers around q Test libraries built on Test::Builder will work together. Test::Exception, Test::Class, Test::MockObject, Test::Inline, Test::Mail, Test::More, Test::Simple q Attend "Writing A Test Library" for more information
  • 87. Passing Tests Should PASS q One must trust their test suite, else it will be ignored. q When it fails, it should indicate a real problem. q "Expected failures" sap that trust. x "Oh, don't worry, that test always fails on Redhat 6.2" x If a failure sometimes isn't really a failure, when do you know a real failure? q "Expected failures" make test automation impossible. x Programs don't know "well, the test failed but it really passed" x Joe CPAN module installer also doesn't know that. q Get your test suite at 100% and keep it there. x That's worth saying again. q STAY AT 100% PASSING!
  • 88. Failure Is An Option q There are three varieties of test failure, and several solutions. x A failure indicating a mistake/bad assumption in the test suite. Ë You fix it. x A real failure indicating a bug or missing feature. Ë You fix it, or... Ë You put off fixing it and... Ë comment out the test (blech) or... Ë declare it "TODO" x A failure due to an assumption about the environment. Ë You can't fix it, so you "skip" it.
  • 89. It'll Never Work q Sometimes, a test just doesn't make sense in certain environments. q Some examples... x Features which require a certain version of perl x Features which require perl configured a certain way (ex. threads) x Features which are platform specific x Features which require optional modules
  • 90. Skipping Tests q Let's assume we have a test for an HTML generator. q Let's also assume that if we have HTML::Lint, we want to lint the generated code. require HTML::Lint; my $lint = HTML::Lint->new; isa_ok( $lint, 'HTML::Lint' ); $lint->parse( $some_html ); is( $lint->errors, 0, 'No errors found in HTML' ); q Since HTML::Lint is optional, this test will fail if you don't have it. x But it's not a real failure, else HTML::Lint isn't really optional. x So the user shouldn't hear about it.
  • 91. # SKIP q You can explicitly skip a set of tests rather than run them. 1..2 ok 1 ok 2 # SKIP no beer x Test #1 passed. x Test #2 was skipped because there is no beer. q A skipped test means the test was never run.
  • 92. SKIP: block q Test::More can cause an entire block of code not to run at all. SKIP: { eval { require HTML::Lint }; skip "HTML::Lint not installed", 2 if $@; my $lint = new HTML::Lint; isa_ok( $lint, "HTML::Lint" ); $lint->parse( $html ); is( $lint->errors, 0, "No errors found in HTML" ); } x if we don't have HTML::Lint, the skip() function is run. x skip() prevents anything further in the SKIP block to be run. x the number indicates how many tests you would have run. q The appropriate number of 'ok's will be output. ok 23 # SKIP HTML::Lint not installed ok 24 # SKIP HTML::Lint not installed
  • 93. skipall q In some cases you want to skip a whole test file. use Test::More; if( $^O eq 'MSWin32' ) { plan tests => 42; } else { plan skip_all => 'Win32 specific test'; } q Test::More will exit at the skip_all. q On non-Win32, the output will be: 1..0 # skip Win32 specific test q Test::Harness will interpret this as a skipped test.
  • 94. Procrastination Codified q It's good to write the test before you add a new feature. q It's good to write a test as soon as you receive a bug report. q It's bad to release code with failing tests. q This would seem to be a contradiction. x Either you fix all your bugs and add all your features immediately x Or you comment out your failing tests. q Option #3, for the professionally lazy: x Declare your failing tests to be "todo" q This allows one to build a test suite without having to fix all the bugs you find right away.
  • 95. TODO Test TODO: { local $TODO = 'URI::Geller not quite working'; my $card = 'Eight of clubs'; is( URI::Geller->your_card, $card, 'Is this your card?' ); my $spoon; URI::Geller->bend($spoon); is( $spoon, 'bent', 'Spoon bending' ); } q Output will be something like: not ok 23 - Is this your card # TODO URI::Geller not quite working not ok 24 - Spoon bending # TODO URI::Geller not quite working
  • 96. Automated TODO List q TODO reverses the sense of the test x 'not ok' will be treated as a quiet success x 'ok' Test::Harness will warn you of an "unexpected success" q It's a TODO list x Write your tests before your feature/bug fix x Each 'unexpected success' is an item off your todo list x Remove the TODO wrapper q You can release at any point and not have to cull your test suite q Keeps users from seeing "expected failures" q Each open bug can have a test. x Sometimes bugs get accidentally fixed
  • 97. Keep Test Scripts Small q Many testing questions start with x "I've got this test script with 1400 tests..." q Big tests are x Hard to maintain x Hard to decouple x Hard to read x Take a long time to run x Have all the same problems as big subroutines q Keep them small & focused. x One function or set of functions per script x One aspect per script x Put complicated tests in their own script x Put slow tests in their own script q Test::Simple/More's tests are a good example
  • 98. Big FTP/XML program example q Common testing problem. You have a big program which... x Downloads an XML file via FTP x Parses the XML x Generates HTML q How do you test that?
  • 99. Programs Are Hard, Libraries Are Easy q The smaller the piece, the better. q The more flexible the piece, the better. q The more hooks into the guts, the better. x Libraries of functions can have small, flexible pieces. x Programs are, by definition, monolithic. q Extract pieces out of your program and put it into a library x Then test the library x Side-benefit, you'll have improved your code q Take the FTP, XML parsing and HTML generation code out of the program.
  • 100. Separate Form And Functionality q HTML is hard to test x It changes a lot x It's hard to parse q Instead of going from XML straight to HTML q ...go from XML -> agnostic format -> HTML x Test the XML -> agnostic part x Test the agnostic -> HTML part q Much easier to test when only one of the input/output pair is formatted. q ...and you'll have improved the flexibility of your code.
  • 101. Mock Code q Sometimes you just can't run a piece of code in a test x Maybe there's no network connection x Maybe the test is destructive (system("/sbin/shutdown now")) q Going to the extreme edge of glassbox testing, replacing code for testing
  • 102. System call / Power manager example q Say you have to test a power management daemon q One of the things it does is puts the computer to sleep q How do you test that? sub should_i_sleep { my($power_remaining) = @_; system("/sbin/snooze") if $power_remaining < $Min_Power; return 1; }
  • 103. First, Isolate The Untestable Part sub should_i_sleep { my($power_remaining) = @_; snooze if $power_remaining < $Min_Power; return 1; } sub snooze { system("/sbin/snooze"); } q Test snooze() by hand once. x It's small, so you can get away with it
  • 104. Then, Replace The Untestable Part { my @snooze_args = (); my $snooze_called = 0; local *Power::Manager::snooze = sub { $snooze_called++; @snooze_args = @_; # trap the arguments return 0; # simulate successful system call }; should_i_sleep($Min_Power - 1); is( $snooze_called, 1, 'snooze called once' ); is( @snooze_args, 0, ' called properly' ); } q Check that it was called. q Check that it got the right arguments q By changing the return value to non-zero we can simulate a failure. q Very, very powerful technique.
  • 105. Forcing Failure q How will your program react if, say, the database connection fails? use DBI; { local *DBI::connect = sub { return 0; }; ...test for graceful failure here... } ...test for graceful recovery here...
  • 106. He's Your Dog, Charlie Brown q Don't leave testing for the QA guys x too much delay x too much animosity q You know your code, you can test it x and you can fix it x and you wrote it, so it's your bug :P
  • 107. Further Reading q perl-qa@perl.org q http://archive.develooper.com/perl-qa@perl.org/ q "Perl Debugged" q "Writing Solid Code"
  • 108. Thanks q Norman Nunley q Andy Lester q Barrie Slaymaker q H. Merijn Brand q Jarkko Hietaniemi q Tatsuhiko Miyagawa q Tels q Rafael Garcia-Suarez q Abhijit Menon-Sen q Curtis Poe & OTI q Beer and Root Beer (fuel of champions)