• Like
Test-Tutorial
Upcoming SlideShare
Loading in...5
×
Uploaded on

 

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
4,340
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
14
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Test::Tutorial chromatic and Michael G Schwern
  • 2. Testing? Why do I care?
  • 3. What Is Testing? q Check that your code does what it's supposed to do. q At varying levels of granularity. q In varying environments. q At various points in its development.
  • 4. What Is Automated Testing? q Programs to check other programs. q As with any rote task, you let the computer do it. x Humans will forget rote tests, computers will not q Press a button and walk away. x No human required (very important) q Manually running tests is a waste of your time. q Tests should run as close to instantaneous as possible x so you won't have an excuse not to run them x so you'll run them as often as possible x Instant feedback
  • 5. Testing Promotes Automation q Testable code is decoupled q Testable code is scriptable
  • 6. Why Test? q no missing functionality q no accidental functionality q when your tests pass, you're done
  • 7. More informative bug reports q Better to get the diagnostic output of your tests than "It doesn't work" q Easily generated by end users x "Run the tests and send me the output" q Helps IMMENSELY with porting (this is why my code works on VMS) x You can often port code without ever using the machine you're porting to
  • 8. More More Reasons q Most of the time spent on a project is debugging and bug fixing. x Worse, it often comes at the end (hidden cost) x "Oh, I'm 99% done, I just need to do some testing" q Testing as you go will increase your development time, but reduce debugging time. x It will let you estimate more realistically x Increased project visibility x Reduced debug time once you get used to it
  • 9. The Real Reason For Writing Tests q Confidence. x No fear of change x No fear of lurking bugs x No fear of breaking old things x No fear that new things don't work Ë Knowing when things don't work. q So you can play and experiment without worry. q Enable refactoring
  • 10. Testing Is Laziness q Take an O(n) amount of work and make it O(1) x Instead of walking through the code by hand at each change x Teach the computer to do that.
  • 11. What to test
  • 12. Textbook Testing q Traditional testing philosophy says things like Test all subroutines Test all branches of all conditions Test the boundary conditions of all inputs ... q This is just big and scary and too much. q We're lazy, and we think we can still be effective with much less work.
  • 13. XP Testing q XP says to write your tests before you write your code. x It's hard enough to get people to write tests at all. x Changing their coding philosophy at the same time is worse. q If you can do Test First, excellent. q If you're not already testing, this is a chance to start some new habits...
  • 14. On Test-First Programming q Think of it as coding to teeny, tiny, mini-iterations. q Break each task into boolean expressions. q Ask "What feature do I need next?" x Test the smallest and most immediate element of the overall task. x Take small steps!
  • 15. The two test-first questions q "How can I prove that this feature works?" x Write the simplest test that will fail unless the feature works. x The test must fail. q "What is the least amount of code I can write to pass the test?" x The simpler the test, the simpler the code you need. x The test must now pass. q This produces known good code and a comprehensive test suite. q Be sure to run the entire test suite after you implement a task. q Don't be afraid of baby steps. That's the point.
  • 16. Test Bugs q Another good philosophy is to test bugs and new features. q Every time you find a bug, write a test for it. q Every time you add a new feature, write a test for it. q In the process, you might test a few other related things. q This is the simplest way to retrofit tests onto existing code.
  • 17. Effects Of Testing Bugs q This has pleasant effects: x Slowly grows your test suite x Focuses tests on the parts of the code which have the most bugs q You're allowed to make mistakes, but ONLY ONCE. Never twice. q A disproportionate amount of bugs use the same logic. x One test can catch lots of bugs
  • 18. Knowing You're Done t/Your-Code.t......ok All tests successful. q For once, your computer is telling you something good. q Instant, positive feedback
  • 19. There Is No Magic q You may have seen this from h2xs. ######## We start with some black magic to print on failure. # Change 1..1 below to 1..last_test_to_print . # (It may become useful if the test is moved to ./t subdirec BEGIN { $| = 1; print "1..1n"; } END {print "not ok 1n" unless $loaded;} use Foo; $loaded = 1; print "ok 1n"; ######## End of black magic. q Testing really isn't this frightening.
  • 20. And Now For Something Completely Different
  • 21. The Most Basic Perl Test Program #!/usr/bin/perl -w print "1..1n"; print 1 + 1 == 2 ? "ok 1n" : "not ok 1n"; q Since 1 + 1 is 2, this prints: 1..1 ok 1 x "1..1" I'm going to run one test. x "ok 1" The first test passed.
  • 22. Perl's Testing Protocol q There are two parts to running a test in Perl. x Your test x Test::Harness q The output of your test is piped to Test::Harness. q Test::Harness interprets your output and reports. $ perl -MTest::Harness -wle 'runtests @ARGV' contrived.t contrived....ok All tests successful. Files=1, Tests=1, 0 wallclock secs ( 0.02 cusr + 0.02 csys
  • 23. There's TMTOWTDI and there's this... q Here's some of the many ways people write their tests: x t/op/sysio.t print 'not ' unless (syswrite(O, $a, 2) == 2); print "ok 20n"; x ext/Cwd/t/cwd.t print +($getcwd eq $start ? "" : "not "), "ok 4n"; x t/pod/plainer.t unless( $returned eq $expected ) { print map { s/^/#/mg; $_; } map {+$_} # to avoid readonly values "EXPECTED:n", $expected, "GOT:n", $returned; print "not "; } printf "ok %dn", ++$test; q Maintenance nightmare.
  • 24. I'm ok, you're ok #!/usr/bin/perl -w use Test::Simple tests => 1; ok( 1 + 1 == 2 ); q "ok" is the backbone of Perl testing. x If the expression is true, the test pass. x False, it fails. q Every conceivable test can be performed just using ok().
  • 25. YOU FAILED!!! #!/usr/bin/perl -w use Test::Simple tests => 2; ok( 1 + 1 == 2 ); ok( 2 + 2 == 5 ); q from that comes: 1..2 ok 1 not ok 2 # Failed test (contrived.t at line 5) # Looks like you failed 1 tests of 2. x "1..2" I'm going to run two tests. x "ok 1" The first test passed. x "not ok 2" The second test failed. x Some helpful commentary from Test::Simple
  • 26. Date::ICal q We're going to write some tests for Date::ICal. x It's real code. x It's sufficiently complex. x Everyone understands dates. Ë Some people even have them.
  • 27. Where To Start? q This is the hardest part. q Retrofitting a test suite onto old code sucks. x Marching through the testing swamps. q Write tests from the start and your life will be easier. q In any event, begin at the beginning.
  • 28. new() q Since Date::ICal is OO, the beginning is when you make an object. x (white-lie: the beginning is when you load the module) #!/usr/bin/perl -w use Test::Simple tests => 2; use Date::ICal; my $ical = Date::ICal->new; # make an object ok( defined $ical ); # check we got something ok( $ical->isa('Date::ICal') ); # and it's the right class q This produces: 1..2 ok 1 ok 2 q This is your first useful test.
  • 29. Names q "ok 2" isn't terribly descriptive. x what if you have 102 tests, what did #64 do? q Each test can be given a little description. ok( defined $ical, 'new() returned something' ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) q This outputs 1..2 ok 1 - new() returned something ok 2 - and it's the right class
  • 30. What's In A Name q Two views on names. x A name is a descriptive tag so you can track the test output back to the code which produced it. (the original purpose) x A name is a short description of what was tested. q There's a subtle difference. q Don't pull your hair out over it. x More importantly, don't pull other people's hair out over it.
  • 31. Test The Manual q Simplest way to build up a test suite is to just test what the manual says it does. x Also a good way to find mistakes/omissions in the docs. x You can take this five steps further and put the tests IN the manual. Test::Inline, later. q If the docs are well written, they should cover usage of your code. x You do have docs, right?
  • 32. SYNOPSIS q A good place to start. x A broad overview of the whole system q Here's a piece of Date::ICal's SYNOPSIS. SYNOPSIS use Date::ICal; $ical = Date::ICal->new( year => 1964, month => 10, day = hour => 16, min => 12, sec => 47, tz => '0530' ); $hour = $ical->hour; $year = $ical->year; q Oddly enough, there is a bug in this.
  • 33. SYNOPSIS test use Test::Simple tests => 8; use Date::ICal; $ical = Date::ICal->new( year => 1964, month => 10, day => 16, hour => 16, min => 12, sec => 47, tz => '0530' ); ok( defined $ical, 'new() returned something' ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) ok( $ical->sec == 47, ' sec()' ); ok( $ical->min == 42, ' min()' ); ok( $ical->hour == 10, ' hour()' ); ok( $ical->day == 16, ' day()' ); ok( $ical->month == 10, ' month()' ); ok( $ical->year == 1964, ' year()' );
  • 34. SYNOPSIS results 1..8 ok 1 - new() returned something ok 2 - and it's the right class ok 3 - sec() not ok 4 - min() # Failed test (ical.t at line 14) not ok 5 - hour() # Failed test (ical.t at line 15) ok 6 - day() ok 7 - month() ok 8 - year() # Looks like you failed 2 tests of 8. q Whoops, failures! q We know what and where it failed, but not much else. q How do you find out more? x Throw in print statements x Run in the debugger. q That sounds like work.
  • 35. Test::More q Test::Simple is deliberately limited to one function. q Test::More does everything Test::Simple does. x You can literally s/use Test::Simple/use Test::More/ q It provides more informative ways to say "ok".
  • 36. is() you is() or is() you isnt() my $baby; q Test::More's is() function: x declares that something is supposed to be something else x "Is this, that?" is( $this, $that ); # From ok( $ical->day == 16, ' day()' ); # To is( $ical->day, 16, ' day()' );
  • 37. ok() to is() q Here's the test with ok() replaced with is() appropriately. use Test::More tests => 8; use Date::ICal; $ical = Date::ICal->new( year => 1964, month => 10, day => 16, hour => 16, min => 12, sec => 47, tz => '+0530' ); ok( defined $ical, 'new() returned something' ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) is( $ical->sec, 47, ' sec()' ); is( $ical->min, 42, ' min()' ); is( $ical->hour, 10, ' hour()' ); is( $ical->day, 16, ' day()' ); is( $ical->month, 10, ' month()' ); is( $ical->year, 1964, ' year()' ); q "Is $ical->sec, 47?" q "Is $ical->min, 12?"
  • 38. Diagnostic Output 1..8 ok 1 - new() returned something ok 2 - and it's the right class ok 3 - sec() not ok 4 - min() # Failed test (- at line 13) # got: '12' # expected: '42' not ok 5 - hour() # Failed test (- at line 14) # got: '21' # expected: '10' ok 6 - day() ok 7 - month() ok 8 - year() # Looks like you failed 2 tests of 8. q $ical->min returned 12 instead of 42. q $ical->hour returned 21 instead of 10.
  • 39. Interpreting The Results q Turns out, there is no 'tz' argument to new()! x And it didn't warn us about bad arguments q The real argument is 'offset' x So the synopsis is wrong. x This is a real bug I found while writing this q Damn those tests.
  • 40. When to use is() q Use instead of ok() when you're testing "this equals that". x Yes, there is an isnt() and isn't(). q is() does a string comparison which 99.99% of the time comes out right. x cmp_ok() exists to test with specific comparison operators
  • 41. Tests Are Sometimes Wrong q The previous example was supposed to be highly contrived to illustrate that tests are sometimes wrong. q When investigating a test failure, look at both the code and the test. q There's a fine line of trusting your testing code. x Too much trust, and you'll be chasing phantoms. x Too little trust, and you'll be changing your tests to cover up bugs.
  • 42. How Can I Be Sure The Test Is Right? q Write the test q Run it and make sure the new test fails q Add the new feature / fix the bug q Run the test and make sure the new test passes. q Some development systems, such as Aegis, can enforce this process. q It's difficult to do this when writing tests for existing code. x Another reason to test as you go
  • 43. Version Control and Testing q VC & testing work well. x Run the tests, make sure they pass x Make sure everything is checked in. x Write tests for the bug / feature. Ë Make sure they fail. x Fix your bug / write your feature x Run the tests. Ë If they pass, commit. You're done. Ë If they fail, look at the diff. The problem is revealed by that change. q The smaller the change, the better this works. q You are using version control, right?
  • 44. Testing vs Brooks's Law q Tests catch damage done by a new programmer immediately q Easier for other developers to help you x They can pre-test their patches. x Even if you write perfect code, the rest of us don't.
  • 45. Testing Lots Of Values q Date handling code is notorious for magic dates that cause problems x 1970, 2038, 1904, 10,000. Leap years. Daylight savings. q So we want to repeat sets of tests with different values.
  • 46. It's Just Programming use Test::More tests => 32; use Date::ICal; my %ICal_Dates = ( '19971024T120000' => # from the docs. [ 1997, 10, 24, 12, 0, 0 ], '20390123T232832' => # after the Unix epoch [ 2039, 1, 23, 23, 28, 32 ], '19671225T000000' => # before the Unix epoch [ 1967, 12, 25, 0, 0, 0 ], '18990505T232323' => # before the MacOS epoch [ 1899, 5, 5, 23, 23, 23 ], ); while( my($ical_str, $expect) = each %ICal_Dates ) { my $ical = Date::ICal->new( ical => $ical_str, offset => 0 ) ok( defined $ical, "new(ical => '$ical_str')" ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) is( $ical->year, $expect->[0], ' year()' ); is( $ical->month, $expect->[1], ' month()' ); is( $ical->day, $expect->[2], ' day()' ); is( $ical->hour, $expect->[3], ' hour()' ); is( $ical->min, $expect->[4], ' min()' ); is( $ical->sec, $expect->[5], ' sec()' ); }
  • 47. The Good News q If you can write good code, you can learn to write good tests. q Just a while loop. q Easy to throw in more dates.
  • 48. The Bad News q You have to keep adjusting the # of tests when you add a date. x use Test::More tests => ##; q There are some tricks: # For each date, we run 8 tests. use Test::More tests => keys %ICal_Dates * 8; q There's also 'no_plan': use Test::More 'no_plan';
  • 49. Plan? There Ain't No Plan! q The plan exists for protection against: x The test dying x Accidentally not printing tests to STDOUT x Exiting early q The first two have other protections, and the third will shortly. x So the plan isn't as useful as it used to be q Newer versions of Test::Harness allow the plan to be at the end: ok 1 ok 2 ok 3 1..3 q This allows Test::More to count your tests for you. x You have to upgrade Test::Harness for this to work.
  • 50. Boundary tests q Almost bad input q Bad input q No input q Lots of input q Input that revealed a bug
  • 51. Bad Input Can Do Bad Things q Garbage in / Error out x graceful exceptions, not perl errors x helpful warnings, not uninitialized value warnings q Make sure bad input causes predictable, graceful failure.
  • 52. Basic bad input example use Test::More tests => 2; local $!; ok( !open(FILE, "I_dont_exist"), 'non-existent file' ); isnt( $!, 0, ' $! set' ); q Note, the exact value of $! is unpredictable.
  • 53. Tests with warnings q Test::More used to have a problem testing undefined values use Test::More tests => 1; is( undef, undef, 'undef is undef' ); q The test will pass, but there would be warnings. x The user will see them, but the test will not. q There's a whole bunch of these in Test-Simple/t/undef.t
  • 54. Catching Warnings q Use $SIG{__WARN__}. my $warnings = ''; local $SIG{__WARN__} = sub { $warnings . join '', @_ }; use Test::More tests => 2; is( undef, undef, 'undef is undef' ); is( $warnings, '', ' no warnings' ); q Use the same technique to check for expected warnings.
  • 55. Dealing With Death q Use eval BLOCK. local $@; eval { croak "Wibble"; }; like( $@, qr/^Wibble/ ); q Use the same technique to check that things didn't die. x Useful for past bugs where certain inputs would cause a fatal error.
  • 56. Acceptance, Regression, Unit, Functional... q Same thing, just a matter of timing. q Unit: Detailed tests of individual parts x Unit tests are easy(er) x So we think of the rest in terms of unit tests q Functional: Tests of your API x Blackbox unit tests q Integration: Testing that the pieces work together x Just unit testing bigger units q Acceptance: Tests defining your requirements x Customer driven unit tests q Regression: Tests for backwards compatibility x Old tests never die, they just become regression tests q All can be done with the same techniques
  • 57. Blackbox vs Glassbox q No, they're not window managers. q Blackbox tests use only the public, documented API. x No cheating x You have to forget how the code is implemented x More closely approximates real world usage x Immune from internal changes x Often forces you to make the public API more flexible q Glassbox tests can use whatever you want. x Cheat, steal, lie, violate encapsulation x Often necessary to test certain 'untestable' parts x May be broken by internal changes, undermines the test suite. q Blackbox is preferred where possible, glassbox is sometimes necessary. x Sometimes you can just peek inside the box.
  • 58. Test::More toys q Test::More has 13 ways to say ok. x It also has a wonderful man page. q Here's some of the most common.
  • 59. like() q Next to is() and ok(), you'll be using like() the most. like( $this, qr/that/ ); q This is the same as: ok( $this =~ /that/ ); q It has nicer diagnostics: not ok 1 # Failed test (contrived.t at line 2) # 'wibble' # doesn't match '(?-xism:woof)' q Because qr// was added in 5.005, it understands a string that looks like a regex for older perls. like( $this, '/that/' ); q There is an unlike() which is the !~ version.
  • 60. isa_ok() q We've been doing this a lot. ok( defined $ical, "new(ical => '$ical_str')" ); ok( $ical->isa('Date::ICal'), " and it's the right class" ) q You do this so much in OO code, there's a special function. isa_ok( $ical, 'Date::ICal' ); q It works on references, too. isa_ok( $foo, 'ARRAY' ); # is $foo an array ref? q It also has nice diagnostics. not ok 1 - The object isa Date::ICal # Failed test (- at line 2) # The object isn't a 'Date::ICal' it's a 'ARRAY'
  • 61. can_ok() q A test for $obj->can($some_method) ok( $obj->can('foo'), 'foo() method inherited' ); q Simple but useful test can be like: # Does the Foo class have these methods? can_ok( 'Foo', qw(this that whatever wibble) ); x Might seem silly, but can catch stupid mistakes like forgetting a "=cut" q Takes an object or a class. q Also useful for checking your functions are exported use Text::Soundex; can_ok(__PACKAGE__, 'soundex');
  • 62. use_ok() q The real first thing you test is if the module loaded. use Test::More tests => 1; BEGIN { use_ok( 'Date::ICal' ); } q Has to be inside a BEGIN block to act like a real 'use'. q Remember the black magic? That's what it was doing.
  • 63. is_deeply() q For comparing complex data structures x Hashes, lists, hash of lists of hashes of lists of scalar references... my %expect = ( this => 42, that => [qw(1 2 3)] ); my %got = some_function(); is_deeply( %got, %expect ); q Will show you where the two structures start to diverge not ok 1 # Failed test (- at line 2) # Structures begin differing at: # $got->{that}[2] = '3' # $expected->{that}[2] = Does not exist q In CS this is really a "shallow comparison" and is() is "deep". x So the name is wrong because Schwern failed CS. q A stopgap measure x Currently doesn't handle circular structures (patches welcome) q Waiting for someone to step up to the plate and write Test::Set
  • 64. diag() q Test::More's functions are pretty good about providing diagnostics. q Sometimes you need more... q diag() lets you display whatever diagnostic information you want. x Guaranteed not to interfere with Test::Harness x Not a test function x Will not display inside a TODO block q Useful for giving suggestions about tricky failures
  • 65. Odd User Reactions q Sometimes users react rather oddly to tests. x won't report failures x will react to failures as if the test caused the bug! x will report "the tests failed" and leave off all the diagnostics x won't run the tests at all
  • 66. Getting People to RUN Your Tests q Once you've gotten people writing tests... q ...your next problem is getting them to RUN them
  • 67. Make It Simple q Preferably ONE command. x no user interaction (or smart defaults) x 'make test' x 'quicktest' CVS integration.
  • 68. Test On Commit q Make running the tests part of your commit policy x Automate with CVS commit actions (CVSROOT/modules) x Use a system such as Aegis
  • 69. Daily Smoke Test q Run the whole battery of tests against the latest code every day, automatically x CPAN::Smoke is one example
  • 70. Test Before Release q Automatically run tests as part of your release process. x 'make disttest' x your release process is automated, right?
  • 71. Testing Is Eating Your Own Dog Food q It forces you to use your own API. q Code that's hard to test may be hard to use. q This often makes your API more flexible. x Tends to get rid of constants and assumptions
  • 72. 'make test' schwern@blackrider:~/src/devel/File-chdir$ make test PERL_DL_NONLAZY=1 /usr/local/bin/perl5.6.1 -Iblib/arch -Iblib/lib -I/usr/local/perl5.6.1/lib/5.6.1/ppc-linux-64int -I/usr/local/perl5.6.1/lib/5.6.1 -e 'use Test::Harness qw(&runtests $verbose); $verbose=0; runtests @ARGV;' t/*.t t/array.............ok t/chdir.............ok t/var...............ok All tests successful. Files=3, Tests=48, 2 wallclock secs ( 1.71 cusr + 0.38 csys = 2.09 CPU) q When you run 'make test' on a CPAN module, you're using: ExtUtils::MakeMaker Test::Harness your test
  • 73. What in the hell is all that mess? PERL_DL_NONLAZY=1 q magic to force XS code to strictly check shared libraries -Iblib/lib -Iblib/lib q Changes @INC to use the module you're about to install -I/usr/local/perl5.6.1/lib/5.6.1/ppc-linux-64int ... q Mistake. Code specific for testing Perl itself that leaked out. q Causes problems with core modules on CPAN. q Fixed in latest versions of MakeMaker.
  • 74. The mess continued... -e 'use Test::Harness qw(&runtests $verbose); q import runtests and $verbose $verbose=0 q This is really $verbose=$(TEST_VERBOSE) runtests @ARGV;' t/*.t q Pass in all your tests to Test::Harness::runtests()
  • 75. Still more mess... t/array.............ok t/chdir.............ok t/var...............ok q Your tests are all ok All tests successful. q It's Miller Time. Files=3, Tests=48, 2 wallclock secs ( 1.71 cusr + 0.38 csys = 2.09 CPU) q Benchmark of how long your tests took to run. May go away.
  • 76. New MakeMaker Is A Little Different $ make test PERL_DL_NONLAZY=1 /usr/local/bin/perl "-MExtUtils::Command::MM" "-e" "test_harness(0, 'blib/lib', 'blib/arch')" t/*.t t/array....ok t/chdir....ok t/var......ok All tests successful. Files=3, Tests=48, 3 wallclock secs ( 2.27 cusr + 0.48 csys = 2.75 CPU) q The -I$(PERL_LIB) -I$(PERL_ARCH) mistake is gone q The hanging Test::Harness wires have been put away q Mostly done for non-Unix platforms.
  • 77. test.pl caveat q Some modules put tests in test.pl. q Do not do that. q 'make test' does not parse the output which means... x 'make test' won't exit with non-zero on failure. x Things like the CPAN shell won't know there was a failure. x Historical accident, MakeMaker predates Test::Harness.
  • 78. Testing and Perl versions q Test::Simple/More will be in 5.8.0. q Test.pm was put in 5.4.5. q Test::Harness has been around so long nobody remembers who wrote it. x pre-5.6.1 will not support TODO tests or no_plan. q They're all available from CPAN. q They all work back to 5.4.0. q They all work on every platform.
  • 79. Testing, CPAN Modules, PREREQ_PM q Some people worry about having too many prereqs on their CPAN modules x Don't want to add prereqs on testing modules q A prereq of Test::More in turn prereqs & upgrades Test::Harness. q Even though Test::More isn't yet in core, it's already widely installed. Acme::ComeFrom, Acme::Magpie, Acme::Time::Asparagus, Acme::USIG, Acme::Your, Alzabo, Apache::ConfigParser, Apache::DefaultCharset, Apache::GuessCharset, Apache::RSS, Apache::Session::CacheAny, Apache::Session::Generate::ModUniqueId, Apache::Session::Generate::ModUsertrack, Apache::Session::PHP, Apache::Session::SQLite, Apache::Singleton, Apache::StickyQuery, App::Info, Archive::Any, Astro::Funtools::Parse, Attribute::Profiles, Attribute::Protected, Attribute::Unimplemented, CPAN, Business::Tax::Vat, Cache::Mmap, Carp::Assert, CDDB::File, CGI::Application, CGI::FormMagick, CGI::Untaint, CGI::Untaint::creditcard, CGI::Untaint::email, CGI::Untaint::uk_postcode, Class::DBI,
  • 80. More modules with Test::Simple/Test::More prerequisites Class::DBI::FromCGI, Class::DBI::Join, Class::DBI::mysql, Class::DBI::SQLite, Class::Factory, Class::Observable, Class::PseudoHash, Class::Trigger, CompBio, File::Random, Crypt::CAST5_PP, Crypt::OOEnigma, Data::BFDump, Data::BT:PhoneBill, Date::Chinese, Date::DayOfWeek, Date::Discordian, Date::Easter, Date::Passover, Date::ICal, Date::ISO, Date::Japanese, Date::Leapyear, Date::Range, Date::Range::Birth, Date::Roman, Date::Set, Date::SundayLetter, Devel::Caller, Devel::LexAlias, Devel::Profiler, Devel::Tinderbox::Reporter, DNS::Singleton, Email::Find, Email::Valid::Loose, Encode::Punycode, Getopt::ArgvFile, GraphViz::Data::Structure, Hash::Merge, HTML::Calendar::Simple, HTML::DWT, HTML::ERuby, HTML::FromANSI, HTML::LBI, HTML::Lint, HTML::TableParser, HTML::Template::JIT, HTML::TextToHTML, I18N::Charset, IDNA::Punycode, Ima::DBI, Image::DS9, Inline::TT, IO::File::Log, Lingua::Pangram, Lingua::SoundChange, Lingua::Zompist::Barakhinei, Lingua::Zompist::Cadhinor, Lingua::Zompist::Kebreni, Lingua::Zombist::Verdurian, Locale::Maketext::Lexicon, Log::Dispatch::Config, Log::Dispatch::DBI, Mail::Address::MobileJp,
  • 81. Everybody's Depending on Us! Mail::Address::Tagged, Mail::ListDetector, Mail::ListDetector::Detector::Fml, MARC::Record, Math::Currency, Module::CoreList, Module::InstalledVersion, SPOPS, Net::DNS, Net::DNS::Zonefile, Net::ICal, Net::IDN::Nameprep, Net::IP::Match, Net::Services, Net::Starnet::DataAccounting, Net::Telnet::Cisco, OutNet::BBS, PerlPoint::Package, PHP::Session, Pod::Coverage, Test::Inline, POE::Component::IKC::ReallySimple, POE::Component::RSS, POE::Component::SubWrapper, POE::Session::Cascading, Proc::InvokeEditor, Regexp::English, Regexp::Network, Spreadsheet::ParseExcel::Simple, Storable, Sub::Context, Sub::Parameters, Term::Cap, Term::TtyRec, Test::Class, Test::Exception, Test::Mail, CGI::Application, Text::Quote, Text::WikiFormat, Tie::Array::Iterable, Tie::Hash::Approx, uny2k, WWW::Automate, WWW::Baseball::NPB, WWW::Page::Author, WWW::Page::Host, WWW::Page::Modified, WWW::Search, XML::Filter::BufferText, XML::SAX::Writer, XML::XPath::Simpl XML::XSLT, XTM, XTM::slides q So the prerequisite will likely already be resolved. q Brought to you by Schwern Of Borg.
  • 82. t/lib trick q If you still don't want to have prerequisites on testing modules x Copy Test/Builder.pm & Test/More.pm into t/lib/ x Slap a "use lib 't/lib'" on your tests x distribute the whole thing q Who does this? x CGI, CPANPLUS, MakeMaker, parrot, Test::Harness q Caveats x You'll be adding to Test::More's takeover of search.cpan.org x Adds 18K to your tarball. x Can't use TODO or no_plan.
  • 83. Make the GUI layer thin q GUIs, CGI programs, etc... are hard to test. q Make the problem as small as possible. x Separate the form from the functionality. x Put as much code into format agnostic libraries as possible x Large, stand-alone programs (especially CGIs) ring alarm bells. q You might wind up with a small amount that still needs to be tested by hand. x At least you don't have to test the whole thing by hand.
  • 84. Testing Web Stuff q WWW::Automate is your friend. x LWP with lots of help. x Easily deals with forms x "Click" on buttons x Follow links x Has a "back" button q Makes simulating a real web site user easier.
  • 85. Domain Specific Test Libraries q WWW::Automate x Technically not a test library, but sooooo useful q Test::Exception q Test::Differences x Testing large blocks of text and complicated structures q Test::Unit x Straight XUnit port to Perl x Great for those used to JUnit & PyUnit q Test::Class x XUnit, but adapted to Perl x Inherited tests q Test::MockObject q Test::Inline x Embed tests in your documentation q Test::Mail
  • 86. Test::Builder q Usually you want Test::More's general functions + domain specific ones. x Unfortunately, sometimes test libraries don't play well together x Who owns the test counter? x Who prints the plan? q Test::Builder is a single backend to solve that problem. x Singleton object to handle the plan and the counter x Test::More-like methods you can write wrappers around q Test libraries built on Test::Builder will work together. Test::Exception, Test::Class, Test::MockObject, Test::Inline, Test::Mail, Test::More, Test::Simple q Attend "Writing A Test Library" for more information
  • 87. Passing Tests Should PASS q One must trust their test suite, else it will be ignored. q When it fails, it should indicate a real problem. q "Expected failures" sap that trust. x "Oh, don't worry, that test always fails on Redhat 6.2" x If a failure sometimes isn't really a failure, when do you know a real failure? q "Expected failures" make test automation impossible. x Programs don't know "well, the test failed but it really passed" x Joe CPAN module installer also doesn't know that. q Get your test suite at 100% and keep it there. x That's worth saying again. q STAY AT 100% PASSING!
  • 88. Failure Is An Option q There are three varieties of test failure, and several solutions. x A failure indicating a mistake/bad assumption in the test suite. Ë You fix it. x A real failure indicating a bug or missing feature. Ë You fix it, or... Ë You put off fixing it and... Ë comment out the test (blech) or... Ë declare it "TODO" x A failure due to an assumption about the environment. Ë You can't fix it, so you "skip" it.
  • 89. It'll Never Work q Sometimes, a test just doesn't make sense in certain environments. q Some examples... x Features which require a certain version of perl x Features which require perl configured a certain way (ex. threads) x Features which are platform specific x Features which require optional modules
  • 90. Skipping Tests q Let's assume we have a test for an HTML generator. q Let's also assume that if we have HTML::Lint, we want to lint the generated code. require HTML::Lint; my $lint = HTML::Lint->new; isa_ok( $lint, 'HTML::Lint' ); $lint->parse( $some_html ); is( $lint->errors, 0, 'No errors found in HTML' ); q Since HTML::Lint is optional, this test will fail if you don't have it. x But it's not a real failure, else HTML::Lint isn't really optional. x So the user shouldn't hear about it.
  • 91. # SKIP q You can explicitly skip a set of tests rather than run them. 1..2 ok 1 ok 2 # SKIP no beer x Test #1 passed. x Test #2 was skipped because there is no beer. q A skipped test means the test was never run.
  • 92. SKIP: block q Test::More can cause an entire block of code not to run at all. SKIP: { eval { require HTML::Lint }; skip "HTML::Lint not installed", 2 if $@; my $lint = new HTML::Lint; isa_ok( $lint, "HTML::Lint" ); $lint->parse( $html ); is( $lint->errors, 0, "No errors found in HTML" ); } x if we don't have HTML::Lint, the skip() function is run. x skip() prevents anything further in the SKIP block to be run. x the number indicates how many tests you would have run. q The appropriate number of 'ok's will be output. ok 23 # SKIP HTML::Lint not installed ok 24 # SKIP HTML::Lint not installed
  • 93. skipall q In some cases you want to skip a whole test file. use Test::More; if( $^O eq 'MSWin32' ) { plan tests => 42; } else { plan skip_all => 'Win32 specific test'; } q Test::More will exit at the skip_all. q On non-Win32, the output will be: 1..0 # skip Win32 specific test q Test::Harness will interpret this as a skipped test.
  • 94. Procrastination Codified q It's good to write the test before you add a new feature. q It's good to write a test as soon as you receive a bug report. q It's bad to release code with failing tests. q This would seem to be a contradiction. x Either you fix all your bugs and add all your features immediately x Or you comment out your failing tests. q Option #3, for the professionally lazy: x Declare your failing tests to be "todo" q This allows one to build a test suite without having to fix all the bugs you find right away.
  • 95. TODO Test TODO: { local $TODO = 'URI::Geller not quite working'; my $card = 'Eight of clubs'; is( URI::Geller->your_card, $card, 'Is this your card?' ); my $spoon; URI::Geller->bend($spoon); is( $spoon, 'bent', 'Spoon bending' ); } q Output will be something like: not ok 23 - Is this your card # TODO URI::Geller not quite working not ok 24 - Spoon bending # TODO URI::Geller not quite working
  • 96. Automated TODO List q TODO reverses the sense of the test x 'not ok' will be treated as a quiet success x 'ok' Test::Harness will warn you of an "unexpected success" q It's a TODO list x Write your tests before your feature/bug fix x Each 'unexpected success' is an item off your todo list x Remove the TODO wrapper q You can release at any point and not have to cull your test suite q Keeps users from seeing "expected failures" q Each open bug can have a test. x Sometimes bugs get accidentally fixed
  • 97. Keep Test Scripts Small q Many testing questions start with x "I've got this test script with 1400 tests..." q Big tests are x Hard to maintain x Hard to decouple x Hard to read x Take a long time to run x Have all the same problems as big subroutines q Keep them small & focused. x One function or set of functions per script x One aspect per script x Put complicated tests in their own script x Put slow tests in their own script q Test::Simple/More's tests are a good example
  • 98. Big FTP/XML program example q Common testing problem. You have a big program which... x Downloads an XML file via FTP x Parses the XML x Generates HTML q How do you test that?
  • 99. Programs Are Hard, Libraries Are Easy q The smaller the piece, the better. q The more flexible the piece, the better. q The more hooks into the guts, the better. x Libraries of functions can have small, flexible pieces. x Programs are, by definition, monolithic. q Extract pieces out of your program and put it into a library x Then test the library x Side-benefit, you'll have improved your code q Take the FTP, XML parsing and HTML generation code out of the program.
  • 100. Separate Form And Functionality q HTML is hard to test x It changes a lot x It's hard to parse q Instead of going from XML straight to HTML q ...go from XML -> agnostic format -> HTML x Test the XML -> agnostic part x Test the agnostic -> HTML part q Much easier to test when only one of the input/output pair is formatted. q ...and you'll have improved the flexibility of your code.
  • 101. Mock Code q Sometimes you just can't run a piece of code in a test x Maybe there's no network connection x Maybe the test is destructive (system("/sbin/shutdown now")) q Going to the extreme edge of glassbox testing, replacing code for testing
  • 102. System call / Power manager example q Say you have to test a power management daemon q One of the things it does is puts the computer to sleep q How do you test that? sub should_i_sleep { my($power_remaining) = @_; system("/sbin/snooze") if $power_remaining < $Min_Power; return 1; }
  • 103. First, Isolate The Untestable Part sub should_i_sleep { my($power_remaining) = @_; snooze if $power_remaining < $Min_Power; return 1; } sub snooze { system("/sbin/snooze"); } q Test snooze() by hand once. x It's small, so you can get away with it
  • 104. Then, Replace The Untestable Part { my @snooze_args = (); my $snooze_called = 0; local *Power::Manager::snooze = sub { $snooze_called++; @snooze_args = @_; # trap the arguments return 0; # simulate successful system call }; should_i_sleep($Min_Power - 1); is( $snooze_called, 1, 'snooze called once' ); is( @snooze_args, 0, ' called properly' ); } q Check that it was called. q Check that it got the right arguments q By changing the return value to non-zero we can simulate a failure. q Very, very powerful technique.
  • 105. Forcing Failure q How will your program react if, say, the database connection fails? use DBI; { local *DBI::connect = sub { return 0; }; ...test for graceful failure here... } ...test for graceful recovery here...
  • 106. He's Your Dog, Charlie Brown q Don't leave testing for the QA guys x too much delay x too much animosity q You know your code, you can test it x and you can fix it x and you wrote it, so it's your bug :P
  • 107. Further Reading q perl-qa@perl.org q http://archive.develooper.com/perl-qa@perl.org/ q "Perl Debugged" q "Writing Solid Code"
  • 108. Thanks q Norman Nunley q Andy Lester q Barrie Slaymaker q H. Merijn Brand q Jarkko Hietaniemi q Tatsuhiko Miyagawa q Tels q Rafael Garcia-Suarez q Abhijit Menon-Sen q Curtis Poe & OTI q Beer and Root Beer (fuel of champions)