A Perl Testing Tutorial



       chromatic
Why to Test
ensure the code does what it should do
q   no missing functionality
q   no accidental functionality
q   when your tests al...
explore boundary conditions
q   how does the code handle bad input?
q   how does the code handle very bad input?
prevent (known) regressions
q   no tested features can be broken accidentally
q   no fixed bugs can reoccur
improve bug reports
q   test output can pinpoint errors
q   easily generated by end users
run this and send me the output
enable refactoring
q   you break it, you know it right away
q   side effects immediately evident
q   well-tested code tend...
improve API
q   hard to test code may be hard to use
q   writing a test first forces you to use the interface before you
 ...
promote automation
x   easy to test code is usually well-decoupled
x   easy to test code is, by definition, scriptable
ease maintainability
q   tests are a safety net
    x   if you don't understand this by now, I'll repeat it once again
q  ...
Don't be Scared!
Don't be Scared!
q   writing good tests exercises the same muscles as writing good
    code
q   if you can write code, you...
Testing Has Psychological Benefits
q   writing tests (especially test first) makes it easier to write tests
q   well-teste...
Perl makes it easy
q   ExtUtils::MakeMaker and the 'test' target
q   Test::* modules hide the Test::Harness expectations
q...
Basic Tests
Thinking "ok"
print "ok";
print "not ok";

q   basis for all tests
q   a simple boolean condition
q   generally numbered s...
More thinking "ok"
q   why printing is a loss:
    x   same arguments as using subroutines (duplicate code)
    x   manual...
The ok() function
ok( 1 == 1 );
q   basis for all Test::Simple, Test::More, Test::Builder tests
q   a simple boolean condi...
Test::Simple and its API
planning tests
q   helps catch too many/not enough test errors
q   helps identify which test fails
q   required by Test::H...
tests => n
q   produces 1..n output for Test::Harness
q   should be set when the test suite is complete
no_plan
q   produces 1..actual output at the end of the test
q   when you don't know how many tests will run
q   very hand...
ok()
q   scalar context of operands
q   first argument always true or false
q   second argument optional, test name
q   co...
ok() passing
ok( 1, '1 should be true' );

ok 1 - 1 should be true
ok() failing
ok( 0, '0 should be true' );

not ok 2 - 0 should be true
#     Failed test (demotest.pl at line 17)
success is not a gray area (individual tests)
  q   pass, or fail, there is no maybe
  q   test the smallest piece possibl...
success is all or nothing (test suite)
q   stamp out expected failures
    x   undermine confidence in the code
    x   cl...
Integrating testing with the development process
catching regressions
q   most bugs aren't new, they happen again and again
q   why let them reoccur?
q   write a test for ...
testing first
q   teeny, tiny, mini-iterations
q   break each task into boolean expressions
q   "What feature do I need ne...
The two test-first questions
q   "How can I prove that this feature works?"
    x   the simplest test that will fail unles...
writing testable code
q   the smaller the unit, the better
q   the simpler the unit, the better
q   the better the test su...
distributing tests
q   Test.pm is easier to distribute
    x   single module, does not depend on Test::Builder
    x   com...
Exploring the Test::More API
graduating from Test::Simple to Test::More, or why
                ok() is not enough
     q   one bit of data -- either i...
matching things - is()
q   asks the question "are these two things equal?"
q   does a pretty good job of guessing at the t...
is() passing
is( 2 + 2, 4, 'Two plus two should be four' );

ok 1 - Two plus two should be four
is() failing
is( 2 + 2, 5, 'Two plus two should be five' );

not ok 2 - Two plus two should be five
#     Failed test (dem...
matching things - isnt()
q   asks the question "are these two things not equal?"
q   essentially the opposite of is()
isnt() passing
isnt( 2 + 2, 5, 'Two plus two should not be five' );

ok 3 - Two plus two should not be five
isnt() failing
isnt( 2 + 2, 4, 'Two plus two should not be four' );

not ok 4 - Two plus two should not be four
#     Fail...
matching things - like()
q   applies a regular expression to a string
q   first argument is the string
q   second argument...
like() passing
like( 'foobar', qr/fo+/,
'+ quantifier should match one or more' );

ok 5 - + quantifier should match one o...
like() failing
like( 'fbar', qr/fo+/,
'+ quantifier should match zero or more' );

not ok 6 - + quantifier should match ze...
matching things - unlike()
q   essentially the opposite of like()
q   the same rules apply
unlike() passing
unlike( 'foobar', qr/baz/, 'foobar should not match baz' );

ok 7 - foobar should not match baz
unlike() failing
unlike( 'foobar', qr/bar/, 'foobar should not match bar' );

not ok 8 - foobar should not match bar
#    ...
exploring behavior - use_ok()
q   (these tests all provide their own test names)
q   see if a module compiles and can be i...
use_ok() passing
use_ok( 'Test::More' );

ok 11 - use Test::More;
use_ok() failing
use_ok( 'No::Module' );

not ok 12 - use No::Module;
#     Failed test (demotest.pl at line 20)
#     Tri...
exploring behavior - require_ok()
q   see if a module compiles
q   load a module, but do not import() it
q   also provides...
require_ok() passing
require_ok( 'Test::Simple' );

ok 13 - require Test::Simple;
require_ok() failing
require_ok( 'No::Module' );

not ok 14 - require No::Module;
#     Failed test (demotest.pl at line 2...
exploring behavior - isa_ok()
q   see if an object is a type of reference
q   can be an object
    x   but not a class
   ...
isa_ok() passing
isa_ok( [], 'ARRAY' );

ok 15 - The object isa ARRAY
isa_ok() failing
isa_ok( {}, 'IO::Socket' );

not ok 16 - The object isa IO::Socket
#     Failed test (demotest.pl at line...
exploring behavior - can_ok()
q   see if an object has named behavior(s)
q   can be an object or a class
q   pass several ...
can_ok() passing
can_ok( 'Test::More', 'can_ok' );

ok 17 - Test::More->can('can_ok')
can_ok() failing
can_ok( 'Test::More', 'autotest' );

not ok 18 - Test::More->can('autotest')
#     Failed test (demotest....
controlling things - skip()
q   skips a specified number of tests
q   generally used for things that will NEVER pass
q   t...
skip() in action
SKIP: {
   skip( 'Never on a Sunday', 2 ) if (localtime)[6] == 0;
can_ok( 'Perl::Porter', 'buy_beer' );
c...
controlling things - todo()
q   marks a specified number of tests as known failures
q   generally used for failures
    x ...
todo() in action
TODO: {
local $TODO = 'The alien overlords have not arrived';
is( $smallpox->status(), 'ready', 'Smallpox...
controlling things - diag()
q   display a (list of) diagnostic message(s)
q   guaranteed not to interfere with the test ha...
diag() in action
diag( "Now that you know a bit about test writing,n",
        "you have no excuse not to test." );

# Now...
controlling things - pass()
q   make a test pass, unconditionally
q   takes a test name -- that's it!
pass() in action
pass( 'Executive fiat should work' );

ok 19 - Executive fiat should work
controlling things - fail()
q   makes a test fail, unconditionally
q   also takes a test name, and that's it
fail() in action
fail( '... but we live in a meritocracy' );

not ok 20 - ... but we live in a meritocracy
#     Failed te...
Upcoming SlideShare
Loading in...5
×

IntroTestMore

1,012

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,012
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

IntroTestMore

  1. 1. A Perl Testing Tutorial chromatic
  2. 2. Why to Test
  3. 3. ensure the code does what it should do q no missing functionality q no accidental functionality q when your tests all pass, you're done
  4. 4. explore boundary conditions q how does the code handle bad input? q how does the code handle very bad input?
  5. 5. prevent (known) regressions q no tested features can be broken accidentally q no fixed bugs can reoccur
  6. 6. improve bug reports q test output can pinpoint errors q easily generated by end users run this and send me the output
  7. 7. enable refactoring q you break it, you know it right away q side effects immediately evident q well-tested code tends to be more modular x I had several pre-existing modules with similar behavior to test. They were only tangentially related. After the second test suite turned out almost the same, I realized I needed a parent class. I moved most of the tests to the parent class and the inherited classes ended up much simpler and very much shorter. Their tests were much simpler, too. This gave me less code to maintain, fewer tests to write, and more flexibility for adding new features.
  8. 8. improve API q hard to test code may be hard to use q writing a test first forces you to use the interface before you create it x simpler interfaces are easier to test x smaller functions are easier to reuse
  9. 9. promote automation x easy to test code is usually well-decoupled x easy to test code is, by definition, scriptable
  10. 10. ease maintainability q tests are a safety net x if you don't understand this by now, I'll repeat it once again q well-written tests serve as a sort of documentation x demonstrate what features the software should have x demonstrate what behavior the code should express
  11. 11. Don't be Scared!
  12. 12. Don't be Scared! q writing good tests exercises the same muscles as writing good code q if you can write code, you can write tests q if you can write good code, you can be taught to write good tests
  13. 13. Testing Has Psychological Benefits q writing tests (especially test first) makes it easier to write tests q well-tested code is more fun to hack q test-first programming gives instant feedback q that's instant positive feedback, most of the time
  14. 14. Perl makes it easy q ExtUtils::MakeMaker and the 'test' target q Test::* modules hide the Test::Harness expectations q if lazy dopes like us can do it, you can too!
  15. 15. Basic Tests
  16. 16. Thinking "ok" print "ok"; print "not ok"; q basis for all tests q a simple boolean condition q generally numbered sequentially
  17. 17. More thinking "ok" q why printing is a loss: x same arguments as using subroutines (duplicate code) x manual test renumbering x no diagnostics unless you provide them Ë what matched Ë what didn't match Ë which test failed and where x no built-in provision for more complex constructs x can be hard to maintain
  18. 18. The ok() function ok( 1 == 1 ); q basis for all Test::Simple, Test::More, Test::Builder tests q a simple boolean condition q why it's a win: x alleviates duplicate code x associates a name with a test x expresses test intent more clearly x avoids portability concerns
  19. 19. Test::Simple and its API
  20. 20. planning tests q helps catch too many/not enough test errors q helps identify which test fails q required by Test::Harness use Test::Simple tests => 10; use Test::Simple 'no_plan';
  21. 21. tests => n q produces 1..n output for Test::Harness q should be set when the test suite is complete
  22. 22. no_plan q produces 1..actual output at the end of the test q when you don't know how many tests will run q very handy while developing a module q not so handy when distributing a module
  23. 23. ok() q scalar context of operands q first argument always true or false q second argument optional, test name q condition is code expression of the test name q test name is a declaration of the intent of the condition
  24. 24. ok() passing ok( 1, '1 should be true' ); ok 1 - 1 should be true
  25. 25. ok() failing ok( 0, '0 should be true' ); not ok 2 - 0 should be true # Failed test (demotest.pl at line 17)
  26. 26. success is not a gray area (individual tests) q pass, or fail, there is no maybe q test the smallest piece possible q test one thing at a time q pass quietly x avoid errors x avoid warnings x test for errors and warnings if you must
  27. 27. success is all or nothing (test suite) q stamp out expected failures x undermine confidence in the code x clutter up screens and logs x hide new problems q keeping changes small helps localize damage q keeping tests working helps localize damage q there are ways to isolate platform-specific tests
  28. 28. Integrating testing with the development process
  29. 29. catching regressions q most bugs aren't new, they happen again and again q why let them reoccur? q write a test for every new bug
  30. 30. testing first q teeny, tiny, mini-iterations q break each task into boolean expressions q "What feature do I need next?" x smallest element of overall task x one small step at a time
  31. 31. The two test-first questions q "How can I prove that this feature works?" x the simplest test that will fail unless the feature works x THE TEST MUST FAIL q "What is the least amount of code I can write to pass the test?" x the simpler the test, the simpler the code you need x THE TEST MUST NOW PASS q this produces known good code and a comprehensive test suite q be sure to run the entire test suite after you implement a task
  32. 32. writing testable code q the smaller the unit, the better q the simpler the unit, the better q the better the test suite, the easier it is to refactor q writing your tests first promotes this!
  33. 33. distributing tests q Test.pm is easier to distribute x single module, does not depend on Test::Builder x compatible with Test::More API q Test::Simple suite can be bundled with module x CGI.pm does this, among others q Test::Simple ships with 5.8 q a module can depend on Test::Simple
  34. 34. Exploring the Test::More API
  35. 35. graduating from Test::Simple to Test::More, or why ok() is not enough q one bit of data -- either it passed or it failed q contains little debugging information x what did you get? x what did you receive? x do you really want to exchange "add a print at line 43" emails? q true laziness is a sign of good testing
  36. 36. matching things - is() q asks the question "are these two things equal?" q does a pretty good job of guessing at the type of equality q takes an optional test name, so use one q special cased for 'undef'
  37. 37. is() passing is( 2 + 2, 4, 'Two plus two should be four' ); ok 1 - Two plus two should be four
  38. 38. is() failing is( 2 + 2, 5, 'Two plus two should be five' ); not ok 2 - Two plus two should be five # Failed test (demotest.pl at line 8) # got: '4' # expected: '5'
  39. 39. matching things - isnt() q asks the question "are these two things not equal?" q essentially the opposite of is()
  40. 40. isnt() passing isnt( 2 + 2, 5, 'Two plus two should not be five' ); ok 3 - Two plus two should not be five
  41. 41. isnt() failing isnt( 2 + 2, 4, 'Two plus two should not be four' ); not ok 4 - Two plus two should not be four # Failed test (demotest.pl at line 10) # '4' # ne # '4' 1..4
  42. 42. matching things - like() q applies a regular expression to a string q first argument is the string q second argument is a regular expression x compiled with qr//, where Perl version supports that x a string that looks like a regex '/foo/', otherwise
  43. 43. like() passing like( 'foobar', qr/fo+/, '+ quantifier should match one or more' ); ok 5 - + quantifier should match one or more
  44. 44. like() failing like( 'fbar', qr/fo+/, '+ quantifier should match zero or more' ); not ok 6 - + quantifier should match zero or more # Failed test (demotest.pl at line 12) # 'fbar' # doesn't match '(?-xism:fo+)'
  45. 45. matching things - unlike() q essentially the opposite of like() q the same rules apply
  46. 46. unlike() passing unlike( 'foobar', qr/baz/, 'foobar should not match baz' ); ok 7 - foobar should not match baz
  47. 47. unlike() failing unlike( 'foobar', qr/bar/, 'foobar should not match bar' ); not ok 8 - foobar should not match bar # Failed test (demotest.pl at line 14) # 'foobar' # matches '(?-xism:bar)' 1..8
  48. 48. exploring behavior - use_ok() q (these tests all provide their own test names) q see if a module compiles and can be import()ed q load a module and import() it q activates at runtime, not compile time! q provides a default test name so you don't have to q allows optional imports to be passed
  49. 49. use_ok() passing use_ok( 'Test::More' ); ok 11 - use Test::More;
  50. 50. use_ok() failing use_ok( 'No::Module' ); not ok 12 - use No::Module; # Failed test (demotest.pl at line 20) # Tried to use 'No::Module'. # Error: Can't locate No/Module.pm in @INC (@INC # contains: /usr/lib/perl5/5.6.1/i386-linux # /usr/lib/perl5/5.6.1 # /usr/lib/perl5/site_perl/5.6.1/i386-linux # /usr/lib/perl5/site_perl/5.6.1 # /usr/lib/perl5/site_perl .) at (eval 6) line 2.
  51. 51. exploring behavior - require_ok() q see if a module compiles q load a module, but do not import() it q also provides a default test name
  52. 52. require_ok() passing require_ok( 'Test::Simple' ); ok 13 - require Test::Simple;
  53. 53. require_ok() failing require_ok( 'No::Module' ); not ok 14 - require No::Module; # Failed test (demotest.pl at line 23) # Tried to require 'No::Module'. # Error: Can't locate No/Module.pm in @INC (@INC # contains: /usr/lib/perl5/5.6.1/i386-linux # /usr/lib/perl5/5.6.1 # /usr/lib/perl5/site_perl/5.6.1/i386-linux # /usr/lib/perl5/site_perl/5.6.1 # /usr/lib/perl5/site_perl .) at (eval 8) line 2.
  54. 54. exploring behavior - isa_ok() q see if an object is a type of reference q can be an object x but not a class x it exists to save you from having to make three extra constructor tests Ë did it return something defined? Ë is that something a reference? Ë is that reference an object? q respects inheritance q can be a reference type (scalar, array, hash, io, code)
  55. 55. isa_ok() passing isa_ok( [], 'ARRAY' ); ok 15 - The object isa ARRAY
  56. 56. isa_ok() failing isa_ok( {}, 'IO::Socket' ); not ok 16 - The object isa IO::Socket # Failed test (demotest.pl at line 26) # The object isn't a 'IO::Socket' it's a 'HASH'
  57. 57. exploring behavior - can_ok() q see if an object has named behavior(s) q can be an object or a class q pass several function or method names to test them all in a single test q loop over a can_ok() call to test them all in multiple tests q respects inheritance, but falls afoul of AUTOLOAD() just like UNIVERSAL::can() does
  58. 58. can_ok() passing can_ok( 'Test::More', 'can_ok' ); ok 17 - Test::More->can('can_ok')
  59. 59. can_ok() failing can_ok( 'Test::More', 'autotest' ); not ok 18 - Test::More->can('autotest') # Failed test (demotest.pl at line 29) # Test::More->can('autotest') failed
  60. 60. controlling things - skip() q skips a specified number of tests q generally used for things that will NEVER pass q takes the reason to skip the tests and the number to skip q should be in a labelled SKIP block q should have a skip condition
  61. 61. skip() in action SKIP: { skip( 'Never on a Sunday', 2 ) if (localtime)[6] == 0; can_ok( 'Perl::Porter', 'buy_beer' ); can_ok( 'Perl::Porter', 'wear_short_pants' ); } ok 21 # skip Never on a Sunday ok 22 # skip Never on a Sunday
  62. 62. controlling things - todo() q marks a specified number of tests as known failures q generally used for failures x use sparingly! x use for features that aren't quite finished x use for bugs that aren't quite fixed x don't let these add up: they undermine the suite q should be in a labelled TODO block q must localize the $TODO variable with the name
  63. 63. todo() in action TODO: { local $TODO = 'The alien overlords have not arrived'; is( $smallpox->status(), 'ready', 'Smallpox ready' ); ok( @cow_decoys > 1000, 'Bait ready...' ); } not ok 23 - Smallpox ready # TODO The alien overlords have not arrived # Failed (TODO) test (demotest.pl at line 46) # got: 'replenishing' # expected: 'ready' not ok 24 - Bait ready... # TODO The alien overlords have not arrived # Failed (TODO) test (demotest.pl at line 47)
  64. 64. controlling things - diag() q display a (list of) diagnostic message(s) q guaranteed not to interfere with the test harness q gives no test number, pass, or fail q useful for giving suggestions on tricky tests
  65. 65. diag() in action diag( "Now that you know a bit about test writing,n", "you have no excuse not to test." ); # Now that you know a bit about test writing, # you have no excuse not to test.
  66. 66. controlling things - pass() q make a test pass, unconditionally q takes a test name -- that's it!
  67. 67. pass() in action pass( 'Executive fiat should work' ); ok 19 - Executive fiat should work
  68. 68. controlling things - fail() q makes a test fail, unconditionally q also takes a test name, and that's it
  69. 69. fail() in action fail( '... but we live in a meritocracy' ); not ok 20 - ... but we live in a meritocracy # Failed test (demotest.pl at line 32)
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×