Short introduction to the basics of Perl testing and some resources for further reading. Includes basics of Test::Simple, Test::More, using Inline for multi-language testing, testing coverage, testing tests.

1. Getting Testy With Perl
Steven Lembark
Workhorse Computing
lembark@wrkhors.com

2. Cultured Perl
Perl's “test culture” is part of the language.
Test::* modules easily added to Perl code.
Smoke testing provided as part of CPAN.
Reporter accumulates results from installers.
Language structure, modules work together.

3. Where there's smoke, there's Perl.
CPAN::Reporter send back success/fail.
CPAN testers: variety of platforms, daily reports.
cpan-testers & cpan-testers-discuss mailing lists.
http://cpantesters.org/
Community effort at testing is unique to Perl.

4. Excellent Reference
Perl Testing: A developer's Notebook,
O'Reilly Press
Wonderful example of a good howto-book.
Includes modules, ways to use them with variations.
Good cover-to-cover read.

5. Perl is easy to test
Test::* modules do most of the work.
Use the wheels, don't re-invent:
Test::Simple, Test::More, Test::Deep,
Test::Builder, Test::Coverage, Object::Exercise
Perl adds introspection.

6. One­stop shopping
Perl as a great glue language.
Use perl to test other programs.
Have other programs output TAP.
Combined with “inline” you can test almost anything!

7. Test your tests!
Devel::Coverage have you tested all of the code?
Test::Builder roll your own tests.
Build you own re-usable test components:
Test::Harness::Straps
Test::Builder::Tester

8. “Test” code can do more
Pre-load fixed data into a database.
Validate the working environment.
Execute daily cleanups in cron.

9. Testing simply
Count hardwired:
use Test::Simple tests => 6;
Difference in count will be an error.

10. Its all “ok”
use Test::Simple tests => 2;
# ok( boolean, message );
ok 1 == 1, “Count: '1' (1)”;
ok 2 == 1, “Count: '2' (1)”;
Output like:
ok 1 ­ Count: '1' (1)
not ok 2 ­ Count: '2' (1)

11. Its all “ok”
use Test::Simple tests => 2;
# ok( boolean, message );
ok 1, 'It passed!';
Output like:
1..2
ok 1 ­ It passed!
# Looks like you planned 2 tests but ran 1.

12. Seen these before?
“not ok 10 ­ Count”
“not ok 11 ­ Format”
“not ok 5 ­ File or directory not found.”
OK, now what?

13. Designing tests:
Test what matters.
Report something useful.

14. Isolate the failure
What to fix:
ok -e $path , “Existing: '$path'”;
ok -d _ , “Directory: '$path'”;
ok -r _ , “Readable: '$path'”;
ok -w _ , “Writeable '$path'”;

15. Q: Why all of the quotes?
ok -e $path, “Existing dir: $path”

16. Q: Why all of the quotes?
ok -e $path, “Existing dir: $path”
not ok 1 - Existing dir: /var/tmp
not ok 1 Existing dir: /var/tmp

17. Q: Why all of the quotes?
ok -e $path, “Existing dir: '$path'”
not ok 1 - Existing dir: '/var/tmp '
not ok 1 Existing dir: '/var/tmp
'

18. Good general format
Include what you found, what you expect.
You'll need them both with “not ok”:
ok $got eq $want, 'Name: '$got' ($want)”;
not ok 99 – Name: '' (Jow Bloe)
not ok 99 – Name: '1 Main St.' (Jow Bloe)
not ok 99 – Name: 'Jow Bloe' ()
not ok 99 – Name: 'Jow Bloe' (User::Name=0x1A...

19. Test::Simple may be enough
Generate a plan.
“ok” generates TAP output.
Test count is fixed.
This works for many day-to-day tests.

20. But Wait! There's More!
Test::More
plan, done_testing test counts
is, isnt, like, unlike stringy “ok”
pass, fail skip the boolean
use_ok, require_ok module test
note, diag, explain messages

21. Flexable count
Set count with “plan”.
Compute from test input:
plan tests => 42;
plan tests => 3 * keys $config;
plan tests => 2 * values $query_output;
Test count varys?
Skip plan.

22. When you're done testing
Just say so:
done_testing;
Generates final report.
Plan count is checked here.
No plan, no count check.

23. Notes show up with “prove ­v”
note “read_config from '$config_path'”;
my $path = read_config 'tmp_dir';
ok -e $path, “Exists: '$path' (tmp_dir)”;
$ prove ­v;
# read_config from './etc/pass1.conf'
not ok 1 – Exists: '/var/tmp/foobar ' (tmp_dir)

24. Diagnostics show why tests failed
Show up without “-v”.
ok … or diag “Oopsie...”;
ok grep /blah/, @test_input, 'Found blah'
or diag “Missing 'blah':”, @test_input;
ok $dbh, “Connected”
or diag “Failed connect: '$dsn'”;

25. Explain shows exactly what failed
“explain” shows nested structure.
Use with “note” to show setup details.
With “diag” shows extra info on failure.
my $dbh = DBI->connect( @argz );
ok $dbh, 'Database connected'
or diag 'Connect args:', explain @argz;

26. Stringy “ok”
like, not_like use regexen.
saves “=~” syntax in the tests:
like $stuff, qr/milk/, “Got: 'milk'?”
or diag “Have: '$stuff' instead”;
not ok 1 ­ Got: 'milk'?
# Have 'very new cheese' instead

27. Variable number of tests
If-logic generates variable number of tests.
Skip “plan”, use “done_testing”.

28. Taking it pass/fail
if( $obj = eval { $class->constructify } )
{
# validate object contents
}
else
{
# nothing more to test
fail “Failed constructify: $@”;
}

29. Test expected errors
eval
{
$obj->new( $junk );
fail "No exception: foo( $junk )?";
1
}
or do
{
# validate error handling
...
}

30. Controlling test cycle
Abort if everything will fail.
Avoid expensive, specialized, un-necesary tests.
Saves extraneous code in all of the tests.

31. BAIL_OUT: Knowing when to give up
Aborts all testing.
Unsafe or guaranteed failure.
Useful in early test for environment, arg checks.

32. BAIL_OUT
BAIL_OUT 'Do not run tests as su!' unless $>;
BAIL_OUT 'Missing $DB' unless $ENV{DB};
-e $test_dir or mkdir $dir, 0777
or BAIL_OUT “Failed mkdir: '$dir', $!”;
-d $test_dir or BAIL_OUT “Missing: '$dir'”;
-r _ or BAIL_OUT “Un-readable: '$dir'”;
-w _ or BAIL_OUT “Un-writeable: '$dir'”;

33. “SKIP” blocks
Skip a block with test count and message.
Adjust the plan test count.
Expensive or specialized tests.

34. Skip external tests
SKIP:
{
skip “No database (TEST_DB)”, 8 if ! $ENV{ TEST_DB };
# or … no network available...
# or … no server handle...
...
}

35. Skip expensive tests
SKIP:
{
skip “Used for internal development only”, 12
unless $ENV{ EXPENSIVE_TESTS };
# test plan count reduced by 12
...
}

36. Skip dangerous tests
SKIP:
{
skip “Unsafe as superuser”, 22 unless $>;
# at this point the tests are not running su.
...
}

37. You'll usually use Test::More
note & diag nearly always worth using.
plan & done_testing makes life easier.
Still have “ok” for the majority of work.

38. Testing Structures
“like” syntax with nested structure:
use Test::Deep;
cmp_deeply $found, $expect, $message;
Great for testing parser or grammar outcome.

39. Devel::Cover: what didn't you check?
All of the else blocks?
All of the “||=” assignments?
All of the “... or die ...” branches?
Devel::Cover bookkeeps running code.
Reports what you didn't test.
Tells you what test to write next.

40. Similar to NYTProf:
Run your program & summarize the results:
$ cover -test;
or
$ perl -MDevel::Cover ./t/foo.t;
$ cover;
Running “cover” generates the report.

41. Introspection simplifies testing
Want to test database connect failure.
Can't assume SQL::Lite.
Flat file databases are messy.
Guarantee that something fails?
You could write an operation that should fail.
Then again, it might not...

42. Making sure you fail
Override class methods.
Override Perl functions.
Result:
Mock objects,
Mock methods,
Mock perl.

43. Mock Objects
Test your wrapper handling failure?
Override DBI::connect with sub { die … }.
No guess: You know it's going to fail.

44. Mock Modules
Your test:
*DBI::connect = sub { die '…' };
my $status = eval { $obj->make_connection };
my $err = $@;
# test $status, $err, $obj...

45. Force an exception
use Symbol qw( qualify_to_ref );
# e.g., force_exception 'Invalid username', 'connect', 'DBI';
# ( name, packcage ) or ( “package::name” )
sub force_exception
{
chomp( my $messsage = shift );
my $ref = qualify_to_ref @_;
undef &{ *$ref };
*{ $ref } = sub { die “$messagen” };
return
}

46. Testing multiple failures
for( @testz )
{
my( $msg, $expect ) = @$_;
force_exception $msg, 'DBI::connect';
my $status = eval { $wrapper->connect };
my $err = $@;
# your tests here
}

47. Avoid caller cleanup
Override with “local”
sub force_exception
{
my ( $proto, $method, $pkg, $name, $msg ) = splice @_, 0, 5;
my $ref = qualify_to_ref $name, $pkg;
local *{ $ref } = sub { die “$msg” };
# exit from block cleans up local override.
# caller checks return, $@, $proto.
eval { $proto->$method( @_ ) }
}

48. Mock Perl
Override perl: *Core::Global::<name>
Is “exit” is called?
my $exits = 0;
*Core::Global::exit = sub { $exits = 1 };
eval { frobnicate $arg };
ok $exits, “Frobnicate exits ($exits)”;

49. Devel::Cover & Mocking
The biggest reason for mock anything:
Force an outcome to test a branch.
Iterate:
Test with Devel::Cover.
See what still needs testing.
Mock object/method/function forces the branch.

50. Automated testing
Lazyness is a virtue.
Avoid cut+paste.
Let Perl do the work!

51. Example: Sanity check modules
Use symlinks.
Validate modules compile.
Check package argument.
require_ok $package; # ok if require-able.
use_ok $package; # ok if use-able.

52. Make the links
Path below ./lib.
Replace slashes with dashes.
Add a leading “01”.
Symlink them all to a generic baseline test.
Symlinks look like:
01­Wcurve­Util.t generic­01­t→

53. use FindBin::libs;
use Test::More;
use File::Basename qw( basename );
my $madness = basename $0, '.t'; # 01-WCurve-Util
$madness =~ s{^ 01-}{}x; # WCurve-Util
$madness =~ s{ W+ }{::}gx; # WCUrve::Util
if( use_ok $madness )
{
# check for correct package argument.
ok $madness->can( 'VERSION' ), “$maddness has a method”;
ok $a = $madness->VERSION, “$madness is '$a'”;
}
done_testing;
./t/generic­01­t

54. ./t/make­generic­links
#!/bin/bash
cd $(dirname $0);
rm 01-*.t;
find ../lib/ -name '*.pm' |
perl -n
-e 'chomp;'
-e 's{^ .+ /lib/}{}x'
-e 's{.pm $}{.t}x'
-e 'tr{/}{-}
-e 'symlink “01-generic-t” => “01-$_”';
exit 0;

55. Similar checks for config files
Add “00-” tests to reads config files.
Combine “ok” with Config::Any.
Now make-generic-links tests config & code.
“prove” runs them all in one pass.

56. The Validation Two­Step
Use with git for sanity check:
git pull &&
./t/make-generic-links &&
prove --jobs=2 –-state=save &&
git tag -a “prove/$date” &&
git push &&
git push –-tag ;

57. Exercise for healthy objects
Data-driven testing for a single object.
Data includes method, data, expected return.
Module iterates the tests, reports results.
When tests or expected values are generated.

58. use Object::Exercise;
my @testz =
(
[
[ qw( name ) ], # method + args
[ q{Jow Bloe} ], # expected return
],
[
[ qw( street ) ],
[ q{1 Nuwere St} ],
],
[
[ qw( street number ) ], # $obj->address( 'number' )
[ qw( 1 ) ],
'Check street number' # hardwired message
],
);
Person->new( 'Jow Blow' => '1 Nuwere St' )->$exercise( @testz );

59. Load fixed data
Flat file -> arrayrefs
“insert” as method, [1] as return.
Load the data with:
$sth->$exercise( @data )
Get “ok” message for each of the data record.

60. Roll your own: Test::Builder
Mirrors Test::More with a singleton object.
my $test = Test::Builder->new;
$test->ok( $boolean, $success_message )
or $test->diag( $more_info );
Spread single test across multiple modules.

61. Testing Testing
Test::Build::Tester wraps your tests.
Forces test to return not-ok in order to test it.
Ignores the output of the test being tested.
Validate the test plan.

62. Getting the most out of prove
Save state: re-run failed tests:
$prove –state=save;
# fix something...
$prove –state=save,failed;
Parallel execution with “--jobs”.
“perldoc” is your friend!

63. Great, but my shop uses <pick one>
Test multiple languages with Inline.
Include C, C++, Python...
Multiple Perl versions.
Mix languages to test interfaces.

64. One­stop teting for your API lib's
Need to test multi-language support?
Use Perl to move data between them.
Unit-test C talking to Java.

65. Example: Testing your C code
use Inline C;
use Test::Simple tests => 1;
my $a = 12 + 34;
my $b = add( 12, 34 );
ok $a == $b, "Add: '$b' ($a)";
__END__
__C__
int add(int x, int y)
{
return x + y;
}

66. Example: Testing your C code
$ prove ­v add.t
add.t ..
1..1
ok 1 ­ Add: '46' (46)
ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs
( 0.05 usr 0.00 sys + 0.09 cusr 0.00 csys = ...
Result: PASS

67. Example: Testing your C code
You can call into a library.
Write a whole program.
Inline builds the interface.

68. Test Anything Protcol
Inline Language Support Modules add languages.
Inline supports:
C, C++, Java, Python, Ruby, Tcl,
Assembler, Basic, Guile, Befunge, Octave,
Awk, BC, TT (Template Toolkit), WebChat,
and (of course) Perl.

69. Really: Any Thing
Say you had a quad-rotor UAV...
You'd write a Perl API for it, wouldn't you?
UAV::Pilot::ARDrone, for example.
But then you'd have to test it...

70. First: Define the test actions
my @actionz =
(
[
takeoff => 10
],
[
wave => 8000
],
[
flip_left => 5000
],
[
land => 5000,
send => $expect
],
);
plan tests => 2 + @actionz;
Basic set of actions:
Takeoff, wobble, flip,
and land.
Final “send” validates
end-of-channel.

71. Control interfaces
BAIL_OUT
avoids running
without controls.
my $driver
= UAV::Pilot::ARDrone::Driver->new
(
{
host => $HOST,
}
)
or BAIL_OUT "Failed construct";
eval
{
$driver->connect;
pass "Connected to '$HOST'";
}
or BAIL_OUT "Failed connect: $@";

72. Error handler for actions
Attempt the last
action (land) on
errors.
sub oops
{
eval { execute $actionz[ -1 ] }
or
BAIL_OUT "Literal crash expected";
}

73. sub execute
{
state $a = $event;
my ( $ctrl_op, $time, $cv_op, $cv_val ) = @$_;
$a = $a->add_timer
(
{
duration => $time,
duration_units => $event->UNITS_MILLISECOND,
cb =>
sub
{
$control->$ctrl_op or die "Failed: $ctrl_op";
$cv->$cv_op( $cv_val ) if $cv_op;
pass "Execute: '$ctrl_op' ($cv_op)";
}
}
)

74. Execute the actions
“note” here
describes what
to expect.
for( @actionz )
{
note "Prepare:n", explain $_;
eval { execute $_ }
or
oops;
}
$event->init_event_loop;

75. Check the end­of­channel
my $found = $cv->recv;
if( $found eq $expect )
{
pass "Recv completed '$found' ($expect)";
}
else
{
fail "Recv incomplete ($found), send emergency land";
oops;
}
done_testing;

76. Execute the tests
$ perl t/01-test-flight.t
1..6
# Connect to: '192.168.1.1'
(UAV::Pilot::ARDrone::Driver)
ok 1 - Connected to '192.168.1.1'
# U::P::EasyEvent has a socket
# Prepare:
# [
# 'takeoff',
# 10
# ]
# Prepare:
# [
# 'wave'

77. “pass” shows what happens
# Prepare:
# [
# 'land',
# 5000,
# 'send',
# '123'
# ]
ok 2 - Execute: 'takeoff' ()

78. “pass” shows what happens
# Prepare:
# [
# 'land',
# 5000,
# 'send',
# '123'
# ]
ok 2 - Execute: 'takeoff' ()
ok 3 - Execute: 'wave' ()

79. “pass” shows what happens
# Prepare:
# [
# 'land',
# 5000,
# 'send',
# '123'
# ]
ok 2 - Execute: 'takeoff' ()
ok 3 - Execute: 'wave' ()
ok 4 - Execute: 'flip_left' ()

80. “pass” shows what happens
# Prepare:
# [
# 'land',
# 5000,
# 'send',
# '123'
# ]
ok 2 - Execute: 'takeoff' ()
ok 3 - Execute: 'wave' ()
ok 4 - Execute: 'flip_left' ()
ok 5 - Execute: 'land' (send)

81. “pass” shows what happens
# Prepare:
# [
# 'land',
# 5000,
# 'send',
# '123'
# ]
ok 2 - Execute: 'takeoff' ()
ok 3 - Execute: 'wave' ()
ok 4 - Execute: 'flip_left' ()
ok 5 - Execute: 'land' (send)
ok 6 - Recv completed '123' (123)

82. Cultured Perls
The Developer's Notebook is a great resource.
POD for Test::More == wheels you don't re-invent.
POD for prove == make life faster, easier.
cpantesters.org: test results for modules by platform.
Stay cultured, be testy: use perl.