Warming up to Modular Testing (YAPC::NA::2009)

Warming up to modular testing
YAPC::NA::2009 June 23rd
Brad Oaks of Plus Three, LP

Who is this guy
First time YAPC speaker.
Not an expert on the subject.
Learned from an existing code base.

What we'll cover
The parts you'll need to move from a t/ directory full of flat .t files to a more modular setup.
A simple harness
An example of putting helper methods in their own module for reuse.
How Test::Class and friends ease inheritance in your tests to parallel inheritance in your main code.

The Harness

Harness to run the show
Runs the tests
Collects TAP formatted output from the tests.
We'll see a simplified example.

./bin/micro_test
#!/usr/bin/env perl use strict; use warnings; use TAP::Harness; my @files = glob("t/pod.t");
my $harness = TAP::Harness->new();
$harness->runtests(@files);

./bin/micro_test
#!/usr/bin/env perl use strict; use warnings; use TAP::Harness; my @files = glob("t/pod.t");
my $harness = TAP::Harness->new(
{verbosity => 1, merge => 0}
); $harness->runtests(@files);

BEGIN {
# Find a ARCOS_ROOT based on path to harness
my @dir = splitdir(canonpath($RealBin));
$ENV{ARCOS_ROOT} ||=
catdir(@dir[0 .. $#dir - 1]);
}

BEGIN {
# use $ARCOS_ROOT/lib for modules
my $lib = catdir($ENV{ARCOS_ROOT}, "lib");
$ENV{PERL5LIB} =
$ENV{PERL5LIB}
? "$ENV{PERL5LIB}:${lib}"
: "${lib}";
unshift @INC, $lib, "$lib/" . $Config{archname};
}

BEGIN {
# use $ARCOS_ROOT/lib for modules
my $lib = catdir($ENV{ARCOS_ROOT}, "lib");
$ENV{PERL5LIB} =
$ENV{PERL5LIB}
? "$ENV{PERL5LIB}:${lib}"
: "${lib}";
unshift @INC, $lib, "$lib/" . $Config{archname};
eval { require Arcos };
die “cannot find Arcos” if $@
}

BEGIN {
# choose the first instance if not set.
unless ($ENV{ARCOS_INSTANCE}) {
require Arcos::Conf;
$ENV{ARCOS_INSTANCE} =
(Arcos::Conf->instances())[0];
}
}

BEGIN {
# choose the first instance if not set.
unless ($ENV{ARCOS_INSTANCE}) {
require Arcos::Conf;
$ENV{ARCOS_INSTANCE} =
(Arcos::Conf->instances())[0];
}
}
use Arcos;
use Arcos::Script;

use Arcos;
use Arcos::Script;
our ($help, $man, $archive);
my $v = 0;
my $files;
GetOptions( 'help' => $help,
'man' => $man,
'verbose' => $v,
'make_verbose=i' => $v,
'files=s' => $files,
'archive' => $archive,
);

TAP::Harness can
try to emit results with color
be verbose or varying levels of quiet
run multiple test jobs at once (aggregate_tests)
see "perldoc TAP::Harness"
Test::Harness is for backwards compatibility.
For new work, you should be starting with TAP::Harness.

Common test functions

Pull commonly used functions out into their own module.
Don't need inheritance for this.
contains:
Convenience methods.
Affecting the environment.
Checking the environment.
Arcos::Test

can_reach_url() - uses LWP::UserAgent and checks response.
js_escape() - when searching for strings in HTML that has already been escaped.
get_server_uri() - composes url from host and port in Arcos::Conf
Arcos::Test Convenience Methods

start/stop daemons
apache
Arcos queue
test SMTP server
These are basically system() calls to ctl scripts
Arcos::Test Affecting the Environment

spread_is_running() -- used in t/island_mode.t
SKIP: {
skip('Spread is not running', 16) unless
Arcos::Test->spread_is_running();
# 16 tests is, isa, ok, like, etc.
}
Arcos::Test Checking the Environment

spread_is_running() -- used in t/island_mode.t
SKIP: {
skip('Spread is not running', 16) unless
Arcos::Test->spread_is_running();
my $island_pp = catfile(ArcosRoot, 'bin',
'arcos_island_postprocessor');
my $result = `$island_pp 2>&1`;
like($result, qr/Island Mode Post-Processing Complete/);
ok($result =~ /Submitted job (d+) : (S+) from (S+)./);
my ($job_id, $amount, $email) = ($1,$2,$3);
. . .
}

apache_is_running() -- used in t/open_handler.t
BEGIN {
if (Arcos::Test->apache_is_running) {
Test::More::plan('no_plan');
} else {
Test::More::plan(skip_all => "Arcos Apache is not running skipping all tests.");
}
}

Test::Class

The methods with Test attribute will be run.
Test::Class Method Attributes Test(4)

The Test attribute specifies the number of tests for this method.

The order they're run is determined alphabetically by name.

sub b_register : Test(15) { . . . }
sub d_short_pass : Test(4) { . . . }
sub e_mismatched_pass : Test(4) { . . . }
sub f_invalid_email : Test(4) { . . . }
sub h_duplicate_email : Test(4) { . . . }
sub h_duplicate_username : Test(4) { . . . }

sub b_register : Test(15) { . . . }
sub d_short_pass : Test(4) { . . . }
sub e_mismatched_pass : Test(4) { . . . }
sub f_invalid_email : Test(4) { . . . }
sub h_duplicate_email : Test(4) { . . . }
sub h_duplicate_username : Test(4) { . . . }
sub b_submit_empty_form : Test(no_plan) { . . . }
You can specify no_plan for a specific sub.
Can localize the uncertainty instead of using no_plan for the entire module.
Test::Class Method Attributes Test(no_plan)

startup / shutdown
run before /after the whole test class
Test::Class Method Attributes Test(startup)

startup / shutdown
setup / teardown
run before / after every test method

startup / shutdown
setup / teardown
sub tc_startup : Test(startup) { my $self = shift; $self->{td} = Arcos::TestData->new;
}

startup / shutdown
setup / teardown
}
sub tc_shutdown : Test(shutdown) { my $self = shift; $self->{td}->cleanup(ignore_deleted => 1);
}

startup can have tests that count towards the plan
sub start : Test(startup => 1) { my $self = shift;
$self->{mail_tmp_dir} =
Arcos::Test->start_test_smtp_server;
use_ok('Arcos::JobQueue::Handler::CommunityInvitation');
}

startup can have tests that count towards the plan.
sub start : Test(startup => 1) { my $self = shift;
$self->{mail_tmp_dir} =
Arcos::Test->start_test_smtp_server;
use_ok('Arcos::JobQueue::Handler::CommunityInvitation');
}
startup can also be specified on more than one method.

Inheritance in your tests

A parent class for many of our tests.
Arcos::Test::Class

Inherit from Test::Class
use base 'Test::Class';
Arcos::Test::Class

Pull in some helper modules.
use base 'Test::Class'; use Arcos::Test; use Arcos::TestData; use Test::Builder qw(ok is_eq); use Test::LongString;
Arcos::Test::Class

Store an Arcos::TestData reference.
}
Arcos::Test::Class

Call cleanup method.
} sub tc_shutdown : Test(shutdown) { my $self = shift; $self->{td}->cleanup(ignore_deleted => 1);
}
Arcos::Test::Class

Arcos::Test::Class
We also provide convenience methods at this level:

contains_url()
lacks_url()
json_contains()
json_lacks()
json_success()
json_failure()
input_value_is()

Arcos::Test::Class
help Test::Builder give the right line number for failures
sub json_success {
my $self = shift;
{
local $Test::Builder::Level =
$Test::Builder::Level + 1;
$self->json_contains('success');
}
}
We want the error to show up where json_success() is called, not way down here in the helper module.

Inherited Forms, with Inherited Tests
Arcos::Form::Test is the parent of
Arcos::Form:: ContactOfficial ::Test
Arcos::Form:: Register ::Test

They each have
use base 'Arcos::Form::Test';

Arcos::Form::Test the parent
Startup and Shutdown
sub tc_startup : Test(startup) { . . . } sub tc_shutdown : Test(shutdown) { . . . } sub x_create_form : Test(startup => 3 ){ . . . }
Intended to be overridden
sub additional_elements { '' } sub form_db_class { '' }
Shared methods for these forms sub mech_shows_custom_fields { . . . }
sub person_has_custom_fields { . . . }

An example of inherited tests
tc_startup can be completely overridden
or augment by calling SUPER

sub tc_startup : Test(startup) {
$self->{td} = Arcos::TestData->new();
return $self->SUPER::tc_startup;
}

Inheritance Example

We still have .t files t/register_form.t
use strict; use warnings; use Arcos::Test; use Arcos::Test::Script; use Arcos::Form::Register::Test; Arcos::Test->needs_running_krang(); Arcos::Test->needs_running_apache(); Arcos::Form::Register::Test->new->runtests;

We still have .t files t/contact_official_form.t
ContactOfficial forms need a little more at startup.
use Arcos::GeoCoder; my $coder = Arcos::GeoCoder->new; if (!$coder->available) { Test::More::plan(skip_all =>
'geocoding databases not available.');
exit;
}

exit;
} unless (Arcos::Test->stop_queue()) { Test::More::plan(skip_all =>
'Could not stop Queue daemon in time.');
exit;
}

exit;
exit;
} Arcos::Form::ContactOfficial::Test->new->runtests;

ContactOfficial forms need a little more more at startup.
exit;
exit;
} Arcos::Form::ContactOfficial::Test->new->runtests;
Arcos::Test->restart_queue;

return to skip
Section of a method not yet applicable
return 'The default templates do not currently have security code; skipping for now.';
Whole method not yet implemented
return 'The deleted form test is not implemented for this form type yet.';

Fail safe check
return 'You have no test payment account.'
unless $self->test_account;

plan to skip
use Arcos::Conf qw(ContributionTestMode);
if (!ContributionTestMode) {
plan(skip_all => 'contributions not in test mode');
} elsif (!Arcos::Test->can_reach_url(
'https://www.vancodev.com:443/cgi-bin/wstest.vps')) {
plan(skip_all => "Can't reach https://www.vancodev.com:443/cgi-bin/wstest.vps");
} else { Arcos::ContributionForm::Test::Vanco->runtests(); }

TEST_METHOD
TEST_METHOD=a_good_file
bin/arcos_test --files=t/admin-listupload.t
TEST_METHOD=e_failed_contributions
bin/arcos_test --files=t/report-contribution.t
TEST_METHOD=k_listing_checkbox
bin/arcos_test --files=t/admin-mailingmessage.t

TEST_METHOD
TEST_METHOD=a_good_file
bin/arcos_test --files=t/admin-listupload.t
TEST_METHOD=e_failed_contributions
bin/arcos_test --files=t/report-contribution.t
TEST_METHOD=k_listing_checkbox
bin/arcos_test --files=t/admin-mailingmessage.t
TEST_METHOD='(a|b)_.*'
bin/arcos_test --files=t/warehouse-loader.t

a lot of reuse
use base 'Arcos::Report::SiteEvent::Test';
Arcos::Report::Volunteer::Test
Arcos::Report::Contribution::Test
Arcos::Report::ContactOfficial::Test
Arcos::Report::Petition::Test
Arcos::Report::Tellafriend::Test
Arcos::Report::Subscription::Test
Arcos::Report::RadioCall::Test
Arcos::Report::LTE::Test
Arcos::Report::ContactUs::Test

Bonus slides

TODO tests
sub live_test : Test {
local $TODO =
"live currently unimplemented";
ok(Object->live, "object live");
};
Skip is so darn easy we haven't used TODO like this.

pausing to look around
warn 'pausing to look around;
hit return to continue:';
my $PAUSE = <STDIN>;

This will keep the test from cleaning up before you've had a chance to look around.
Add a few warns of URL values and you can bring up the reports in a browser to get a better idea of what's going on.
Litter this within a specific method, or in your method with the Test(shutdown) attribute.

calling plan from Arcos::Test
I added a needs_run_daily method to bail out gracefully.
if ($ENV{'RUN_DAILY_TESTS'}) { if (
$ENV{'OVERRIDE_DAILY_TESTS'}
or $class->_needs_daily_run()
) { $class->_update_run_daily_timestamp(); Test::More::plan(@args); }
} else { Test::More::plan( skip_all => 'RUN_DAILY_TESTS environment variable not set; skipping daily tests.'); }

Additional Resources

Additional Resources
#perl-qa on the irc.perl.org server
http://search.cpan.org/~adie/Test-Class-0.31/lib/Test/Class.pm
http://search.cpan.org/~mschwern/Test-Simple-0.88/lib/Test/Builder.pm
Organizing Test Suites with Test::Class
http://www.modernperlbooks.com/mt/2009/03/organizing-test-suites-with-testclass.html

Thank you for your time!

Warming up to Modular Testing (YAPC::NA::2009)

by bradoaks on Jun 27, 2009

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 6

Statistics

Warming up to Modular Testing (YAPC::NA::2009) — Presentation Transcript