• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Warming up to Modular Testing (YAPC::NA::2009)
 

Warming up to Modular Testing (YAPC::NA::2009)

on

  • 2,459 views

This talk will help you get started arranging your tests into modules. We'll cover setting up a simple TAP::Harness to run your tests. Then we'll see how Plus Three has used Test::Class to divide up ...

This talk will help you get started arranging your tests into modules. We'll cover setting up a simple TAP::Harness to run your tests. Then we'll see how Plus Three has used Test::Class to divide up and reuse code in our test suite.

Separating tests out from a large .t file into modules and subroutines has helped me confirm more quickly that a code change has not introduced a regression. Developers save time by only running the relevant subset of tests before committing a change to the code or a change to the tests themselves.

I'll offer a few tips on checking preconditions in your testing environment (e.g. is a daemon running, is an external service url reachable) and
either bailing out gracefully or trying to remedy the situation.

You can ease into this modularization adventure. With Test::Class your shinier new tests can work right beside the venerable dustier ones letting you rework them as they need it.

Statistics

Views

Total Views
2,459
Views on SlideShare
2,453
Embed Views
6

Actions

Likes
0
Downloads
16
Comments
0

1 Embed 6

http://www.slideshare.net 6

Accessibility

Categories

Upload Details

Uploaded via as OpenOffice

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • grab a list of .t files then create a TAP::Harness object now have the harness run the tests
  • the harness object can take arguments in new(). can try with verbose -- ok for every passing test.   The merge option on new merges STDOUT and STDERR -- this is most useful when using TAP::Harness::Archive -- you know, for submitting to a Smolder server.
  • We do some setup in our harness before we get to running any test code. If ARCOS_ROOT isn't set, calculate it from harness path. We modify the PERL5LIB to add our application lib. Try loading Arcos module and bail if cannot.
  • We do some setup in our harness before we get to running any test code. If ARCOS_ROOT isn't set, calculate it from harness path. We modify the PERL5LIB to add our application lib. Try loading Arcos module and bail if cannot.
  • We do some setup in our harness before we get to running any test code. If ARCOS_ROOT isn't set, calculate it from harness path. We modify the PERL5LIB to add our application lib. Try loading Arcos module and bail if cannot.
  • Go ahead and add more BEGIN blocks Our code also needs pointed to an instance (any instance) within our configuration. Pick the first one if none is already set.
  • Arcos sets a version string. This Arcos Script will see if we're running as the right system user and become them with calls to setuid() and setgid() We do more here, but that's beyond the scope of this talk.
  • You've all seen Getopt::Long's GetOptions() We use it to pull in command line parameters to our harness. There are a handful more than are listed here database passwords a test_pollution flag
  • “ try” -- Colorized results depends on terminal support. if your tests can be run in parallel without interfering with one another, TAP::Harness can handle it.
  • Spread is only part of what island_mode.t exercises, so skip instead of skip_all
  • Here is a sample showing the individual test assertions you're used to seeing : ok, like, etc. We capture the stdout and stderr with a backticks execute and check the results. We use some of the data to pull a job from the database, retrieve the associated contribution, and check its contact information.
  • open_handler.t tests that we properly handle the image requests from end users opening emails. no_plan is still a plan. If apache is not running, none of this test is going to be useful. so we skip_all.
  • .
  • .
  • use base 'Test::Class'; json_success and json_failure are calls to json_contains('success') and json_lacks('success) respectively.
  • Here we can see the common definition of methods that will be shared by the children.   x_create_form has the Test(startup) attribute, so it will be run before the regular test methods.  The 3 parameter notes how many tests are within that method. The form_db_class() method is used in the x_create_form() call to setup the form before the tests begin.  Here it is empty string, the children must override with a proper value. ContactOfficial forms need some additional elements, but Register forms do not.
  • Because it is inherited, the tc_startup method can be completely overridden, or merely augmented if the call to SUPER is included.
  • This script performs a couple checks before calling runtests. These are common across many of our tests and are defined in Arcos::Test. needs_running_apache checks if apache is running and attempts to start it if not. If apache cannot be started, the test would skip with the following code: Test::More::plan(skip_all => 'Could not start Arcos Apache/mod_perl server');
  • We also stop our Arcos job queue daemon for this test. We want to be able to inspect jobs as they're inserted into the queue, before they're processed.
  • Now we restart the queue so that it functions normally for other tests.
  • While developing tests it may be convenient to go ahead and skip particular test methods.     Calls to return() will be interpreted as a skip.     A string can be passed to provide a reason for skipping the rest of that method.
  • In run scripts it's often best to just skip the whole module worth of tests if some conditions do not exist. ContributionTestMode is imported from Arcos::Conf Do add descriptive instructions to your messages, if you're doing something unusual in part of your tests.
  • This environment variable can save you time while developing your tests.   Its value restricts which of the subroutines with the Test attribute are run. The startup and shutdown subroutines are still going to be called. In a large file with many tests, this can help you iterate quickly when making and checking modifications to specific subroutines.
  • You can treat it as a string, but it is a regex and can perform more complicated matching. Here we're running any Test methods that either start with a_ or with b_ in their names. (possibly show how often I use it: “ history | grep TEST_METHOD”)
  • Many of our report modules have a common ancestor in Arcos::Report::SiteEvents These reports all provide Listing of forms, Downloading CSV data, It was only natural to use inheritance to gain some reuse in the testing code too.  
  • We don't do much with marking tests as TODO in our suite. Here is how you can.
  • Lots of our tests are interacting with the web interface of Arcos. Often it is nice to get a failing test to stop for a bit before cleaning up and removing all of the evidence during shutdown. I find myself adding this little fragment all the time. Put the pause code in your shutdown method if you want to be able to use it with any routine you're putting into TEST_METHOD. Examples: * a new column is added to a listing screen. * the test isn't including a timestamp into the form names, and is choking because the database hasn't been cleared between runs.
  • Our run scripts make use of some reused routines in Arcos::Test.  needs_run_daily is one of those cases. Be nice and put the instructions into the skip messages for running the conditional parts of your unusual tests.   The external vendor had a limitation on the number of requests they could receive before an over-limit error was returned. We didn't want these tests running on each developers laptops even once daily. We wanted these tests to run a maximum of once daily in our automated test environment.
  • My resource for learning this material was an existing codebase. Hopefully these other resources will be helpful to you.

Warming up to Modular Testing (YAPC::NA::2009) Warming up to Modular Testing (YAPC::NA::2009) Presentation Transcript

  • Warming up to  modular testing
      • YAPC::NA::2009 June 23rd
      • Brad Oaks of Plus Three, LP
  • Who is this guy
    • First time YAPC speaker.
    • Not an expert on the subject.
    • Learned from an existing code base.
  • What we'll cover
    • The parts you'll need to move from a t/ directory full of flat .t files to a more modular setup.
    • A simple harness
    • An example of putting helper methods in their own module for reuse.
    • How Test::Class and friends ease inheritance in your tests to parallel inheritance in your main code.
  • The Harness
  • Harness to run the show
    • Runs the tests
    • Collects TAP formatted output from the tests.
    • We'll see a simplified example.
  • ./bin/micro_test
    • #!/usr/bin/env perl use strict; use warnings; use TAP::Harness; my @files = glob("t/pod.t");
    • my $harness = TAP::Harness->new();
    • $harness->runtests(@files);
  • ./bin/micro_test
    • #!/usr/bin/env perl use strict; use warnings; use TAP::Harness; my @files = glob("t/pod.t");
    • my $harness = TAP::Harness->new(
    •     {verbosity => 1, merge => 0}
    • ); $harness->runtests(@files);
    • BEGIN {
    • # Find a ARCOS_ROOT based on path to harness
    • my @dir = splitdir(canonpath($RealBin));
    • $ENV{ARCOS_ROOT} ||=
    • catdir(@dir[0 .. $#dir - 1]);
    • }
    • BEGIN {
    • # Find a ARCOS_ROOT based on path to harness
    • my @dir = splitdir(canonpath($RealBin));
    • $ENV{ARCOS_ROOT} ||=
    • catdir(@dir[0 .. $#dir - 1]);
    • # use $ARCOS_ROOT/lib for modules
    • my $lib = catdir($ENV{ARCOS_ROOT}, "lib");
    • $ENV{PERL5LIB} =
    • $ENV{PERL5LIB}
    • ? "$ENV{PERL5LIB}:${lib}"
    • : "${lib}";
    • unshift @INC, $lib, "$lib/" . $Config{archname};
    • }
    • BEGIN {
    • # Find a ARCOS_ROOT based on path to harness
    • my @dir = splitdir(canonpath($RealBin));
    • $ENV{ARCOS_ROOT} ||=
    • catdir(@dir[0 .. $#dir - 1]);
    • # use $ARCOS_ROOT/lib for modules
    • my $lib = catdir($ENV{ARCOS_ROOT}, "lib");
    • $ENV{PERL5LIB} =
    • $ENV{PERL5LIB}
    • ? "$ENV{PERL5LIB}:${lib}"
    • : "${lib}";
    • unshift @INC, $lib, "$lib/" . $Config{archname};
    • eval { require Arcos };
    • die “cannot find Arcos” if $@
    • }
    • BEGIN {
    • # choose the first instance if not set.
    • unless ($ENV{ARCOS_INSTANCE}) {
    • require Arcos::Conf;
    • $ENV{ARCOS_INSTANCE} =
    • (Arcos::Conf->instances())[0];
    • }
    • }
    • BEGIN {
    • # choose the first instance if not set.
    • unless ($ENV{ARCOS_INSTANCE}) {
    • require Arcos::Conf;
    • $ENV{ARCOS_INSTANCE} =
    • (Arcos::Conf->instances())[0];
    • }
    • }
    • use Arcos;
    • use Arcos::Script;
    • use Arcos;
    • use Arcos::Script;
    • our ($help, $man, $archive);
    • my $v = 0;
    • my $files;
    • GetOptions( 'help' => $help,
    • 'man' => $man,
    • 'verbose' => $v,
    • 'make_verbose=i' => $v,
    • 'files=s' => $files,
    • 'archive' => $archive,
    • );
  • TAP::Harness can
      • try to emit results with color
      • be verbose or varying levels of quiet
      • run multiple test jobs at once (aggregate_tests)
    • see "perldoc TAP::Harness"
    • Test::Harness is for backwards compatibility.  
    • For new work, you should be starting with TAP::Harness.
  • Common test functions
    • Pull commonly used functions out into their own module.
    • Don't need inheritance for this.
    • contains:
    • Convenience methods.
    • Affecting the environment.
    • Checking the environment.
    Arcos::Test
    • can_reach_url() - uses LWP::UserAgent and checks response.
    • js_escape() - when searching for strings in HTML that has already been escaped.
    • get_server_uri() - composes url from host and port in Arcos::Conf
    Arcos::Test Convenience Methods
    • start/stop daemons
      • apache
      • Arcos queue
      • test SMTP server
    • These are basically system() calls to ctl scripts
    Arcos::Test Affecting the Environment
    • spread_is_running() -- used in t/island_mode.t
    • SKIP: {
    • skip('Spread is not running', 16) unless
    • Arcos::Test->spread_is_running();
    • # 16 tests is, isa, ok, like, etc.
    • }
    Arcos::Test Checking the Environment
    • spread_is_running() -- used in t/island_mode.t
    • SKIP: {
    • skip('Spread is not running', 16) unless
    • Arcos::Test->spread_is_running();
    • my $island_pp = catfile(ArcosRoot, 'bin',
    • 'arcos_island_postprocessor');
    • my $result = `$island_pp 2>&1`;
    • like($result, qr/Island Mode Post-Processing Complete/);
    • ok($result =~ /Submitted job (d+) : (S+) from (S+)./);
    • my ($job_id, $amount, $email) = ($1,$2,$3);
    • . . .
    • }
    Arcos::Test Checking the Environment
    • apache_is_running() -- used in t/open_handler.t
    • BEGIN {
    • if (Arcos::Test->apache_is_running) {
    • Test::More::plan('no_plan');
    • } else {
    • Test::More::plan(skip_all => "Arcos Apache is not running skipping all tests.");
    • }
    • }
    Arcos::Test Checking the Environment
  • Test::Class
    • The methods with Test attribute will be run.
    Test::Class Method Attributes Test(4)
    • The methods with Test attribute will be run.
    • The Test attribute specifies the number of tests for this method.
    Test::Class Method Attributes Test(4)
    • The methods with Test attribute will be run.
    • The Test attribute specifies the number of tests for this method.
    • The order they're run is determined alphabetically by name.
    Test::Class Method Attributes Test(4)
    • The methods with Test attribute will be run.
    • The Test attribute specifies the number of tests for this method.
    • The order they're run is determined alphabetically by name.
    • sub b_register : Test(15) { . . . }
    • sub d_short_pass : Test(4) { . . . }
    • sub e_mismatched_pass : Test(4) { . . . }
    • sub f_invalid_email : Test(4) { . . . }
    • sub h_duplicate_email : Test(4) { . . . }
    • sub h_duplicate_username : Test(4) { . . . }
    Test::Class Method Attributes Test(4)
    • The methods with Test attribute will be run.
    • The Test attribute specifies the number of tests for this method.
    • The order they're run is determined alphabetically by name.
    • sub b_register : Test(15) { . . . }
    • sub d_short_pass : Test(4) { . . . }
    • sub e_mismatched_pass : Test(4) { . . . }
    • sub f_invalid_email : Test(4) { . . . }
    • sub h_duplicate_email : Test(4) { . . . }
    • sub h_duplicate_username : Test(4) { . . . }
    • sub b_submit_empty_form : Test(no_plan) { . . . }
    • You can specify no_plan for a specific sub.
    • Can localize the uncertainty instead of using no_plan for the entire module.
    Test::Class Method Attributes Test(no_plan)
    • startup / shutdown
      • run before /after the whole test class
    Test::Class Method Attributes Test(startup)
    • startup / shutdown
      • run before /after the whole test class
    • setup / teardown
      • run before / after every test method
    Test::Class Method Attributes Test(startup)
    • startup / shutdown
      • run before /after the whole test class
    • setup / teardown
      • run before / after every test method
    • sub tc_startup : Test(startup) {     my $self = shift;     $self->{td} = Arcos::TestData->new;
    • }
    Test::Class Method Attributes Test(startup)
    • startup / shutdown
      • run before /after the whole test class
    • setup / teardown
      • run before / after every test method
    • sub tc_startup : Test(startup) {     my $self = shift;     $self->{td} = Arcos::TestData->new;
    • }
    • sub tc_shutdown : Test(shutdown) {     my $self = shift;     $self->{td}->cleanup(ignore_deleted => 1);
    • }
    Test::Class Method Attributes Test(startup)
    • startup can have tests that count towards the plan
    • sub start : Test(startup => 1) {     my $self = shift;
    • $self->{mail_tmp_dir} =
    • Arcos::Test->start_test_smtp_server;
    • use_ok('Arcos::JobQueue::Handler::CommunityInvitation');
    • }
    Test::Class Method Attributes Test(startup)
    • startup can have tests that count towards the plan.
    • sub start : Test(startup => 1) {     my $self = shift;
    • $self->{mail_tmp_dir} =
    • Arcos::Test->start_test_smtp_server;
    • use_ok('Arcos::JobQueue::Handler::CommunityInvitation');
    • }
    • startup can also be specified on more than one method.
    Test::Class Method Attributes Test(startup)
  • Inheritance in your tests
    • A parent class for many of our tests.
    Arcos::Test::Class
    • Inherit from Test::Class
    • use base 'Test::Class';
    Arcos::Test::Class
    • Pull in some helper modules.
    • use base 'Test::Class'; use Arcos::Test; use Arcos::TestData; use Test::Builder qw(ok is_eq); use Test::LongString;
    Arcos::Test::Class
    • Store an Arcos::TestData reference.
    • use base 'Test::Class'; use Arcos::Test; use Arcos::TestData; use Test::Builder qw(ok is_eq); use Test::LongString;
    • sub tc_startup : Test(startup) {     my $self = shift;     $self->{td} = Arcos::TestData->new;
    • }
    Arcos::Test::Class
    • Call cleanup method.
    • use base 'Test::Class'; use Arcos::Test; use Arcos::TestData; use Test::Builder qw(ok is_eq); use Test::LongString;
    • sub tc_startup : Test(startup) {     my $self = shift;     $self->{td} = Arcos::TestData->new;
    • } sub tc_shutdown : Test(shutdown) {     my $self = shift;     $self->{td}->cleanup(ignore_deleted => 1);
    • }
    Arcos::Test::Class
  • Arcos::Test::Class
    • We also provide convenience methods at this level:
    •  
      • contains_url()
      • lacks_url()
      • json_contains()
      • json_lacks()
      • json_success()
      • json_failure()
      • input_value_is()
  • Arcos::Test::Class
    • help Test::Builder give the right line number for failures
      • sub json_success {
      • my $self = shift;
      • {
      • local $Test::Builder::Level =
      • $Test::Builder::Level + 1;
      • $self->json_contains('success');
      • }
      • }
    • We want the error to show up where json_success() is called, not way down here in the helper module.
  • Inherited Forms, with Inherited Tests
    • Arcos::Form::Test is the parent of
      • Arcos::Form:: ContactOfficial ::Test
      • Arcos::Form:: Register ::Test
    •  
    • They each have
    • use base 'Arcos::Form::Test';
  • Arcos::Form::Test the parent
    • Startup and Shutdown
    • sub tc_startup : Test(startup) { . . . } sub tc_shutdown : Test(shutdown) { . . . } sub x_create_form : Test(startup => 3 ){ . . . }  
    • Intended to be overridden
    • sub additional_elements { '' } sub form_db_class { '' }
    • Shared methods for these forms sub mech_shows_custom_fields { . . . }
    • sub person_has_custom_fields { . . . }
  • An example of inherited tests
    • tc_startup can be completely overridden
    • or augment by calling SUPER
    •  
    • sub tc_startup : Test(startup) {
    •      $self->{td} = Arcos::TestData->new();
    •      return $self->SUPER::tc_startup;
    • }
  • Inheritance Example
  • We still have .t files t/register_form.t
    • use strict; use warnings; use Arcos::Test; use Arcos::Test::Script; use Arcos::Form::Register::Test; Arcos::Test->needs_running_krang(); Arcos::Test->needs_running_apache(); Arcos::Form::Register::Test->new->runtests;
  • We still have .t files t/contact_official_form.t
    • ContactOfficial forms need a little more at startup.
    • use Arcos::GeoCoder; my $coder = Arcos::GeoCoder->new; if (!$coder->available) {     Test::More::plan(skip_all => 
    •          'geocoding databases not available.');
    • exit;
    • }
  • We still have .t files t/contact_official_form.t
    • ContactOfficial forms need a little more at startup.
    • use Arcos::GeoCoder; my $coder = Arcos::GeoCoder->new; if (!$coder->available) {     Test::More::plan(skip_all => 
    •          'geocoding databases not available.');
    • exit;
    • } unless (Arcos::Test->stop_queue()) {     Test::More::plan(skip_all => 
    •              'Could not stop Queue daemon in time.');
    • exit;
    • }
  • We still have .t files t/contact_official_form.t
    • ContactOfficial forms need a little more at startup.
    • use Arcos::GeoCoder; my $coder = Arcos::GeoCoder->new; if (!$coder->available) {     Test::More::plan(skip_all => 
    •          'geocoding databases not available.');
    • exit;
    • } unless (Arcos::Test->stop_queue()) {     Test::More::plan(skip_all => 
    •          'Could not stop Queue daemon in time.');
    • exit;
    • } Arcos::Form::ContactOfficial::Test->new->runtests;
  • We still have .t files t/contact_official_form.t
    • ContactOfficial forms need a little more more at startup.
    • use Arcos::GeoCoder; my $coder = Arcos::GeoCoder->new; if (!$coder->available) {     Test::More::plan(skip_all => 
    •          'geocoding databases not available.');
    • exit;
    • } unless (Arcos::Test->stop_queue()) {     Test::More::plan(skip_all => 
    •          'Could not stop Queue daemon in time.');
    • exit;
    • } Arcos::Form::ContactOfficial::Test->new->runtests;
    • Arcos::Test->restart_queue;
  • return to skip
    • Section of a method not yet applicable
    • return 'The default templates do not currently have security code; skipping for now.';
    • Whole method not yet implemented
    • return 'The deleted form test is not implemented for this form type yet.';
    •     
    • Fail safe check
    • return 'You have no test payment account.' 
    •      unless $self->test_account;
  • plan to skip
    • use Arcos::Conf qw(ContributionTestMode);
    • if (!ContributionTestMode) {
    •     plan(skip_all => 'contributions not in test mode');
    • } elsif (!Arcos::Test->can_reach_url(
    •     'https://www.vancodev.com:443/cgi-bin/wstest.vps')) {
    •     plan(skip_all => "Can't reach https://www.vancodev.com:443/cgi-bin/wstest.vps");
    • } else {     Arcos::ContributionForm::Test::Vanco->runtests(); }
  • TEST_METHOD
    • TEST_METHOD=a_good_file
    • bin/arcos_test --files=t/admin-listupload.t
    • TEST_METHOD=e_failed_contributions
    • bin/arcos_test --files=t/report-contribution.t
    • TEST_METHOD=k_listing_checkbox
    • bin/arcos_test --files=t/admin-mailingmessage.t
  • TEST_METHOD
    • TEST_METHOD=a_good_file
    • bin/arcos_test --files=t/admin-listupload.t
    • TEST_METHOD=e_failed_contributions
    • bin/arcos_test --files=t/report-contribution.t
    • TEST_METHOD=k_listing_checkbox
    • bin/arcos_test --files=t/admin-mailingmessage.t
    • TEST_METHOD='(a|b)_.*'
    • bin/arcos_test --files=t/warehouse-loader.t
  • a lot of reuse
    • use base 'Arcos::Report::SiteEvent::Test';
    • Arcos::Report::Volunteer::Test
    • Arcos::Report::Contribution::Test
    • Arcos::Report::ContactOfficial::Test
    • Arcos::Report::Petition::Test
    • Arcos::Report::Tellafriend::Test
    • Arcos::Report::Subscription::Test
    • Arcos::Report::RadioCall::Test
    • Arcos::Report::LTE::Test
    • Arcos::Report::ContactUs::Test
  • Bonus slides
  • TODO tests
    • sub live_test : Test {
    • local $TODO =
    • "live currently unimplemented";
    • ok(Object->live, "object live");
    • };
    • Skip is so darn easy we haven't used TODO like this.
  • pausing to look around
    • warn 'pausing to look around; 
    •      hit return to continue:';
    • my $PAUSE = <STDIN>;
    •  
    • This will keep the test from cleaning up before you've had a chance to look around.
    • Add a few warns of URL values and you can bring up the reports in a browser to get a better idea of what's going on.
    • Litter this within a specific method, or in your method with the Test(shutdown) attribute.
  • calling plan from Arcos::Test
    • I added a needs_run_daily method to bail out gracefully.
    • if ($ENV{'RUN_DAILY_TESTS'}) {     if (
    • $ENV{'OVERRIDE_DAILY_TESTS'}
    • or $class->_needs_daily_run()
    • ) {         $class->_update_run_daily_timestamp();         Test::More::plan(@args);     }
    • } else {     Test::More::plan(        skip_all => 'RUN_DAILY_TESTS environment variable not set; skipping daily tests.'); }
  • Additional Resources
  • Additional Resources
    • #perl-qa on the irc.perl.org server
    • http://search.cpan.org/~adie/Test-Class-0.31/lib/Test/Class.pm
    • http://search.cpan.org/~mschwern/Test-Simple-0.88/lib/Test/Builder.pm
    • Organizing Test Suites with Test::Class
    • http://www.modernperlbooks.com/mt/2009/03/organizing-test-suites-with-testclass.html
  • Thank you for your time!