Turbo Charged
Test Suites
Tuesday, 7 October 2008

Curtis “Ovid” Poe
• CPAN Author
• “Perl Hacks” Contributor
• Perl Foundation Member
Tuesday, 7 October 2008

Test Suite Performance
• Computer
• Developer
Tuesday, 7 October 2008

Faster Tests
Tuesday, 7 October 2008

Considerations
• Why speed things up?
• Trade-offs
Tuesday, 7 October 2008

Before You Start
• Set a goal
• No tests may fail
• No non-test output
Tuesday, 7 October 2008

Databases
• Don’t drop them
• Don’t fake it with transactions
Tuesday, 7 October 2008

Databases
• Static tables -- country_codes
• Dynamic tables -- orders
Tuesday, 7 October 2008

CREATE TABLE changed_table (
changed_table_id INT(11)
NOT NULL AUTO_INCREMENT PRIMARY KEY,
table_name VARCHAR(30) NOT NULL,
is_static INT NOT NULL DEFAULT 0,
inserts INT NOT NULL DEFAULT 0,
updates INT NOT NULL DEFAULT 0,
deletes INT NOT NULL DEFAULT 0
)
-- then insert table records
Tuesday, 7 October 2008

foreach my $action (qw/insert update delete/) {
$dbh->do(<<\" END_SQL\");
CREATE TRIGGER tr_${action}_$table
BEFORE $action ON $table
FOR EACH ROW UPDATE changed_table
SET ${action}s = ${action}s + 1
WHERE table_name = '$table';
END_SQL
}
Tuesday, 7 October 2008

db.disable_foreign_keys()
for table in changed_tables()
table.truncate()
if table.is_static()
table.rebuild()
db.enable_foreign_keys()
Tuesday, 7 October 2008

Databases
• Before: 80 minutes
• After: 22 minutes
Tuesday, 7 October 2008

Aggregation
• Only load things once
• Dangers of shared state
Tuesday, 7 October 2008

use My::YAML::Test;
My::YAML::Test->run;
# t/api/customer.t
# t/api/customer.yml
Tuesday, 7 October 2008

use My::YAML::Test;
foreach (yaml_tests()) {
diag “running $_”;
My::YAML::Test->run($_);
}
sub yaml_tests {
return @ARGV if @ARGV;
# or return all YAML files ...
}
Tuesday, 7 October 2008

YAML Aggregation
• Before: 22 minutes
• After: 16 minutes
Tuesday, 7 October 2008

use Test::Aggregate;
my $tests = Test::Aggregate->new({
dirs => ‘aggtests’,
});
$tests->run;
Tuesday, 7 October 2008

use Test::Aggregate;
my $tests = Test::Aggregate->new({
dirs => 'aggtests',
dump => $dump,
set_filenames => 1,
shuffle => 1,
startup => \\&startup,
shutdown => \\&shutdown,
});
Tuesday, 7 October 2008

Generic Aggregation
• Before: 16 minutes
• After: 12 minutes
Tuesday, 7 October 2008

Better Aggregation
• use Test::Class
Tuesday, 7 October 2008

OK to Fail is OK
• POD tests
• Perl::Critic
• Anything “non-functional”
Tuesday, 7 October 2008

OK to Fail is OK
• Move them to xt/
• But they must be run!
Tuesday, 7 October 2008

Lessons Learned
• Not dropping your database is tricky
• Aggregating tests finds bugs in tests
• Aggregating tests finds bugs in code
• *CORE::GLOBAL:: is evil
• Don’t touch UNIVERSAL::
Tuesday, 7 October 2008

Faster Programmers
Tuesday, 7 October 2008

Custom Test Modules
Tuesday, 7 October 2008

use Test::More tests => 13;
use Test::Exception;
use Test::XML;
use Test::JSON;
use Test::Differences;
Tuesday, 7 October 2008

package Our::Test::More;
use Test::Builder::Module;
our ( @ISA, @EXPORT );
use Test::More;
use Test::Exception;
BEGIN {
@ISA = qw(Test::Builder::Module);
@EXPORT = (
@Test::More::EXPORT,
@Test::Exception::EXPORT,
);
}
1;
Tuesday, 7 October 2008

package My::Custom::Tests;
use Test::Kit (
'Test::More',
'Test::XML',
'Test::Differences',
'+explain',
);
1;
Tuesday, 7 October 2008

Popularity Contest
Test::More 44461
Test 8937
Test::Exception 1397
Test::Simple 731
Test::Base 316
Test::Builder::Tester 193
Test::NoWarnings 174
Test::Differences 146
Test::MockObject 139
Test::Deep 127
Tuesday, 7 October 2008

use Test::Most tests => 4, 'die';
ok 1, 'one is true';
is 2, 2, '... and two is two';
eq_or_diff [3], [4],
“... but three ain’t four”;
throws_ok { $foo/0 }
qr/Illegal division by zero/,
'and no-constant folding with vars';
Tuesday, 7 October 2008

Test From Your Editor
Tuesday, 7 October 2008

“ in your .vimrc
function! PerlMappings()
noremap K :!perldoc <cword> <bar><bar> \\
perldoc -f <cword><cr>
endfunction
function! PerlTestMappings()
noremap <buffer> ,t :!prove -vl %<CR>
endfunction
“ remember My::Test::YAML?
function! YAMLTestMappings()
noremap <buffer> ,t :!prove -vl t/yaml.t :: %<CR>
endfunction
au! FileType perl :call PerlMappings()
au! FileType yaml :call YAMLTestMappings()
au! BufRead,BufNewFile *.t :call PerlTestMappings()
Tuesday, 7 October 2008

vim $(ack -l --perl 'api/v1/episode' t/)
map <leader>tb :call RunTestsInBuffers()<cr>
function! RunTestsInBuffers()
let i = 1
let tests = ''
while (i <= bufnr(\"$\"))
let filename = bufname(i)
if match(filename, '\\.t$') > -1
let tests = tests . ' \"' . filename . '\"'
endif
let i = i+1
endwhile
if !strlen(tests)
echo \"No tests found in buffers\"
else
execute ':!prove ' . tests
endif
endfunction
Tuesday, 7 October 2008

Advanced “prove”
Tuesday, 7 October 2008

Test-Harness $ prove -l t --state=hot,fast,save --timer
[20:57:19] t/yamlish-output.........ok 40 ms
[20:57:20] t/console................ok 45 ms
[20:57:20] t/utils..................ok 48 ms
<snip>
[20:57:23] t/harness................ok 300 ms
[20:57:23] t/process................ok 1020 ms
[20:57:24] t/prove..................ok 1017 ms
[20:57:25] t/regression.............ok 4217 ms
[20:57:29]
All tests successful.
Files=32, Tests=10326, 10 wallclock secs ( 1.25 usr
0.24 sys + 5.60 cusr 1.83 csys = 8.92 CPU)
Result: PASS
Tuesday, 7 October 2008

Test-Harness $ prove -l t --state=hot,slow,save -j 9
t/callbacks.............. ok
t/nofork................. ok
t/proverc................ ok
<snip>
t/yamlish................ ok
t/prove.................. ok
t/regression............. ok
All tests successful.
Files=32, Tests=10326, 6 wallclock secs ( 1.34 usr
0.24 sys + 5.63 cusr 1.83 csys = 9.04 CPU)
Result: PASS
Tuesday, 7 October 2008

# slow running tests from App::Prove::State
# http://use.perl.org/~Ovid/journal/35831
Generation 18
Number of test programs: 58
Total runtime approximately 17 minutes 35 seconds
Five slowest tests:
482.7 seconds -> t/acceptance.t
234.4 seconds -> t/aggregate.t
96.3 seconds -> t/standards/strict.t
66.6 seconds -> t/unit/db/migrations.t
56.7 seconds -> t/unit/piptest/pprove/testdb.t
Tuesday, 7 October 2008

Devel::CoverX::Covered
Tuesday, 7 October 2008

Devel::CoverX::Covered
• What tests cover this file?
• What files are covered by this test?
• (Soon) What tests cover this line?
• (Soon) What tests cover this subroutine?
• ... and more ...
Tuesday, 7 October 2008

function! PerlMappings()
noremap <buffer> ,cv :call Coverage()<cr>
endfunction
function! PerlTestMappings()
noremap <buffer> ,t :!prove -vl --norc %<CR>
endfunction
function! Coverage()
let file = bufname('%')
if match(filename, '\\.t$') > -1
execute '!covered by --test_file=\"'.file.'\"'
else
execute '!covered covering --source_file=\"'.file.'\"'
end
endfunction
au! FileType perl :call PerlMappings()
au! BufRead,BufNewFile *.t :call PerlTestMappings()
Tuesday, 7 October 2008

Questions?
Tuesday, 7 October 2008