Turbo Charged
                            Test Suites



Tuesday, 7 October 2008
Curtis “Ovid” Poe

                    • CPAN Author
                    • “Perl Hacks” Contributor
                    • ...
Test Suite Performance


                    • Computer
                    • Developer


Tuesday, 7 October 2008
Faster Tests




Tuesday, 7 October 2008
Considerations


                    • Why speed things up?
                    • Trade-offs


Tuesday, 7 October 2008
Before You Start

                    • Set a goal
                    • No tests may fail
                    • No non-te...
Databases


                    • Don’t drop them
                    • Don’t fake it with transactions


Tuesday, 7 Octob...
Databases


                    • Static tables -- country_codes
                    • Dynamic tables -- orders


Tuesday,...
CREATE TABLE changed_table (
                    changed_table_id INT(11)
                      NOT NULL AUTO_INCREMENT PR...
foreach my $action (qw/insert update delete/) {
                    $dbh->do(<<quot;    END_SQLquot;);
                   ...
db.disable_foreign_keys()
                for table in changed_tables()
                  table.truncate()
               ...
Databases


                    • Before: 80 minutes
                    • After: 22 minutes


Tuesday, 7 October 2008
Aggregation


                    • Only load things once
                    • Dangers of shared state


Tuesday, 7 Octob...
use My::YAML::Test;
                My::YAML::Test->run;
                # t/api/customer.t
                # t/api/custom...
use My::YAML::Test;
                foreach (yaml_tests()) {
                    diag “running $_”;
                    My...
YAML Aggregation


                    • Before: 22 minutes
                    • After: 16 minutes


Tuesday, 7 October 2...
use Test::Aggregate;
                my $tests = Test::Aggregate->new({
                    dirs => ‘aggtests’,
          ...
use Test::Aggregate;
                my $tests = Test::Aggregate->new({
                    dirs          => 'aggtests',
 ...
Generic Aggregation


                    • Before: 16 minutes
                    • After: 12 minutes


Tuesday, 7 Octobe...
Better Aggregation


                    • use Test::Class



Tuesday, 7 October 2008
OK to Fail is OK

                    • POD tests
                    • Perl::Critic
                    • Anything “non-f...
OK to Fail is OK


                    • Move them to xt/
                    • But they must be run!


Tuesday, 7 October...
Lessons Learned

                    • Not dropping your database is tricky
                    • Aggregating tests finds b...
Faster Programmers




Tuesday, 7 October 2008
Custom Test Modules




Tuesday, 7 October 2008
use       Test::More tests => 13;
                use       Test::Exception;
                use       Test::XML;
        ...
package Our::Test::More;

                use       Test::Builder::Module;
                our       ( @ISA, @EXPORT );
  ...
package My::Custom::Tests;

                use Test::Kit (
                    'Test::More',
                    'Test::X...
Popularity Contest
                           Test::More              44461
                           Test               ...
use Test::Most tests => 4, 'die';

                ok 1,    'one is true';
                is 2, 2, '... and two is two';
...
Test From Your Editor




Tuesday, 7 October 2008
“ in your .vimrc

                function! PerlMappings()
                    noremap K :!perldoc <cword> <bar><bar> 
   ...
vim $(ack -l --perl 'api/v1/episode' t/)



                map <leader>tb :call RunTestsInBuffers()<cr>
                f...
Advanced “prove”




Tuesday, 7 October 2008
Test-Harness $ prove -l t --state=hot,fast,save   --timer
                [20:57:19] t/yamlish-output.........ok       40 ...
Test-Harness $ prove -l t --state=hot,slow,save -j 9
                t/callbacks.............. ok
                t/nofork...
# slow running tests from App::Prove::State
                # http://use.perl.org/~Ovid/journal/35831

                Gen...
Devel::CoverX::Covered




Tuesday, 7 October 2008
Devel::CoverX::Covered

                    • What tests cover this file?
                    • What files are covered by th...
function! PerlMappings()
                  noremap <buffer> ,cv :call Coverage()<cr>
                endfunction

        ...
Questions?




Tuesday, 7 October 2008
Upcoming SlideShare
Loading in …5
×

Turbo Charged Test Suites

4,031
-1

Published on

Techniques for Perl programmers to speed up test suite and developer performance.

Published in: Technology, Business
1 Comment
4 Likes
Statistics
Notes
No Downloads
Views
Total Views
4,031
On Slideshare
0
From Embeds
0
Number of Embeds
22
Actions
Shares
0
Downloads
36
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide

Turbo Charged Test Suites

  1. 1. Turbo Charged Test Suites Tuesday, 7 October 2008
  2. 2. Curtis “Ovid” Poe • CPAN Author • “Perl Hacks” Contributor • Perl Foundation Member Tuesday, 7 October 2008
  3. 3. Test Suite Performance • Computer • Developer Tuesday, 7 October 2008
  4. 4. Faster Tests Tuesday, 7 October 2008
  5. 5. Considerations • Why speed things up? • Trade-offs Tuesday, 7 October 2008
  6. 6. Before You Start • Set a goal • No tests may fail • No non-test output Tuesday, 7 October 2008
  7. 7. Databases • Don’t drop them • Don’t fake it with transactions Tuesday, 7 October 2008
  8. 8. Databases • Static tables -- country_codes • Dynamic tables -- orders Tuesday, 7 October 2008
  9. 9. CREATE TABLE changed_table ( changed_table_id INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY, table_name VARCHAR(30) NOT NULL, is_static INT NOT NULL DEFAULT 0, inserts INT NOT NULL DEFAULT 0, updates INT NOT NULL DEFAULT 0, deletes INT NOT NULL DEFAULT 0 ) -- then insert table records Tuesday, 7 October 2008
  10. 10. foreach my $action (qw/insert update delete/) { $dbh->do(<<quot; END_SQLquot;); CREATE TRIGGER tr_${action}_$table BEFORE $action ON $table FOR EACH ROW UPDATE changed_table SET ${action}s = ${action}s + 1 WHERE table_name = '$table'; END_SQL } Tuesday, 7 October 2008
  11. 11. db.disable_foreign_keys() for table in changed_tables() table.truncate() if table.is_static() table.rebuild() db.enable_foreign_keys() Tuesday, 7 October 2008
  12. 12. Databases • Before: 80 minutes • After: 22 minutes Tuesday, 7 October 2008
  13. 13. Aggregation • Only load things once • Dangers of shared state Tuesday, 7 October 2008
  14. 14. use My::YAML::Test; My::YAML::Test->run; # t/api/customer.t # t/api/customer.yml Tuesday, 7 October 2008
  15. 15. use My::YAML::Test; foreach (yaml_tests()) { diag “running $_”; My::YAML::Test->run($_); } sub yaml_tests { return @ARGV if @ARGV; # or return all YAML files ... } Tuesday, 7 October 2008
  16. 16. YAML Aggregation • Before: 22 minutes • After: 16 minutes Tuesday, 7 October 2008
  17. 17. use Test::Aggregate; my $tests = Test::Aggregate->new({ dirs => ‘aggtests’, }); $tests->run; Tuesday, 7 October 2008
  18. 18. use Test::Aggregate; my $tests = Test::Aggregate->new({ dirs => 'aggtests', dump => $dump, set_filenames => 1, shuffle => 1, startup => &startup, shutdown => &shutdown, }); Tuesday, 7 October 2008
  19. 19. Generic Aggregation • Before: 16 minutes • After: 12 minutes Tuesday, 7 October 2008
  20. 20. Better Aggregation • use Test::Class Tuesday, 7 October 2008
  21. 21. OK to Fail is OK • POD tests • Perl::Critic • Anything “non-functional” Tuesday, 7 October 2008
  22. 22. OK to Fail is OK • Move them to xt/ • But they must be run! Tuesday, 7 October 2008
  23. 23. Lessons Learned • Not dropping your database is tricky • Aggregating tests finds bugs in tests • Aggregating tests finds bugs in code • *CORE::GLOBAL:: is evil • Don’t touch UNIVERSAL:: Tuesday, 7 October 2008
  24. 24. Faster Programmers Tuesday, 7 October 2008
  25. 25. Custom Test Modules Tuesday, 7 October 2008
  26. 26. use Test::More tests => 13; use Test::Exception; use Test::XML; use Test::JSON; use Test::Differences; Tuesday, 7 October 2008
  27. 27. package Our::Test::More; use Test::Builder::Module; our ( @ISA, @EXPORT ); use Test::More; use Test::Exception; BEGIN { @ISA = qw(Test::Builder::Module); @EXPORT = ( @Test::More::EXPORT, @Test::Exception::EXPORT, ); } 1; Tuesday, 7 October 2008
  28. 28. package My::Custom::Tests; use Test::Kit ( 'Test::More', 'Test::XML', 'Test::Differences', '+explain', ); 1; Tuesday, 7 October 2008
  29. 29. Popularity Contest Test::More 44461 Test 8937 Test::Exception 1397 Test::Simple 731 Test::Base 316 Test::Builder::Tester 193 Test::NoWarnings 174 Test::Differences 146 Test::MockObject 139 Test::Deep 127 Tuesday, 7 October 2008
  30. 30. use Test::Most tests => 4, 'die'; ok 1, 'one is true'; is 2, 2, '... and two is two'; eq_or_diff [3], [4], “... but three ain’t four”; throws_ok { $foo/0 } qr/Illegal division by zero/, 'and no-constant folding with vars'; Tuesday, 7 October 2008
  31. 31. Test From Your Editor Tuesday, 7 October 2008
  32. 32. “ in your .vimrc function! PerlMappings() noremap K :!perldoc <cword> <bar><bar> perldoc -f <cword><cr> endfunction function! PerlTestMappings() noremap <buffer> ,t :!prove -vl %<CR> endfunction “ remember My::Test::YAML? function! YAMLTestMappings() noremap <buffer> ,t :!prove -vl t/yaml.t :: %<CR> endfunction au! FileType perl :call PerlMappings() au! FileType yaml :call YAMLTestMappings() au! BufRead,BufNewFile *.t :call PerlTestMappings() Tuesday, 7 October 2008
  33. 33. vim $(ack -l --perl 'api/v1/episode' t/) map <leader>tb :call RunTestsInBuffers()<cr> function! RunTestsInBuffers() let i = 1 let tests = '' while (i <= bufnr(quot;$quot;)) let filename = bufname(i) if match(filename, '.t$') > -1 let tests = tests . ' quot;' . filename . 'quot;' endif let i = i+1 endwhile if !strlen(tests) echo quot;No tests found in buffersquot; else execute ':!prove ' . tests endif endfunction Tuesday, 7 October 2008
  34. 34. Advanced “prove” Tuesday, 7 October 2008
  35. 35. Test-Harness $ prove -l t --state=hot,fast,save --timer [20:57:19] t/yamlish-output.........ok 40 ms [20:57:20] t/console................ok 45 ms [20:57:20] t/utils..................ok 48 ms <snip> [20:57:23] t/harness................ok 300 ms [20:57:23] t/process................ok 1020 ms [20:57:24] t/prove..................ok 1017 ms [20:57:25] t/regression.............ok 4217 ms [20:57:29] All tests successful. Files=32, Tests=10326, 10 wallclock secs ( 1.25 usr 0.24 sys + 5.60 cusr 1.83 csys = 8.92 CPU) Result: PASS Tuesday, 7 October 2008
  36. 36. Test-Harness $ prove -l t --state=hot,slow,save -j 9 t/callbacks.............. ok t/nofork................. ok t/proverc................ ok <snip> t/yamlish................ ok t/prove.................. ok t/regression............. ok All tests successful. Files=32, Tests=10326, 6 wallclock secs ( 1.34 usr 0.24 sys + 5.63 cusr 1.83 csys = 9.04 CPU) Result: PASS Tuesday, 7 October 2008
  37. 37. # slow running tests from App::Prove::State # http://use.perl.org/~Ovid/journal/35831 Generation 18 Number of test programs: 58 Total runtime approximately 17 minutes 35 seconds Five slowest tests: 482.7 seconds -> t/acceptance.t 234.4 seconds -> t/aggregate.t 96.3 seconds -> t/standards/strict.t 66.6 seconds -> t/unit/db/migrations.t 56.7 seconds -> t/unit/piptest/pprove/testdb.t Tuesday, 7 October 2008
  38. 38. Devel::CoverX::Covered Tuesday, 7 October 2008
  39. 39. Devel::CoverX::Covered • What tests cover this file? • What files are covered by this test? • (Soon) What tests cover this line? • (Soon) What tests cover this subroutine? • ... and more ... Tuesday, 7 October 2008
  40. 40. function! PerlMappings() noremap <buffer> ,cv :call Coverage()<cr> endfunction function! PerlTestMappings() noremap <buffer> ,t :!prove -vl --norc %<CR> endfunction function! Coverage() let file = bufname('%') if match(filename, '.t$') > -1 execute '!covered by --test_file=quot;'.file.'quot;' else execute '!covered covering --source_file=quot;'.file.'quot;' end endfunction au! FileType perl :call PerlMappings() au! BufRead,BufNewFile *.t :call PerlTestMappings() Tuesday, 7 October 2008
  41. 41. Questions? Tuesday, 7 October 2008
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×