Turbo Charged
                            Test Suites



Tuesday, 7 October 2008
Curtis “Ovid” Poe

                    • CPAN Author
                    • “Perl Hacks” Contributor
                    • ...
Test Suite Performance


                    • Computer
                    • Developer


Tuesday, 7 October 2008
Faster Tests




Tuesday, 7 October 2008
Considerations


                    • Why speed things up?
                    • Trade-offs


Tuesday, 7 October 2008
Before You Start

                    • Set a goal
                    • No tests may fail
                    • No non-te...
Databases


                    • Don’t drop them
                    • Don’t fake it with transactions


Tuesday, 7 Octob...
Databases


                    • Static tables -- country_codes
                    • Dynamic tables -- orders


Tuesday,...
CREATE TABLE changed_table (
                    changed_table_id INT(11)
                      NOT NULL AUTO_INCREMENT PR...
foreach my $action (qw/insert update delete/) {
                    $dbh->do(<<quot;    END_SQLquot;);
                   ...
db.disable_foreign_keys()
                for table in changed_tables()
                  table.truncate()
               ...
Databases


                    • Before: 80 minutes
                    • After: 22 minutes


Tuesday, 7 October 2008
Aggregation


                    • Only load things once
                    • Dangers of shared state


Tuesday, 7 Octob...
use My::YAML::Test;
                My::YAML::Test->run;
                # t/api/customer.t
                # t/api/custom...
use My::YAML::Test;
                foreach (yaml_tests()) {
                    diag “running $_”;
                    My...
YAML Aggregation


                    • Before: 22 minutes
                    • After: 16 minutes


Tuesday, 7 October 2...
use Test::Aggregate;
                my $tests = Test::Aggregate->new({
                    dirs => ‘aggtests’,
          ...
use Test::Aggregate;
                my $tests = Test::Aggregate->new({
                    dirs          => 'aggtests',
 ...
Generic Aggregation


                    • Before: 16 minutes
                    • After: 12 minutes


Tuesday, 7 Octobe...
Better Aggregation


                    • use Test::Class



Tuesday, 7 October 2008
OK to Fail is OK

                    • POD tests
                    • Perl::Critic
                    • Anything “non-f...
OK to Fail is OK


                    • Move them to xt/
                    • But they must be run!


Tuesday, 7 October...
Lessons Learned

                    • Not dropping your database is tricky
                    • Aggregating tests finds b...
Faster Programmers




Tuesday, 7 October 2008
Custom Test Modules




Tuesday, 7 October 2008
use       Test::More tests => 13;
                use       Test::Exception;
                use       Test::XML;
        ...
package Our::Test::More;

                use       Test::Builder::Module;
                our       ( @ISA, @EXPORT );
  ...
package My::Custom::Tests;

                use Test::Kit (
                    'Test::More',
                    'Test::X...
Popularity Contest
                           Test::More              44461
                           Test               ...
use Test::Most tests => 4, 'die';

                ok 1,    'one is true';
                is 2, 2, '... and two is two';
...
Test From Your Editor




Tuesday, 7 October 2008
“ in your .vimrc

                function! PerlMappings()
                    noremap K :!perldoc <cword> <bar><bar> 
   ...
vim $(ack -l --perl 'api/v1/episode' t/)



                map <leader>tb :call RunTestsInBuffers()<cr>
                f...
Advanced “prove”




Tuesday, 7 October 2008
Test-Harness $ prove -l t --state=hot,fast,save   --timer
                [20:57:19] t/yamlish-output.........ok       40 ...
Test-Harness $ prove -l t --state=hot,slow,save -j 9
                t/callbacks.............. ok
                t/nofork...
# slow running tests from App::Prove::State
                # http://use.perl.org/~Ovid/journal/35831

                Gen...
Devel::CoverX::Covered




Tuesday, 7 October 2008
Devel::CoverX::Covered

                    • What tests cover this file?
                    • What files are covered by th...
function! PerlMappings()
                  noremap <buffer> ,cv :call Coverage()<cr>
                endfunction

        ...
Questions?




Tuesday, 7 October 2008
Upcoming SlideShare
Loading in...5
×

Turbo Charged Test Suites

3,896

Published on

Techniques for Perl programmers to speed up test suite and developer performance.

Published in: Technology, Business
1 Comment
4 Likes
Statistics
Notes
  • It's worth noting that the slides regarding rebuilding tables and avoiding transactions assume that your test suite is constrained to run in a single process. If you parallelize your test suite, transactions are recommended.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
3,896
On Slideshare
0
From Embeds
0
Number of Embeds
22
Actions
Shares
0
Downloads
36
Comments
1
Likes
4
Embeds 0
No embeds

No notes for slide

Turbo Charged Test Suites

  1. 1. Turbo Charged Test Suites Tuesday, 7 October 2008
  2. 2. Curtis “Ovid” Poe • CPAN Author • “Perl Hacks” Contributor • Perl Foundation Member Tuesday, 7 October 2008
  3. 3. Test Suite Performance • Computer • Developer Tuesday, 7 October 2008
  4. 4. Faster Tests Tuesday, 7 October 2008
  5. 5. Considerations • Why speed things up? • Trade-offs Tuesday, 7 October 2008
  6. 6. Before You Start • Set a goal • No tests may fail • No non-test output Tuesday, 7 October 2008
  7. 7. Databases • Don’t drop them • Don’t fake it with transactions Tuesday, 7 October 2008
  8. 8. Databases • Static tables -- country_codes • Dynamic tables -- orders Tuesday, 7 October 2008
  9. 9. CREATE TABLE changed_table ( changed_table_id INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY, table_name VARCHAR(30) NOT NULL, is_static INT NOT NULL DEFAULT 0, inserts INT NOT NULL DEFAULT 0, updates INT NOT NULL DEFAULT 0, deletes INT NOT NULL DEFAULT 0 ) -- then insert table records Tuesday, 7 October 2008
  10. 10. foreach my $action (qw/insert update delete/) { $dbh->do(<<quot; END_SQLquot;); CREATE TRIGGER tr_${action}_$table BEFORE $action ON $table FOR EACH ROW UPDATE changed_table SET ${action}s = ${action}s + 1 WHERE table_name = '$table'; END_SQL } Tuesday, 7 October 2008
  11. 11. db.disable_foreign_keys() for table in changed_tables() table.truncate() if table.is_static() table.rebuild() db.enable_foreign_keys() Tuesday, 7 October 2008
  12. 12. Databases • Before: 80 minutes • After: 22 minutes Tuesday, 7 October 2008
  13. 13. Aggregation • Only load things once • Dangers of shared state Tuesday, 7 October 2008
  14. 14. use My::YAML::Test; My::YAML::Test->run; # t/api/customer.t # t/api/customer.yml Tuesday, 7 October 2008
  15. 15. use My::YAML::Test; foreach (yaml_tests()) { diag “running $_”; My::YAML::Test->run($_); } sub yaml_tests { return @ARGV if @ARGV; # or return all YAML files ... } Tuesday, 7 October 2008
  16. 16. YAML Aggregation • Before: 22 minutes • After: 16 minutes Tuesday, 7 October 2008
  17. 17. use Test::Aggregate; my $tests = Test::Aggregate->new({ dirs => ‘aggtests’, }); $tests->run; Tuesday, 7 October 2008
  18. 18. use Test::Aggregate; my $tests = Test::Aggregate->new({ dirs => 'aggtests', dump => $dump, set_filenames => 1, shuffle => 1, startup => &startup, shutdown => &shutdown, }); Tuesday, 7 October 2008
  19. 19. Generic Aggregation • Before: 16 minutes • After: 12 minutes Tuesday, 7 October 2008
  20. 20. Better Aggregation • use Test::Class Tuesday, 7 October 2008
  21. 21. OK to Fail is OK • POD tests • Perl::Critic • Anything “non-functional” Tuesday, 7 October 2008
  22. 22. OK to Fail is OK • Move them to xt/ • But they must be run! Tuesday, 7 October 2008
  23. 23. Lessons Learned • Not dropping your database is tricky • Aggregating tests finds bugs in tests • Aggregating tests finds bugs in code • *CORE::GLOBAL:: is evil • Don’t touch UNIVERSAL:: Tuesday, 7 October 2008
  24. 24. Faster Programmers Tuesday, 7 October 2008
  25. 25. Custom Test Modules Tuesday, 7 October 2008
  26. 26. use Test::More tests => 13; use Test::Exception; use Test::XML; use Test::JSON; use Test::Differences; Tuesday, 7 October 2008
  27. 27. package Our::Test::More; use Test::Builder::Module; our ( @ISA, @EXPORT ); use Test::More; use Test::Exception; BEGIN { @ISA = qw(Test::Builder::Module); @EXPORT = ( @Test::More::EXPORT, @Test::Exception::EXPORT, ); } 1; Tuesday, 7 October 2008
  28. 28. package My::Custom::Tests; use Test::Kit ( 'Test::More', 'Test::XML', 'Test::Differences', '+explain', ); 1; Tuesday, 7 October 2008
  29. 29. Popularity Contest Test::More 44461 Test 8937 Test::Exception 1397 Test::Simple 731 Test::Base 316 Test::Builder::Tester 193 Test::NoWarnings 174 Test::Differences 146 Test::MockObject 139 Test::Deep 127 Tuesday, 7 October 2008
  30. 30. use Test::Most tests => 4, 'die'; ok 1, 'one is true'; is 2, 2, '... and two is two'; eq_or_diff [3], [4], “... but three ain’t four”; throws_ok { $foo/0 } qr/Illegal division by zero/, 'and no-constant folding with vars'; Tuesday, 7 October 2008
  31. 31. Test From Your Editor Tuesday, 7 October 2008
  32. 32. “ in your .vimrc function! PerlMappings() noremap K :!perldoc <cword> <bar><bar> perldoc -f <cword><cr> endfunction function! PerlTestMappings() noremap <buffer> ,t :!prove -vl %<CR> endfunction “ remember My::Test::YAML? function! YAMLTestMappings() noremap <buffer> ,t :!prove -vl t/yaml.t :: %<CR> endfunction au! FileType perl :call PerlMappings() au! FileType yaml :call YAMLTestMappings() au! BufRead,BufNewFile *.t :call PerlTestMappings() Tuesday, 7 October 2008
  33. 33. vim $(ack -l --perl 'api/v1/episode' t/) map <leader>tb :call RunTestsInBuffers()<cr> function! RunTestsInBuffers() let i = 1 let tests = '' while (i <= bufnr(quot;$quot;)) let filename = bufname(i) if match(filename, '.t$') > -1 let tests = tests . ' quot;' . filename . 'quot;' endif let i = i+1 endwhile if !strlen(tests) echo quot;No tests found in buffersquot; else execute ':!prove ' . tests endif endfunction Tuesday, 7 October 2008
  34. 34. Advanced “prove” Tuesday, 7 October 2008
  35. 35. Test-Harness $ prove -l t --state=hot,fast,save --timer [20:57:19] t/yamlish-output.........ok 40 ms [20:57:20] t/console................ok 45 ms [20:57:20] t/utils..................ok 48 ms <snip> [20:57:23] t/harness................ok 300 ms [20:57:23] t/process................ok 1020 ms [20:57:24] t/prove..................ok 1017 ms [20:57:25] t/regression.............ok 4217 ms [20:57:29] All tests successful. Files=32, Tests=10326, 10 wallclock secs ( 1.25 usr 0.24 sys + 5.60 cusr 1.83 csys = 8.92 CPU) Result: PASS Tuesday, 7 October 2008
  36. 36. Test-Harness $ prove -l t --state=hot,slow,save -j 9 t/callbacks.............. ok t/nofork................. ok t/proverc................ ok <snip> t/yamlish................ ok t/prove.................. ok t/regression............. ok All tests successful. Files=32, Tests=10326, 6 wallclock secs ( 1.34 usr 0.24 sys + 5.63 cusr 1.83 csys = 9.04 CPU) Result: PASS Tuesday, 7 October 2008
  37. 37. # slow running tests from App::Prove::State # http://use.perl.org/~Ovid/journal/35831 Generation 18 Number of test programs: 58 Total runtime approximately 17 minutes 35 seconds Five slowest tests: 482.7 seconds -> t/acceptance.t 234.4 seconds -> t/aggregate.t 96.3 seconds -> t/standards/strict.t 66.6 seconds -> t/unit/db/migrations.t 56.7 seconds -> t/unit/piptest/pprove/testdb.t Tuesday, 7 October 2008
  38. 38. Devel::CoverX::Covered Tuesday, 7 October 2008
  39. 39. Devel::CoverX::Covered • What tests cover this file? • What files are covered by this test? • (Soon) What tests cover this line? • (Soon) What tests cover this subroutine? • ... and more ... Tuesday, 7 October 2008
  40. 40. function! PerlMappings() noremap <buffer> ,cv :call Coverage()<cr> endfunction function! PerlTestMappings() noremap <buffer> ,t :!prove -vl --norc %<CR> endfunction function! Coverage() let file = bufname('%') if match(filename, '.t$') > -1 execute '!covered by --test_file=quot;'.file.'quot;' else execute '!covered covering --source_file=quot;'.file.'quot;' end endfunction au! FileType perl :call PerlMappings() au! BufRead,BufNewFile *.t :call PerlTestMappings() Tuesday, 7 October 2008
  41. 41. Questions? Tuesday, 7 October 2008
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×