Turbo Charged Test Suites

  • 3,672 views
Uploaded on

Techniques for Perl programmers to speed up test suite and developer performance.

Techniques for Perl programmers to speed up test suite and developer performance.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • It's worth noting that the slides regarding rebuilding tables and avoiding transactions assume that your test suite is constrained to run in a single process. If you parallelize your test suite, transactions are recommended.
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
3,672
On Slideshare
0
From Embeds
0
Number of Embeds
22

Actions

Shares
Downloads
34
Comments
1
Likes
4

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Turbo Charged Test Suites Tuesday, 7 October 2008
  • 2. Curtis “Ovid” Poe • CPAN Author • “Perl Hacks” Contributor • Perl Foundation Member Tuesday, 7 October 2008
  • 3. Test Suite Performance • Computer • Developer Tuesday, 7 October 2008
  • 4. Faster Tests Tuesday, 7 October 2008
  • 5. Considerations • Why speed things up? • Trade-offs Tuesday, 7 October 2008
  • 6. Before You Start • Set a goal • No tests may fail • No non-test output Tuesday, 7 October 2008
  • 7. Databases • Don’t drop them • Don’t fake it with transactions Tuesday, 7 October 2008
  • 8. Databases • Static tables -- country_codes • Dynamic tables -- orders Tuesday, 7 October 2008
  • 9. CREATE TABLE changed_table ( changed_table_id INT(11) NOT NULL AUTO_INCREMENT PRIMARY KEY, table_name VARCHAR(30) NOT NULL, is_static INT NOT NULL DEFAULT 0, inserts INT NOT NULL DEFAULT 0, updates INT NOT NULL DEFAULT 0, deletes INT NOT NULL DEFAULT 0 ) -- then insert table records Tuesday, 7 October 2008
  • 10. foreach my $action (qw/insert update delete/) { $dbh->do(<<quot; END_SQLquot;); CREATE TRIGGER tr_${action}_$table BEFORE $action ON $table FOR EACH ROW UPDATE changed_table SET ${action}s = ${action}s + 1 WHERE table_name = '$table'; END_SQL } Tuesday, 7 October 2008
  • 11. db.disable_foreign_keys() for table in changed_tables() table.truncate() if table.is_static() table.rebuild() db.enable_foreign_keys() Tuesday, 7 October 2008
  • 12. Databases • Before: 80 minutes • After: 22 minutes Tuesday, 7 October 2008
  • 13. Aggregation • Only load things once • Dangers of shared state Tuesday, 7 October 2008
  • 14. use My::YAML::Test; My::YAML::Test->run; # t/api/customer.t # t/api/customer.yml Tuesday, 7 October 2008
  • 15. use My::YAML::Test; foreach (yaml_tests()) { diag “running $_”; My::YAML::Test->run($_); } sub yaml_tests { return @ARGV if @ARGV; # or return all YAML files ... } Tuesday, 7 October 2008
  • 16. YAML Aggregation • Before: 22 minutes • After: 16 minutes Tuesday, 7 October 2008
  • 17. use Test::Aggregate; my $tests = Test::Aggregate->new({ dirs => ‘aggtests’, }); $tests->run; Tuesday, 7 October 2008
  • 18. use Test::Aggregate; my $tests = Test::Aggregate->new({ dirs => 'aggtests', dump => $dump, set_filenames => 1, shuffle => 1, startup => &startup, shutdown => &shutdown, }); Tuesday, 7 October 2008
  • 19. Generic Aggregation • Before: 16 minutes • After: 12 minutes Tuesday, 7 October 2008
  • 20. Better Aggregation • use Test::Class Tuesday, 7 October 2008
  • 21. OK to Fail is OK • POD tests • Perl::Critic • Anything “non-functional” Tuesday, 7 October 2008
  • 22. OK to Fail is OK • Move them to xt/ • But they must be run! Tuesday, 7 October 2008
  • 23. Lessons Learned • Not dropping your database is tricky • Aggregating tests finds bugs in tests • Aggregating tests finds bugs in code • *CORE::GLOBAL:: is evil • Don’t touch UNIVERSAL:: Tuesday, 7 October 2008
  • 24. Faster Programmers Tuesday, 7 October 2008
  • 25. Custom Test Modules Tuesday, 7 October 2008
  • 26. use Test::More tests => 13; use Test::Exception; use Test::XML; use Test::JSON; use Test::Differences; Tuesday, 7 October 2008
  • 27. package Our::Test::More; use Test::Builder::Module; our ( @ISA, @EXPORT ); use Test::More; use Test::Exception; BEGIN { @ISA = qw(Test::Builder::Module); @EXPORT = ( @Test::More::EXPORT, @Test::Exception::EXPORT, ); } 1; Tuesday, 7 October 2008
  • 28. package My::Custom::Tests; use Test::Kit ( 'Test::More', 'Test::XML', 'Test::Differences', '+explain', ); 1; Tuesday, 7 October 2008
  • 29. Popularity Contest Test::More 44461 Test 8937 Test::Exception 1397 Test::Simple 731 Test::Base 316 Test::Builder::Tester 193 Test::NoWarnings 174 Test::Differences 146 Test::MockObject 139 Test::Deep 127 Tuesday, 7 October 2008
  • 30. use Test::Most tests => 4, 'die'; ok 1, 'one is true'; is 2, 2, '... and two is two'; eq_or_diff [3], [4], “... but three ain’t four”; throws_ok { $foo/0 } qr/Illegal division by zero/, 'and no-constant folding with vars'; Tuesday, 7 October 2008
  • 31. Test From Your Editor Tuesday, 7 October 2008
  • 32. “ in your .vimrc function! PerlMappings() noremap K :!perldoc <cword> <bar><bar> perldoc -f <cword><cr> endfunction function! PerlTestMappings() noremap <buffer> ,t :!prove -vl %<CR> endfunction “ remember My::Test::YAML? function! YAMLTestMappings() noremap <buffer> ,t :!prove -vl t/yaml.t :: %<CR> endfunction au! FileType perl :call PerlMappings() au! FileType yaml :call YAMLTestMappings() au! BufRead,BufNewFile *.t :call PerlTestMappings() Tuesday, 7 October 2008
  • 33. vim $(ack -l --perl 'api/v1/episode' t/) map <leader>tb :call RunTestsInBuffers()<cr> function! RunTestsInBuffers() let i = 1 let tests = '' while (i <= bufnr(quot;$quot;)) let filename = bufname(i) if match(filename, '.t$') > -1 let tests = tests . ' quot;' . filename . 'quot;' endif let i = i+1 endwhile if !strlen(tests) echo quot;No tests found in buffersquot; else execute ':!prove ' . tests endif endfunction Tuesday, 7 October 2008
  • 34. Advanced “prove” Tuesday, 7 October 2008
  • 35. Test-Harness $ prove -l t --state=hot,fast,save --timer [20:57:19] t/yamlish-output.........ok 40 ms [20:57:20] t/console................ok 45 ms [20:57:20] t/utils..................ok 48 ms <snip> [20:57:23] t/harness................ok 300 ms [20:57:23] t/process................ok 1020 ms [20:57:24] t/prove..................ok 1017 ms [20:57:25] t/regression.............ok 4217 ms [20:57:29] All tests successful. Files=32, Tests=10326, 10 wallclock secs ( 1.25 usr 0.24 sys + 5.60 cusr 1.83 csys = 8.92 CPU) Result: PASS Tuesday, 7 October 2008
  • 36. Test-Harness $ prove -l t --state=hot,slow,save -j 9 t/callbacks.............. ok t/nofork................. ok t/proverc................ ok <snip> t/yamlish................ ok t/prove.................. ok t/regression............. ok All tests successful. Files=32, Tests=10326, 6 wallclock secs ( 1.34 usr 0.24 sys + 5.63 cusr 1.83 csys = 9.04 CPU) Result: PASS Tuesday, 7 October 2008
  • 37. # slow running tests from App::Prove::State # http://use.perl.org/~Ovid/journal/35831 Generation 18 Number of test programs: 58 Total runtime approximately 17 minutes 35 seconds Five slowest tests: 482.7 seconds -> t/acceptance.t 234.4 seconds -> t/aggregate.t 96.3 seconds -> t/standards/strict.t 66.6 seconds -> t/unit/db/migrations.t 56.7 seconds -> t/unit/piptest/pprove/testdb.t Tuesday, 7 October 2008
  • 38. Devel::CoverX::Covered Tuesday, 7 October 2008
  • 39. Devel::CoverX::Covered • What tests cover this file? • What files are covered by this test? • (Soon) What tests cover this line? • (Soon) What tests cover this subroutine? • ... and more ... Tuesday, 7 October 2008
  • 40. function! PerlMappings() noremap <buffer> ,cv :call Coverage()<cr> endfunction function! PerlTestMappings() noremap <buffer> ,t :!prove -vl --norc %<CR> endfunction function! Coverage() let file = bufname('%') if match(filename, '.t$') > -1 execute '!covered by --test_file=quot;'.file.'quot;' else execute '!covered covering --source_file=quot;'.file.'quot;' end endfunction au! FileType perl :call PerlMappings() au! BufRead,BufNewFile *.t :call PerlTestMappings() Tuesday, 7 October 2008
  • 41. Questions? Tuesday, 7 October 2008