TAP-Harness + friends
Upcoming SlideShare
Loading in...5
×

Like this? Share it with your network

Share

TAP-Harness + friends

  • 7,333 views
Uploaded on

TAP::Harness implements the Test Anything Protocol. This talk covers: ...

TAP::Harness implements the Test Anything Protocol. This talk covers:

* What’s TAP?
* TAP::Harness Overview
* Plugins & Module::Build integration
* Common use cases:
* TAP::Formatter::HTML
* TAP & Hudson CI

Presented at Ottawa.pm, Jan 26 2012.

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
7,333
On Slideshare
7,331
From Embeds
2
Number of Embeds
2

Actions

Shares
Downloads
14
Comments
0
Likes
3

Embeds 2

http://a0.twimg.com 1
http://www.redditmedia.com 1

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. TAP::Harness (+ friends) by Steve PurkisFriday, 27 January 2012
  • 2. About Steve... • Software developer & manager • Sporadic TAP::Harness dev • did a lot of refactoring • redesigned the plugin framework w/David Wheeler • wrote the HTML formatterFriday, 27 January 2012
  • 3. What we’ll cover... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  • 4. Unit Tests? • Surely you all write tests by now... • right?Friday, 27 January 2012
  • 5. Why Test? • “Kwalitee” • measurable approximation of quality (http://qa.perl.org/phalanx/kwalitee.html) • prevent regressions • gives others confidence what you’ve written is working • developer aid • break solutions down into chunks • test your assumptions • gives you confidence each chunk is working (so you can forget about it and move onto the next chunk!) • team co-ordination • gives your team confidence what you’re committing is working • right up there with version control, bug tracking, dev environments & build toolsFriday, 27 January 2012
  • 6. What’s TAP? • “Test Anything Protocol” • Grew from Perl’s test output format: 1..7 ok 1 - use Test::Approx; ok 2 - equal strings ok 3 - equal numbers ok 4 - completely different strings ok 5 - similar strings, 20% threshold ok 6 - similar strings, 10% threshold ok 7 - big strings, default threshold See: http://testanything.org/Friday, 27 January 2012
  • 7. What’s TAP? Origins =head1 AUTHORS Either Tim Bunce or Andreas Koenig, we dont know. What we know for sure is, that it was inspired by Larry Walls TEST script that came with perl distributions for ages. Numerous anonymous contributors exist. Andreas Koenig held the torch for many years, and then Michael G Schwern. - http://cpansearch.perl.org/src/PETDANCE/Test-Harness-2.56/lib/Test/Harness.pm For more details: http://testanything.org/wiki/index.php/TAP_History See: http://testanything.org/Friday, 27 January 2012
  • 8. What’s TAP? • Now available in many languages! • A few projects I find interesting: • node.tap • MyTAP • pgTAP • PHPUnitFriday, 27 January 2012
  • 9. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 10. TAP Structure Example 1 Example 2 TAP version 13 Optional TAP version, must come 1st 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 11. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 12. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl Comments start with # not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 13. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 14. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 Test Plan # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 15. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 16. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 test status & identifier # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 17. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 18. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test test description # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 19. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 20. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! non-TAP: ignored by parser # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 21. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 22. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented TODO directive # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 23. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 24. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! SKIP directive ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 25. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 26. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info custom directive # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 27. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 28. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore stop testing & parsing Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 29. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  • 30. TAP Lifecycle http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_StandardFriday, 27 January 2012
  • 31. Why TAP? • Advantages: • Simple • Human-readable • Easy to generate & parse • Widely used by the Perl communityFriday, 27 January 2012
  • 32. What else is there? • Similar output formats: • JUnit report (XML generated by most xUnit libs & other tools like googletest) • SubUnit - probably many more that I don’t know of... • Complementary tools: • A QA team ;-) • CI & Build tools (CPAN, Smolder, Hudson, buildbot, etc) - probably many more that I don’t know of...Friday, 27 January 2012
  • 33. Up Next... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  • 34. What’s TAP::Harness? • A set of extensible tools that make it easy to produce & consume TAP - prove cmdline utility - Module::Build & ExtUtils::MakeMaker hooks - Parser, Source Handlers & Formatters - Plugin frameworkFriday, 27 January 2012
  • 35. Using TAP::Harness? • Youre already using it! • Several years ago Test::Harness became a compatibility wrapper around TAP::Harness • The old Test::Harness has been retired: it was getting hard to maintain & extend • Thanks to these fine folk: Michael G. Schwern Andy Lester Ovid Andy Armstrong Eric Wilhelm & more!Friday, 27 January 2012
  • 36. Using TAP::Harness? prove from Perl prove -l t use TAP::Harness; my @tests = glob( t/*.t ); Module::Build my $harness = TAP::Harness->new({ formatter_class => perl Build.PL TAP::Formatter::JUnit, ./Build test merge => 1 }); $harness->runtests( @tests ); ExtUtils::MakeMaker perl Makefile.PL make testFriday, 27 January 2012
  • 37. Under the hood... TAP ... prove Formatter Console File Module::Build Harness Color plugins Backwards compatibility EU::MakeMaker Parser Parser SourceHandler Aggregator Test::Harness Perl Executable Lots of hidden complexity here! File Handle RawTAP pluginsFriday, 27 January 2012
  • 38. Class diagram notes Class diagram Red: plugins are availableseparate plugin systems! prove & App::Prove TAP::Harness cmdline utility to run tests & display results main interface to TAP::* parses args loads formatter & sourcehandler plugins loads App::Prove plugins creates parsers kicks off TAP::Harness aggregates results formats data Parser Parser::Aggregator SourceHandlers 1 parser per stream (ie: test script) collate results across many TAP streams what actually run your tests has lots of helpers, not shown here because: 1 parser per stream! or open a file... opens TAP stream stream input into the parser parses input as it arrives sends events to the aggregator Friday, 27 January 2012
  • 39. Customising? • Write Plugins • Sub-classFriday, 27 January 2012
  • 40. Available Extensions • Plugins: • Formatters: JUnit, HTML, Spreadsheet, TextMate, TeamCity • Sources: pgTAP, MyTAP, PHP • Prove: Idempotent • Known Subclasses • TAP::Harness::JUnit • TAP::Harness::Multiple • Other tools • TAP::Filter • All as of Jan 2012...Friday, 27 January 2012
  • 41. Where’s T::H Going? • Where is TAP::Harness going? • Obviously: ongoing maintenance releases • I’d like to see: • Multiple Formatters • Arbitrary plugin args • The rest is up to you! • write plugins! • find bugs! • send patches! • get involved.Friday, 27 January 2012
  • 42. Up Next... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  • 43. App::Prove plugin... package App::Prove::Plugin::Foo; # From http://search.cpan.org/dist/Test-Harness/lib/App/Prove.pm#Sample_Plugin # Sample plugin, try running with: # prove -PFoo=bar -r -j3 # prove -PFoo -Q # prove -PFoo=bar,My::Formatter use strict; use warnings; sub load { my ($class, $p) = @_; my @args = @{ $p->{args} }; my $app = $p->{app_prove}; print "loading plugin: $class, args: ", join(, , @args ), "n"; # turn on verbosity $app->verbose( 1 ); # set the formatter? $app->formatter( $args[1] ) if @args > 1; # print some of App::Proves state: for my $attr (qw( jobs quiet really_quiet recurse verbose )) { my $val = $app->$attr; $val = undef unless defined( $val ); print "$attr: $valn"; } return 1; } 1;Friday, 27 January 2012
  • 44. Formatter plugin... package TAP::Formatter::Text; package TAP::Formatter::Text::Session; use strict; use strict; use warnings; use warnings; use TAP::Formatter::Text::Session; sub new { my ( $class, $args ) = @_; sub new { my $self = bless {%$args}, $class; my ( $class, $args ) = @_; } my $self = bless {}, $class; } sub header { my ( $self ) = @_; sub prepare { print "Test start: $self->{name}n"; my ( $self ) = @_; } print "getting prepared!n"; } sub result { my ( $self, $result ) = @_; sub open_test { my ( $self, $test, $parser ) = @_; my $str = ; my $session = TAP::Formatter::Text::Session->new({ if ($result->is_test) { name => $test $str = $result->is_ok ? + : !!; }); } elsif ($result->is_comment) { $session->header; $str = #; return $session; } } if ($result->has_directive) { sub summary { $str .= ->; my ( $self, $test, $parser ) = @_; } print "Text plugin summary!n"; print " $strt" . $result->as_string . "n"; } } # shouldnt really need to create these... sub close_test { for (qw( directives verbosity timer failures comments my ( $self, $test, $parser ) = @_; errors stdout color show_count normalize )) { print "Session closed!n"; eval "sub $_ {}"; } } 1; 1;Friday, 27 January 2012
  • 45. SourceHandler plugin... package TAP::Parser::SourceHandler::Web; sub make_iterator { use strict; my ($class, $source) = @_; use warnings; $class->_croak($source->raw must be a scalar ref) use File::Slurp qw( read_file ); unless $source->meta->{is_scalar}; use LWP::UserAgent; use TAP::Parser::IteratorFactory; my $file = ${ $source->raw }; use TAP::Parser::Iterator::Array; my $uri = read_file( $file ); chomp $uri; use base qw( TAP::Parser::SourceHandler ); my $ua = LWP::UserAgent->new; TAP::Parser::IteratorFactory->register_handler( __PACKAGE__ ); $ua->agent("TAP-Parser-SourceHandler-Web/0.1 "); my $req = HTTP::Request->new(GET => $uri); sub can_handle { my $res = $ua->request($req); my ( $class, $src ) = @_; my $meta = $src->meta; # Check the outcome of the response my $config = $src->config_for( $class ); $class->_croak("couldnt GET $uri: " . $res->status_line) unless $res->is_success; return 0 unless $meta->{is_file}; if (my $ext = $meta->{file}->{ext}) { my $tap = [ split(/n/, $res->content) ]; return 0.99 if $ext =~ /.ur[il]$/i; my $iterator = TAP::Parser::Iterator::Array->new($tap); } return $iterator; return 0; } } 1;Friday, 27 January 2012
  • 46. Module::Build • Using custom sources & formatters my $build = Module::Build->new ( ... build_requires => { TAP::Harness => 3.22, Test::More => 0.01, Module::Build => 0.30, }, # TAP::Harness configuration test_file_exts => [qw(.t .tap .txt )], use_tap_harness => 1, tap_harness_args => { sources => { Web => { }, File => { extensions => [.tap, .txt], }, }, color => 1, formatter_class => TAP::Formatter::Text, }, );Friday, 27 January 2012
  • 47. Up Next... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  • 48. TAP::Formatter::HTML Interpret your test results with HTML Based on Yuval Kogman’s Test::TAP::HTMLMatrixFriday, 27 January 2012
  • 49. Friday, 27 January 2012
  • 50. Friday, 27 January 2012
  • 51. Common uses • Integration tests • Nightly smokes • Being lazy • avoid scrolling through tonnes of outputFriday, 27 January 2012
  • 52. Demo • Download from CPAN • runprove -m -Q -P HTML=outfile:output.htmlFriday, 27 January 2012
  • 53. Up Next... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  • 54. Meet Hudson • Free Java CI servlet - http://hudson-ci.org/ • Consumes JUnit XML reports • Provides useful stats & reportsFriday, 27 January 2012
  • 55. JUnit XML? • Lots of build tools unit test frameworks use it! • Current options for TAP::Harness: • TAP::Harness::JUnit • TAP::Formatter::JUnit (alpha)Friday, 27 January 2012
  • 56. Demo • 3 build tasks: • TAP::Harness::JUnit • TAP::Formatter::JUnit • TAP::Formatter::HTMLFriday, 27 January 2012
  • 57. Hudson config TAP::Harness::JUnit: Execute: cd /Users/spurkis/dev/Perl-accessors # For DEMO ONLY: # avoid running perl here; defaults to junit_output.xml # so just move the file to Hudsons workspace prove -l --harness TAP::Harness::JUnit || echo 1 mv junit_output.xml "$WORKSPACE/test-report-$BUILD_ID.xml" Publish JUnit test report: test-report*.xml TAP::Formatter::JUnit: Execute: cd /Users/spurkis/dev/Perl-accessors prove -l --formatter TAP::Formatter::JUnit > "$WORKSPACE/test-report-$BUILD_ID.xml" Publish JUnit test report: test-report*.xml TAP::Formatter::HTML: Prep: cp /usr/local/perl-5.10.1/lib/site_perl/5.10.1/TAP/Formatter/HTML/*.[cj]s* $HUDSON_HOME/userContent Execute: cd /Users/spurkis/dev/Perl-accessors prove -lm -P HTML=outfile:$WORKSPACE/test-report-$BUILD_ID.html,force_inline_css:0,css_uri:/userContent/default_page.css,css_uri:/ userContent/default_report.css,js_uri:/userContent/jquery-1.4.2.min.js,js_uri:/userContent/jquery.tablesorter-2.0.3.min.js,js_uri:/ userContent/default_report.js Archive artifacts: test-report*.htmlFriday, 27 January 2012
  • 58. That’s all... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  • 59. See Alsohttp://testanything.org/http://search.cpan.org/dist/Test-Harness/http://search.cpan.org/dist/Test-Harness/lib/TAP/Harness/Beyond.podhttp://search.cpan.org/dist/TAP-Formatter-HTML/http://search.cpan.org/dist/TAP-Formatter-JUnit/http://search.cpan.org/dist/TAP-Harness-JUnit/http://search.cpan.org/dist/Smolder/http://hudson-ci.org/Friday, 27 January 2012
  • 60. Questions Test::Harness Developers tapx-dev@hexten.net patches welcome! Steve Purkis spurkis@cpan.orgFriday, 27 January 2012
  • 61. Ideas: Future Topics • Extending TAP: adding a custom TAP::Parser::Result • Creating a new TAP::Parser::Grammar • Map of TAP::Parser internalsFriday, 27 January 2012