0
TAP::Harness                                 (+ friends)                             by Steve PurkisFriday, 27 January 2012
About Steve...                     • Software developer & manager                     • Sporadic TAP::Harness dev         ...
What we’ll cover...                     • What’s TAP?                     • TAP::Harness Overview                     • Pl...
Unit Tests?                     • Surely you all write tests by now...                     • right?Friday, 27 January 2012
Why Test?                     •    “Kwalitee”                           •   measurable approximation of quality (http://qa...
What’s TAP?                     • “Test Anything Protocol”                     • Grew from Perl’s test output format:     ...
What’s TAP?                Origins                =head1 AUTHORS                Either Tim Bunce or Andreas Koenig, we don...
What’s TAP?                     • Now available in                          many languages!                     •    A few...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                            Example 2         TAP version 1...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                            Example 2         TAP version 1...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Structure         Example 1                                                          Example 2         TAP version 13 ...
TAP Lifecycle                             http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_StandardFriday, 27 Jan...
Why TAP?                     • Advantages:                          •   Simple                          •   Human-readable...
What else is there?                     • Similar output formats:                          •   JUnit report (XML generated...
Up Next...                     • What’s TAP?                     • TAP::Harness Overview                     • Plugins & M...
What’s TAP::Harness?                     • A set of extensible tools that make it easy                          to produce...
Using TAP::Harness?                     •    Youre already using it!                          •   Several years ago Test::...
Using TAP::Harness?                          prove                 from Perl                            prove -l t        ...
Under the hood...                                                                             TAP                         ...
Class diagram notes   Class diagram   Red: plugins are availableseparate plugin systems!   prove & App::Prove             ...
Customising?                     • Write Plugins                     • Sub-classFriday, 27 January 2012
Available Extensions                     • Plugins:                          •   Formatters: JUnit, HTML, Spreadsheet, Tex...
Where’s T::H Going?                     • Where is TAP::Harness going?                           •   Obviously: ongoing ma...
Up Next...                     • What’s TAP?                     • TAP::Harness Overview                     • Plugins & M...
App::Prove plugin...                          package App::Prove::Plugin::Foo;                            # From http://se...
Formatter plugin...         package TAP::Formatter::Text;                            package TAP::Formatter::Text::Session...
SourceHandler plugin...    package TAP::Parser::SourceHandler::Web;                                                       ...
Module::Build                • Using custom sources & formatters                          my $build = Module::Build->new  ...
Up Next...                     • What’s TAP?                     • TAP::Harness Overview                     • Plugins & M...
TAP::Formatter::HTML                          Interpret your test results with HTML                               Based on...
Friday, 27 January 2012
Friday, 27 January 2012
Common uses                     • Integration tests                     • Nightly smokes                     • Being lazy ...
Demo                     • Download from CPAN                     • runprove -m -Q -P HTML=outfile:output.htmlFriday, 27 J...
Up Next...                     • What’s TAP?                     • TAP::Harness Overview                     • Plugins & M...
Meet Hudson                     • Free Java CI                           servlet                          - http://hudson-...
JUnit XML?                     • Lots of build tools unit test frameworks                          use it!                ...
Demo                     • 3 build tasks:                      • TAP::Harness::JUnit                      • TAP::Formatter...
Hudson config                TAP::Harness::JUnit:                Execute:                  cd /Users/spurkis/dev/Perl-acces...
That’s all...                     • What’s TAP?                     • TAP::Harness Overview                     • Plugins ...
See Alsohttp://testanything.org/http://search.cpan.org/dist/Test-Harness/http://search.cpan.org/dist/Test-Harness/lib/TAP/...
Questions                          Test::Harness Developers                            tapx-dev@hexten.net                ...
Ideas: Future Topics                • Extending TAP: adding a custom                          TAP::Parser::Result         ...
Upcoming SlideShare
Loading in...5
×

TAP-Harness + friends

7,669

Published on

TAP::Harness implements the Test Anything Protocol. This talk covers:

* What’s TAP?
* TAP::Harness Overview
* Plugins & Module::Build integration
* Common use cases:
* TAP::Formatter::HTML
* TAP & Hudson CI

Presented at Ottawa.pm, Jan 26 2012.

Published in: Technology, Business
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
7,669
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
18
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Transcript of "TAP-Harness + friends"

  1. 1. TAP::Harness (+ friends) by Steve PurkisFriday, 27 January 2012
  2. 2. About Steve... • Software developer & manager • Sporadic TAP::Harness dev • did a lot of refactoring • redesigned the plugin framework w/David Wheeler • wrote the HTML formatterFriday, 27 January 2012
  3. 3. What we’ll cover... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  4. 4. Unit Tests? • Surely you all write tests by now... • right?Friday, 27 January 2012
  5. 5. Why Test? • “Kwalitee” • measurable approximation of quality (http://qa.perl.org/phalanx/kwalitee.html) • prevent regressions • gives others confidence what you’ve written is working • developer aid • break solutions down into chunks • test your assumptions • gives you confidence each chunk is working (so you can forget about it and move onto the next chunk!) • team co-ordination • gives your team confidence what you’re committing is working • right up there with version control, bug tracking, dev environments & build toolsFriday, 27 January 2012
  6. 6. What’s TAP? • “Test Anything Protocol” • Grew from Perl’s test output format: 1..7 ok 1 - use Test::Approx; ok 2 - equal strings ok 3 - equal numbers ok 4 - completely different strings ok 5 - similar strings, 20% threshold ok 6 - similar strings, 10% threshold ok 7 - big strings, default threshold See: http://testanything.org/Friday, 27 January 2012
  7. 7. What’s TAP? Origins =head1 AUTHORS Either Tim Bunce or Andreas Koenig, we dont know. What we know for sure is, that it was inspired by Larry Walls TEST script that came with perl distributions for ages. Numerous anonymous contributors exist. Andreas Koenig held the torch for many years, and then Michael G Schwern. - http://cpansearch.perl.org/src/PETDANCE/Test-Harness-2.56/lib/Test/Harness.pm For more details: http://testanything.org/wiki/index.php/TAP_History See: http://testanything.org/Friday, 27 January 2012
  8. 8. What’s TAP? • Now available in many languages! • A few projects I find interesting: • node.tap • MyTAP • pgTAP • PHPUnitFriday, 27 January 2012
  9. 9. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  10. 10. TAP Structure Example 1 Example 2 TAP version 13 Optional TAP version, must come 1st 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  11. 11. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  12. 12. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl Comments start with # not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  13. 13. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  14. 14. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 Test Plan # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  15. 15. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  16. 16. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 test status & identifier # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  17. 17. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  18. 18. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test test description # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  19. 19. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  20. 20. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! non-TAP: ignored by parser # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  21. 21. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  22. 22. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented TODO directive # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  23. 23. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  24. 24. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! SKIP directive ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  25. 25. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  26. 26. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info custom directive # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  27. 27. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  28. 28. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore stop testing & parsing Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  29. 29. TAP Structure Example 1 Example 2 TAP version 13 1..3 # Generated from test.pl not ok 1 - get remote # TODO not yet implemented 1..13 # Failed (TODO) test get remote ok 1 # at test.pl line 18. ok 2 - a good test not ok 2 - a bad test not ok 3 - a bad test # Failed test a bad test # Failed test a bad test # at test.pl line 11. # at test.pl line 11. Bail out! I cant take it anymore Oops, I printed to STDOUT! ok 3 - a good test Oops, I printed to STDERR! # Looks like you failed 1 test of 3 run. not ok 4 - get remote # TODO not yet implemented # Failed (TODO) test get remote # at test.pl line 18. ok 5 # skip to the loo! ok 6 - open connection to server # MyDirective with more info # Looks like you planned 13 tests but ran 6. # Looks like you failed 1 test of 6 run. http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_Standard#GrammarFriday, 27 January 2012
  30. 30. TAP Lifecycle http://testanything.org/wiki/index.php/TAP_at_IETF:_Draft_StandardFriday, 27 January 2012
  31. 31. Why TAP? • Advantages: • Simple • Human-readable • Easy to generate & parse • Widely used by the Perl communityFriday, 27 January 2012
  32. 32. What else is there? • Similar output formats: • JUnit report (XML generated by most xUnit libs & other tools like googletest) • SubUnit - probably many more that I don’t know of... • Complementary tools: • A QA team ;-) • CI & Build tools (CPAN, Smolder, Hudson, buildbot, etc) - probably many more that I don’t know of...Friday, 27 January 2012
  33. 33. Up Next... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  34. 34. What’s TAP::Harness? • A set of extensible tools that make it easy to produce & consume TAP - prove cmdline utility - Module::Build & ExtUtils::MakeMaker hooks - Parser, Source Handlers & Formatters - Plugin frameworkFriday, 27 January 2012
  35. 35. Using TAP::Harness? • Youre already using it! • Several years ago Test::Harness became a compatibility wrapper around TAP::Harness • The old Test::Harness has been retired: it was getting hard to maintain & extend • Thanks to these fine folk: Michael G. Schwern Andy Lester Ovid Andy Armstrong Eric Wilhelm & more!Friday, 27 January 2012
  36. 36. Using TAP::Harness? prove from Perl prove -l t use TAP::Harness; my @tests = glob( t/*.t ); Module::Build my $harness = TAP::Harness->new({ formatter_class => perl Build.PL TAP::Formatter::JUnit, ./Build test merge => 1 }); $harness->runtests( @tests ); ExtUtils::MakeMaker perl Makefile.PL make testFriday, 27 January 2012
  37. 37. Under the hood... TAP ... prove Formatter Console File Module::Build Harness Color plugins Backwards compatibility EU::MakeMaker Parser Parser SourceHandler Aggregator Test::Harness Perl Executable Lots of hidden complexity here! File Handle RawTAP pluginsFriday, 27 January 2012
  38. 38. Class diagram notes Class diagram Red: plugins are availableseparate plugin systems! prove & App::Prove TAP::Harness cmdline utility to run tests & display results main interface to TAP::* parses args loads formatter & sourcehandler plugins loads App::Prove plugins creates parsers kicks off TAP::Harness aggregates results formats data Parser Parser::Aggregator SourceHandlers 1 parser per stream (ie: test script) collate results across many TAP streams what actually run your tests has lots of helpers, not shown here because: 1 parser per stream! or open a file... opens TAP stream stream input into the parser parses input as it arrives sends events to the aggregator Friday, 27 January 2012
  39. 39. Customising? • Write Plugins • Sub-classFriday, 27 January 2012
  40. 40. Available Extensions • Plugins: • Formatters: JUnit, HTML, Spreadsheet, TextMate, TeamCity • Sources: pgTAP, MyTAP, PHP • Prove: Idempotent • Known Subclasses • TAP::Harness::JUnit • TAP::Harness::Multiple • Other tools • TAP::Filter • All as of Jan 2012...Friday, 27 January 2012
  41. 41. Where’s T::H Going? • Where is TAP::Harness going? • Obviously: ongoing maintenance releases • I’d like to see: • Multiple Formatters • Arbitrary plugin args • The rest is up to you! • write plugins! • find bugs! • send patches! • get involved.Friday, 27 January 2012
  42. 42. Up Next... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  43. 43. App::Prove plugin... package App::Prove::Plugin::Foo; # From http://search.cpan.org/dist/Test-Harness/lib/App/Prove.pm#Sample_Plugin # Sample plugin, try running with: # prove -PFoo=bar -r -j3 # prove -PFoo -Q # prove -PFoo=bar,My::Formatter use strict; use warnings; sub load { my ($class, $p) = @_; my @args = @{ $p->{args} }; my $app = $p->{app_prove}; print "loading plugin: $class, args: ", join(, , @args ), "n"; # turn on verbosity $app->verbose( 1 ); # set the formatter? $app->formatter( $args[1] ) if @args > 1; # print some of App::Proves state: for my $attr (qw( jobs quiet really_quiet recurse verbose )) { my $val = $app->$attr; $val = undef unless defined( $val ); print "$attr: $valn"; } return 1; } 1;Friday, 27 January 2012
  44. 44. Formatter plugin... package TAP::Formatter::Text; package TAP::Formatter::Text::Session; use strict; use strict; use warnings; use warnings; use TAP::Formatter::Text::Session; sub new { my ( $class, $args ) = @_; sub new { my $self = bless {%$args}, $class; my ( $class, $args ) = @_; } my $self = bless {}, $class; } sub header { my ( $self ) = @_; sub prepare { print "Test start: $self->{name}n"; my ( $self ) = @_; } print "getting prepared!n"; } sub result { my ( $self, $result ) = @_; sub open_test { my ( $self, $test, $parser ) = @_; my $str = ; my $session = TAP::Formatter::Text::Session->new({ if ($result->is_test) { name => $test $str = $result->is_ok ? + : !!; }); } elsif ($result->is_comment) { $session->header; $str = #; return $session; } } if ($result->has_directive) { sub summary { $str .= ->; my ( $self, $test, $parser ) = @_; } print "Text plugin summary!n"; print " $strt" . $result->as_string . "n"; } } # shouldnt really need to create these... sub close_test { for (qw( directives verbosity timer failures comments my ( $self, $test, $parser ) = @_; errors stdout color show_count normalize )) { print "Session closed!n"; eval "sub $_ {}"; } } 1; 1;Friday, 27 January 2012
  45. 45. SourceHandler plugin... package TAP::Parser::SourceHandler::Web; sub make_iterator { use strict; my ($class, $source) = @_; use warnings; $class->_croak($source->raw must be a scalar ref) use File::Slurp qw( read_file ); unless $source->meta->{is_scalar}; use LWP::UserAgent; use TAP::Parser::IteratorFactory; my $file = ${ $source->raw }; use TAP::Parser::Iterator::Array; my $uri = read_file( $file ); chomp $uri; use base qw( TAP::Parser::SourceHandler ); my $ua = LWP::UserAgent->new; TAP::Parser::IteratorFactory->register_handler( __PACKAGE__ ); $ua->agent("TAP-Parser-SourceHandler-Web/0.1 "); my $req = HTTP::Request->new(GET => $uri); sub can_handle { my $res = $ua->request($req); my ( $class, $src ) = @_; my $meta = $src->meta; # Check the outcome of the response my $config = $src->config_for( $class ); $class->_croak("couldnt GET $uri: " . $res->status_line) unless $res->is_success; return 0 unless $meta->{is_file}; if (my $ext = $meta->{file}->{ext}) { my $tap = [ split(/n/, $res->content) ]; return 0.99 if $ext =~ /.ur[il]$/i; my $iterator = TAP::Parser::Iterator::Array->new($tap); } return $iterator; return 0; } } 1;Friday, 27 January 2012
  46. 46. Module::Build • Using custom sources & formatters my $build = Module::Build->new ( ... build_requires => { TAP::Harness => 3.22, Test::More => 0.01, Module::Build => 0.30, }, # TAP::Harness configuration test_file_exts => [qw(.t .tap .txt )], use_tap_harness => 1, tap_harness_args => { sources => { Web => { }, File => { extensions => [.tap, .txt], }, }, color => 1, formatter_class => TAP::Formatter::Text, }, );Friday, 27 January 2012
  47. 47. Up Next... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  48. 48. TAP::Formatter::HTML Interpret your test results with HTML Based on Yuval Kogman’s Test::TAP::HTMLMatrixFriday, 27 January 2012
  49. 49. Friday, 27 January 2012
  50. 50. Friday, 27 January 2012
  51. 51. Common uses • Integration tests • Nightly smokes • Being lazy • avoid scrolling through tonnes of outputFriday, 27 January 2012
  52. 52. Demo • Download from CPAN • runprove -m -Q -P HTML=outfile:output.htmlFriday, 27 January 2012
  53. 53. Up Next... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  54. 54. Meet Hudson • Free Java CI servlet - http://hudson-ci.org/ • Consumes JUnit XML reports • Provides useful stats & reportsFriday, 27 January 2012
  55. 55. JUnit XML? • Lots of build tools unit test frameworks use it! • Current options for TAP::Harness: • TAP::Harness::JUnit • TAP::Formatter::JUnit (alpha)Friday, 27 January 2012
  56. 56. Demo • 3 build tasks: • TAP::Harness::JUnit • TAP::Formatter::JUnit • TAP::Formatter::HTMLFriday, 27 January 2012
  57. 57. Hudson config TAP::Harness::JUnit: Execute: cd /Users/spurkis/dev/Perl-accessors # For DEMO ONLY: # avoid running perl here; defaults to junit_output.xml # so just move the file to Hudsons workspace prove -l --harness TAP::Harness::JUnit || echo 1 mv junit_output.xml "$WORKSPACE/test-report-$BUILD_ID.xml" Publish JUnit test report: test-report*.xml TAP::Formatter::JUnit: Execute: cd /Users/spurkis/dev/Perl-accessors prove -l --formatter TAP::Formatter::JUnit > "$WORKSPACE/test-report-$BUILD_ID.xml" Publish JUnit test report: test-report*.xml TAP::Formatter::HTML: Prep: cp /usr/local/perl-5.10.1/lib/site_perl/5.10.1/TAP/Formatter/HTML/*.[cj]s* $HUDSON_HOME/userContent Execute: cd /Users/spurkis/dev/Perl-accessors prove -lm -P HTML=outfile:$WORKSPACE/test-report-$BUILD_ID.html,force_inline_css:0,css_uri:/userContent/default_page.css,css_uri:/ userContent/default_report.css,js_uri:/userContent/jquery-1.4.2.min.js,js_uri:/userContent/jquery.tablesorter-2.0.3.min.js,js_uri:/ userContent/default_report.js Archive artifacts: test-report*.htmlFriday, 27 January 2012
  58. 58. That’s all... • What’s TAP? • TAP::Harness Overview • Plugins & Module::Build integration • TAP::Formatter::HTML • TAP & Hudson CIFriday, 27 January 2012
  59. 59. See Alsohttp://testanything.org/http://search.cpan.org/dist/Test-Harness/http://search.cpan.org/dist/Test-Harness/lib/TAP/Harness/Beyond.podhttp://search.cpan.org/dist/TAP-Formatter-HTML/http://search.cpan.org/dist/TAP-Formatter-JUnit/http://search.cpan.org/dist/TAP-Harness-JUnit/http://search.cpan.org/dist/Smolder/http://hudson-ci.org/Friday, 27 January 2012
  60. 60. Questions Test::Harness Developers tapx-dev@hexten.net patches welcome! Steve Purkis spurkis@cpan.orgFriday, 27 January 2012
  61. 61. Ideas: Future Topics • Extending TAP: adding a custom TAP::Parser::Result • Creating a new TAP::Parser::Grammar • Map of TAP::Parser internalsFriday, 27 January 2012
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×