TAP In Depth

TAP in Depth Michael Peters Plus Three, LP YAPC::NA 10 Pittsburgh, PA

TAP Test Anything Protocol ● And we do mean *anything* ● Mainly used for regression testing code ● Could be used for systems monitoring, performance testing, etc ● 1987, current version 13 ● Patient Zero was Perl, but the infection has since spread to lots of other languages ● Ongoing work to make an IETF standard

Definitions Test ● or an "assertion" (ok/not ok) ● smallest granule of measurement ● did it do what we wanted Stream ● or a "set" ● a group of tests ● usually output of running a single test file, but is not necessarily tied to a file or any specific output. Test Suite ● A group of TAP streams

Definitions Harness ● A process which collects a group of TAP streams Plan ● number of tests that are expected to be run in a given stream Directive ● Changes the meaning of a given test ● TODO/SKIP Reason ● Explains the "why" of a directive

Definitions Description ● What is the test asserting? What functionality is being tested? Producer ● Anything that generates TAP Consumer ● Anything that processes TAP ● Could just blindly echo it ● Usually parses it

Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed

Document Structure Line Oriented Protocol ● Everything is separated by a new-line (CRLF) ● Not everything fits on 1 line ● Everything starts and ends on it's own line Encoding ● Everything is UTF-8 ● Unless the Producer/Consumer agree otherwise 1 Version, 1* plans, 0+ tests, 0* comments, 0* ignored elements

Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed

Pass vs Fail Tests ● "ok" == pass ● "not ok" == fail ● Yay! binary!

Pass vs Fail Tests ● "ok" == pass ● "not ok" == fail ● If only it were that simple ● Directives! Normal TODO SKIP ok pass * pass not ok pass fail ** * a passing TODO might indicate that a test shouldn't be a TODO anymore ** SKIP means the test wasn't run, so it shouldn't be able to fail

Pass vs Fail Streams ● Can fail if any of the following are true (1) Any tests in this stream failed (2) Tests run did not agree with the plan (3) Other factors ● process exit codes (non-zero) ● HTTP status code ● Passed only if none of the above are true and (1) There was at least 1 passing test - or - (2) There was a "skip all" ● 1..0 # SKIP some module not installed

Pass vs Fail Test Suites ● Fails if there are any streams that failed ● Passes if nothing failed and there's at least 1 stream ● Pass or fail at the suite level is boolean ● Tools can show percentages ● But even a 99.999% pass is still a fail

Producing Perl Modules ● Test::More (classic and for good reasons)

Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder

Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder ● Test::Most

Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder ● Test::Most ● Test::Class

Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder ● Test::Most ● Test::Class ● Test::LongString, Test::WWW::Mechanize, Test::Exception, Test::File::Contents, Test::HTML::Content, Test::MockObject, Test::Pod, Test::Warn, Test::WWW::Selenium, Test::NoWarnings, Test::Output, Test::Number::Delta, Test::Perl::Critic

Producing Other Languages ● Javascript →Test.Simple ● C/C++ →libtap, libtap++ ● PHP → phpt, PHPUnit, SimpleTest or Apache-Test ● Java →JTap ● Ruby →taptap, Bacon ● Python → PyTAP ● Also Haskell, Erlang, Forth, C#, Shell, Postgres

Consuming Perl ● Test::Harness ● prove ● TAP::Harness ● TAP::Harness::Archive ● GUI Test Runner ● Smolder

Comming Soon to a terminal near you Nested TAP ● Group related blocks of TAP together in the same stream ● Mimic xUnit style ● Aggregate TAP streams

Comming Soon to a terminal near you Nested TAP 1..3 1..3 ok 1 - guests arrived on time ok 2 - table was available ok 3 - order correct ok 1 - guests served 1..2 ok 1 - chicken is fried not ok - steak is medium well not ok 1 - food prepared properly 1..2 ok 1 # SKIP guests did not come back ok 2 # SKIP guests did not come back ok 1 # SKIP guests did not come back

Comming Soon to a terminal near you Metadata ● Everything is crammed into the descriptions or comments not ok - fooey, Bug #123 # Failed test 'fooey, Bug #123' # at barry.pl line 20. # got: foo bar # expected: bar foo ● If you want to do anything useful with the information in that message you need to parse it. Parsing error messages is evil

Comming Soon to a terminal near you Metadata ●Data should be structured (YAML-ish) not ok - fooey --- file : barry.pl line : 20 want : foo bar have : bar foo display : > Failed test 'fooey' at barry.pl line 20. want: "foo bar" have: "bar foo" Bug-ID : 123 ...

Comming Soon to a terminal near you Metadata ● Opens up a whole world of possibilities ● Visual diffs on want/have ● Tagging ● Linking between systems (source code, specs, issue tracking) ● Consumers can show or hide what ever they want or understand. ● Unknown uses (standardization vs growth)

Usage Scenarios Typical Perl tests

Usage Scenarios Automated Javascript Testing * example from Michael Schwern - http://use.perl.org/~schwern/journal/39092

Usage Scenarios Automated Javascript Testing

TAP in Depth Michael Peters Plus Three, LP Thanks! Questions?

TAP In Depth

by Michael Peters on Jun 23, 2009

Accessibility

Categories

Upload Details

Usage Rights

Statistics

TAP In Depth — Presentation Transcript