TAP in Depth
Michael Peters
Plus Three, LP
YAPC::NA 10
Pittsburgh, PA

TAP
Test Anything Protocol
● And we do mean *anything*
● Mainly used for regression testing code
● Could be used for systems monitoring,
performance testing, etc
● 1987, current version 13
● Patient Zero was Perl, but the infection has since
spread to lots of other languages
● Ongoing work to make an IETF standard

Definitions
Test
● or an "assertion" (ok/not ok)
● smallest granule of measurement
● did it do what we wanted
Stream
● or a "set"
● a group of tests
● usually output of running a single test file, but is not
necessarily tied to a file or any specific output.
Test Suite
● A group of TAP streams

Definitions
Harness
● A process which collects a group of TAP streams
Plan
● number of tests that are expected to be run in a given
stream
Directive
● Changes the meaning of a given test
● TODO/SKIP
Reason
● Explains the "why" of a directive

Definitions
Description
● What is the test asserting? What functionality is
being tested?
Producer
● Anything that generates TAP
Consumer
● Anything that processes TAP
● Could just blindly echo it
● Usually parses it

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
Line Oriented Protocol
● Everything is separated by a new-line (CRLF)
● Not everything fits on 1 line
● Everything starts and ends on it's own line
Encoding
● Everything is UTF-8
● Unless the Producer/Consumer agree otherwise
1 Version, 1* plans, 0+ tests,
0* comments, 0* ignored elements

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

Pass vs Fail
Tests
● "ok" == pass
● "not ok" == fail
● Yay! binary!

Pass vs Fail
Tests
● "ok" == pass
● "not ok" == fail
● If only it were that simple
● Directives!
Normal TODO SKIP
ok pass * pass
not ok pass fail **
* a passing TODO might indicate that a test shouldn't be a TODO anymore
** SKIP means the test wasn't run, so it shouldn't be able to fail

Pass vs Fail
Streams
● Can fail if any of the following are true
(1) Any tests in this stream failed
(2) Tests run did not agree with the plan
(3) Other factors
● process exit codes (non-zero)
● HTTP status code
● Passed only if none of the above are true and
(1) There was at least 1 passing test - or -
(2) There was a "skip all"
● 1..0 # SKIP some module not installed

Pass vs Fail
Test Suites
● Fails if there are any streams that failed
● Passes if nothing failed and there's at least 1 stream
● Pass or fail at the suite level is boolean
● Tools can show percentages
● But even a 99.999% pass is still a fail

Producing
Perl Modules
● Test::More (classic and for good reasons)

Producing
Perl Modules
● Test::More (classic and for good reasons)
● Test::Builder

Producing
Perl Modules
● Test::More (classic and for good reasons)
● Test::Builder
● Test::Most

Producing
Perl Modules
● Test::More (classic and for good reasons)
● Test::Builder
● Test::Most
● Test::Class

Producing
Perl Modules
● Test::More (classic and for good reasons)
● Test::Builder
● Test::Most
● Test::Class
● Test::LongString, Test::WWW::Mechanize,
Test::Exception, Test::File::Contents,
Test::HTML::Content, Test::MockObject, Test::Pod,
Test::Warn, Test::WWW::Selenium,
Test::NoWarnings, Test::Output,
Test::Number::Delta, Test::Perl::Critic

Producing
Other Languages
● Javascript →Test.Simple
● C/C++ →libtap, libtap++
● PHP → phpt, PHPUnit, SimpleTest or Apache-Test
● Java →JTap
● Ruby →taptap, Bacon
● Python → PyTAP
● Also Haskell, Erlang, Forth, C#, Shell, Postgres

Consuming
Perl
● Test::Harness
● prove
● TAP::Harness
● TAP::Harness::Archive
● GUI Test Runner
● Smolder

Comming Soon to a terminal near you
Nested TAP
● Group related blocks of TAP together in the same
stream
● Mimic xUnit style
● Aggregate TAP streams

Comming Soon to a terminal near you
Nested TAP
1..3
1..3
ok 1 - guests arrived on time
ok 2 - table was available
ok 3 - order correct
ok 1 - guests served
1..2
ok 1 - chicken is fried
not ok - steak is medium well
not ok 1 - food prepared properly
1..2
ok 1 # SKIP guests did not come back
ok 2 # SKIP guests did not come back
ok 1 # SKIP guests did not come back

Comming Soon to a terminal near you
Metadata
● Everything is crammed into the descriptions or
comments
not ok - fooey, Bug #123
# Failed test 'fooey, Bug #123'
# at barry.pl line 20.
# got: foo bar
# expected: bar foo
● If you want to do anything useful with the information
in that message you need to parse it. Parsing error
messages is evil

Comming Soon to a terminal near you
Metadata
●Data should be structured (YAML-ish)
not ok - fooey
---
file : barry.pl
line : 20
want : foo bar
have : bar foo
display : >
Failed test 'fooey' at barry.pl line 20.
want: "foo bar"
have: "bar foo"
Bug-ID : 123
...

Comming Soon to a terminal near you
Metadata
● Opens up a whole world of possibilities
● Visual diffs on want/have
● Tagging
● Linking between systems (source code, specs, issue
tracking)
● Consumers can show or hide what ever they want or
understand.
● Unknown uses (standardization vs growth)

Usage Scenarios
Typical Perl tests

Usage Scenarios
Automated Javascript Testing
* example from Michael Schwern - http://use.perl.org/~schwern/journal/39092

Usage Scenarios
Automated Javascript Testing

Usage Scenarios
Automated Javascript Testing

TAP in Depth
Michael Peters
Plus Three, LP
Thanks!
Questions?