The document discusses the Test Anything Protocol (TAP), which is a simple text-based interface for reporting test results. TAP can be used for regression testing code, systems monitoring, and performance testing. It originated for testing Perl code but has since been adopted by many other languages. TAP outputs test results in a structured format with directives to indicate test status, plans, and other metadata. Producers generate TAP output and consumers process and analyze the results. Popular Perl modules for producing and consuming TAP are discussed, as are implementations in other languages. Future enhancements to TAP like nested output and structured metadata are also outlined.

1. TAP in Depth
Michael Peters
Plus Three, LP
YAPC::NA 10
Pittsburgh, PA

2. TAP
Test Anything Protocol
● And we do mean *anything*
● Mainly used for regression testing code
● Could be used for systems monitoring,
performance testing, etc
● 1987, current version 13
● Patient Zero was Perl, but the infection has since
spread to lots of other languages
● Ongoing work to make an IETF standard

3. Definitions
Test
● or an "assertion" (ok/not ok)
● smallest granule of measurement
● did it do what we wanted
Stream
● or a "set"
● a group of tests
● usually output of running a single test file, but is not
necessarily tied to a file or any specific output.
Test Suite
● A group of TAP streams

4. Definitions
Harness
● A process which collects a group of TAP streams
Plan
● number of tests that are expected to be run in a given
stream
Directive
● Changes the meaning of a given test
● TODO/SKIP
Reason
● Explains the "why" of a directive

5. Definitions
Description
● What is the test asserting? What functionality is
being tested?
Producer
● Anything that generates TAP
Consumer
● Anything that processes TAP
● Could just blindly echo it
● Usually parses it

6. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

7. Document Structure
Line Oriented Protocol
● Everything is separated by a new-line (CRLF)
● Not everything fits on 1 line
● Everything starts and ends on it's own line
Encoding
● Everything is UTF-8
● Unless the Producer/Consumer agree otherwise
1 Version, 1* plans, 0+ tests,
0* comments, 0* ignored elements

8. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

9. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

10. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

11. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

12. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

13. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

14. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

15. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

16. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

17. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

18. Document Structure
TAP version 13
1..7
ok 1 - Input file opened
not ok 2 - First line of the input valid
ok 3 - Read the rest of the file
# Can't load Digest::MD5. This modules is needed
# to run some optional tests
ok 4 # SKIP MD5 library not installed
ok 5 # SKIP MD5 library not installed
not ok 6 - Correct summary # TODO Not written yet
ok 7 - File closed

19. Pass vs Fail
Tests
● "ok" == pass
● "not ok" == fail
● Yay! binary!

20. Pass vs Fail
Tests
● "ok" == pass
● "not ok" == fail
● If only it were that simple
● Directives!
Normal TODO SKIP
ok pass * pass
not ok pass fail **
* a passing TODO might indicate that a test shouldn't be a TODO anymore
** SKIP means the test wasn't run, so it shouldn't be able to fail

21. Pass vs Fail
Streams
● Can fail if any of the following are true
(1) Any tests in this stream failed
(2) Tests run did not agree with the plan
(3) Other factors
● process exit codes (non-zero)
● HTTP status code
● Passed only if none of the above are true and
(1) There was at least 1 passing test - or -
(2) There was a "skip all"
● 1..0 # SKIP some module not installed

22. Pass vs Fail
Test Suites
● Fails if there are any streams that failed
● Passes if nothing failed and there's at least 1 stream
● Pass or fail at the suite level is boolean
● Tools can show percentages
● But even a 99.999% pass is still a fail

23. Producing
Perl Modules
● Test::More (classic and for good reasons)

24. Producing
Perl Modules
● Test::More (classic and for good reasons)
● Test::Builder

25. Producing
Perl Modules
● Test::More (classic and for good reasons)
● Test::Builder
● Test::Most

26. Producing
Perl Modules
● Test::More (classic and for good reasons)
● Test::Builder
● Test::Most
● Test::Class

27. Producing
Perl Modules
● Test::More (classic and for good reasons)
● Test::Builder
● Test::Most
● Test::Class
● Test::LongString, Test::WWW::Mechanize,
Test::Exception, Test::File::Contents,
Test::HTML::Content, Test::MockObject, Test::Pod,
Test::Warn, Test::WWW::Selenium,
Test::NoWarnings, Test::Output,
Test::Number::Delta, Test::Perl::Critic

28. Producing
Other Languages
● Javascript →Test.Simple
● C/C++ →libtap, libtap++
● PHP → phpt, PHPUnit, SimpleTest or Apache-Test
● Java →JTap
● Ruby →taptap, Bacon
● Python → PyTAP
● Also Haskell, Erlang, Forth, C#, Shell, Postgres

29. Consuming
Perl
● Test::Harness
● prove
● TAP::Harness
● TAP::Harness::Archive
● GUI Test Runner
● Smolder

30. Comming Soon to a terminal near you
Nested TAP
● Group related blocks of TAP together in the same
stream
● Mimic xUnit style
● Aggregate TAP streams

31. Comming Soon to a terminal near you
Nested TAP
1..3
1..3
ok 1 - guests arrived on time
ok 2 - table was available
ok 3 - order correct
ok 1 - guests served
1..2
ok 1 - chicken is fried
not ok - steak is medium well
not ok 1 - food prepared properly
1..2
ok 1 # SKIP guests did not come back
ok 2 # SKIP guests did not come back
ok 1 # SKIP guests did not come back

32. Comming Soon to a terminal near you
Metadata
● Everything is crammed into the descriptions or
comments
not ok - fooey, Bug #123
# Failed test 'fooey, Bug #123'
# at barry.pl line 20.
# got: foo bar
# expected: bar foo
● If you want to do anything useful with the information
in that message you need to parse it. Parsing error
messages is evil

33. Comming Soon to a terminal near you
Metadata
●Data should be structured (YAML-ish)
not ok - fooey
---
file : barry.pl
line : 20
want : foo bar
have : bar foo
display : >
Failed test 'fooey' at barry.pl line 20.
want: "foo bar"
have: "bar foo"
Bug-ID : 123
...

34. Comming Soon to a terminal near you
Metadata
● Opens up a whole world of possibilities
● Visual diffs on want/have
● Tagging
● Linking between systems (source code, specs, issue
tracking)
● Consumers can show or hide what ever they want or
understand.
● Unknown uses (standardization vs growth)

35. Usage Scenarios
Typical Perl tests

36. Usage Scenarios
Automated Javascript Testing
* example from Michael Schwern - http://use.perl.org/~schwern/journal/39092

37. Usage Scenarios
Automated Javascript Testing

38. Usage Scenarios
Automated Javascript Testing

39. TAP in Depth
Michael Peters
Plus Three, LP
Thanks!
Questions?