TAP In Depth

2,958 views

Published on

An in depth look at the Test Anything Protocol

Published in: Technology, Education
1 Comment
1 Like
Statistics
Notes
No Downloads
Views
Total views
2,958
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
35
Comments
1
Likes
1
Embeds 0
No embeds

No notes for slide

TAP In Depth

  1. 1. TAP in Depth Michael Peters Plus Three, LP YAPC::NA 10 Pittsburgh, PA
  2. 2. TAP Test Anything Protocol ● And we do mean *anything* ● Mainly used for regression testing code ● Could be used for systems monitoring, performance testing, etc ● 1987, current version 13 ● Patient Zero was Perl, but the infection has since spread to lots of other languages ● Ongoing work to make an IETF standard
  3. 3. Definitions Test ● or an "assertion" (ok/not ok) ● smallest granule of measurement ● did it do what we wanted Stream ● or a "set" ● a group of tests ● usually output of running a single test file, but is not necessarily tied to a file or any specific output. Test Suite ● A group of TAP streams
  4. 4. Definitions Harness ● A process which collects a group of TAP streams Plan ● number of tests that are expected to be run in a given stream Directive ● Changes the meaning of a given test ● TODO/SKIP Reason ● Explains the "why" of a directive
  5. 5. Definitions Description ● What is the test asserting? What functionality is being tested? Producer ● Anything that generates TAP Consumer ● Anything that processes TAP ● Could just blindly echo it ● Usually parses it
  6. 6. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  7. 7. Document Structure Line Oriented Protocol ● Everything is separated by a new-line (CRLF) ● Not everything fits on 1 line ● Everything starts and ends on it's own line Encoding ● Everything is UTF-8 ● Unless the Producer/Consumer agree otherwise 1 Version, 1* plans, 0+ tests, 0* comments, 0* ignored elements
  8. 8. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  9. 9. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  10. 10. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  11. 11. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  12. 12. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  13. 13. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  14. 14. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  15. 15. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  16. 16. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  17. 17. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  18. 18. Document Structure TAP version 13 1..7 ok 1 - Input file opened not ok 2 - First line of the input valid ok 3 - Read the rest of the file # Can't load Digest::MD5. This modules is needed # to run some optional tests ok 4 # SKIP MD5 library not installed ok 5 # SKIP MD5 library not installed not ok 6 - Correct summary # TODO Not written yet ok 7 - File closed
  19. 19. Pass vs Fail Tests ● "ok" == pass ● "not ok" == fail ● Yay! binary!
  20. 20. Pass vs Fail Tests ● "ok" == pass ● "not ok" == fail ● If only it were that simple ● Directives! Normal TODO SKIP ok pass * pass not ok pass fail ** * a passing TODO might indicate that a test shouldn't be a TODO anymore ** SKIP means the test wasn't run, so it shouldn't be able to fail
  21. 21. Pass vs Fail Streams ● Can fail if any of the following are true (1) Any tests in this stream failed (2) Tests run did not agree with the plan (3) Other factors ● process exit codes (non-zero) ● HTTP status code ● Passed only if none of the above are true and (1) There was at least 1 passing test - or - (2) There was a "skip all" ● 1..0 # SKIP some module not installed
  22. 22. Pass vs Fail Test Suites ● Fails if there are any streams that failed ● Passes if nothing failed and there's at least 1 stream ● Pass or fail at the suite level is boolean ● Tools can show percentages ● But even a 99.999% pass is still a fail
  23. 23. Producing Perl Modules ● Test::More (classic and for good reasons)
  24. 24. Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder
  25. 25. Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder ● Test::Most
  26. 26. Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder ● Test::Most ● Test::Class
  27. 27. Producing Perl Modules ● Test::More (classic and for good reasons) ● Test::Builder ● Test::Most ● Test::Class ● Test::LongString, Test::WWW::Mechanize, Test::Exception, Test::File::Contents, Test::HTML::Content, Test::MockObject, Test::Pod, Test::Warn, Test::WWW::Selenium, Test::NoWarnings, Test::Output, Test::Number::Delta, Test::Perl::Critic
  28. 28. Producing Other Languages ● Javascript →Test.Simple ● C/C++ →libtap, libtap++ ● PHP → phpt, PHPUnit, SimpleTest or Apache-Test ● Java →JTap ● Ruby →taptap, Bacon ● Python → PyTAP ● Also Haskell, Erlang, Forth, C#, Shell, Postgres
  29. 29. Consuming Perl ● Test::Harness ● prove ● TAP::Harness ● TAP::Harness::Archive ● GUI Test Runner ● Smolder
  30. 30. Comming Soon to a terminal near you Nested TAP ● Group related blocks of TAP together in the same stream ● Mimic xUnit style ● Aggregate TAP streams
  31. 31. Comming Soon to a terminal near you Nested TAP 1..3 1..3 ok 1 - guests arrived on time ok 2 - table was available ok 3 - order correct ok 1 - guests served 1..2 ok 1 - chicken is fried not ok - steak is medium well not ok 1 - food prepared properly 1..2 ok 1 # SKIP guests did not come back ok 2 # SKIP guests did not come back ok 1 # SKIP guests did not come back
  32. 32. Comming Soon to a terminal near you Metadata ● Everything is crammed into the descriptions or comments not ok - fooey, Bug #123 # Failed test 'fooey, Bug #123' # at barry.pl line 20. # got: foo bar # expected: bar foo ● If you want to do anything useful with the information in that message you need to parse it. Parsing error messages is evil
  33. 33. Comming Soon to a terminal near you Metadata ●Data should be structured (YAML-ish) not ok - fooey --- file : barry.pl line : 20 want : foo bar have : bar foo display : > Failed test 'fooey' at barry.pl line 20. want: "foo bar" have: "bar foo" Bug-ID : 123 ...
  34. 34. Comming Soon to a terminal near you Metadata ● Opens up a whole world of possibilities ● Visual diffs on want/have ● Tagging ● Linking between systems (source code, specs, issue tracking) ● Consumers can show or hide what ever they want or understand. ● Unknown uses (standardization vs growth)
  35. 35. Usage Scenarios Typical Perl tests
  36. 36. Usage Scenarios Automated Javascript Testing * example from Michael Schwern - http://use.perl.org/~schwern/journal/39092
  37. 37. Usage Scenarios Automated Javascript Testing
  38. 38. Usage Scenarios Automated Javascript Testing
  39. 39. TAP in Depth Michael Peters Plus Three, LP Thanks! Questions?

×