Testing Toolbox


Published on

Presentation at OSCON 2007 by Michael Peters about various web testing tools.

Published in: Technology
  • Be the first to comment

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Testing Toolbox

  1. 1. Updating Your Testing Toolbox Michael Peters PlusThree, LP http://www.plusthree.com/ OSCON 2007
  2. 2. Difficulties <ul><li>Hard to get time scheduled </li></ul><ul><li>Hard to get non-programmers involved </li></ul><ul><li>Hard to see progress </li></ul>
  3. 3. Testing <ul><li>During Development </li></ul><ul><li>Performance </li></ul><ul><li>Regression (Smoke) </li></ul>
  4. 4. ...
  5. 5. During Development <ul><li>Firebug </li></ul><ul><li>Firefox Plugin </li></ul><ul><li>JS Console and Debugger </li></ul><ul><li>HTML/CSS/DOM Inspector </li></ul><ul><li>XHR inspector </li></ul>
  6. 6. During Development <ul><li>Firebug </li></ul><ul><li>Firefox Only </li></ul><ul><li>Firebug Lite ( http://www.getfirebug.com/lite.html ) </li></ul><ul><li>Safari 3 will have Web Inspector </li></ul><ul><li>IE has VWD, but it's not even close </li></ul>
  7. 7. During Development <ul><li>LiveHTTPHeaders </li></ul><ul><li>Another Firefox plugin </li></ul><ul><li>All HTTP headers for All requests </li></ul><ul><li>Cookies, X-JSON, Cache, etc </li></ul>
  8. 8. During Development <ul><li>View Formatted Source </li></ul><ul><li>View Source is static </li></ul>
  9. 9. During Development <ul><li>View Formatted Source </li></ul><ul><li>View Source is static </li></ul><ul><li>View Formatted Source is ALIVE! </li></ul>
  10. 10. During Development <ul><li>PageDiff </li></ul><ul><li>Capture the state of the current page </li></ul><ul><li>View the difference after some actions </li></ul>
  11. 11. ...
  12. 12. Performance <ul><li>Load Time Analyzer </li></ul><ul><li>Why does this page take so long to load? </li></ul><ul><li>Does it really matter how big this JS file is? </li></ul>
  13. 13. Performance Load Time Analyzer <ul><li>vs Built-in Firebug Net Inspector </li></ul><ul><li>LTA is a printable </li></ul><ul><li>LTA gives more information about JS execution time after it's loaded </li></ul>Pingdom – a similar online tool
  14. 14. Performance Yslow vs GTA <ul><li>More than just time </li></ul><ul><li>Gives tips and hints </li></ul><ul><li>Summarized by topic </li></ul>
  15. 15. Performance benchmark.js <ul><li>http://gfx.neohub.com/benchmark/ </li></ul><ul><li>Similar to Benchmark in Perl </li></ul><ul><li>Compare similar techniques to find the fastest </li></ul><ul><li>See performance differences between browsers </li></ul><ul><li>Compare different JS libs that do similar things </li></ul>
  16. 16. Performance benchmark.js <ul><li>vs Firebug's built-in JS profiler </li></ul><ul><li>Benchmark is for smaller examples </li></ul><ul><li>Profiler for real code/pages </li></ul><ul><li>benchmark.js works in multiple browsers </li></ul>
  17. 17. ...
  18. 18. Regression (Smoke) Selenium RC <ul><li>http://www.openqa.org/selenium-rc/ </li></ul><ul><li>Selenium Server </li></ul><ul><ul><li>Java program that can start/stop/control a browser </li></ul></ul><ul><ul><ul><ul><li>IE 6, 7 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>FF 1.5, 2.0 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Opera 8.5, 9.0 </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Safari 1.3, 2.0+ </li></ul></ul></ul></ul><ul><li>Selenium Core </li></ul><ul><ul><li>Javascript libraries to control a browser's behavior and make requests </li></ul></ul>
  19. 19. Regression (Smoke) Selenium RC <ul><li>http://www.openqa.org/selenium-rc/ </li></ul><ul><li>WWW::Selenium </li></ul><ul><ul><ul><li>Control the browser </li></ul></ul></ul><ul><li>Test::WWW::Selenium </li></ul><ul><ul><ul><li>subclasses WWW::Selenium to add testing methods </li></ul></ul></ul>
  20. 20. Regression (Smoke) http://www.openqa.org/selenium-ide/ Selenium IDE <ul><li>Record actions in the browser (Firefox) </li></ul><ul><li>Add assertions </li></ul><ul><li>Save test case (a simple HTML format) </li></ul><ul><li>Run test cases </li></ul>
  21. 21. Regression (Smoke) Selenium RC + IDE <ul><li>Anyone can record test cases </li></ul><ul><li>Save those cases in a shared location </li></ul><ul><li>Anyone can run those cases </li></ul><ul><ul><li>Can be done so that cases are run on different OSes/browsers </li></ul></ul><ul><li>Very powerful if you allow non-programmers to reset test data </li></ul>
  22. 22. Regression (Smoke) Smolder <ul><li>At PlusThree we have some large projects with large test suites </li></ul><ul><li>15,000+ tests (200+ .t files) </li></ul><ul><li>50+ minutes </li></ul><ul><li>Doesn't let you use that box for anything else </li></ul><ul><li>Run automated tests on a dedicated machine periodically if there have been changes </li></ul>
  23. 23. Regression (Smoke) Smolder <ul><li>Tests can be commented, categoried and invalidated </li></ul><ul><li>Trends can be viewed with graphs </li></ul><ul><li>Individual test runs have interactive reports </li></ul>
  24. 24. Regression (Smoke) Smolder <ul><li>Can host multiple projects </li></ul><ul><ul><li>public and private </li></ul></ul><ul><li>Developers can customize email they receive </li></ul><ul><ul><li>frequency and format </li></ul></ul><ul><li>Public projects can also provide data feeds </li></ul><ul><ul><li>Atom </li></ul></ul>
  25. 25. Regression (Smoke) TAP <ul><li>Test Anything Protocol </li></ul><ul><li>Simple, line oriented </li></ul><ul><li>Almost 20 years old </li></ul><ul><li>Language Agnostic </li></ul>C, C++, Forth, Javascript, PHP, Perl, PostgreSQL,Python, Ruby, PIR
  26. 26. Regression (Smoke) TAP <ul><li>In Perl </li></ul><ul><li>Test::Builder, Test::More, etc </li></ul><ul><li>Test::Harness </li></ul><ul><li>TAP::Parser </li></ul>
  27. 27. Regression (Smoke) TAP + Smolder <ul><li>TAP Archive </li></ul><ul><li>An archive of .tap files: one for each test run </li></ul><ul><li>An optional YAML file containing meta information about the test run </li></ul><ul><ul><li>Start and end times </li></ul></ul><ul><ul><li>What order the files were run in </li></ul></ul><ul><li>TestRunner::Multi </li></ul><ul><ul><li>runs tests from source control </li></ul></ul>
  28. 28. All Together Now <ul><li>Developers write tests </li></ul><ul><ul><ul><li>Perl – Test::More, Test::WWW::Mechanize, et c </li></ul></ul></ul><ul><li>Automated test harness runs them </li></ul><ul><li>QA/Acceptance Testers, Product Managers record tests </li></ul><ul><ul><ul><li>Browser – Selenium IDE </li></ul></ul></ul><ul><li>Automated test harness runs them </li></ul><ul><li>TAP is produced </li></ul><ul><li>Smolder creates pretty reports, graphs and email </li></ul><ul><li>Everyone feels involved </li></ul><ul><li>= Shiny Happy Users :) </li></ul>
  29. 29. ...
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.