Class 10

870 views
793 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
870
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
23
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Class 10

  1. 1. CES 592 Telecommunications System Product Verification Sonoma State University <ul><li>Class Lecture 10: </li></ul><ul><li>Software Testing Tools </li></ul>Fall 2004
  2. 2. Instructors <ul><li>France Antelme </li></ul><ul><li>Ario Bigattini </li></ul><ul><li>Jaseem Masood </li></ul><ul><li>Steven Woody </li></ul><ul><li>Coordinator: Prof Ravi Kumar </li></ul>Website http://www.sonoma.edu/engineering/courses/CES592.shtml
  3. 3. Software Testing Tools <ul><li>Software Testing Tools </li></ul><ul><li>Code Coverage Analysis </li></ul><ul><li>Testing Metrics </li></ul><ul><li>Automation </li></ul>
  4. 4. Software Testing Tools <ul><li>Coverage Analysis </li></ul><ul><li>Code coverage analysis tools measure how much of the software was executed </li></ul><ul><li>Used to measure test coverage </li></ul><ul><li>Can measure by statement, block/path, or function/procedure </li></ul>
  5. 5. Software Testing Tools <ul><li>Coverage Analysis </li></ul><ul><li>Requires the coverage tool to be compiled with the application software </li></ul><ul><li>Creates a summary file of the coverage analysis </li></ul><ul><li>10-15% performance hit for block coverage. The more measurement points, the greater the hit. </li></ul>
  6. 6. Software Testing Tools <ul><li>Coverage Analysis </li></ul><ul><li>70-75% coverage typical goal </li></ul><ul><li>100% coverage usually not attainable </li></ul><ul><li>100% coverage does not mean 100% tested </li></ul>
  7. 7. Software Testing Tools <ul><li>Example Coverage Analysis tools: </li></ul><ul><li>Bullseye Coverage </li></ul><ul><li>Cantata++ </li></ul><ul><li>CodeTEST </li></ul><ul><li>LOGISCOPE </li></ul><ul><li>Panorama C/C++ </li></ul><ul><li>Rational PureCoverage </li></ul><ul><li>TCAT C/C++ </li></ul><ul><li>GCT </li></ul><ul><li>Reference: http://testingfaqs.org/t-eval.html </li></ul>
  8. 8. Software Testing Tools <ul><li>Software Testing Tools </li></ul><ul><li>Coverage Analysis </li></ul><ul><li>Testing Metrics </li></ul><ul><li>Automation </li></ul>
  9. 9. Software Testing Metrics <ul><li>Metrics (Measurements) </li></ul><ul><li>What is quality? </li></ul><ul><li>How can you measure it? </li></ul><ul><li>Can it be measured directly? </li></ul><ul><li>Project metrics vs Product metrics </li></ul><ul><li>Use objective measurements to improve your testing process & product quality </li></ul><ul><li>Anything can be measured, and any measurement is better than no measurement </li></ul>
  10. 10. Software Testing Metrics <ul><li>Suggested metrics </li></ul><ul><li>Dates </li></ul><ul><ul><li>Test requirements approved </li></ul></ul><ul><ul><li>Test plan approved </li></ul></ul><ul><ul><li>Test bed ready </li></ul></ul><ul><ul><li>Testing cycle complete (multiple cycles) </li></ul></ul><ul><ul><li>Regression testing scripts complete </li></ul></ul><ul><ul><li>Track original vs actual dates </li></ul></ul>
  11. 11. Software Testing Metrics <ul><li>Suggested metrics </li></ul><ul><li>Effort (Person-weeks) </li></ul><ul><ul><li>Test prep </li></ul></ul><ul><ul><li>Test execution </li></ul></ul><ul><ul><li>Test script automation </li></ul></ul><ul><ul><li>Test script maintenance </li></ul></ul><ul><ul><li>Test script breakage due to software changes </li></ul></ul>
  12. 12. Software Testing Metrics <ul><li>Suggested metrics </li></ul><ul><li>Costs / Expenses </li></ul><ul><ul><li>Prototype requirements </li></ul></ul><ul><ul><li>Equipment costs: test equipment </li></ul></ul><ul><ul><li>Software licences </li></ul></ul>
  13. 13. Software Testing Metrics <ul><li>Suggested metrics </li></ul><ul><li>Quantity of testing </li></ul><ul><ul><li>Number of tests performed </li></ul></ul><ul><ul><li>Number of tests passed / failed </li></ul></ul><ul><ul><li>Number of tests automated </li></ul></ul>
  14. 14. Software Testing Metrics <ul><li>Suggested metrics </li></ul><ul><li>Quality of testing </li></ul><ul><ul><li>Code coverage </li></ul></ul><ul><ul><li>Number of defects found in test </li></ul></ul><ul><ul><li>Number of defects found by customers </li></ul></ul><ul><ul><li>Severity of defects found per test phase </li></ul></ul><ul><ul><li>Rate of defect discovery </li></ul></ul><ul><ul><li>Rate of defect resolution </li></ul></ul>
  15. 15. Software Testing Metrics <ul><li>A Word of Caution… </li></ul><ul><li>Never use metrics to evaluate an individual (i.e. “Number of defects found by tester x” or “Number of defects caused by developer y”) </li></ul><ul><li>People will either stop reporting numbers that can be used against them, or they will only report numbers that will make them look good. </li></ul>
  16. 16. Software Testing Tools <ul><li>Software Testing Tools </li></ul><ul><li>Coverage Analysis </li></ul><ul><li>Testing Metrics </li></ul><ul><li>Automation </li></ul>
  17. 17. Software Testing Tools <ul><li>Automation is a requirement </li></ul><ul><li>Manual testing does not scale </li></ul><ul><li>Consider that the testing effort would double (at least) without automation if: </li></ul><ul><ul><li>A maintenance version is needed for all customers </li></ul></ul><ul><ul><li>A “special” version is needed by one customer </li></ul></ul><ul><ul><li>The common control card hardware is modified </li></ul></ul><ul><ul><li>A new platform is introduced </li></ul></ul><ul><ul><li>Software is compiled with a different option </li></ul></ul>
  18. 18. Software Testing Tools <ul><li>However, very few test automation efforts are successful. </li></ul><ul><li>To be successful: </li></ul><ul><li>Know your goals </li></ul><ul><li>Get a shared vision </li></ul><ul><li>Be realistic </li></ul><ul><li>Start small </li></ul><ul><li>Invest the resources </li></ul><ul><li>Don’t falter and lose your investment </li></ul>
  19. 19. Software Testing Tools <ul><li>Are you ready to automate? </li></ul><ul><li>Do you know how to test it manually? </li></ul><ul><li>Are organized testing procedures in place? </li></ul><ul><li>Is there time to learn the automation tool? </li></ul><ul><li>Will there be ongoing tool support? </li></ul><ul><li>Is the application stable enough? </li></ul><ul><li>Is the budget adequate? </li></ul><ul><li>Will the organization support the effort? </li></ul><ul><li>Adapted from Software Testing and Quality Assurance byRoss Collard </li></ul>
  20. 20. Software Testing Tools <ul><li>Questions to ask ahead of time: </li></ul><ul><li>Who maintains the tools? </li></ul><ul><li>Who writes the libraries? </li></ul><ul><li>How to track and report what has been tested? </li></ul><ul><li>How will known bugs be handled? </li></ul>
  21. 21. Software Testing Tools <ul><li>When to automate? </li></ul><ul><li>Prior to software delivery, using product specifications? </li></ul><ul><li>From the start of software development? </li></ul><ul><li>After software development is complete? </li></ul><ul><li>During manual testing? </li></ul><ul><li>After manual testing? </li></ul><ul><li>Don’t spend a lot of time automating a feature that is still evolving </li></ul>
  22. 22. Software Testing Tools <ul><li>How to automate? </li></ul><ul><li>“ A good manual testing regime is a firm foundation on which to build test automation.” </li></ul><ul><li>Centralized vs. Decentralized approach </li></ul><ul><ul><li>Tool expert automates the test, vs. </li></ul></ul><ul><ul><li>Testers automate the tests. </li></ul></ul><ul><ul><li>Adapted from Chapter 8: Metrics - Software Test Automation, Fewster & Graham </li></ul></ul>
  23. 23. Software Testing Tools <ul><li>What to automate? </li></ul><ul><li>Determine where the risk is and focus there </li></ul><ul><li>Automate “the long pole in the tent” </li></ul><ul><li>Don’t aim for 100% automation </li></ul><ul><li>Automate new features as they arrive in test – don’t wait for that magical free month </li></ul>
  24. 24. Software Testing Tools <ul><li>Perl </li></ul><ul><li>TCL / Expect </li></ul><ul><li>Python </li></ul><ul><li>Ruby </li></ul><ul><li>VBScript </li></ul><ul><li>Use the right tool for the job. </li></ul>
  25. 25. References <ul><li>Software Test Automation, Fewster & Graham </li></ul><ul><li>Software Testing In The Real World by Edward Kit </li></ul><ul><li>Testing Computer Software, by C. Kaner, J. Falk, and H. Nguyen </li></ul><ul><li>Managing the Testing Process, Rex Black </li></ul><ul><li>Black-Box Testing: Techniques for Functional Testing of Software and Systems, B. Beizer </li></ul><ul><li>Code Complete, Steve McConnell </li></ul><ul><li>Software Reliability Engineering, John Musa </li></ul><ul><li>“ Software Testing and Quality Assurance”, Ross Collard </li></ul><ul><li>IEEE Standard for Software Test Documentation, Std 829 - 1998 </li></ul><ul><li>Classic Testing Mistakes, Brian Marick: http :// www.testing.com/writings/classic/mistakes.pdf </li></ul><ul><li>Software QA / Test Resource Center: http://www.softwareqatest.com/index.html </li></ul>

×