Software Testing and Reliability Software Test Process Aditya P. Mathur Purdue University August 12-16 @ Guidant Corporati...
Learning Objectives <ul><li>What is software life cycle? </li></ul><ul><li>How to mesh various types of testing into the d...
References <ul><li>Software Testing in the Real World , Edward Kit, Addison Wesley, 1995. </li></ul>
Life Cycle <ul><li>There are many models of Life Cycle. </li></ul><ul><li>Software Development Cycle is a sequence of step...
Testing and the Life Cycle <ul><li>The key questions with which most process managers struggle are: </li></ul><ul><li>It i...
Meshing the Test Process with the Life Cycle Test the requirements Test the modules Test the subsystems Test the system Te...
Test Tools and Techniques:  Requirements Phase <ul><li>Inspection and walkthroughs : Ensure completeness, unambiguity, pre...
Test Tools and Techniques:  Design Phase <ul><li>Inspection and walkthroughs  [ Rational: Rose ] </li></ul><ul><li>Simulat...
Test Tools and Techniques:  Coding Phase <ul><li>Inspection and walkthroughs:  Ensure that the code matches the design. If...
Test Tools and Techniques:  Integration Phase <ul><li>Perform integration testing : Test each subsystem  thoroughly . [ Cl...
Test Tools and Techniques:  System Integration <ul><li>Perform system testing : Test the system to ensure correct function...
Test Tools and Techniques:  System Changes <ul><li>Perform regression testing : Test the modified system to ensure that ex...
Test Tools and Techniques:  In-target system <ul><li>Perform system testing : Test the application for all functionality, ...
Test Tools and Techniques:  Usability testing <ul><li>Perform usability testing : Have a sample of the potential users tes...
Test Tools and Techniques:  Security testing <ul><li>Perform security and safety testing : Security and safety testing mus...
Responsibilities <ul><li>Planning : (High level) Test team  </li></ul><ul><li>Unit tests : Developer. (Who checks the deve...
Release Criteria <ul><li>Units : Code coverage based. Example: all statements, decisions, and conditions must be covered. ...
Standards for Test Documents <ul><li>SQAP:  : Software Quality Assurance Plan, IEEE/ANSI, 1989 [Std 730-1989].  </li></ul>...
Standards for Test Documents (contd.) <ul><li>TDS:  : Test Design Specification, IEEE/ANSI, 1983 [Std 829-1983], one per D...
Structure of Standards for Test Documents SQAP SVVP MTP VTP VTP VTP DTP DTP DTP DTP TDS TDS TDS TDS TCS TCS TPS TC TC TPS
Defect Tracking and Classification <ul><li>Defect tracking and classification [ Applied Innovation management: Bug/Defect ...
Why collect data? <ul><li>Assists with improving the test process; correlations between other process related metrics, suc...
Summary <ul><li>Software development cycle </li></ul><ul><li>Test process and the software development cycle </li></ul><ul...
Upcoming SlideShare
Loading in …5
×

Software Testing and Reliability Software Test Process

1,714 views

Published on

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,714
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
116
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Software Testing and Reliability Software Test Process

  1. 1. Software Testing and Reliability Software Test Process Aditya P. Mathur Purdue University August 12-16 @ Guidant Corporation Minneapolis/St Paul, MN Last update: August 9, 2002 Graduate Assistants : Ramkumar Natarajan Baskar Sridharan
  2. 2. Learning Objectives <ul><li>What is software life cycle? </li></ul><ul><li>How to mesh various types of testing into the development cycle? </li></ul><ul><li>What documents are generated for and as a result of testing? </li></ul><ul><li>What test-metrics needs to be collected and why? </li></ul>
  3. 3. References <ul><li>Software Testing in the Real World , Edward Kit, Addison Wesley, 1995. </li></ul>
  4. 4. Life Cycle <ul><li>There are many models of Life Cycle. </li></ul><ul><li>Software Development Cycle is a sequence of steps in which a software system is developed. We shall refer to this sequence as, simply, Life Cycle . </li></ul><ul><li>The waterfall model </li></ul><ul><li>Iterative refinement model </li></ul><ul><li>Incremental development model </li></ul><ul><li>Spiral model </li></ul><ul><li>Unified Process model </li></ul>
  5. 5. Testing and the Life Cycle <ul><li>The key questions with which most process managers struggle are: </li></ul><ul><li>It is widely believed that testing is necessary in each phase of the life cycle regardless of which model is adopted. </li></ul><ul><li>How should the test process be “meshed” with the development cycle? </li></ul><ul><li>What tools and techniques should be used in each phase? </li></ul><ul><li>What data should be collected to evaluate the effectiveness of the test process? </li></ul>
  6. 6. Meshing the Test Process with the Life Cycle Test the requirements Test the modules Test the subsystems Test the system Test for acceptance Run regression Tests Test the design Requirements specification Design specification Code Integration System Integration System Deployment System Modification
  7. 7. Test Tools and Techniques: Requirements Phase <ul><li>Inspection and walkthroughs : Ensure completeness, unambiguity, precision, consistency, testability, and traceability. [ Rational: Rose and Analyst Studio, I-Logix: Rhapsody] </li></ul><ul><li>Simulation : Use this to improve the understanding of the system to be built; where possible, work with the customer. </li></ul><ul><li>Generate test procedures, plans, and test cases : Develop procedures, plans and test cases to be used for testing each requirement. Identify what tools will be used in each phase, who will use these, and what data will be collected. Determine product release criteria. </li></ul>
  8. 8. Test Tools and Techniques: Design Phase <ul><li>Inspection and walkthroughs [ Rational: Rose ] </li></ul><ul><li>Simulation : Check for the correctness of scenarios. [ I-Logix: Rhapsody ] </li></ul><ul><li>Refine test procedures and test cases : Use the artifacts created during design to generate additional test cases and refine or correct the existing ones. </li></ul><ul><li>Risk analysis : Use the application architecture to perform risk analysis. [ Rational: ROOM ] </li></ul>
  9. 9. Test Tools and Techniques: Coding Phase <ul><li>Inspection and walkthroughs: Ensure that the code matches the design. If not then at leant one of the two needs to be modified to achieve consistency. [ Rational: Rose ] </li></ul><ul><li>Perform unit testing : Test each module thoroughly . [ CleanScape: TestWise; Rational: Purify and Visual Test; AMC: CodeTest ] </li></ul>
  10. 10. Test Tools and Techniques: Integration Phase <ul><li>Perform integration testing : Test each subsystem thoroughly . [ CleanScape: TestWise; Rational: Purify and Visual Test; Applied Microsystems: CodeTest; Windriver: Sniff+ ] </li></ul>
  11. 11. Test Tools and Techniques: System Integration <ul><li>Perform system testing : Test the system to ensure correct functionality with respect to use cases, do performance, stress, and robustness tests. [ CMU: Ballista; Telcordia: AETG; Applied Microsystems: CodeTest; Windriver: Diab RTA suite and Sniff+ ] </li></ul>
  12. 12. Test Tools and Techniques: System Changes <ul><li>Perform regression testing : Test the modified system to ensure that existing functions perform as intended. Note that system testing for the added features ought to be performed. [ CleanScape: Testwise for test minimization] </li></ul>
  13. 13. Test Tools and Techniques: In-target system <ul><li>Perform system testing : Test the application for all functionality, its performance, and robustness while the application runs embedded in the target platform. [ Advanced Microsystems: CodeTest; Home-grown test generators; Rational: Purify ] </li></ul>
  14. 14. Test Tools and Techniques: Usability testing <ul><li>Perform usability testing : Have a sample of the potential users test the application and the documents that accompany it. The goal is to ensure that (a) the documents are understandable, consistent, and correct and (b) the GUI satisfies the requirements of the potential users. </li></ul>Usability testing can begin when the GUI is ready for use, i.e. developed and tested for correctness, but the remainder of the application code might not be.
  15. 15. Test Tools and Techniques: Security testing <ul><li>Perform security and safety testing : Security and safety testing must be done when an application is accessible from the “outside” and might compromise the user’s privacy and safety, </li></ul>The objective of security and safety testing is to demonstrate that privacy and safety cannot be compromised when the assumptions regarding the use of the application are met. <ul><li>Security and safety testing is done with respect to the requirements for security and safety. Traditional testing techniques are useful but might have to be augmented with techniques specific to the application environment. </li></ul>
  16. 16. Responsibilities <ul><li>Planning : (High level) Test team </li></ul><ul><li>Unit tests : Developer. (Who checks the developers work?) </li></ul><ul><li>Integration tests : Developer and/or test team depending on the complexity of the sub-system to be tested. </li></ul><ul><li>System level tests : Test team. These tests will generally require communication with the developers. This must be a part of the detail plan built at the beginning of the development cycle. </li></ul><ul><li>Acceptance and usability tests : Test team in collaboration with the end user (or a sample of the end user set). </li></ul>
  17. 17. Release Criteria <ul><li>Units : Code coverage based. Example: all statements, decisions, and conditions must be covered. </li></ul><ul><li>Integration tests : Code coverage and function based. Low level code coverage might not be usable if the integration produces a “large” component. </li></ul><ul><li>System level tests: System reliability measured in terms of the number and type of failures detected. </li></ul><ul><li>Acceptance and usability tests: Determined by the user primarily based on correct functionality, robustness, performance, and usability. </li></ul>
  18. 18. Standards for Test Documents <ul><li>SQAP: : Software Quality Assurance Plan, IEEE/ANSI, 1989 [Std 730-1989]. </li></ul><ul><li>SVVP : Software Verification and Validation Plan, IEEE/ANSI, 1986 [Std 1012-1986]; one per SQAP. </li></ul><ul><li>VTP : Verification Test Plan, one per verification activity. </li></ul><ul><li>MVP : Master Validation Test Plan, IEEE/ANSI, 1983 [Std 829-1983]; one per SVVP. </li></ul><ul><li>DTP : Detailed Validation Test Plan, IEEE/ANSI, 1983 [Std 829-1983]; one or more per activity. </li></ul>
  19. 19. Standards for Test Documents (contd.) <ul><li>TDS: : Test Design Specification, IEEE/ANSI, 1983 [Std 829-1983], one per DTP. </li></ul><ul><li>TCS : Test Case Specification IEEE/ANSI, 1983 [Std 829-1983]; one or more per TDS/TPS. </li></ul><ul><li>TPS : Test Procedure Specification, IEEE/ANSI, 1983 [Std 829-1983]; one or more per TDS. </li></ul><ul><li>TC : Test Case, one per TCS. </li></ul>
  20. 20. Structure of Standards for Test Documents SQAP SVVP MTP VTP VTP VTP DTP DTP DTP DTP TDS TDS TDS TDS TCS TCS TPS TC TC TPS
  21. 21. Defect Tracking and Classification <ul><li>Defect tracking and classification [ Applied Innovation management: Bug/Defect Tracking Expert; defectx: defectX ] </li></ul><ul><li>Defect estimation: Use inspection data for defect estimation; design metrics; past data on similar projects. </li></ul><ul><li>Defects open vs. closed </li></ul><ul><li>Cross-project and cross version comparison of the number of defects. </li></ul><ul><li>Cost of repairing a defect: time and manpower resources needed. </li></ul>
  22. 22. Why collect data? <ul><li>Assists with improving the test process; correlations between other process related metrics, such as code coverage, and defects can be established. </li></ul><ul><li>Helps improve product quality. </li></ul><ul><li>Useful in setting up a reward system ( no penalty please !). </li></ul>
  23. 23. Summary <ul><li>Software development cycle </li></ul><ul><li>Test process and the software development cycle </li></ul><ul><li>Assignment of responsibilities </li></ul><ul><li>Standards </li></ul><ul><li>Metrics </li></ul>

×