Component Testing: Part I


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Component Testing: Part I

  1. 1. Component Testing: Part I Ednaldo Dilorenzo de Souza Filho April 16, 2010 1
  2. 2. Summary  Introduction  Fundaments of Testing  Software Testability  Test Case Projects  Environment Testing, Architecture and Specialized Applications  Software Test Strategies  Component Testing April 16, 2010 2
  3. 3. Introduction  Testing is an important process to support assurance;  As software becomes more pervasive and is used more often to perform critical tasks, it will be required to be of higher quality;  Because testing requires the execution of the software, it is often called dynamic analysis [Harrold];  Testing consists in compare the outputs from the executions with expected results to identify those test cases on which the software failed;  Testing cannot show the absence of faults, if can show only their presence. April 16, 2010 3
  4. 4. Fundaments of Testing  Testing goals:  Find faults in a software program;  A good test case have a high probability to find errors;  A good test process should find errors;  Testing fundamentals:  Testing should be connected with client requirements;  Testing should be planned before testing execution;  Isolation of suspected components;  Testing should start in individual components;  Complete testing is imposible;  Testing shouldn’t be executed for developers; April 16, 2010 4
  5. 5. Software Testability  “Software Testability means how easy the software is for testing” [PRESS];  There are lots of characteristics of a testable software:  Operability;  Observables;  Controllability;  Decouplability;  Simplicity;  Stability;  Understandably; April 16, 2010 5
  6. 6. Software Testability  How to design for software testability and how to measure the degree to which you have achieve it?  Software testability is the process of investigate where software faults are and how to execute [Voas];  Testability provide some guidance for testing;  Testability can give you confidence of correctness in fewer tests; April 16, 2010 6
  7. 7. Software Testability – Design for Testability  Testability’s goal is to assess software accurately enough to demonstrate whether or not it has high quality;  There are two ways to reduce the number of required tests:  Select tests that have a great ability to reveal faults;  Design software that has a greater ability to fail when faults do exist (design for testability);  There are several criteria on program design:  More of the code must be exercised for each input;  Programs must contain constructs that are likely to cause the state of the program to become incorrect if the constructs are themselves incorrect;  Programs must be able to propagate incorrect states into software failures; April 16, 2010 7
  8. 8. Software Testability – Design for Testability  Information Loss;  Implicit Information Loss;  Explicit Information Loss;  Design Heuristics;  Specification Decomposition;  Minimizations of variation reuse;  Analysis; April 16, 2010 8
  9. 9. Software Testability – Sensitive Analysis  “Sensitive Analysis quantifies behavioral information about the likelihood that faults are hiding” [Voas];  Repeatedly executes the original program and mutations of its source code and data states using two assumptions:  Single-fault assumption;  Simple-fault assumption;  The specific purpose of sensitivity analysis is to provide information that suggests how small the program’s smallest faults are likely to be;  The strength of sensitivity analysis is that your prediction is based on observed effects from actual faults;  The weakness is that the faults injected and observed are only a small set from what might be an infinite class of faults; April 16, 2010 9
  10. 10. Software Testability – Sensitive Analysis  Execution Analysis;  Infection Analysis;  Propagation Analysis;  Implementation; April 16, 2010 10
  11. 11. Test Case Projects  Main goal of Test Projects is having a great probability of finding faults in software;  Hitachi Software, has attained such a high quality software that only 0.02 percent of all bugs in software program emerge at user’s site [Yamaura]; April 16, 2010 11
  12. 12. Test Case Projects  Documenting Test Cases - Benefits  Designing test cases gives you a chance to analyze the specification from a different angle;  You can repeat the same test cases;  Somebody else can execute the test cases for you;  You can easily validate the quality of the test cases;  You can estimate the quality of the target software early on;  Schedule, Cost, and Personnel  An average project might apportion 12 months spend 2 months in testing by quality assurance team;  The test-case density;  A project with 100,000 LOC needs approximately 10,000 test cases – 1,000 per programmer; April 16, 2010 12
  13. 13. Test Case Projects  Steps for Debugging and Testing:  Design a set of test cases;  Check the test cases;  Do code inspection based on the test cases;  Do machine debugging based on the test cases;  Collect quality-related data during debugging;  Analyze the data during debugging; April 16, 2010 13
  14. 14. Test Case Projects  Main ways of testing a software:  White-box Testing;  Basic Way Testing;  Graph Flow Notation;  Cyclomatic Complexity;  Test Case Derivation;  Graph Matrix;  Control Structure Testing;  Condition Testing;  Data Flow Testing;  Cycle Testing;  Black-box Testing;  Graph Based Testing Method;  Partition Equivalence;  Comparison Testing;  Orthogonal Matrix Testing; April 16, 2010 14
  15. 15. Environment Testing, Architecture and Specialized Applications  GUI Testing;  Client/Server Architecture Testing;  Documentation and Help Devices Testing;  Real Time Systems Testing; April 16, 2010 15
  16. 16. Software Test Strategies  Integration of test case projects methods in well planed steps, resulting in a good software construction;  It should be sufficiently flexible for holding test method and hard for holding planning and management;  A software strategy should execute low and high level tests; April 16, 2010 16
  17. 17. Software Test Strategies  A Strategic Approach for Testing Software  Verification and Validation  Verification means the activities that assure the software implements a function correctly;  Software Test Organization  Software developers should test execute only unit tests;  Independent test group (ITG) is responsible for execute destructive tests in the software integration;  Software developers should be available for correcting system’s faults;  A Software Testing Strategy  Unit Test;  Integration Test;  Validation Test;  System Test; April 16, 2010 17
  18. 18. Software Test Strategies  A Strategic Approach for Testing Software  Testing Completion Criterions  When the test phase is finished?  Based on Statistical Criterions [PRESS]: f(t) = (1/p) ln[l0 pt + 1] April 16, 2010 18
  19. 19. Software Test Strategies  Strategically Aspects  Specify product requirements in a quantifiable way before test;  Show test goals in a clear way;  Understand users software and develop a profile for each user category;  Develop a test plan that emphasize a fast cycle test;  Build a robust software projected for testing itself;  Use effective technical formal revisions as a filter before testing;  Conduct technical formal revisions for evaluate a test strategy and test cases;  Develop an approach for improve the test process; April 16, 2010 19
  20. 20. Software Test Strategies  Unit Test  Used for testing software components separately ;  It uses white-box test for exercising code lines;  Test cases should be planned with the expected results for comparison; April 16, 2010 20
  21. 21. Software Test Strategies  Integration Test  If separated components work, why should I make integration tests?  Loosing data through an interface;  A module should have a not expected effect;  Top-Down Integration;  Bottom-up Integration;  Regression Test;  Smoke Test; April 16, 2010 21
  22. 22. Software Test Strategies  Top-Down Integration  First-In-Depth Integration;  First-In-Width Integration;  The main control module is used first for testing;  The next modules are being added;  Tests are being executed while components are added;  Regression tests can be executed for assure new errors are not added; M1 M2 M3 M4 M5 M6 M7 M8 April 16, 2010 22
  23. 23. Software Test Strategies  Bottom-up Integration  Atomic modules are added before in the system;  Components are being placed when low level components are tested; Mc Ma Mb D1 D2 D3 April 16, 2010 23
  24. 24. Software Test Strategies  Regression Test  All components added in the software may cause errors in components added before;  Regression test strategy test components tested before to assure new components didn’t affected it;  Some tools can be used for execute regression test;  It should be separated in modules;  Analyzing Regression Test Selection Techniques [Rothernel]  Regression Test Selection for Fault Detection;  Framework for Analyzing Regression Test Selection Techniques  Inclusiveness;  Precision;  Efficiency;  Generality;  Tradeoffs; April 16, 2010 24
  25. 25. Software Test Strategies  Analyzing Regression Test Selection Techniques  An Analysis of Regression Test Selection Techniques April 16, 2010 25
  26. 26. Software Test Strategies  Analyzing Regression Test Selection Techniques  An Analysis of Regression Test Selection Techniques April 16, 2010 26
  27. 27. Software Test Strategies  Analyzing Regression Test Selection Techniques  An Analysis of Regression Test Selection Techniques April 16, 2010 27
  28. 28. Software Test Strategies  Smoke Test  Used in software products developed fast;  Modules built by components are tested;  Modules are integrated and tested daily:  Integration risk;  Final product quality;  Errors diagnose are simplified;  Easy progress evaluation; April 16, 2010 28
  29. 29. Software Test Strategies  Validation Tests  Started just after integration tests;  Used for verify results based on requirements of the system;  An important element in validation test is called configuration revision;  Customer should validate requirements using system;  Alfa tests are tests executed in the user environment in the developer presence;  Beta tests are executed by the user without developer’s presence;  Validation, Verification, and Testing: Diversity Rules [Kitchenham]  Practical Problems with Operational Testing  It assumes that the most frequently occurring operations will have the highest manifestation of faults;  Transition situations are often the most complete error-prone; April 16, 2010 29
  30. 30. Software Test Strategies  Validation, Verification, and Testing: Diversity Rules [Kitchenham]  Testing Critical Function  Critical functions have extremely severe consequences if they fail;  This leads to a problem with identifying the reliability of the system as a whole;  The product reliability can be defined in terms of failure in a given period of time;  For critical functions, you are likely to obtain a measure of reliability related of failure on demand;  The Need For Non-Execution-Based Testing  You can only ignore nonoperational testing if:  All faults found by execution testing have exactly the same cost to debug and correct  All faults can be detected, and  The relative size of faults is constant across different operations for all time April 16, 2010 30
  31. 31. Software Test Strategies  Validation, Verification, and Testing: Diversity Rules [Kitchenham]  Diminishing Returns  Operational test becomes less and less useful for fault detection when it have been removed faults from the most commonly used functions;  A systems engineer should mix some techniques, including:  Additional testing boundary and transition conditions and of critical functions  Design inspections and reviews to identify specification and design faults early in development process  Proofs for functions whose correct outcome cannot otherwise be verified, and  Operational testing for those initial tests aimed at identifying the largest faults and assessing reliability. April 16, 2010 31
  32. 32. Software Test Strategies  System Test  Should exercise the system as a whole for find errors;  Recovery Test:  Enforce the system to fail;  Verify the system recovery;  Security Test:  Verify if hackers can affect the system;  Testers should try to obtain secure data from the system;  Stress Test  Execute system in a way it should require almost all resources;  Sensibility Test  Try to find data that the system fail;  Performance Test  For systems need to provide performance requirements;  Executed with Stress Test; April 16, 2010 32
  33. 33. Software Test Strategies  Debugging  Process started after the tester find an error and try to remove it;  Relation between a fault and its cause;  Two possible results:  The cause is found and corrected;  The cause is not found;  Why debugging is so difficult?  Error and cause geographically remote;  Error can disappear when another error was corrected;  Error can be caused by non-error;  Error can be cause by an human mistake;  Error can be a time problem;  Error can be difficult to be found;  Error can be caused by distributed reasons;  Error can be found in a component; April 16, 2010 33
  34. 34. Component Testing  Testing Component-Based Software: A Cautionary Tale [Weyuker]  The Ariane 5 Lesson  In June 1996, during the maiden voyage of the Ariane 5 launch vehicle, the launcher veered off course and exploded less than one minute after take-off;  The explosion resulted from insufficiently tested software reused from Ariane 4 launcher;  Testing Newly Developed Software  Unit Testing;  Integration Testing;  System Testing; April 16, 2010 34
  35. 35. Component Testing  Testing Component-Based Software: A Cautionary Tale [Weyuker]  Problems with systems built from reusable components:  Performance problems;  Fitting selected components together;  Absence of low-level understanding;  Developing a component for a particular project with no expectation of reuse, testing proceed as usual;  Significant additional testing should be done for changing priority;  Modify or not the source code?  Facilitating Reusable Component Testing  Associate a person or a team for maintaining each component;  Add software specification;  Add test suite;  Add pointers between the specification and parts of the test suite; April 16, 2010 35
  36. 36. Component Testing  Third-Party Testing and the Quality of Software Components [Councill]  10 contribuitors devoted 70 person-hour to developing a definition of component;  Ariane Project;  Certification business introduced a Maturity Model focused on large organizations;  99 percent of all US businesses are small businesses;  Producers provide the developing and testing procedures, purchasers or their agents conduct second-party testing and independent testing organizations perform third-party testing; April 16, 2010 36
  37. 37. References  [PRESS] Pressman, Roger S., Engenharia de Software, 5° Ed., Rio de Janeiro, MacGraw-Hill, 2002.  [Councill] William T. Councill, Third-Part Testing and the Quality of Software Components, IEEE Software, Quality Time, July 1999.  [Adams] Tom Adams, Functionally Effective Functional Testing Primer, IEEE Software, bookshelf, September 1995.  [Rothermel] G. Rothermel and M. J. Harrold, Analyzing Refression Test Selection Techniques, IEEE Transactions of Software Engineering, Vol. 22, N° 8, August 1996.  [Stocks] P. Stocks and D. Carrington, A Framework for Specification-Based Testing, IEEE Transactions on Software Engineering, Vol. 22, N° 11, November 1996.  [Harrold] M. J. Harrold, Testing a Roadmap, College of Computing, Georgia Institute Technology, 2000. April 16, 2010 37
  38. 38. References  [Franki] P. G. Franki and R. G. Hamlet, Evaluating Testing Methods by Delivered Reliability, IEEE Transactions on Software Engineering, Vol. 24, N° 8, August 1998.  [Kitchenham] B. Kitchenham and S. Linkman, Validation, Verification, and Testing: Diversity Rules, IEEE Software, August 1998.  [Voas] J. M. Voas and K. W. Miller, Software Testability: The New Verification, IEEE Software, May 1995.  [Yamaura] T. Yamaura, How to Design Practical Test Cases, IEEE Software, December 1998.  [Weyuker] E. J. Weyuker, Testing Component-Based Software: A Cautionary Tale, IEEE Software, October 1998. April 16, 2010 38