Your SlideShare is downloading. ×

Component Testing: Part I


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

No notes for slide


  • 1. Component Testing: Part I Ednaldo Dilorenzo de Souza Filho [email_address]
  • 2. Summary
    • Introduction
    • Fundaments of Testing
    • Software Testability
    • Test Case Projects
    • Environment Testing, Architecture and Specialized Applications
    • Software Test Strategies
    • Component Testing
  • 3. Introduction
    • Testing is an important process to support assurance ;
    • As software becomes more pervasive and is used more often to perform critical tasks, it will be required to be of higher quality ;
    • Because testing requires the execution of the software, it is often called dynamic analysis [Harrold];
    • Testing consists in compare the outputs from the executions with expected results to identify those test cases on which the software failed;
    • Testing cannot show the absence of faults, if can show only their presence.
  • 4. Fundaments of Testing
    • Testing goals:
      • Find faults in a software program;
      • A good test case have a high probability to find errors;
      • A good test process should find errors;
    • Testing fundamentals:
      • Testing should be connected with client requirements;
      • Testing should be planned before testing execution;
      • Isolation of suspected components ;
      • Testing should start in individual components;
      • Complete testing is imposible;
      • Testing shouldn’t be executed for developers;
  • 5. Software Testability
    • “ Software Testability means how easy the software is for testing” [PRESS];
    • There are lots of characteristics of a testable software:
      • Operability ;
      • Observables ;
      • Controllability ;
      • Decouplability ;
      • Simplicity ;
      • Stability ;
      • Understandably ;
  • 6.
    • How to design for software testability and how to measure the degree to which you have achieve it?
    • Software testability is the process of investigate where software faults are and how to execute [Voas];
    • Testability provide some guidance for testing;
    • Testability can give you confidence of correctness in fewer tests;
    Software Testability
  • 7. Software Testability – Design for Testability
    • Testability’s goal is to assess software accurately enough to demonstrate whether or not it has high quality ;
    • There are two ways to reduce the number of required tests:
      • Select tests that have a great ability to reveal faults;
      • Design software that has a greater ability to fail when faults do exist (design for testability);
    • There are several criteria on program design:
      • More of the code must be exercised for each input;
      • Programs must contain constructs that are likely to cause the state of the program to become incorrect if the constructs are themselves incorrect;
      • Programs must be able to propagate incorrect states into software failures;
  • 8.
    • Information Loss;
    • Implicit Information Loss;
    • Explicit Information Loss;
    • Design Heuristics;
    • Specification Decomposition;
    • Minimizations of variation reuse;
    • Analysis;
    Software Testability – Design for Testability
  • 9.
    • “ Sensitive Analysis quantifies behavioral information about the likelihood that faults are hiding” [Voas];
    • Repeatedly executes the original program and mutations of its source code and data states using two assumptions:
      • Single-fault assumption;
      • Simple-fault assumption;
    • The specific purpose of sensitivity analysis is to provide information that suggests how small the program’s smallest faults are likely to be;
    • The strength of sensitivity analysis is that your prediction is based on observed effects from actual faults;
    • The weakness is that the faults injected and observed are only a small set from what might be an infinite class of faults;
    Software Testability – Sensitive Analysis
  • 10.
    • Execution Analysis;
    • Infection Analysis;
    • Propagation Analysis;
    • Implementation;
    Software Testability – Sensitive Analysis
  • 11.
    • Main goal of Test Projects is having a great probability of finding faults in software;
    • Hitachi Software, has attained such a high quality software that only 0.02 percent of all bugs in software program emerge at user’s site [Yamaura];
    Test Case Projects
  • 12. Test Case Projects
    • Documenting Test Cases - Benefits
      • Designing test cases gives you a chance to analyze the specification from a different angle;
      • You can repeat the same test cases;
      • Somebody else can execute the test cases for you;
      • You can easily validate the quality of the test cases;
      • You can estimate the quality of the target software early on;
    • Schedule, Cost, and Personnel
      • An average project might apportion 12 months spend 2 months in testing by quality assurance team;
      • The test-case density ;
      • A project with 100,000 LOC needs approximately 10,000 test cases – 1,000 per programmer;
  • 13. Test Case Projects
    • Steps for Debugging and Testing:
      • Design a set of test cases;
      • Check the test cases;
      • Do code inspection based on the test cases;
      • Do machine debugging based on the test cases;
      • Collect quality-related data during debugging;
      • Analyze the data during debugging;
  • 14.
    • Main ways of testing a software:
      • White-box Testing;
        • Basic Way Testing;
          • Graph Flow Notation;
          • Cyclomatic Complexity;
          • Test Case Derivation;
          • Graph Matrix;
        • Control Structure Testing;
          • Condition Testing;
          • Data Flow Testing;
          • Cycle Testing;
      • Black-box Testing;
        • Graph Based Testing Method;
        • Partition Equivalence;
        • Comparison Testing;
        • Orthogonal Matrix Testing;
    Test Case Projects
  • 15. Environment Testing, Architecture and Specialized Applications
    • GUI Testing;
    • Client/Server Architecture Testing;
    • Documentation and Help Devices Testing;
    • Real Time Systems Testing;
  • 16. Software Test Strategies
    • Integration of test case projects methods in well planed steps, resulting in a good software construction;
    • It should be sufficiently flexible for holding test method and hard for holding planning and management;
    • A software strategy should execute low and high level tests;
  • 17. Software Test Strategies
    • A Strategic Approach for Testing Software
      • Verification and Validation
        • Verification means the activities that assure the software implements a function correctly;
      • Software Test Organization
        • Software developers should test execute only unit tests;
        • Independent test group (ITG) is responsible for execute destructive tests in the software integration;
        • Software developers should be available for correcting system’s faults;
      • A Software Testing Strategy
        • Unit Test;
        • Integration Test;
        • Validation Test;
        • System Test;
  • 18. Software Test Strategies
    • A Strategic Approach for Testing Software
      • Testing Completion Criterions
        • When the test phase is finished?
        • Based on Statistical Criterions [PRESS]:
    f(t) = (1/p) ln[l 0 pt + 1]
  • 19. Software Test Strategies
    • Strategically Aspects
      • Specify product requirements in a quantifiable way before test;
      • Show test goals in a clear way;
      • Understand users software and develop a profile for each user category;
      • Develop a test plan that emphasize a fast cycle test;
      • Build a robust software projected for testing itself;
      • Use effective technical formal revisions as a filter before testing;
      • Conduct technical formal revisions for evaluate a test strategy and test cases;
      • Develop an approach for improve the test process;
  • 20. Software Test Strategies
    • Unit Test
      • Used for testing software components separately ;
      • It uses white-box test for exercising code lines;
      • Test cases should be planned with the expected results for comparison;
  • 21. Software Test Strategies
    • Integration Test
      • If separated components work, why should I make integration tests?
        • Loosing data through an interface;
        • A module should have a not expected effect;
      • Top-Down Integration;
      • Bottom-up Integration;
      • Regression Test;
      • Smoke Test;
  • 22. Software Test Strategies
    • Top-Down Integration
      • First-In-Depth Integration;
      • First-In-Width Integration;
      • The main control module is used first for testing;
      • The next modules are being added;
      • Tests are being executed while components are added;
      • Regression tests can be executed for assure new errors are not added;
    M1 M2 M7 M5 M6 M3 M8 M4
  • 23. Software Test Strategies
    • Bottom-up Integration
      • Atomic modules are added before in the system;
      • Components are being placed when low level components are tested;
    Mc Ma Mb D1 D2 D3
  • 24. Software Test Strategies
    • Regression Test
      • All components added in the software may cause errors in components added before;
      • Regression test strategy test components tested before to assure new components didn’t affected it;
      • Some tools can be used for execute regression test;
      • It should be separated in modules;
    • Analyzing Regression Test Selection Techniques [Rothernel]
      • Regression Test Selection for Fault Detection;
      • Framework for Analyzing Regression Test Selection Techniques
        • Inclusiveness;
        • Precision;
        • Efficiency;
        • Generality;
        • Tradeoffs;
  • 25. Software Test Strategies
    • Analyzing Regression Test Selection Techniques
      • An Analysis of Regression Test Selection Techniques
  • 26. Software Test Strategies
    • Analyzing Regression Test Selection Techniques
      • An Analysis of Regression Test Selection Techniques
  • 27. Software Test Strategies
    • Analyzing Regression Test Selection Techniques
      • An Analysis of Regression Test Selection Techniques
  • 28. Software Test Strategies
    • Smoke Test
      • Used in software products developed fast;
      • Modules built by components are tested;
      • Modules are integrated and tested daily:
        • Integration risk;
        • Final product quality;
        • Errors diagnose are simplified;
        • Easy progress evaluation;
  • 29. Software Test Strategies
    • Validation Tests
      • Started just after integration tests;
      • Used for verify results based on requirements of the system;
      • An important element in validation test is called configuration revision;
      • Customer should validate requirements using system;
      • Alfa tests are tests executed in the user environment in the developer presence;
      • Beta tests are executed by the user without developer’s presence;
    • Validation, Verification, and Testing: Diversity Rules [Kitchenham]
      • Practical Problems with Operational Testing
        • It assumes that the most frequently occurring operations will have the highest manifestation of faults;
        • Transition situations are often the most complete error-prone;
  • 30. Software Test Strategies
    • Validation, Verification, and Testing: Diversity Rules [Kitchenham]
      • Testing Critical Function
        • Critical functions have extremely severe consequences if they fail;
        • This leads to a problem with identifying the reliability of the system as a whole;
        • The product reliability can be defined in terms of failure in a given period of time;
        • For critical functions, you are likely to obtain a measure of reliability related of failure on demand;
      • The Need For Non-Execution-Based Testing
        • You can only ignore nonoperational testing if:
          • All faults found by execution testing have exactly the same cost to debug and correct
          • All faults can be detected, and
          • The relative size of faults is constant across different operations for all time
  • 31. Software Test Strategies
    • Validation, Verification, and Testing: Diversity Rules [Kitchenham]
      • Diminishing Returns
        • Operational test becomes less and less useful for fault detection when it have been removed faults from the most commonly used functions;
        • A systems engineer should mix some techniques, including:
          • Additional testing boundary and transition conditions and of critical functions
          • Design inspections and reviews to identify specification and design faults early in development process
          • Proofs for functions whose correct outcome cannot otherwise be verified, and
          • Operational testing for those initial tests aimed at identifying the largest faults and assessing reliability.
  • 32. Software Test Strategies
    • System Test
      • Should exercise the system as a whole for find errors;
      • Recovery Test:
        • Enforce the system to fail;
        • Verify the system recovery;
      • Security Test:
        • Verify if hackers can affect the system;
        • Testers should try to obtain secure data from the system;
    • Stress Test
      • Execute system in a way it should require almost all resources;
      • Sensibility Test
        • Try to find data that the system fail;
    • Performance Test
      • For systems need to provide performance requirements;
      • Executed with Stress Test;
  • 33. Software Test Strategies
    • Debugging
      • Process started after the tester find an error and try to remove it;
      • Relation between a fault and its cause;
      • Two possible results:
        • The cause is found and corrected;
        • The cause is not found;
      • Why debugging is so difficult?
        • Error and cause geographically remote;
        • Error can disappear when another error was corrected;
        • Error can be caused by non-error;
        • Error can be cause by an human mistake;
        • Error can be a time problem;
        • Error can be difficult to be found;
        • Error can be caused by distributed reasons;
        • Error can be found in a component;
  • 34. Component Testing
    • Testing Component-Based Software: A Cautionary Tale [Weyuker]
      • The Ariane 5 Lesson
        • In June 1996, during the maiden voyage of the Ariane 5 launch vehicle, the launcher veered off course and exploded less than one minute after take-off;
        • The explosion resulted from insufficiently tested software reused from Ariane 4 launcher;
      • Testing Newly Developed Software
        • Unit Testing;
        • Integration Testing;
        • System Testing;
  • 35. Component Testing
    • Testing Component-Based Software: A Cautionary Tale [Weyuker]
      • Problems with systems built from reusable components:
        • Performance problems;
        • Fitting selected components together;
        • Absence of low-level understanding;
      • Developing a component for a particular project with no expectation of reuse, testing proceed as usual;
      • Significant additional testing should be done for changing priority;
      • Modify or not the source code?
      • Facilitating Reusable Component Testing
        • Associate a person or a team for maintaining each component;
        • Add software specification;
        • Add test suite;
        • Add pointers between the specification and parts of the test suite;
  • 36. Component Testing
    • Third-Party Testing and the Quality of Software Components [Councill]
      • 10 contribuitors devoted 70 person-hour to developing a definition of component ;
      • Ariane Project ;
      • Certification business introduced a Maturity Model focused on large organizations;
      • 99 percent of all US businesses are small businesses;
      • Producers provide the developing and testing procedures, purchasers or their agents conduct second-party testing and independent testing organizations perform third-party testing;
  • 37. References
    • [PRESS] Pressman, Roger S., Engenharia de Software , 5° Ed., Rio de Janeiro, MacGraw-Hill, 2002.
    • [Councill] William T. Councill, Third-Part Testing and the Quality of Software Components , IEEE Software, Quality Time, July 1999.
    • [Adams] Tom Adams, Functionally Effective Functional Testing Primer , IEEE Software, bookshelf, September 1995.
    • [Rothermel] G. Rothermel and M. J. Harrold, Analyzing Refression Test Selection Techniques , IEEE Transactions of Software Engineering, Vol. 22, N° 8, August 1996.
    • [Stocks] P. Stocks and D. Carrington, A Framework for Specification-Based Testing , IEEE Transactions on Software Engineering, Vol. 22, N° 11, November 1996.
    • [Harrold] M. J. Harrold, Testing a Roadmap , College of Computing, Georgia Institute Technology, 2000.
  • 38. References
    • [Franki] P. G. Franki and R. G. Hamlet, Evaluating Testing Methods by Delivered Reliability , IEEE Transactions on Software Engineering, Vol. 24, N° 8, August 1998.
    • [Kitchenham] B. Kitchenham and S. Linkman, Validation, Verification, and Testing: Diversity Rules , IEEE Software, August 1998.
    • [Voas] J. M. Voas and K. W. Miller, Software Testability: The New Verification , IEEE Software, May 1995.
    • [Yamaura] T. Yamaura, How to Design Practical Test Cases , IEEE Software, December 1998.
    • [Weyuker] E. J. Weyuker, Testing Component-Based Software: A Cautionary Tale , IEEE Software, October 1998.