Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

VV07

271 views

Published on

  • Be the first to comment

  • Be the first to like this

VV07

  1. 1. Testing Automation CFICSE VV07 October 1999 Dr. Terry Shepard Royal Military College of Canada Electrical and Computer Engineering 613-541-6000 ext. 6031
  2. 2. Testing Automation Topics <ul><li>Motivation for Automation </li></ul><ul><li>Design for Testability </li></ul><ul><li>What can be automated? </li></ul><ul><li>Context and Needs </li></ul><ul><li>Intrusive vs. non-intrusive </li></ul><ul><li>Software Test Technologies Report </li></ul><ul><ul><li>Classification of testing tools </li></ul></ul>
  3. 3. Motivation for Testing Automation <ul><li>reduce costs </li></ul><ul><li>ensure repeatability of tests </li></ul><ul><li>ensure that test scenarios are followed exactly </li></ul><ul><li>increase number of tests that can be run by a deadline </li></ul><ul><li>improve reliability estimation </li></ul>
  4. 4. Design for Testability <ul><li>Common for computer hardware </li></ul><ul><li>Primarily means </li></ul><ul><ul><li>Controllability </li></ul></ul><ul><ul><li>Observability </li></ul></ul><ul><ul><li>Well behaved interfaces between modules </li></ul></ul><ul><li>Determines what can easily be automated </li></ul><ul><ul><li>e.g. instrumentation code built into source code </li></ul></ul>
  5. 5. What can be automated? <ul><li>Test execution </li></ul><ul><li>Comparison of actual and expected results </li></ul><ul><li>Logging of state at the time of a failed test </li></ul><ul><ul><li>recovery for debugging purposes </li></ul></ul><ul><li>Management of test logs </li></ul><ul><li>Collection of metrics </li></ul><ul><li>Version control (in part) </li></ul><ul><li>Test case selection: hard! </li></ul>
  6. 6. Automated support can be provided for other tasks <ul><li>Test planning and budgeting </li></ul><ul><li>Test setup procedures </li></ul><ul><li>Test adequacy decisions </li></ul><ul><li>Test scenarios with external dependencies </li></ul>
  7. 7. Context and Need <ul><li>In a given project, the context will determine candidate testing tools </li></ul><ul><ul><li>constraints include language, platform, OS, … </li></ul></ul><ul><li>Given a set of constraints, a market search and evaluation can be undertaken </li></ul><ul><li>Broad range of applicability and variety of tools makes needs assessment hard </li></ul><ul><li>Cost justification should be applied </li></ul>
  8. 8. Intrusive vs. Non-Intrusive <ul><li>Intrusion is generally not possible in an operational setting </li></ul><ul><li>Build black box recorder capabilities into the software? </li></ul><ul><li>Continuing hardware performance improvements will make this increasingly feasible </li></ul>
  9. 9. Software Test Technologies Report <ul><li>August 1994 </li></ul><ul><li>Software Technology Support Center, Hill AFB, Air Logistics Command, Ogden, Utah </li></ul><ul><ul><li>Online at www.stsc.hill.af.mil </li></ul></ul><ul><ul><li>STSC has an Annual Conference, publishes Crosstalk, … </li></ul></ul><ul><li>No update in classification since 1994 </li></ul>
  10. 10. 1994 STSC Test Tools Classification <ul><li>Test Resource Management </li></ul><ul><li>Requirements and Design Test Support </li></ul><ul><li>Implementation and Maintenance Test Support </li></ul><ul><ul><li>Compilers </li></ul></ul><ul><ul><li>Source Code Static Analyzers </li></ul></ul><ul><ul><li>Test Preparation </li></ul></ul><ul><ul><li>Test Execution (Dynamic Analyzers) </li></ul></ul><ul><ul><li>Test Evaluators </li></ul></ul>
  11. 11. Test Resource Management Tools <ul><li>Configuration Managers </li></ul><ul><ul><li>test version control </li></ul></ul><ul><ul><li>defect list management </li></ul></ul><ul><ul><li>change control management </li></ul></ul><ul><li>Project Managers </li></ul><ul><ul><li>can help to elevate the visibility of testing early in a project </li></ul></ul>
  12. 12. Requirements and Design Test Support Tools <ul><li>Analyzers for Software Plans, Requirements, and Designs </li></ul><ul><li>System/Prototype Simulators </li></ul><ul><li>Requirements Tracers </li></ul><ul><li>Requirements-Based Test Case Generators </li></ul><ul><li>Test Planners </li></ul>
  13. 13. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  14. 14. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  15. 15. Source Code Static Analyzers <ul><li>Auditors (enforce style manuals) </li></ul><ul><li>Complexity Measurers </li></ul><ul><li>Cross Referencing Tools </li></ul><ul><ul><li>e.g. to verify uses </li></ul></ul><ul><li>Size Measurers </li></ul><ul><li>Structure Checkers </li></ul><ul><li>Syntax and Semantics Analyzers </li></ul>
  16. 16. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  17. 17. Test Preparation Tools <ul><li>Data Extractors </li></ul><ul><li>Requirements-Based Test Case Generators </li></ul><ul><li>Test Data Generators </li></ul><ul><li>Test Planners </li></ul>
  18. 18. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  19. 19. Test Execution Tools (Dynamic Analyzers) - Part 1 <ul><li>Assertion Analyzers </li></ul><ul><li>Capture-Replay Tools </li></ul><ul><li>Coverage/Frequency Analyzers </li></ul><ul><li>Debuggers </li></ul><ul><li>Emulators </li></ul><ul><li>Network Analyzers </li></ul><ul><li>Performance/Timing Analyzers </li></ul>
  20. 20. Test Execution Tools (Dynamic Analyzers) - Part 2 <ul><li>Run-Time Error Checkers </li></ul><ul><li>Simulators </li></ul><ul><li>Status Displayer/Session Documenters </li></ul><ul><li>Test Execution Managers </li></ul><ul><li>Validation Suites </li></ul>
  21. 21. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  22. 22. Test Evaluators <ul><li>Comparators </li></ul><ul><li>Data Reducers and Analyzers </li></ul><ul><li>Defect/Change Trackers </li></ul>
  23. 23. Other sources for testing tool info <ul><li>comp.software.testing </li></ul><ul><li>www.cs.queensu.ca/shepard </li></ul><ul><li>www.soft.com </li></ul><ul><li>www.sqe.com </li></ul><ul><li>www.methods-tools.com </li></ul><ul><li>article by Robert Poston: A Guided Tour of Software Testing Tools </li></ul>
  24. 24. Testing Tool Market (Ovum - www.ovum.com) <ul><li>Currently $450M, growing at 30% per year </li></ul><ul><li>dominant players (with 60% of the total market) are: </li></ul><ul><ul><li>Mercury Interactive </li></ul></ul><ul><ul><li>Rational </li></ul></ul><ul><ul><li>Compuware </li></ul></ul><ul><ul><li>Segue </li></ul></ul>

×