VV07

393 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
393
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

VV07

  1. 1. Testing Automation CFICSE VV07 October 1999 Dr. Terry Shepard Royal Military College of Canada Electrical and Computer Engineering 613-541-6000 ext. 6031
  2. 2. Testing Automation Topics <ul><li>Motivation for Automation </li></ul><ul><li>Design for Testability </li></ul><ul><li>What can be automated? </li></ul><ul><li>Context and Needs </li></ul><ul><li>Intrusive vs. non-intrusive </li></ul><ul><li>Software Test Technologies Report </li></ul><ul><ul><li>Classification of testing tools </li></ul></ul>
  3. 3. Motivation for Testing Automation <ul><li>reduce costs </li></ul><ul><li>ensure repeatability of tests </li></ul><ul><li>ensure that test scenarios are followed exactly </li></ul><ul><li>increase number of tests that can be run by a deadline </li></ul><ul><li>improve reliability estimation </li></ul>
  4. 4. Design for Testability <ul><li>Common for computer hardware </li></ul><ul><li>Primarily means </li></ul><ul><ul><li>Controllability </li></ul></ul><ul><ul><li>Observability </li></ul></ul><ul><ul><li>Well behaved interfaces between modules </li></ul></ul><ul><li>Determines what can easily be automated </li></ul><ul><ul><li>e.g. instrumentation code built into source code </li></ul></ul>
  5. 5. What can be automated? <ul><li>Test execution </li></ul><ul><li>Comparison of actual and expected results </li></ul><ul><li>Logging of state at the time of a failed test </li></ul><ul><ul><li>recovery for debugging purposes </li></ul></ul><ul><li>Management of test logs </li></ul><ul><li>Collection of metrics </li></ul><ul><li>Version control (in part) </li></ul><ul><li>Test case selection: hard! </li></ul>
  6. 6. Automated support can be provided for other tasks <ul><li>Test planning and budgeting </li></ul><ul><li>Test setup procedures </li></ul><ul><li>Test adequacy decisions </li></ul><ul><li>Test scenarios with external dependencies </li></ul>
  7. 7. Context and Need <ul><li>In a given project, the context will determine candidate testing tools </li></ul><ul><ul><li>constraints include language, platform, OS, … </li></ul></ul><ul><li>Given a set of constraints, a market search and evaluation can be undertaken </li></ul><ul><li>Broad range of applicability and variety of tools makes needs assessment hard </li></ul><ul><li>Cost justification should be applied </li></ul>
  8. 8. Intrusive vs. Non-Intrusive <ul><li>Intrusion is generally not possible in an operational setting </li></ul><ul><li>Build black box recorder capabilities into the software? </li></ul><ul><li>Continuing hardware performance improvements will make this increasingly feasible </li></ul>
  9. 9. Software Test Technologies Report <ul><li>August 1994 </li></ul><ul><li>Software Technology Support Center, Hill AFB, Air Logistics Command, Ogden, Utah </li></ul><ul><ul><li>Online at www.stsc.hill.af.mil </li></ul></ul><ul><ul><li>STSC has an Annual Conference, publishes Crosstalk, … </li></ul></ul><ul><li>No update in classification since 1994 </li></ul>
  10. 10. 1994 STSC Test Tools Classification <ul><li>Test Resource Management </li></ul><ul><li>Requirements and Design Test Support </li></ul><ul><li>Implementation and Maintenance Test Support </li></ul><ul><ul><li>Compilers </li></ul></ul><ul><ul><li>Source Code Static Analyzers </li></ul></ul><ul><ul><li>Test Preparation </li></ul></ul><ul><ul><li>Test Execution (Dynamic Analyzers) </li></ul></ul><ul><ul><li>Test Evaluators </li></ul></ul>
  11. 11. Test Resource Management Tools <ul><li>Configuration Managers </li></ul><ul><ul><li>test version control </li></ul></ul><ul><ul><li>defect list management </li></ul></ul><ul><ul><li>change control management </li></ul></ul><ul><li>Project Managers </li></ul><ul><ul><li>can help to elevate the visibility of testing early in a project </li></ul></ul>
  12. 12. Requirements and Design Test Support Tools <ul><li>Analyzers for Software Plans, Requirements, and Designs </li></ul><ul><li>System/Prototype Simulators </li></ul><ul><li>Requirements Tracers </li></ul><ul><li>Requirements-Based Test Case Generators </li></ul><ul><li>Test Planners </li></ul>
  13. 13. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  14. 14. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  15. 15. Source Code Static Analyzers <ul><li>Auditors (enforce style manuals) </li></ul><ul><li>Complexity Measurers </li></ul><ul><li>Cross Referencing Tools </li></ul><ul><ul><li>e.g. to verify uses </li></ul></ul><ul><li>Size Measurers </li></ul><ul><li>Structure Checkers </li></ul><ul><li>Syntax and Semantics Analyzers </li></ul>
  16. 16. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  17. 17. Test Preparation Tools <ul><li>Data Extractors </li></ul><ul><li>Requirements-Based Test Case Generators </li></ul><ul><li>Test Data Generators </li></ul><ul><li>Test Planners </li></ul>
  18. 18. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  19. 19. Test Execution Tools (Dynamic Analyzers) - Part 1 <ul><li>Assertion Analyzers </li></ul><ul><li>Capture-Replay Tools </li></ul><ul><li>Coverage/Frequency Analyzers </li></ul><ul><li>Debuggers </li></ul><ul><li>Emulators </li></ul><ul><li>Network Analyzers </li></ul><ul><li>Performance/Timing Analyzers </li></ul>
  20. 20. Test Execution Tools (Dynamic Analyzers) - Part 2 <ul><li>Run-Time Error Checkers </li></ul><ul><li>Simulators </li></ul><ul><li>Status Displayer/Session Documenters </li></ul><ul><li>Test Execution Managers </li></ul><ul><li>Validation Suites </li></ul>
  21. 21. Implementation and Maintenance Test Support Tools <ul><li>Compilers (syntax checking) </li></ul><ul><li>Source Code Static Analyzers </li></ul><ul><li>Test Preparation Tools </li></ul><ul><li>Test Execution Tools (Dynamic Analyzers) </li></ul><ul><li>Test Evaluators </li></ul>
  22. 22. Test Evaluators <ul><li>Comparators </li></ul><ul><li>Data Reducers and Analyzers </li></ul><ul><li>Defect/Change Trackers </li></ul>
  23. 23. Other sources for testing tool info <ul><li>comp.software.testing </li></ul><ul><li>www.cs.queensu.ca/shepard </li></ul><ul><li>www.soft.com </li></ul><ul><li>www.sqe.com </li></ul><ul><li>www.methods-tools.com </li></ul><ul><li>article by Robert Poston: A Guided Tour of Software Testing Tools </li></ul>
  24. 24. Testing Tool Market (Ovum - www.ovum.com) <ul><li>Currently $450M, growing at 30% per year </li></ul><ul><li>dominant players (with 60% of the total market) are: </li></ul><ul><ul><li>Mercury Interactive </li></ul></ul><ul><ul><li>Rational </li></ul></ul><ul><ul><li>Compuware </li></ul></ul><ul><ul><li>Segue </li></ul></ul>

×