#22 Software Testing, FEEDS Presention #2 by Mr. Loren Stroup ...
Upcoming SlideShare
Loading in...5
×
 

#22 Software Testing, FEEDS Presention #2 by Mr. Loren Stroup ...

on

  • 1,323 views

 

Statistics

Views

Total Views
1,323
Views on SlideShare
1,323
Embed Views
0

Actions

Likes
0
Downloads
22
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

#22 Software Testing, FEEDS Presention #2 by Mr. Loren Stroup ... #22 Software Testing, FEEDS Presention #2 by Mr. Loren Stroup ... Presentation Transcript

  • Software Testing: Tools for Software Test Automation Robert V. Binder 2000 EEL6887: Software Engineering Life-Cycle Control Loren Stroup March 22, 2006
  • Reference Sources
    • Textbook: Testing Object-Oriented Systems: Models, Patterns, and Tools, by Robert V. Binder, Addison Wesley, 2000
    • Website: Are Automated Test Tools For Real?, by Alan R. Earls, May 1, 2004, http://www.adtmag.com/article.aspx?id=9307
    • Website: A Sampling of Automated Testing Tools, by Lana Gates, May 1, 2004, http://www.adtmag.com/article.aspx?id=9308
  • Overview
    • Test Automation Defined
    • Advantages to Automated Testing
    • Limitations and Caveats to Automated Testing
    • Test Harness Design
      • Test Case/Suite Method
      • Test Case/Suite Class
    • Automated Testing and the CMM
    • Views on Automated Test Tools
    • Sampling of Automated Test Tools
    • Summary
    • Conclusion
  • Test Automation
    • Definition:
      • Test automation is software that automates any aspect of testing of an application system
    • Test automation includes capabilities to generate test inputs and expected results, it has no manual intervention and it can evaluate pass/no pass
    • Automated testing depends on testing goals, budget, software process, kind of application under development, and particulars of the development and target environment
    • As project nears completion, two undesirable options are faced:
      • Rerun fewer baseline test cases and focus on the tests necessary to validate the final increments
      • Add more people to enter and judge the increasing number of baseline test cases
      • Both options involve risk: The first option has risk concerning error detection, the tester is hoping that software fixes have not introduced errors in other test cases. The second option has risk concerning adding more staff on a budget that cannot change and adding test cases that increases the testing schedule.
  • Test Automation (cont)
    • Six main steps to automated testing (post design/development of test driver):
      • Exercise the Implementation Under Test (IUT) with the automated test suite
      • Repair faults revealed by failures
      • Rerun the test suite on the revised IUT
      • Evaluate test suite coverage
      • Enhance the test suite to achieve the coverage goal
      • Rerun these suites to support regression testing or other forms of testing that require the ability to compare baseline results
    • Last step is the focal point of automated testing, it delivers quick results on how stable the baseline is
  • Advantages to Automated Testing
    • Permits quick and efficient verification of bug fixes
    • Speeds up debugging and reduces “bad fixes” by allowing the developer to rerun a test case to reproduce the problem and then verify that the bug fix works
    • Allows consistent capture and analysis of test results
    • Tester productivity is improved by allowing the time spent entering and reentering tests to be used on designing tests that achieve greater coverage
    • Manual tests are always error-prone while automated tests won’t type a wrong input or have other distractions
    • Automated comparison is a repeatable and efficient way to evaluate a large quantity of output, testers can become fatigued after scanning pages of output
  • Limitations and Caveats
    • Automating a test is typically more expensive than just running the application, if no need to repeat the tests is evident, then automation is probably not justified
    • When testing applications with significant user interaction, the tester can use this knowledge to try features and combinations that might be hard to identify when designing a GUI script
    • The cost of maintenance may be quite high if the requirements and implementation of the IUT change frequently. If the test automation is not maintained, it will become unusable and the expected return will not be realized
  • Test Harness Design
    • Definition:
      • A test harness is a system that supports effective and repeatable automated testing
    • The test harness is the system that starts the IUT, send test messages to it, and then evaluates test pass/no pass results
    • A test harness provides some of the following capabilities
      • Test case/test suite implementation
      • Interface to the IUT by which test messages may be sent
      • Test execution control
      • Automated comparator
      • Result logging
      • Regression test support – selection of a test suite subset and creation of the necessary test configuration
  • Test Harness Design (cont)
    • Test Harness design patterns and architecture
      • Test Cases – How should test message be implemented?
      • Test Control – How should the servers of the IUT and its environment be controlled and observed?
      • Test drivers – How should test code be organized so that it is modular, has a meaningful correspondence to the IUT, and achieves reuse where possible? How can drivers support controllability and observability?
      • Test Framework – How should test cases, test suites, drivers, and a tester interface be organized for an entire application system?
  • Test Harness Architecture
  • Test Case/Test Suite Method
    • Intent
      • Implement an API scope test case as a method of a driver class
    • Applicability
      • Useful at class and cluster scope for unit and integration testing when developing a driver to exercise a class, a class cluster, or other component that will be tested using its API
    • Advantages
      • Implementation is simple and easy to code
      • May be used with any kind of API driver class, can be inherited and overridden
    • Disadvantages
      • Pattern required programmer skills to design, develop, debug, and maintain
      • Test code is susceptible to the same problems as application code (bugs, low maintainability, low portability)
  • Test Case/Test Suite Method Pattern Structure
  • Test Case/Test Suite Class
    • Intent
      • Implement test cases or test suites as objects of class TestCase
    • Applicability
      • Useful at class and cluster scopes for unit and integration testing – developing a driver to exercise a class, class cluster, or other component that will be tested using its API
    • Advantages
      • Traceability and related information are easily represented and may be automatically updated as tests are run and rerun
      • The full power of the implementation language can be used to automate all aspects of test execution
    • Disadvantages
      • Pattern required programmer skills to design, develop, debug, and maintain
      • Test code is susceptible to the same problems as application code (bugs, low maintainability, low portability)
  • Test Case/Test Suite Class Pattern Structure
  • Automated Test and the CMM
    • Deciding when automated testing is appropriate involves reviewing past metrics on various programs, thus, a CMMI level of at least 3 is ideal
    • Automated testing can be used at any level of the CMM, however, recording and following process standards is evident at level 3 and higher, which makes automated testing easier to implement
    • At CMM levels 4 and 5, automated testing can be firmly established in the company’s processes and standards and the organization can predict trends in product quality and how automated testing will achieve this or if it is cost effective
    • Automated testing may involve training staff members and CMM levels of 3 and higher have training processes in place and have been used many times on other projects
  • Other Views on Automated Testing
    • Downside (Kanglin Li)
      • Vendors tools are impressive during demonstrations of basic things, however, when the tester needs to test software in depth, more functionality needs to be added to generated test scripts which can be complex and time consuming
      • Automated test tools can be expensive and will only be a good investment if the automated test scripts are reused frequently
    • Considerations (Theresa Lanowitz)
      • When considering to adopt automated tools, ensure there are skilled professionals in place and a good process
      • Automated tools are only useful when proper training is given to bring staffing up to speed on the tools
      • Documented, repeatable development and test process should be already established (i.e. CMMI level 3 and higher is valuable)
      • Test automation can significantly reduce risk only if processes and standards are followed version after version
  • Sampling of Automated Test Tools
    • QACenter Performance Edition 5.0 – Compuware Corp.
      • Used mainly for load and performance testing
      • http://www.compuware.com
    • QuickTest Professional – Mercury Interactive Corp.
      • Good for functional and automated regression test
      • Uses the concept of Keyword-driven testing to radically simplify test creation and maintenance
      • http://www.mercuryinteractive.com
    • WebLoad Analyzer – RadView Software Inc.
      • Good for Web application performance testing
      • http://www.radview.com
  • Sampling of Automated Test Tools (cont)
    • Unified TestPro (UTP) 7.0 – SDT Technologies
      • Good for functional testing; Test Transformation Suite – test methodology, tools, best practices, services
      • http://sdtcorp.com
    • SilkPerformer Component Test Edition – Segue Software Inc.
      • Good for remote component testing
      • SilkPerformer Lite is good for Web application testing
      • http://www.segue.com
  • Summary
    • Automated testing depends on testing goals, budget, software process, kind of application under development, and particulars of the development and target environment
    • Automated Testing has many advantages and disadvantages and must be used wisely and when it is deemed appropriate
    • A test harness is a system that supports effective and repeatable automated testing
    • Two kinds of test harness design are Test Case/Test Suite Method and Test Case/Test Suite Class
    • There are many automated test tools available through many companies and vendors
  • Conclusion
    • It’s up to an organization to decide whether automated testing is applicable for them
    • A CMMI level of 3 or higher will have the proper training and documentation on how to handle automated testing
    • Automated test tools can be expensive and will only be a good investment if the automated test scripts are reused frequently
    • Test harness design is a viable option for helping to automate testing at an organization
    • Automated testing is very powerful for regression testing efforts when complete systems need a full run through of test cases to ensure no other failures
    • Automated testing can greatly reduce risk if the processes and documentation at an organization are followed consistently