Your SlideShare is downloading. ×
Automation testing  strategy, approach & planning
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Automation testing strategy, approach & planning

25,082
views

Published on

Published in: Education

0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
25,082
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
1,055
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. IBM Global Services – Testing Competency IBM Confidential © Copyright IBM Corporation 200501/23/2005 Strategy, Approach & Planning
  • 2. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 2 2 IBM Confidential Agenda  Introduction  Effective Automation  Fundamental Approach  Real Facts & Think Factor  Considerations for Automation  Automation Start-up Strategies to Avoid  Guidelines & Approach  Automation Methodologies  Overview on Automation Tools  References
  • 3. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 3 3 IBM Confidential Introduction  GUI automation is always viewed as the SILVER BULLET, which will save money, time and resources  Test automation raises our hopes, yet often frustrates & disappoints us  Promises to deliver us from a tough situation, yet can create as many problems as it solves "Embarking on a successful automation test project is similar to embarking on a successful software development project“
  • 4. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 4 4 IBM Confidential Effective Automation  Reduced utilization of manual resources  More efficient use of scarce hardware resources  Reduced test cycle time  Reduced cost of testing, after initial cost of implementation  Improved reliability of the testing  Repeatable tests, thus overcoming boredom-induced missed bugs  Improved test coverage, since testing may be done more frequently  Development of programming skills  May allow the use of less skilled staff to do the actual testing
  • 5. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 5 5 IBM Confidential Fundamental Approach  Understand the need / requirement for automation  Point out pitfalls to avoid  Promote an evolving implementation  Begin simple and increase in complexity as the team's skills increase "Learning from others is the most effective way to grow your automation expertise"
  • 6. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 6 6 IBM Confidential Real Facts & Think factors Real Fact  Automation is initially expensive and it may not do a better job than manual testing in some situations Think Factors 1. How much will automating and running a test cost, versus running it manually? 2. Automated test has a limited lifetime in which it must recoup the additional cost. Will this test be reusable long enough? What factors will prevent it from being reusable? 3. The automated test will find bugs it was designed to test for and in addition it will find bugs that are not directly related to the test. What is the likelihood that this test will find additional bugs?
  • 7. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 7 7 IBM Confidential Real Facts & Think factors Real Fact  Automation is initially expensive and it may not do a better job than manual testing in some situations Think Factors 1. How much will automating and running a test cost, versus running it manually? 2. Automated test has a limited lifetime in which it must recoup the additional cost. Will this test be reusable long enough? What factors will prevent it from being reusable? 3. The automated test will find bugs it was designed to test for and in addition it will find bugs that are not directly related to the test. What is the likelihood that this test will find additional bugs?
  • 8. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 8 8 IBM Confidential Considerations for Automation  Test cases those are long running, repetitive, and non-subjective  Projects / Products that have a working, relatively stable GUI  Projects / Products that will span multiple releases.
  • 9. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 9 9 IBM Confidential Guidelines & Approach Record / Playback initially using the Testing tool’s wizard or toolbar will be easier but hard to maintain because they represent not only WHAT needs to be tested, but HOW it is to be tested Guideline: 1 Once familiarity in scripting is achieved, continue using the scripting language of an automation tool to provide your own implementation logic (framework) that is separate from your test case scripts  Requires more investment upfront, but the code will be less expensive to maintain over the life span of multiple product releases Example: Imagine having recorded test case scripts for a banking application that initially allowed the user to simply click "OK" to commit a sensitive transaction, then the application developers changed the navigational logic requiring the user to type "yes" to confirm these transactions. Every recorded test case that has hard coded "OK" clicks relating to the sensitive transaction commits would require maintenance.
  • 10. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 10 10 IBM Confidential Guidelines & Approach (Contd..) Automation code without any Discipline - Tendency to jump in and start developing new test scripts without collaborating with other team's members and without establishing a plan, structure, and/or standards is natural. However, in the long term this undisciplined approach is not productive. Any test case scripts and frameworks that are produced in this manner tend to lack such characteristics as: maintainability, reusability, readability, flexibility, reliability, effectiveness, and efficiency Guideline: 2 Establish complete Automation plan, Structure, and Standards “Quality automation effort should follow a software development process”
  • 11. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 11 11 IBM Confidential Guidelines & Approach (Contd..) Guideline: 3 “Separating the What from the How” An approach to separate what needs to be tested from how it is to be tested. To accomplish this separation, guideline would suggest to have the test script writers write the test scripts to identify what needs to be tested and automation developers write a framework to handle the implementation (i.e., the navigation logic, window processing logic, and verification logic) of your tests. Framework - "a supporting or enclosing structure”
  • 12. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 12 12 IBM Confidential Guideline: 4 Use the 6 phases for implementing the best course of Automated Testing action - Evaluation - Planning - Design - Implementation - Deployment - Review Iteration Cycle – Plan for future changes Guidelines & Approach (Contd..)
  • 13. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 13 13 IBM Confidential Guidelines & Approach (Contd..) Guideline: 5 A. Define Automation Goals & Prioritize At every point in the specification process, you can check the "testability" of the requirement or specification by ensuring it is S.M.A.R.T. S - Specific M - Measurable A - Attainable or Achievable R - Realistic T - Timely B. Define Automation Strategy - GUI - Roundtrip / Segment, Define objects, Synchronization - Types of Testing - Strategy Considerations 1. Partial automation 2. Time cost 3. Testing delay 4. Inertia 5. Risk of missed bugs
  • 14. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 14 14 IBM Confidential Guidelines & Approach (Contd..) Guideline: 6 Define Users and Skill Level Required / Desired - Test Script Writers - Automation Developers Guideline: 7 Resource Planning / Source Code Management - Hardware - Software - People
  • 15. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 15 15 IBM Confidential Guidelines & Approach (Contd..) Guideline: 8 Define Design Goals & Prioritize - Maintainability - Structured - Well Documented - Reviewed - Framework Specific Goals - Modularity - Reusability - Usability - Scalability - Reusability - Portability - Robustness - Reliability - Efficiency - Flexibility
  • 16. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 16 16 IBM Confidential Guidelines & Approach (Contd..) Guideline: 9 Determine the Automation Interface Architecture - Data Driven Interface - Programming Interface Guideline: 10 Determine the Automation Framework Architecture - Error Recovery - Support of Custom Controls - Product Testability Issues - NLS – Localization Guideline: 11 Define Standards – Naming & Code Conventions
  • 17. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 17 17 IBM Confidential Guidelines & Approach (Contd..) Components of Automated Test System Test Script Writers Automation Developers Test Scripts “WHAT” Interface Framework “HOW” Documents  Framework Usage Notes  Guidelines  Naming Conventions
  • 18. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 18 18 IBM Confidential Automation Methodologies  Functional Decomposition Method This methodology is to reduce all the test cases to their most fundamental tasks, and write User-Defined Functions, Business Function Scripts, and "Sub-routine" or "Utility" Scripts which perform these tasks independently of one another  Key-Word Driven or Test Plan Driven Method This method uses the actual Test Case document developed by the tester using a spreadsheet containing special "Key-Words". This method preserves most of the advantages of the "Functional Decomposition" method, while eliminating most of the disadvantages. In this method, the entire process is data-driven, including functionality. The Keyword controls the entire processing
  • 19. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 19 19 IBM Confidential Overview on Automation Tools  Most of the Automated Tools in the market remain as an integrated suite for Test Documentations, Scripting, Executions & Defect Tracking  Automation tool does exist for specific testing needs such as Performance Testing Code Coverage Memory Leak Detection  Automation Tools available in the market which are more on focus today includes  Rational Tools – Robot, Test Manager, ClearQuest  Mercury Interactive – WinRunner, LoadRunner, QTP, Astra Load Test  Segue – Silk Performer
  • 20. IBM Global ServicesIBM Global Services – Testing Competency © Copyright IBM Corporation 2005 20 20 IBM Confidential References Web Sites for Additional Resources 1. http://www.io.com/~wazmo/qa/ Software Testing Hotlist, maintained by Bret Pettichord, has links to many of the articles and papers mentioned in this document 2. http://www.softwareqatest.com/index.html Software QA and Testing Resource Center, maintained by Rick Hower, links 3. http://qaforums.com/ QA Forums, maintained by individual moderators, links to forums on Quality Assurance, Tools, Consulting & Employment Bibliography 1. Dustin, Elfriede, Rashka, Jeff, and Paul, John (1999) Automated Software Testing: Introduction, Management and Performance. Addison-Wesley Longman, Inc. Reading, Massachusetts 2. Fewster, Mark and Graham, Dorothy (1999) Software Test Automation: Effective use of test execution tools. Addison-Wesley: ACM Press, New York 3. Kaner, Cem, Falk, Jack, and Nguyen, Hung Quoc (1999) Testing Computer Software. Wiley Computer Publishing, New York
  • 21. IBM Global Services – Testing Competency IBM Confidential © Copyright IBM Corporation 200501/23/2005