1. Software Test Automation Overview Rohan Bhattarai
2. Contents Overview of STA BEP and ROI Tools Automation Framework
3. Contents Overview of STA  Choosing the right tool – The Strategic  History Steps  Myths and Truths  Tools Classification  Why Automation Projects Fail?  Short List Tools – Functional  What is TA?  Short List Tools – Performance  Why TA is needed?  Evaluate Vendors  Manual vs. Automated  Functional Test Tools - Analysis  Pros & Cons  Feasibility Analysis  What to Automate?  Automation Framework  What Not to Automate?  Where to start? BEP and ROI  What is AF?  BEP  Benefits of AF  How does it work?  BEP - Example  Architecture  ROI  Framework Approaches  Classic ROI  Record and Playback  Real ROI  Script Based Approach  Benefits of ROI  Keyword Driven Approach  ROI Calculator  Data Driven Approach  Hybrid Keyword and Data Driven Approach Tools  Choosing the right tool – The Strategic Approach
4. Contents Overview  History  Myths and Truths  Why Automation Projects Fail?  What is TA?  Why TA is needed?  Manual vs. Automated  Pros & Cons  What to Automate?  What Not to Automate?
5. History Moving swiftly past the hype Historically Automation is perceived as a “Silver Bullet” of the Testing world  “The term has been adopted into a general metaphor, where “silver bullet” refers to any straight forward solution perceived to have extreme effectiveness. The phrase typically appears with an expectation that some new technology or practice will easily cure a major prevailing problem”.
7. Myths and Truths1. Test Automation is simple, that every tester can do it  Promoted by the Sales people by simply saying:  Record the script  Enhance the script by adding functions and data driving  Run the scripts  Report results  Under this influence QA manager can proudly say “All our testers are doing test automation”.
8. Myths and Truths But in Reality: TA is a software development task  It should be designed, developed and tested  Need to have some kind of a programming background to implement test automation. TA is not as complex as C++/C#/Java development.  TA components are assets that should be treated like application source code Don’t fall into tool vendor sales pitch …remember Record & Playback is not real test automation
9. Myths and Truths2. Commercial TA tools are expensive  Under the influence of this myth some companies, especially the small ones:  Try to develop their own test automation tools  Use scripting languages like Perl and Ruby  Use shareware test tools  Do not consider test automation at all
10. Myths and Truths But in Reality: Commercial TA tools are not that expensive  Per seat license for the most expensive automation tool is $8K, which can be used for 5 years.  Maintenance/Support fees are 20% of tool cost or $1,600 per year  The cost of this tool is $8K/5+$1,600 = $3,200 per year  The automation developer cost with overhead is $100K per year  The cost of this tool is just 3% of the person who uses it, but productivity gain can be very significant
11. Myths and Truths Learning from past experience  Truth: 92% FAIL to meet target ROI Automation Projects 8% 40% Working ROI Failure 60% 32% 2004 2010 (Estimated) Industry: Test Automation $1 Billion $ 6.3 Billion (Net Worth) Automation Projects $ 0.6 Billion $ 3.8 Billion (Failure Cost)Source 1: http://www.nytimes.com/2006/07/26/technology/26hewlett.htmlSource 2: http://www.slideshare.net/Jonathon_Wright/hybrid-keyword-data-driven-automation-frameworks-jonathon-wright
12. Why do Automation Projects typically Fail? IDT study(www.idtus.com) Misc Lack of Time Tool 15% 37% Incompatibility 11% Lack Of Budget 17% Lack of Expertise 20%
13. Why do Automation Projects typically Fail? Lack of defined automation methodology Automation is not treated as a legitimate project with the necessary planning / resources Test Automation is typically performed at the end of the SDLC After the initial success the automation scripts are not maintained for future builds
14. Why do Automation Projects typically Fail? Testers are typically untrained in test tools and programming techniques No modularization (reusable functions) in automation scripts Automated tests cases are usually designed based on front end functionality (black box testing)
15. What is Software Test Automation? Test Automation is the use of software to execute tests without Human intervention It refers to the activities and efforts that intend to automate engineering tasks and operations in a software test process using well-defined strategies and systematic solutions.Not like Rube Goldbergcartoons
16. Why TA is needed? Objectives:  To free engineers from tedious and redundant manual testing operations  To speed up a software testing process, and to reduce software testing cost and time during a software life cycle  To increase the quality and effectiveness of a software test process by achieving pre-defined adequate test criteria in a limited schedule
17. Key to SuccessTo reduce manual testing activitiesand redundant test operations using a systematic solution to achieve a better testing coverage.
18. Manual vs. Automated Testing Manual Testing:  Testing time is consuming and tedious  Inefficient in today’s shorter SDLC  Delay the ability to thoroughly test an application  Critical bugs escape undetected  What happens when multiple platforms involved Automated Testing:  Higher efficiency that accelerate the testing cycle and promote software quality  Optimizes software quality and testing efficiency by delivering  Reusability  Predictability and Consistency  Productivity  Enables accurate assessment of quality level
19. Pros and Cons Pros:  Speed  Reusability  Accuracy  Run Anytime  Efficiency
20. Pros and Cons Cons:  SignificantInvestment  Maintenance  Not as Robust  Error Detection  Cannot Think
21. What to Automate? Regression Tests: Stabilized tests that verify stabilized functionality Tests rerun often: Tests that are executed regularly vs. rarely Tests that will not expire shortly: Most tests have a finite lifetime during which its automated script must recoup the additional cost required for its automation Tedious/Boring tests:  tests with many calculations and number verifications  repetitive tests performing the same operations over and over  tests requiring many performance measurements  Just plain boring tests Reliably repeatable
22. What NOT to Automate? Unstable functionality: Not reliably repeatable Rarely executed tests: poor Return-On-Investment Tests that will soon expire: poor Return-On-Investment Requiring in-depth business analysis:  some tests require so much business specific knowledge that it becomes prohibitive time wise to include every verification required to make its automated script robust enough to be effective  exceedingly complex tests are sometimes not possible to automate because computers cannot think
23. Contents BEP and ROI  BEP  BEP - Example  ROI  PayBack Period  Classic ROI  Real ROI  Benefits of ROI  ROI Calculator
24. BEP Break-Even Point (BEP) is the point at which cost or expenses and revenue are equal: there is no net loss or gain
26. ROI Return on Investment ROI = BENEFIT/COST ROI = (total benefit – total cost) / (total cost) ROI = (cost of manual – cost of automation) / cost of automation  Where,  Automation Cost = Price Of HW + Price of SW + Development Cost + Maintenance Cost + Execution Cost  Manual Testing Cost = Development Cost + Maintenance Cost + Execution Cost Looks right, Doesn’t it?
27. Classic ROI Problems with classic ROI calculation:  You can’t compare Automated Testing and Manual Testing. They are not the same and they provide different information about the AUT.  You can’t compare cost of multiple execution of automated tests vs. manual tests. You would never dream of executing that many test cases manually So then…what is real ROI?
28. Real ROI ROI value is not the value of Automation vs. Cost of executing these tests manually Automation ROI value is the benefit of this type of testing, and it can be:  Reducing Time to Market  Increased Test Efficiency (Productivity)  Increased Test Effectiveness
29. Benefits of ROI Reduced Time to Market  Products delivered quickly  Makes people available to work on other projects  Higher margins, if no competitive products in market Productivity and Effectiveness  More testing gets done faster, increasing the odds of finding defects  Defects found early have better chances of being fixed  Manual Testers can concentrate on clever ways to finding defects, instead of typing test inputs and verify output.
30. Benefits of ROI About 7% of bug fixes create new bugs, sometimes in already tested parts of the system. With automation you can rerun tests for those modules. This almost never happens when testing is done manually.
31. ROI Calculator ROI Calculator:  Source 1: http://www.aspiresys.com/testautomationroi/index.php  Source 2: http://www.elbrus.com/services/test_automation_roi_ calc/
32. ROI Calculator
33. ROI Calculator
34. Contents Tools  Choosing the right tool – The Strategic Approach  Choosing the right tool – The Strategic Steps  Tools Classification  Short List Tools – Functional  Short List Tools – Performance  Evaluate Vendors  Functional Test Tools - Analysis  Feasibility Analysis
35. Tools There is no single best testing tool; rather, different tools are more or less appropriate in different environments.
36. Tools Over 300 Test Tools are available (http://www.softwareqatest.com)  Load/Performance tools – 54  Web Functional/Regression – 60  Java Test tools - 48  Other Web tools – 76Which tool is right for you?
37. Choosing the right tool – The StrategicApproach• Is there an organizational methodology for test automation?• Which applications/processes?• What is the impact to current project schedules?• What is the effort in maintaining automated tests?• What are the costs?• What about tools integration?• What about Continuous Integration?
38. Choosing the right tool – The Strategic Steps• Step 1: Define and Refine Requirements• Step 2: Communicate the Impact• Step 3: Develop Evaluation Methodology• Step 4: Select, Procure, and Implement
39. Step 1: Define and refine requirement Create a list of organizational requirements  What problems do you want the tool to solve?  What capabilities will the tool need to be effective in your environment? Other lifecycle tools?  What constraints, budgetary or otherwise? Identify compatibility issues  What operating systems does your application support?  What is the development environment?  Does your application integrate with third-party software?  Does the application use custom controls?
40. Step 1: Define and refine requirement Identify tool audience  Who will use the tool on a day-to-day basis?  What is the level and mix of user skill levels?  Is your organization willing to invest in training? Define technical or business requirements  Does your organization have additional requirements?  Software standards  Technical standards  Procurement rules  Preferred vendor rules
41. Step 1: Define and refine requirement Identify budget constraints  How much can we afford?  How much is this worth? New requirements may surface based on research  Did not know about  Forgot to include
42. Step 2: Communicate the Impact Automated testing is part of the larger strategic application development endeavor  Communicate the effects of implementing a tool  Chance to discuss and mitigate concerns  How tool may change job description  Commitment to training  Implementation strategy  Discussion may imply additional requirements  Business, functional, technical, or operational
43. Step 3: Develop the evaluation methodology How will tools be compared? Are there specific features that may differentiate one tool from another? Are there specific things that can eliminate a tool from consideration?  Preferred vendor list can reduce evaluation scope  Demos and evaluations are time-consuming  Identify a representative set of activities to accomplish with the tool during the evaluation
44. Step 4: Select, procure, and implement Make an informed selection Follow organization’s procurement process Develop the implementation plan  What  When  Why  Who  How
45. Step 4: Select, procure, and implement Develop an implementation plan  Enterprise applications requiring multiple releases  Applications that must produce a consistent set of results using stable data  These characteristics fully leverage reusability and predictability benefits of automated testing Take implementation one step at a time  Take time for training  Keep focus on staff issues and reactions
46. Step 4: Select, procure, and implement Develop a test plan  Describes scope, approach, resources and schedule for all automated and manual activities  Rule of Thumb: (Test scripts) 40% manual - 60% automated Create and deploy your automated tests  Be selective with the automation of test scripts  Verify the most critical functionality  Are the most likely to expose defects  Are expensive or impossible to perform manually  Use the first automated suites you build for  Smoke testing  Regression testing
47. Decision? How to calculate the cost of functional test automation Labor costs of Labor costs ofCost of test automation Cost of tool(s) script creation script maintenanceIf a test script will be run every week for the next 2 years, automate the test if thecost of automation is less than the cost of manually executing the test 104 times. Automate if Cost of manually executing the test as many Cost of automation times as the automated test will be executed
48. Tools Classification Test Tool Types Basic Descriptions of Different Types of Test ToolsTest Information Systematic solutions and tools support test engineers and quality assurance people toManagement create, update, and maintain diverse test information, including test cases, test scripts, test data, test results, and discovered problems.Test Execution and Systematic solutions and tools help engineer set up and run tests, and collect andControl validate test results.Test Generation Systematic solutions and tools generate program tests in an automatic way.Test Coverage Systematic solutions and tools analyze the test coverage during a test process basedAnalysis on selected test criteria.Performance Testing Systematic solutions and tools support program performance testing and performanceand Measurement measurement.Software Simulators Programs are developed to simulate the functions and behaviors of external systems, or dependent subsystems/components for a under test program.Regression Testing Test tools support the automation performance of regression testing and activities, including test recording and re-playing.
49. Tools Classification Types of Test Tools Test Tool Vendors Test ToolsProblem Management Tools Rational Inc. ClearQuest, ClearDDTS Microsoft Corp. PVCS Tracker Imbus AG Imbus FehlerdatenbankTest Information Management Rautional Inc. TestManagerTools Mercury Interactive TestDirectoryTest Suite Management Tools Evalid TestSuiter Rational Inc. TestFactory SUN JavaTest, JavaHarnessWhite-Box Test Tools McCabe & Associates McCabe IQ2 Junit IBM IBM COBOL Unit Tester IBM ATC - Coverage Assistant - Source Audit Assistant - Distillation Assistant - Unit Test Assistant
50. Tools ClassificationTest Execution Tools OC Systems Aprob Softbridge ATF/TestWright AutoTester AutoTester Rational Inc. Visual Test, Rational Functional Tester SQA Robot Mercury Interactive WinRunner, Quick Test Prof Sterling Software Vision TestPro Compuware QARun Seque Software SilkTest RSW Software Inc. e-Test Cyrano Gmbh Cyrano RobotCode Coverage Analysis Tools Case Consult Corp. Analyzer, Analyzer Java OC Systems Aprob IPL Software Product Group Cantata/Cantata++ ATTOL Testware SA Coverage Compuware NuMega TruCoverage Software Research TestWorks Coverage Rational Inc PureCoverage SUN JavaScope ParaSoft TCA Software Automation Inc Panorama
51. Tools ClassificationLoad Test and Performance Tools Rational Inc. Rational Performance Tester InterNetwork AG sma@rtTest Compuware QA-Load Mercury Interactive LoadRunner RSW Software Inc. e- Load SUN JavaLoad Seque Software SilkPerformer Client/Server Solutions, Inc. Benchmark FactoryRegression Testing Tools IBM Regression Testing Tool(ARTT) Distillation AssistantGUI Record/Replay Software Research eValid Mercury Interactive Xrunner Astra Astra QuickTest AutoTester AutoTester, AutoTester One
52. Short List Tools - FunctionalVendor Tool Test Suite - Companion ToolsCompuware TestPartner QACenter Enterprise Edition+Empirix e-Tester e-TEST suite Rational FunctionalIBM Rational Suite TesterMercury QuickTest Professional Quality CenterRadView WebFT TestView SuiteSeapine QA Wizard Pro TestTrack ProBorland SilkTest SilkCentral Test Manager(Segue)
53. Short List Tools - PerformanceVendor Tool Test Suite - Companion ToolsCompuware QALoad QACenter Enterprise Edition+Empirix e-Load e-TEST suite Rational Performance Rational SuiteIBM TesterMercury LoadRunner Quality CenterRadView WebLOAD TestView SuiteFacilita Forecast ForecastWeb, ForecastNet, ForecastDBBorland SilkPerformer SilkCentral Test Manager(Segue)
55. Functional Test Tools - Analysis Tool Pros ConsIBM/Rational •Built as Eclipse Plug-In with full IDE •Insufficient browserFunctional Tester and Java support support(RFT) •Supports Web 2.0, Java or .NET •Licensed product applications •Full GUI Object Map repositoryHP/Mercury Quick •Supports Web 2.0, Java or .NET •VisualBasic scripting isTest Pro (QTP) applications limited •Full GUI Object Map repository •No IDE (may change in •Seamless integration with new release) QualityCenter •Licensed ProductSelenium RC & •Good browser support •No GUI Object repositoryIDE •Good language support (Java, •Only web application Ruby,C# ) support •Can be easily extended as JUnit suite •Open-source
59. Contents Automation Framework  Where to start?  What is AF?  Benefits of AF  How does it work?  Architecture  Framework Approaches  Record and Playback  Script Based Approach  Keyword Driven Approach  Data Driven Approach  Hybrid Keyword and Data Driven Approach
60. Where to Start?“Start Quick winsSMALL should be NEVER expectthink BIG” avoided to automate 100% First find out Then you can work out What needs to be tested? What needs be automated? What can be tested? What can be automated? What could be tested? What could be automated? Focus on key critical“Under Keep it businesspromise, simple, processesOver deliver?” wherever possible
61. What is Automation Framework? Framework – independent of application or environment under test A Test Automation Framework is a set of assumptions, concepts and tools that provide support for Automated Software Testing. A reusable set of libraries or classes for a software system (or subsystem). A correctly implemented Test Automation Framework can further improve ROI by reducing the development and maintenance costs.
62. Benefits of Framework Ease of Use – easy to learn and easy to use Time – faster than capture/replay and scripting approach Maintainability – significantly reduces the test maintenance effort Reusability – due to modularity of test cases and library functions Manageability - effective test design, execution, and traceability Accessibility – to design, develop & modify tests whilst executing Availability – scheduled execution can run unattended on a 24/7 basis Reliability – due to advanced error handling and scenario recovery Flexibility – framework independent of AUT or environment Measurability – customizable reporting of test results ensure
63. How does it work? Different Implementations One Example of Keyword Driven Framework could be:  Spreadsheets, Spreadsheets, Spreadsheets  Test Objects  Keywords and Methods = Toast!  Parameters  Description or Call the 911?
65. Framework Approaches Record and Playback Script Based Approach Keyword Driven Approach Data Driven Approach Hybrid Keyword and Data Driven Approach
66. Manual Testing – Looking back + easy & cheap to start + flexible testing - expensive every execution - no auto regression testing - less coverage measurement
67. Record and Playback+ flexible testing- expensive first execution+ auto regression testing- fragile tests break easily- less coverage measurement
68. Script Based Approach +/- test impl. = programming + automatic execution + auto regression testing - fragile tests break easily? (depends on abstraction) - less coverage measurement
69. Data Driven Approach Automation is data-centric User defines just data sets to drive tests with Will have an external data source (DB tables, Excel spreadsheets, XML for data sets) Flow control (navigation) is normally done by the test script not by the data sourceEx: data set exercises creation of new sales accounts functionality; stored in a DB table account_data CompanyName PrimarySalesPerson Street Zip City State Genesis Inc. Phil Collins 5775 Main st 30075 Atlanta GA RollingStones Inc. Mick Jagger Jr. 2332 Washington st 02111 Boston MA
71. Keyword/Action Driven Approach Automation is action-centric De-compose your test cases/modules into granular re-usable keywords The idea is for non-coders to be able to create automated test cases with action keywords User defines flow control of the test via action keywords Example: Test Case “Verify Checking Account Balance” 1. Enter Username and Password and Click submit button  step 1 is action Login 2. Enter “Phil Collins” as a Sales Person and Click Submit button 3. Verify the Sales Person was successfully created and Logout So you may want to choose the following re-usable action keywords: EnterText, Click, Login, VerifyExists
72. Benefits of Keyword Driven Approach This Framework addresses the most common problem with test automation: Automation Engineers do not have domain knowledge and the End Users (Subject Matter Experts/Test Engineers) usually do not have automation expertise. When properly implemented and maintained, it presents a superior ROI because each business event is designed, automated and maintained as a discrete entity. Keywords can then be used to design test cases, but the design and automation overhead for the keyword has already been paid.
73. Benefits of Keyword Driven Approach Reduced the cost and time spent maintaining and updating tests The modular structure of keyword-driven testing means that new tests can easily be created from pre-existing modules The test team is capable of entirely automating tests, even without programming knowledge Can be easily modified to use with different test tool Reusability across different projects Classic Example: Object Action Data Textfield (username) Enter Text <username>
74. Keyword/Action Driven Approach  May have an external data source (DB tables, Excel spreadsheets, XML for data sets) with action keywordsSte Descriptiop n Page Action Module Type Object Expected UserLo .id:=LoginSu .text:=Login1 Login Home Login gin N/A bmit Successful Enter New CreateS Sales alesPers .text:=Sales .value:=Phil2 Person data on EnterText Field PerName Collins CreateS Click alesPers Butto .id:=SubmitS .url:=.*createdSal3 Submit on Click n alesPer esPerStatus.html Verify Sales CreateS Person alesPers .value:=User Creation onStatus VerifyExist .id:=Creation Created4 Successful s DIV Status Successfully
75. Hybrid Keyword and Data Driven Approach Combines the best of both worlds  User defines data sets to drive tests with  User also defines flow control of the test via action keywords  May have an external data source (DB tables, Excel spreadsheets, XML for data sets) with action keywords in addition to generic and test case specific data sets
77. Open2Test Automation Framework
78. Open2Test Automation Framework
79. Open2Test Automation Framework
80. Model Based Approach+ abstract tests+ automatic execution+ auto regression testing+ auto design of tests+ systematic coverage+ measure coverage of model andrequirements- modelling overhead Emerging Approach
82. Summarizing… Overview of STA  Choosing the right tool – The Strategic  History Steps  Myths and Truths  Tools Classification  Why Automation Projects Fail?  Short List Tools – Functional  What is TA?  Short List Tools – Performance  Why TA is needed?  Evaluate Vendors  Manual vs. Automated  Functional Test Tools - Analysis  Pros & Cons  Feasibility Analysis  What to Automate?  Automation Framework  What Not to Automate?  Where to start? BEP and ROI  What is AF?  BEP  Benefits of AF  How does it work?  BEP - Example  Architecture  ROI  Framework Approaches  Classic ROI  Record and Playback  Real ROI  Script Based Approach  Benefits of ROI  Keyword Driven Approach  ROI Calculator  Data Driven Approach  Hybrid Keyword and Data Driven Approach Tools  Choosing the right tool – The Strategic Approach