Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Evaluating Functional Testing Solutions


Published on

  • Be the first to comment

  • Be the first to like this

Evaluating Functional Testing Solutions

  1. 1. June 27, 2006. Call in at 12:55 pm Eastern Time Carey Schwaber Analyst Forrester Research Teleconference Evaluating Functional Testing Solutions: The Forrester Wave ™ Q2 2006
  2. 2. Agenda <ul><li>The what and why of functional testing solutions </li></ul><ul><li>How Forrester evaluated functional testing solutions </li></ul><ul><li>The findings of Forrester’s evaluation </li></ul><ul><li>Recommendations and WIM </li></ul>
  3. 3. Agenda <ul><li>The what and why of functional testing solutions </li></ul><ul><li>How Forrester evaluated functional testing solutions </li></ul><ul><li>The findings of Forrester’s evaluation </li></ul><ul><li>Recommendations and WIM </li></ul>
  4. 4. Definitions <ul><li>Functional tests: </li></ul><ul><li>Tests to verify that application functionality conforms to predefined specifications </li></ul>
  5. 5. What are the key ingredients of a functional testing solution? <ul><li>In scope: </li></ul><ul><ul><li>Manual testing </li></ul></ul><ul><ul><li>Functional test automation </li></ul></ul><ul><ul><li>Test management </li></ul></ul><ul><li>Out of scope: </li></ul><ul><ul><li>Unit testing </li></ul></ul><ul><ul><li>Static analysis </li></ul></ul><ul><ul><li>Performance testing </li></ul></ul><ul><ul><li>Application monitoring </li></ul></ul>
  6. 6. “Does your shop conduct any automation of functional test scripts?” Base: 74 IT decision-makers at North American and European companies
  7. 7. Pros and cons of manual testing <ul><li>Pros </li></ul><ul><li>Cost of test case design is minimal </li></ul><ul><ul><li>No required tools or tool expertise </li></ul></ul><ul><ul><li>No automation necessary </li></ul></ul><ul><ul><li>No need for lead time prior to test execution </li></ul></ul><ul><li>Can be scripted, exploratory, or both </li></ul><ul><ul><li>Simultaneous learning, test design, and test execution (Bach) </li></ul></ul><ul><ul><li>Useful before, during, and after design and execution of standard manual test scripts </li></ul></ul><ul><li>Cons </li></ul><ul><li>Cost of test execution is high </li></ul><ul><ul><li>Cost = execution time X labor rate </li></ul></ul><ul><ul><li>Execution time is high </li></ul></ul><ul><ul><li>No efficiencies when execution is repeated </li></ul></ul><ul><li>Scripted test execution is tedious </li></ul><ul><ul><li>No creativity required </li></ul></ul><ul><li>All forms are highly error-prone </li></ul><ul><ul><li>Quality depends on tester’s attention to detail over time </li></ul></ul><ul><ul><li>Documentation of test results is another potential source of errors </li></ul></ul>
  8. 8. Specialized tool support can improve efficiency of scripted manual testing <ul><li>Tool support helps manual testers: </li></ul><ul><ul><li>Store test plans, test scripts, and test results in a single, secure location </li></ul></ul><ul><ul><li>Share test components (e.g., “login”) across test cases </li></ul></ul><ul><ul><li>Automate data entry and data verification </li></ul></ul>
  9. 9. Pros and cons of test automation <ul><li>Pros </li></ul><ul><li>Frees testers for more intelligent types of testing (e.g., exploratory testing) </li></ul><ul><li>Drives down time, cost of test execution </li></ul><ul><li>Permits shops to expand scope of testing efforts </li></ul><ul><li>Cons </li></ul><ul><li>Increases up-front investment in test design </li></ul><ul><li>Easy to waste time automating the “wrong” tests — or the right tests in the wrong way </li></ul><ul><li>Requires more technical expertise, specialized tools support than manual testing </li></ul>
  10. 10. A simplified view of the economics of test automation If a test script will be run every week for the next two years, automate the test if the cost of automation is less than the cost of manually executing the test 104 times. Choose to automate when: To calculate the cost of automating a test script: Cost of automation Cost of manually executing the test as many times as the automated test will be executed Cost of test automation Cost of tool(s) Labor costs of script creation Labor costs of script maintenance
  11. 11. “Why does your company not perform any test automation?” Base: 38 North American and European IT decision-makers at companies that don’t perform any test automation
  12. 12. The right balance varies by test effort <ul><li>Testing team composition </li></ul><ul><ul><li>Programming skills vs. subject matter expertise </li></ul></ul><ul><ul><li>Division of labor that leverages each team members’ strengths </li></ul></ul><ul><ul><li>Assessment of development team’s own testing efforts </li></ul></ul><ul><li>Nature of the application under test </li></ul><ul><ul><li>Application technology </li></ul></ul><ul><ul><li>Application stability </li></ul></ul><ul><li>Timeline </li></ul><ul><ul><li>Time available for creation of automated test scripts </li></ul></ul><ul><ul><li>Expected lifetime of the application </li></ul></ul>
  13. 13. The benefits of an integrated test management solution for manual and automated testing <ul><li>Common interface for planning and monitoring all test activities </li></ul><ul><li>Change management for manual and automated test assets </li></ul><ul><li>Submission of defects from manual testing and test automation tool directly to test management tool </li></ul><ul><li>Incremental automation of portions of the test suite </li></ul>
  14. 14. Agenda <ul><li>The what and why of functional testing solutions </li></ul><ul><li>How Forrester evaluated functional testing solutions </li></ul><ul><li>The findings of Forrester’s evaluation </li></ul><ul><li>Recommendations and WIM </li></ul>
  15. 15. How did we decide which vendors to evaluate? <ul><li>Criteria for inclusion: </li></ul><ul><li>$10M in annual revenues </li></ul><ul><li>Support for manual testing, test automation, and test management </li></ul><ul><li>Some excluded vendors: </li></ul><ul><li>RadView and Seapine </li></ul><ul><ul><li>Both included in last year’s Wave of test automation tools </li></ul></ul><ul><li>Worksoft, SDT, and LogiGear </li></ul><ul><ul><li>Focus on keyword-driven test automation </li></ul></ul><ul><li>Agitar and Parasoft </li></ul><ul><ul><li>Focus on developer testing </li></ul></ul>
  16. 16. Evaluated vendors and their product offerings Borland Software SilkTest, SilkCentral Test Manager, and SilkCentral Issue Manager Compuware QACenter Enterprise Edition: TestPartner, QARun, QADirector, TrackRecord, Reconcile Empirix e-Tester, e-Manager Enterprise IBM Rational Functional Tester, Rational Robot, Rational Manual Tester, Rational ClearQuest Mercury Interactive QuickTest Professional, WinRunner, Quality Center
  17. 17. The Forrester Wave ™ evaluation process <ul><li>Evaluation conducted between February and May of 2006 </li></ul><ul><ul><li>Based on product capabilities generally available by June 1, 2006 </li></ul></ul><ul><li>Open process to select 87 evaluation criteria: </li></ul><ul><ul><li>Interviewed vendors, experts, outsourcers, and users </li></ul></ul><ul><li>Vendor self-evaluations </li></ul><ul><ul><li>Evaluation relies in part on data provided by vendors </li></ul></ul><ul><li>Interviews on vendor strategy </li></ul><ul><ul><li>Conversations with executives to determine how vendors will enhance their offerings in the future </li></ul></ul><ul><li>Product demonstrations </li></ul><ul><ul><li>Validate our understanding of product capabilities </li></ul></ul><ul><li>Extensive fact-checking with customer references </li></ul><ul><ul><li>Determine how vendor offerings work in practice as well as in theory </li></ul></ul>
  18. 18. Evaluation criteria <ul><li>Forrester evaluated these five vendors’ solutions across 87 criteria </li></ul><ul><li>These criteria fell into three categories (and 19 subcategories): </li></ul><ul><ul><li>Current offerings </li></ul></ul><ul><ul><li>Strategy </li></ul></ul><ul><ul><li>Market presence </li></ul></ul>
  19. 19. Current offerings criteria Solution architecture What is the architecture of the vendor’s functional testing solution? (Operating systems, databases, directory servers) Life-cycle integration With what life-cycle tools do the products in the vendor’s functional testing solution integrate? (Requirements management, issue management, SCM, performance testing, SOA management, integration interfaces) Manual testing How strong are the manual testing capabilities of the vendor’s functional testing solution? (Test case creation, attachments, reuse, impact analysis) Test automation How strong are the test automation capabilities of the vendor’s functional testing solution? (Testable applications, object recognition, script creation) Test management How strong are the test management capabilities of the vendor’s functional testing solution? (Scalability, supported test assets, test planning, test execution, workflow, reporting and analytics, collaboration)
  20. 20. Strategy criteria Product direction How well does the vendor’s product strategy position it for market leadership? (Product vision, product roadmap) Focus What percentage of the vendor’s resources are focused on the products in its functional testing solution? R&D How significantly is the vendor investing in R&D for the products in its functional testing solution? (R&D resources, R&D expense ratio) Price What is the price of the vendor’s functional testing solution? (Test automation price per user, manual testing price per user, test management price per user, solution price for a typical deployment, maintenance price, pricing structure) Financial resources to execute on strategy What is the ratio of the vendor’s current assets to its current liabilities?
  21. 21. Market presence criteria Installed base How large is the vendor's installed base in terms of unique companies? (Total installed base, functional testing installed base, new functional testing customers, functional testing reference customers) Financial strength How financially healthy is the vendor? (Revenues, revenue growth) Employees How many employees does the vendor have? Training What training is available for the products in the vendor’s functional testing solution? Who are the vendor's partners for its functional testing solution? (Technology partners, systems integrators, channel partners) Partnerships
  22. 22. Agenda <ul><li>The what and why of functional testing solutions </li></ul><ul><li>How Forrester evaluated functional testing solutions </li></ul><ul><li>The findings of Forrester’s evaluation </li></ul><ul><li>Recommendations and WIM </li></ul>
  23. 23. Findings <ul><li>Overall rankings </li></ul><ul><li>(50% current offering, 50% strategy) </li></ul>One leader Three strong performers One contender Empirix 2.11 Vendor Mercury IBM Compuware Borland Ranking 4.39 3.23 2.65 2.64
  24. 24. How to create a custom ranking <ul><li>Determine how much each of the evaluation criteria matters to you. </li></ul><ul><li>Weight the evaluation criteria accordingly. </li></ul><ul><li>Read the score explanation text to familiarize yourself with these tools and vendors. </li></ul><ul><li>Follow up with demos, trials, and pilots. </li></ul>
  25. 25. Forrester Wave™: Functional Testing Solutions, Q2 ’06 May 2006, Tech Choices “The Forrester Wave™: Functional Testing Solutions, Q2 2006”
  26. 26. Vendor profile: Borland <ul><li>Strengths: </li></ul><ul><ul><li>Life-cycle integration </li></ul></ul><ul><ul><li>Reporting and analytics </li></ul></ul><ul><li>Weaknesses: </li></ul><ul><ul><li>Manual testing </li></ul></ul><ul><ul><li>Automated test script creation </li></ul></ul><ul><ul><li>Environment support </li></ul></ul><ul><li>Best for: </li></ul><ul><ul><li>Shops with testers who have programming skills. </li></ul></ul><ul><ul><li>Shops that use other Borland life-cycle management products </li></ul></ul>
  27. 27. Vendor profile: Compuware <ul><li>Strengths: </li></ul><ul><ul><li>Overall breadth — though not depth — of capabilities </li></ul></ul><ul><ul><li>Built-in support for risk-based testing </li></ul></ul><ul><li>Weaknesses: </li></ul><ul><ul><li>Weak support for hand-coding and graphical modification of test scripts </li></ul></ul><ul><ul><li>Core test management capabilities available only to CARS customers </li></ul></ul><ul><ul><li>Too many disparate interfaces </li></ul></ul><ul><li>Best for: </li></ul><ul><ul><li>Project-level testing efforts </li></ul></ul><ul><ul><li>Shops that use other Compuware products </li></ul></ul>
  28. 28. Vendor profile: Empirix <ul><li>Strengths: </li></ul><ul><ul><li>Strong support for Web environments </li></ul></ul><ul><ul><li>Specialized support for Web services </li></ul></ul><ul><ul><li>XML-based APIs </li></ul></ul><ul><li>Weaknesses: </li></ul><ul><ul><li>e-Tester has extremely limited environment support </li></ul></ul><ul><ul><li>e-Tester does not serve technical or nontechnical testers well </li></ul></ul><ul><ul><li>e-Manager Enterprise has minimal support for manual testing </li></ul></ul><ul><ul><li>e-Manager Enterprise offers only basic capabilities when it comes to test management </li></ul></ul><ul><ul><li>The solution as a whole falls down on life-cycle integration </li></ul></ul><ul><li>Best for: </li></ul><ul><ul><li>Project-level testing efforts </li></ul></ul><ul><ul><li>Web applications and services testing efforts </li></ul></ul>
  29. 29. Vendor profile: IBM <ul><li>Strengths: </li></ul><ul><ul><li>Support for manual testing </li></ul></ul><ul><ul><li>Support for custom coding of test scripts </li></ul></ul><ul><ul><li>A platform for test management </li></ul></ul><ul><li>Weaknesses: </li></ul><ul><ul><li>Nonprogrammers don't get much help on test automation </li></ul></ul><ul><ul><li>Environment support is still limited, although it is improving </li></ul></ul><ul><ul><li>Test execution capabilities are primitive </li></ul></ul><ul><ul><li>The functional testing solution itself is in need of better integration </li></ul></ul><ul><li>Best for: </li></ul><ul><ul><li>Using other IBM Rational tools </li></ul></ul><ul><ul><li>Doing a great deal of manual testing </li></ul></ul><ul><ul><li>Having testers with programming experience </li></ul></ul>
  30. 30. Vendor profile: Mercury <ul><li>Strengths: </li></ul><ul><ul><li>Enhanced user productivity through ease of use </li></ul></ul><ul><ul><li>Top-notch environment support </li></ul></ul><ul><ul><li>Proven scalability across multiple dimensions </li></ul></ul><ul><li>Weaknesses: </li></ul><ul><ul><li>Weak scripting language and scripting environment </li></ul></ul><ul><ul><li>Limited management of changes to reused manual test components </li></ul></ul><ul><ul><li>Corporate instability </li></ul></ul><ul><li>Best for: </li></ul><ul><ul><li>Centralized testing organizations </li></ul></ul><ul><ul><li>Companies that use other Mercury products </li></ul></ul>
  31. 31. Agenda <ul><li>The what and why of functional testing solutions </li></ul><ul><li>How Forrester evaluated functional testing solutions </li></ul><ul><li>The findings of Forrester’s evaluation </li></ul><ul><li>Recommendations and WIM </li></ul>
  32. 32. Factors to consider when selecting a functional testing solution <ul><li>Application technologies in use </li></ul><ul><ul><li>Legacy 4GL, Web services, ERP/CRM, custom controls </li></ul></ul><ul><li>Skill sets </li></ul><ul><ul><li>Strong knowledge of the business, programming experience and/or aptitude </li></ul></ul><ul><li>Organizational structure </li></ul><ul><ul><li>Centralized test organization, testers on development teams, offshore testing </li></ul></ul><ul><li>Development life-cycle tools in use </li></ul><ul><ul><li>Tools for developer testing, requirements definition and management, issue management, software configuration management </li></ul></ul><ul><li>IT operations tools in use </li></ul><ul><ul><li>Tools for deployment, performance monitoring, SOA management </li></ul></ul><ul><li>IT management tools in use </li></ul>
  33. 33. How will vendors improve their offerings? <ul><li>Better enable incremental automation of manual test cases </li></ul><ul><li>Provide better facilities for graphical creation and modification of test cases </li></ul><ul><li>Improve support for testing in an SOA environment </li></ul><ul><li>Do more to facilitate geographically distributed testing efforts </li></ul><ul><li>Improve integrations with development, operations, and management tools </li></ul><ul><li>Continue to explore open standards </li></ul>
  34. 34. The next area of innovation: SOA testing July 2006 (Upcoming) Trends “SOA Raises The Stakes For Software Quality”
  35. 35. Manual and automated functional testing are good candidates for offshore outsourcing March 2006, Trends “How To Benefit From Offshore Testing Services”
  36. 36. Thank you Carey Schwaber +1 617/613-6260 [email_address]
  37. 37. Selected bibliography <ul><li>May 31, 2006, Tech Choices “The Forrester Wave, Q2 2006: Functional Testing Solutions” </li></ul><ul><li>March 8, 2006, Trends “How To Benefit From Offshore Testing Services” </li></ul><ul><li>September 15, 2005, Quick Take “Take Careful Inventory Before Adopting Standalone Code Quality Tools” </li></ul><ul><li>May 16, 2005, Best Practices “Software Quality Is Everybody’s Business” </li></ul><ul><li>February 3, 2005, Tech Choices “Evaluating Automated Functional Testing Tools” </li></ul>