Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

NASSCOM Quality forum - Trends In Sw Testing


Published on

Software Testing today had undergone a huge change in the last decade and interestingly enough, the next 10 years hence, would also change the landscape completely. Some of the factors contributing to the changing trends:

• Technology – SOA, Web Technology, Distributed technology were all evidence of the change
• Methodology – Rapid to Agile Testing
• Customer / Business Expectations, based on cost, time, reliability and robustness
• Business Complexities and
• The advent of Globalization

Published in: Business, Technology
  • Be the first to comment

NASSCOM Quality forum - Trends In Sw Testing

  1. 1. Trends in Software Testing - Ajay Kumar Chhokra
  2. 2. Content <ul><li>About UnitedHealth Group </li></ul><ul><li>Factors Contribute in Trends </li></ul><ul><li>Shift in Company Technology Priorities </li></ul><ul><li>Business Drivers and Expectations </li></ul><ul><li>Key Drivers </li></ul><ul><li>Test Reduction Techniques </li></ul><ul><ul><li>Risk based Testing </li></ul></ul><ul><ul><li>Pair-wise Testing </li></ul></ul><ul><ul><li>Cycle based Testing </li></ul></ul><ul><ul><li>Adaptive Testing </li></ul></ul><ul><li>How to Apply Adaptive Techniques </li></ul><ul><li>Benefits and Challenges of Adaptive Techniques </li></ul><ul><li>Case Study on Adaptive Testing </li></ul><ul><li>Optimization </li></ul><ul><li>Test Automation - Effort ROI Computation </li></ul><ul><li>Summary </li></ul>
  3. 3. About UnitedHealth Group <ul><li>UHG is ranked Number 21 in the Latest Fortune 500 rankings for 2009 (revenues wise) </li></ul><ul><li>Headquartered in Minnetonka, Minnesota, USA </li></ul><ul><li>Serves 71 million lives </li></ul><ul><li>Annual revenue greater than US # 80 Billion </li></ul><ul><li>Total number of employees: 60,000+ </li></ul>
  4. 4. Factors Contribute in Trends <ul><li>Trends change regularly - Software Testing Today is different than what it was 10 years ago and will be different 10 years from now…….. </li></ul><ul><li>Factors contribute in Trends are </li></ul>Technology SOA, Distributed Technology, Web Methodology Rapid Development, Agile Customer /Business Expectation Cost, Time, Reliability and Robustness Business Complexities Globalization
  5. 5. Shift in Company Technology Priorities <ul><li>UHG 2009 Priorities </li></ul><ul><ul><li>Strengthen Release Quality – On Time, On Quality, on Cost, On Benefit, On Scope </li></ul></ul><ul><ul><li>Improve Our Service – Operational Improvements </li></ul></ul><ul><ul><li>Simplify our Technology Landscape – System Investments </li></ul></ul><ul><ul><li>Invest in our Future – People and Leadership </li></ul></ul><ul><li>Shift is happening </li></ul><ul><ul><li>No more Technology Focus </li></ul></ul><ul><ul><li>No more Methodology Focus </li></ul></ul><ul><ul><li>No one is talking about Zero Defect……but releasing with Known Defects </li></ul></ul><ul><ul><li>Demanding more Dashboards - Progress Visibility </li></ul></ul><ul><ul><li>It is more and more of Business Focus </li></ul></ul>
  6. 6. Business Drivers and Expectations <ul><li>Visibility </li></ul><ul><li>Confidence in Release Quality </li></ul>Quality Gates, Metrics and Dash Boards Governance <ul><li>Pressure on reducing Testing Effort </li></ul><ul><li>Profitability </li></ul>Manage Peaks and Valleys Cost <ul><li>Employee Morale </li></ul>Smart Testing Employee <ul><li>Customer Satisfaction </li></ul><ul><li>Market Share </li></ul><ul><li>Revenue </li></ul>Predictability, Defect Density, and High Test Coverage Quality <ul><li>Revenue </li></ul><ul><li>Cost </li></ul><ul><li>Competitive Edge </li></ul>Reduce Cycle Time Time to Market Effects on Expectations Drivers
  7. 7. Key Drivers - Reduce Cycle Time
  8. 8. Key Drivers <ul><li>- Reduce Cycle Time </li></ul><ul><li>Smart Testing </li></ul>
  9. 9. Key Drivers <ul><li>- Reduce Cycle Time </li></ul><ul><li>Smart Testing </li></ul><ul><li>Visibility </li></ul>
  10. 10. Key Drivers <ul><li>- Reduce Cycle Time </li></ul><ul><li>Smart Testing </li></ul><ul><li>Visibility </li></ul>Test Reduction Techniques Test Automation Optimization
  11. 11. Key Drivers <ul><li>- Reduce Cycle Time </li></ul><ul><li>Smart Testing </li></ul><ul><li>- Visibility </li></ul>Test Reduction Techniques Test Automation Optimization People
  12. 12. Key Drivers - Reduce Cycle Time - Smart Testing - Visibility Test Reduction Techniques Test Automation Optimization People Process
  13. 13. Key Drivers - Reduce Cycle Time - Smart Testing - Visibility Test Reduction Techniques Test Automation Optimization People Process Tools
  14. 14. Test Reduction Techniques <ul><li>Risk based Testing </li></ul><ul><li>Pair-wise Testing </li></ul><ul><li>Cycle Based Testing </li></ul>
  15. 15. Test Reduction Techniques -Risk based Testing <ul><li>Identifying the business risks precisely is difficult ( Requires clear understanding of applications and business and detailed knowledge of requirements ) </li></ul><ul><li>Technique focuses on the key risks and often ignores the low impact areas. </li></ul><ul><li>Technique is not foolproof; it is more or less intuitive in nature. </li></ul><ul><li>Focus on the key business areas/critical business processes </li></ul><ul><li>Eliminates the major risks </li></ul><ul><li>Quick to apply if the tester and business have good exposure to business domain </li></ul>Challenges Benefits
  16. 16. Test Reduction Techniques - Pair-wise Testing <ul><li>Chance of undetected defects at lower level interaction is more than higher order interaction </li></ul><ul><li>With number of interaction levels increases, the number of cases increases very significantly </li></ul><ul><li>Pairwise testing fails when the parameters and levels have not been appropriately identified. </li></ul><ul><li>Pairwise testing (primarily focusing on 2 –way implementation) does not cover all input space. </li></ul><ul><li>Reduce the test cases drastically at 2 and 3-level interactions </li></ul><ul><li>Helpful for complex systems that involve large number of parameters and their combination </li></ul><ul><li>Suitable for regression cases where coverage is to be maintained high with minimal cases count </li></ul><ul><li>The defect detection rate versus the applied effort is high. </li></ul>Challenges Benefits
  17. 17. Test Reduction Techniques - Cycle Based Testing <ul><li>Technique is discouraged when individual test cycles are too long. </li></ul><ul><li>May involve overheads if repeated data setup is required in every subsequent cycle. </li></ul><ul><li>Too many cycles in single release delays the process. </li></ul><ul><li>Tracking the execution, defects and cross-impact is sometimes difficult when using this technique. </li></ul><ul><li>Gives good visibility on Testing progress </li></ul><ul><li>Helps to define logical milestone for testing. The defect fixes or patches applied are tested in subsequent cycles. </li></ul><ul><li>Very helpful when applying Incremental Integration techniques </li></ul>Challenges Benefits 80% pass 70% Executed Cycle 1 90% pass 100% Executed Cycle 2 100% pass 100% Executed Cycle 3 Sample Cycle Based Testing Criteria
  18. 18. Adaptive Testing <ul><li>A Technique developed to “adapt” to various project needs. </li></ul><ul><li>Well suited for specific project needs – reducing time or increasing </li></ul><ul><li>test coverage etc. </li></ul><ul><li>Adaptive techniques can be combination of various market </li></ul><ul><li>proven techniques such as Risk Based testing, Pairwise testing etc </li></ul><ul><li>tailored per project needs. </li></ul>Adaptive Testing
  19. 19. How to Apply Adaptive Techniques ? <ul><li>The Mix-n-Match of the individual techniques can help project have different FLAVORS of adaptive testing. Each Case below is project specific and business objective based. </li></ul><ul><li>Case 1: Risk Based + Pairwise Test </li></ul><ul><ul><li>Define the parameters and levels of interaction </li></ul></ul><ul><ul><li>Identify the high impact condition using RBT </li></ul></ul><ul><ul><li>Apply Pairwise technique on less/ low impacting parameters </li></ul></ul><ul><li>Case 2: Risk Based + Cycle Based Test </li></ul><ul><ul><li>Identify the high impact and low impact business condition </li></ul></ul><ul><ul><li>Define multiple cycles </li></ul></ul><ul><ul><li>Distribute the high impact and low impact functionalities over cycles based on priorities </li></ul></ul><ul><li>Case 3: Pairwise Test + Cycle Based Test </li></ul><ul><ul><li>Identify parameters and levels </li></ul></ul><ul><ul><li>Apply Pairwise technique to arrive at reduced test set </li></ul></ul><ul><ul><li>Prioritize the test cases </li></ul></ul><ul><ul><li>Distribute the test cases in various cycles – sanity, priority and retest </li></ul></ul>
  20. 20. Benefits and Challenges of Adaptive Techniques <ul><li>Can be applied to varied project scenarios or type of testing – regression, enhancement, system, integration testing etc </li></ul><ul><li>Adaptive techniques are fool proof and help to identify hidden defects </li></ul><ul><li>Focus is on high coverage and critical business functions simultaneously </li></ul><ul><li>Can be universally applied in product based releases cycles </li></ul><ul><li>High probability of defect detection . </li></ul>Benefits <ul><li>Requires good knowledge of system & market techniques (Pairwise, combinatorial, risk based techniques etc) </li></ul><ul><li>Need analytical capabilities in determining how to Mix-n-Match or tailor the techniques </li></ul><ul><li>Changes in business process necessitate revision of test cases and re-assess risk associated </li></ul>Challenges
  21. 21. Test Reduction Techniques – Adaptive Testing Case Study Exhaustive Test Case Count = 1152 UHG India Experience – Case Study 97.2% 98.6% 97.2% Test Reduction (%) 5 16 Pairwise Technique 9 8 Defects Found 32 32 Reduced Test Cases Adaptive Technique Risk-Based Technique
  22. 22. Key Drivers - Reduce Cycle Time - Smart Testing - Visibility Test Reduction Techniques Test Automation Optimization People Process Tools
  23. 23. App 3 App 1 App 2 Business Group 1 Business Group 2 Current State <ul><li>Attributes </li></ul><ul><li>Heavy onshore </li></ul><ul><li>Lot of productive time going in </li></ul><ul><li>interfacing and coordination </li></ul><ul><li>Challenges </li></ul><ul><li>Test Teams working in Silos </li></ul><ul><li>Have to struggle with peaks and Valleys </li></ul><ul><li>No leverage of Best Practices across </li></ul><ul><li>Applications </li></ul><ul><li>Limited Test Governance </li></ul><ul><li>Hardly any Test Standardization </li></ul><ul><li>High Cost of Operation </li></ul><ul><li>Limited Career Growth </li></ul><ul><li>Lot of dependency on Vendor partners </li></ul>App 6 App 4 App 5 Onsite Employee Offshore Employee Onsite Contractor Offshore Contractor Standards Governance Optimization
  24. 24. App 3 App 1 App 2 Business Group 1 End State Onsite Employee Offshore Employee Onsite Contractor Offshore Contractor Standards Governance Automation Regression Functional Testing Test Strategy End-to-End Testing UAT Testing Center of Excellence K I N M O P W A L R E T D I G N E G App 6 App 4 App 5 DB Testing Performance Testing DW Testing UI Testing Process Specialist Facet Testing T G E O S V T E R N A N C E Business Group 2 Trainer <ul><li>Attributes </li></ul><ul><li>Lean onshore and </li></ul><ul><li>focusing on Business </li></ul><ul><li>Interface and defining Test </li></ul><ul><li>Requirements </li></ul><ul><li>TCOE owns Test Delivery </li></ul><ul><li>Responsibility </li></ul><ul><li>TCoE coordinating with </li></ul><ul><li>Vendor partners </li></ul><ul><li>Optimal utilization of </li></ul><ul><li>offshore </li></ul><ul><li>Standardization and Test </li></ul><ul><li>Governance </li></ul><ul><li>Good Career Opportunity </li></ul><ul><li>Challenges </li></ul><ul><li>Resource Mapping </li></ul><ul><li>Cross Training </li></ul>Optimization
  25. 25. CoE Benefits Upgrade Skills of Test professionals Develop SMEs on Applications, Technology, Processes and Tools Minimize IT Risk Suggesting and Reviewing Testing Approach, solutions and Frameworks, Helping in establishing Quality Gates; Metrics Collection thus giving predictable results and quality delivery Give a Jump Start Provide skilled resources for Architecting test solutions or solving complex problems. Reduce Cost of Operation Promote and help in reusing the valuable Assets created in various engagements thus giving high ROI and lower rework effort. Capture and Drive Best Practices Assign SMEs to train and mentor other Testing associates across projects on Assets, Best Practices and Tools .
  26. 26. Key Drivers - Reduce Cycle Time - Smart Testing - Visibility Test Reduction Techniques Test Automation Optimization People Process Tools
  27. 27. Test Automation <ul><ul><li>Pros </li></ul></ul><ul><ul><ul><li>Frees testers for more intelligent types of testing </li></ul></ul></ul><ul><ul><ul><li>Low cost of Execution </li></ul></ul></ul><ul><ul><ul><li>More coverage </li></ul></ul></ul><ul><ul><ul><li>High efficiency and consistency for repeated execution </li></ul></ul></ul><ul><ul><ul><li>Improved Quality of testing </li></ul></ul></ul><ul><ul><li>Cons </li></ul></ul><ul><ul><ul><li>Increased up-front investment in Test Development </li></ul></ul></ul><ul><ul><ul><li>Easy to waste time in automating wrong tests or right tests in wrong way </li></ul></ul></ul><ul><ul><ul><li>Require more technical expertise </li></ul></ul></ul><ul><ul><ul><li>Specialized tool support required than manual testing </li></ul></ul></ul>
  28. 28. Effort ROI Computation <ul><li>Effort ROI = </li></ul>88% Test Execution Saving Break Even +ve ROI Time spent on the Maintenance of the scripts, generally considered from the second iteration cycle onwards Main Effort Spent on Script Development, Automation Strategy Development effort, Debugging Automation Execution Effort Data Preparation Result Analysis Effort to Execute Test Cases Manually Definition 750(450+300)Hrs Dev 10 Hrs Auto 81 Hrs for 270 Test Cases Man Effort Item 25.20 13 15.47 12 5.73 11 -4.00 10 -13.73 9 -23.47 8 -33.20 7 -42.93 6 -52.67 5* -62.40 4* -72.13 3* -81.87 2* -90.27 1* Effort ROI Regression Iteration
  29. 29. Summary - <ul><ul><li>Trend Setter </li></ul></ul><ul><ul><li>Shorter Time to Market by Adaptive Techniques </li></ul></ul><ul><ul><li>Resource Optimization and Specialization by Center of Excellence </li></ul></ul><ul><ul><li>Test Governance and Dash Boards for Visibility and Control </li></ul></ul><ul><ul><li>Doing More with Less Cost by Reusing Assets </li></ul></ul><ul><ul><li>Test Automation if positive RoI </li></ul></ul>
  30. 30. Thank You