Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs

1,232 views
1,028 views

Published on

This presentation on Hypothesis Based Testing (HBT) was delivered by Mr Satvik Kini, Associate Quality Manager, Suite Test Centre, SAP Labs India Pvt. Ltd at STeP-IN Forum webinar on Dec 19, 2013.

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,232
On SlideShare
0
From Embeds
0
Number of Embeds
466
Actions
Shares
0
Downloads
9
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Hypothesis Based Testing – Application and Adaptation for testing Enterprise Software - at SAP Labs

  1. 1. Hypothesis Based Testing - Application and Adaptation for testing Enterprise Software Satvik Kini, Suite Test Center, SAP Labs India Pvt Ltd., Bangalore December 19, 2013 Public
  2. 2. Agenda  Hypothesis Based Tests – Overview  Traditional Test Approach vs. HBT  Stages in HBT Approach – (Detailed Explanation with a case study)  Challenges and Benefits  Key factors for success  Summary  Q&A © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 2
  3. 3. Hypothesis Based Tests - Overview
  4. 4. Hypothesis Based Tests - Overview Hypothesis : Tentative explanation for a certain behavior, phenomenon, events that have occurred or will occur • Null Hypothesis – H0 – Eg: Heavy Smoking is the cause of Lung Cancer • Alternate Hypothesis – H1 – Eg: Heavy Smoking is not the cause of Lung Cancer Hypothesis Based test:* Goal focused approach, test methodology based on solid scientific principles built on the core theme of hypothesizing potential defects. * HBT is the IP of STAG Software. © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 4
  5. 5. Hypothesis Based Tests - Overview The Scientific Method Form a Question Does my software work ? Research Understand Context Hypothesis Hypothesize Defects Experiment Develop & Execute Tests Analyze Results Pass or Fail ? Draw Conclusions Does my software work ? © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 5
  6. 6. Traditional Test Approach Vs. HBT
  7. 7. Traditional Test Approach Vs. HBT Goal Activities Manual Test Cases Automated Test Cases ---------------------------------------------------------------------------------Hopefully results in Defects Determines Risks/ Potential Defects Drives Activities Test Levels Test Design Test Execution ------------------------------------------------------Defect Elimination Focus: Coverage, Automation, Test Scripts © 2013 SAP AG or an SAP affiliate company. All rights reserved. Clean Software Focus: Defect Elimination based approach for cleanliness Public 7
  8. 8. Traditional Test Approach - Limitations Limitations of Traditional Approach • • • • Often Software is passed with X% OK Rate Assumption – existing test cases cover desired parts of the software Hopeful encounter of bugs Focus on Quantitative reporting © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 8
  9. 9. Stages in HBT
  10. 10. HBT – An Example Example 1 : Clean Water Goal: Having Clean Water Potential Defects in water: color, odour, suspended particles, bacteria Cleaning Process: • Conduct activities to first determine if each of the potential defects exists • If a potential defect exists, then take steps to clean that defect • Once all defects are removed, re-test the water for all potential defects and certify that the water is clean Result: Clean Water (goal is achieved) © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 10
  11. 11. HBT - Stages Define Cleanliness Criteria Define Potential Defect Types (PDT) Perform Testing Analyze and Conclude Understand End user expectation, System Behavior Consider All possible different defect types and identify each with test quality levels Use of different test techniques and tools to test each of the PDTs Document and Analyze the results and get an overall picture on Quality © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 11
  12. 12. Stages in HBT – Case Study Scenario – Student Course Registration and Completion User Types Equivalency Determination • Course Registration Student • Admin • Record External Achievements Academic Advisor/Instructor Attendance Tracking Market Places Produce Transcript Grading Examination US Region • Europe Region • © 2013 SAP AG or an SAP affiliate company. All rights reserved. • APJ Region Public 12
  13. 13. HBT – Cleanliness Criteria (Goal) Example 2 : End to End (E2E) Scenario tests Define Cleanliness Criteria Goal : Define Potential Defect Types Perform Testing Analyze and Conclude The E2E Scenario Student Course Registration and Completion works fine © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 13
  14. 14. HBT – Define Potential Defect Types Define Cleanliness Criteria Define Potential Defect Types © 2013 SAP AG or an SAP affiliate company. All rights reserved. Perform Testing Analyze and Conclude Public 14
  15. 15. HBT – Test Levels Level 5 Expert Level E2E Tests Level 4 Advanced E2E Scenario Tests Level 3 Integration tests Level 2 Primary E2E Use Cases Level 1 Transactional Coverage workflows; role based tests;Accessibility Integration across scenarios Multiple Users; Timezone based; Less Used features Integration defects; Reporting usability Key E2E Requirements Browser compatibility Performance Obvious Errors Runtime Errors Commonly used E2E Use cases • • • Each level corresponds to a set of potential Detect Types (PDTs) A purposeful detection of PDTs Only once all PDTs are tested for, a level is called to be reached © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 15
  16. 16. HBT – Test Matrix Creation of Report Card - Derive Test Matrix Test levels Potential Defect Types Level 1 Runtime Errors Obvious Functional Errors Commonly used E2E usecases Level 2 Broswer Compatibility Calculations Performance Level 3 Integration Reports Usability Level 4 Currency conversion Less used features Timezone based Level 5 Workflows Multiple Users Accessibility © 2013 SAP AG or an SAP affiliate company. All rights reserved. Integration to other scenarios Public 16
  17. 17. HBT – Perform Testing Define Cleanliness Criteria Define Potential Defect Types Perform Testing Analyze and Conclude I. Use of Existing Test Techniques/Tools : 1. Test case driven testing or scripted testing Automated test cases (only scripted tests) b. Manual test cases (also giving the tester liberty of ad-hoc tests) a. 2. All pairs testing 3. Exploratory testing Usage of Specific Test tools 4. II. Preparation of Charters © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 17
  18. 18. HBT – Analyze and Conclude Define Cleanliness Criteria Define Potential Defect Types Perform Testing Analyze and Conclude • Document the results using charters/protocols • Check the test results for each of the PDTs • Update Report Card/Test Matrix - get a overall picture on Product Quality © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 18
  19. 19. HBT – Test Matrix Update Report Card Test levels Potential Defects Types Level 1 Runtime Errors Obvious Functional Errors Commonly used E2E usecases Level 2 Broswer Compatibility Calculations Performance Level 3 Integration Reports Usability Level 4 Currency conversion Less used features Timezone based Level 5 Workflows Multiple Users Accessibility © 2013 SAP AG or an SAP affiliate company. All rights reserved. Integration to other scenarios Public 19
  20. 20. Summary - HBT Realization Define Cleanliness Criteria Prepare PDT List Derive Test Matrix Prepare Charters Test Using Charters Protocol Logging Update Report Card © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 20
  21. 21. Challenges, Benefits and Key Factors for Success
  22. 22. Challenges Challenges • Understanding different user contexts – market places • Existence of custom code on top of standard Ads • © 2013 SAP AG or an SAP affiliate company. All rights reserved. Highly configurable software – customization and Business Functions Public 22
  23. 23. Benefits Benefits • Confidence on what is tested • Exploration of new scenarios as we do the tests • Reduction in customer messages Ads • Test Matrix gives an overall picture of quality • Higher Probability of finding defects as the goal is to check existence of PDT • Improvement - Code Coverage © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 23
  24. 24. Key Factors for Success Key Factors for Success • Success mainly depends on Understanding internal and external factors and list of PDTs • Identifying each PDT and its corresponding level Ads • For each of the PDT, choosing right set of scenarios/risks • Choosing right set of tools and test techniques to test each PDT © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 24
  25. 25. Summary
  26. 26. Summary Goal focused approach, test methodology based on solid scientific principles built on the core theme of hypothesizing potential defects. © 2013 SAP AG or an SAP affiliate company. All rights reserved. Public 26
  27. 27. Q&A
  28. 28. Thank you Contact information: Satvik Kini Associate Quality Manager SAP Labs India Pvt. Ltd, Bangalore +91 98863 89654 satvik.kini@sap.com © 2013 SAP AG or an SAP affiliate company. All rights reserved.
  29. 29. APPENDIX © 2013 SAP AG or an SAP affiliate company. All rights reserved.
  30. 30. Stages in HBT - Realization BACK Example of Charter : Scope PDT Risks To Manage How Test Techniques Tester All Business Processes Common Used E2E use cases Check the straight forward E2E scenario is working Run Automated Test cases Automated test cases Garbage Collector Tour Browser Compatibility Check Attendance Tracking UI is compatible with IE9, Chrome Version 30, Firefox, Safari Check All the Tabs, Popups, F4 Helps, Result List, Status Icons, Calendar Display Exploratory Tours : Super Model Tour, Garbage Collector Tour Mr. B Equivalency Determination Obvious Functional Errors Mapping of External Achievements to Internal subjects Try different achievements with different status and validity dates Exploratory Tour : Intellectual Tour Additional Comments Mr. A Attendance Tracking UI Messages Mr. C © 2013 SAP AG or an SAP affiliate company. All rights reserved. All pair Testing Public 30
  31. 31. Stages in HBT - Realization BACK Update each Charter after testing each of the PDTs Scope PDT Risks To Manage How Test Techniques Tester Messages Additional Comments All Business Processes Common Used E2E use cases Check the straight forward E2E scenario is working Run Automated Test cases Automated test cases Landmark Tour Mr. A None All Business processes are working fine Attendance Tracking UI Browser Compatibility Check Attendance Tracking UI is compatible with IE9, Chrome Version 30, Firefox, Safari Check All the Tabs, Popups, F4 Helps, Result List, Status Icons, Calendar Display Exploratory Tours : Super Model Tour, Garbage Collector Mr. B 23348743 32476764 Checked browser compatible for the IE9,chrome,Safari . Yet to check for Firefox Equivalency Determination Obvious Functional Errors Mapping of External Achievements to Internal subjects Try different achievements with different status and validity dates Exploratory Tour : Intellectual Tour Mr. C 43234234 Runtime Error while clicking on the Sort Button Blocked © 2013 SAP AG or an SAP affiliate company. All rights reserved. All pair Testing Public 31

×