Software Testing Strategies Cliff Andrews & Iain McCowatt
Introduction
Introduction <ul><li>Welcome </li></ul><ul><li>Disclaimer: </li></ul><ul><ul><li>The views expressed during this seminar a...
Introduction  What are the  OBJECTIVES  of Testing? <ul><li>The purpose of testing is to obtain and communicate informatio...
Introduction  What are the  CHALLENGES  of testing? <ul><li>Testing is fundamentally challenging: </li></ul><ul><ul><li>We...
Introduction  What is a  TEST STRATEGY   and how can it  HELP ? <ul><li>A test strategy  is a set of  organising principle...
Introduction  Common Strategies <ul><li>There are many types of Test Strategy 1 . </li></ul><ul><li>Analytical: </li></ul>...
Introduction  Tonight’s Focus <ul><li>Tonight we will look at some of these strategies in more detail: </li></ul><ul><ul><...
Requirements Based Testing
Requirements Based Testing  Overview (What are requirements?) <ul><li>A  requirement  is a statement that identifies a cap...
<ul><li>What is Requirements based testing? </li></ul><ul><ul><li>An analytical strategy where test cases are derived from...
Requirements Based Testing Mechanics: Test Design <ul><li>Test design should start early in the cycle </li></ul><ul><ul><l...
Requirements Based Testing Mechanics: Coverage <ul><li>Every requirement must be tested* </li></ul><ul><ul><li>Does not re...
Requirements Based Testing Mechanics: Techniques <ul><li>How to identify test cases </li></ul><ul><ul><li>Read the require...
Requirements Based Testing Mechanics: No requirements Document? <ul><li>What can you do if there is no formal requirements...
Requirements Based Testing   Mechanics: Heuristics <ul><li>Heuristics: </li></ul><ul><ul><li>Higher priority requirements ...
Requirements Based Testing:  Strengths and Weaknesses <ul><li>Strengths </li></ul><ul><ul><li>Provides definitive understa...
Risk Based Testing
Risk Based Testing  Overview <ul><li>Testing is generally concerned with finding problems 1   </li></ul><ul><li>A  product...
Risk Based Testing  What is Risk? <ul><li>Risk is a function of  likelihood  and  impact </li></ul><ul><li>Many factors af...
Risk Based Testing Mechanics: Process Analysis Control Identification
Risk Based Testing Mechanics: Risk Identification <ul><li>Process: </li></ul><ul><ul><li>Brainstorm potential risks </li><...
Risk Based Testing Mechanics: Risk Analysis <ul><li>Process: </li></ul><ul><ul><li>Assess each risk for likelihood and imp...
Risk Based Testing Mechanics: Risk Control <ul><li>What to test: </li></ul><ul><ul><li>Determine test scope based on risks...
Risk Based Testing:  Strengths & Weaknesses <ul><li>Strengths </li></ul><ul><ul><li>Actively engages stakeholders in testi...
Dynamic Strategies
Dynamic Test Strategies: Overview <ul><li>There are a variety of dynamic approaches: </li></ul><ul><ul><li>Error Guessing ...
Dynamic Test Strategies: Exploratory Testing <ul><li>Exploratory testing is simultaneous learning, test design, and test e...
Dynamic Test Strategies: Mechanics <ul><li>What to test: </li></ul><ul><ul><li>Targeting heuristics 1 </li></ul></ul><ul><...
Dynamic Testing:  Strengths & Weaknesses <ul><li>Strengths </li></ul><ul><ul><li>High yield in terms of finding important ...
Blended Strategies
Blended Strategies: Overview <ul><li>Every test strategy has its strengths and weaknesses </li></ul><ul><li>Context is the...
Blended Strategies: Example Risk Automation Exploratory
Blended Strategies: Example <ul><li>Risk is the motivating force behind our testing.  It helps determine: </li></ul><ul><u...
Blended Strategies: Example <ul><li>Automated regression testing: </li></ul><ul><ul><li>Helps provide rapid feedback on ch...
Blended Strategies: Example <ul><li>Exploratory testing: </li></ul><ul><ul><li>A creative yet structured investigation int...
Upcoming SlideShare
Loading in …5
×

Software Testing Strategies Cliff Andrews

1,839 views

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,839
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
33
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • 1 For more examples, see Lesson 2 in Kaner, Bach and Pettichord : Lessons Learned in Software Testing
  • 1 Rex Black : Surveying Test Strategies: A Guide to Smart Selection and Blending, from Testing Experience Magazine June, 2008 http://www.testingexperience.com/index.html
  • 1 A. Terry Bahill, Frank F.Dean Chapter 4, Discovering Requirements in the Handbook of Systems Engineering and Management , A.P Sage and W.B Rouse http://www.sie.arizona.edu/sysengr/publishedPapers/requirements.pdf
  • 1 Software Errors Cost U.S. Economy $59.5 Billion Annually (NIST) ( http://www.nist.gov/public_affairs/releases/n02-10.htm ) 2 Scott McEwen: Requirements: An introduction ( http://www.ibm.com/developerworks/rational/library/4166.html ) 3 Dean Leffingwell: Calculating your return on investment from more effect requirements management (http://www.ibm.com/developerworks/rational/library/347.html)
  • 1 Gary E. Mogyorodi Requirements-Based Testing – Cause-Effect Graphing, from Testing Experience Magazine, March 3, 2009 http://www.testingexperience.com/index.html
  • 1 Robin F. Goldsmith: Pros and Cons of requirements-based software testing - http://searchsoftwarequality.techtarget.com/tip/0,289483,sid92_gci1347997,00.html
  • 1 James Bach: http://www.satisfice.com/articles/hrbt.pdf
  • 1 Rex Black: Advanced Software Testing - Vol. 2
  • 1 Rex Black: Advanced Software Testing - Vol. 2 2 For example: ISO9126 3 Boris Beizer: Software Testing Techniques
  • 1 Rex Black: Advanced Software Testing - Vol. 2
  • 1 Myers: The Art of Software Testing 2 Boris Beizer: Software Testing Techniques 3 James Whittaker: How to Break Software 4 Cem Kaner: Testing Computer Software
  • 1 Cem Kaner : http://www.kaner.com/pdfs/QAIExploring.pdf (slide 36) 2 Cem Kaner &amp; James Bach: http://www.testingeducation.org/a/nature.pdf (slide 19) 3 Jon Bach : http://www.satisfice.com/articles/sbtm.pdf 4 James Whittaker: Exploratory Software Testing
  • 1 Targeting heuristics I use: The What’s the Mission? Heuristic. The That Scares Me Stupid Heuristic. The Target the Clusters Heuristic. 2 Intensity heuristics I use: The Impact of Failure Heuristic. The History of Failure Heuristic. 3 Early coverage heuristics, Cem Kaner &amp; James Bach: http://www.testingeducation.org/a/nature.pdf (slide 19) Examples: Test capability before reliability Test single features and single variables before combinations Test visible / accessible features before hidden or esoteric ones Test common use cases before unusual ones 4 James Whittacker: How to Break Software 5 Boris Beizer: Software Testing Techniques 6 An excellent set of heuristics is available at: http://testobsessed.com/wordpress/wp-content/uploads/2007/02/testheuristicscheatsheetv1.pdf 7 Stopping heuristics, Michael Bolton: http://www.developsense.com/2009/09/when-do-we-stop-test.html Examples: The Time&apos;s Up! Heuristic. The Piñata Heuristic. The Dead Horse Heuristic. The Mission Accomplished Heuristic. The Mission Revoked Heuristic.
  • Software Testing Strategies Cliff Andrews

    1. 1. Software Testing Strategies Cliff Andrews & Iain McCowatt
    2. 2. Introduction
    3. 3. Introduction <ul><li>Welcome </li></ul><ul><li>Disclaimer: </li></ul><ul><ul><li>The views expressed during this seminar are not intended to be authoritative </li></ul></ul><ul><ul><li>Testing is context dependant </li></ul></ul><ul><ul><li>What works for us may not work for you </li></ul></ul><ul><li>Slides, citations and jumping off points to other literature </li></ul><ul><ul><li>See notes pages for citations & links </li></ul></ul><ul><li>Approach for the evening </li></ul><ul><li>Group introductions </li></ul>
    4. 4. Introduction What are the OBJECTIVES of Testing? <ul><li>The purpose of testing is to obtain and communicate information </li></ul><ul><li>Such information might include 1 ; </li></ul><ul><ul><li>Information about important bugs </li></ul></ul><ul><ul><li>Information about readiness for release </li></ul></ul><ul><ul><li>Information about compliance to requirements or specifications </li></ul></ul><ul><ul><li>Information about compliance to regulations or standards </li></ul></ul>
    5. 5. Introduction What are the CHALLENGES of testing? <ul><li>Testing is fundamentally challenging: </li></ul><ul><ul><li>We cannot test everything </li></ul></ul><ul><ul><li>We cannot find all the bugs </li></ul></ul><ul><ul><li>Time and resources are limited </li></ul></ul>
    6. 6. Introduction What is a TEST STRATEGY and how can it HELP ? <ul><li>A test strategy is a set of organising principles for achieving the objectives of testing </li></ul><ul><li>A test strategy can help you decide: </li></ul><ul><ul><li>what to test </li></ul></ul><ul><ul><li>how aggressively to test </li></ul></ul><ul><ul><li>what order to test in </li></ul></ul><ul><ul><li>what types of bugs to test for </li></ul></ul><ul><ul><li>when to stop </li></ul></ul>
    7. 7. Introduction Common Strategies <ul><li>There are many types of Test Strategy 1 . </li></ul><ul><li>Analytical: </li></ul><ul><ul><li>Requirements based </li></ul></ul><ul><ul><li>Risk based </li></ul></ul><ul><li>Model Based: </li></ul><ul><ul><li>Scenario based </li></ul></ul><ul><li>Methodical: </li></ul><ul><ul><li>Function Based </li></ul></ul><ul><ul><li>Standards Based </li></ul></ul><ul><li>There is no “one size fits all” strategy, each has strengths and weaknesses </li></ul><ul><li>Dynamic: </li></ul><ul><ul><li>Defect Taxonomy </li></ul></ul><ul><ul><li>Error Guessing </li></ul></ul><ul><ul><li>Attacks </li></ul></ul><ul><ul><li>Exploratory </li></ul></ul><ul><li>Process Based: </li></ul><ul><ul><li>Historical </li></ul></ul><ul><ul><li>Lifecycle </li></ul></ul><ul><li>Regression: </li></ul><ul><ul><li>Continuous Integration </li></ul></ul>
    8. 8. Introduction Tonight’s Focus <ul><li>Tonight we will look at some of these strategies in more detail: </li></ul><ul><ul><li>Requirements based </li></ul></ul><ul><ul><li>Risk based </li></ul></ul><ul><ul><li>Dynamic strategies </li></ul></ul><ul><ul><li>Blended strategies </li></ul></ul>
    9. 9. Requirements Based Testing
    10. 10. Requirements Based Testing Overview (What are requirements?) <ul><li>A requirement is a statement that identifies a capability or function that is needed by a system in order to satisfy its customer’s needs 1 </li></ul><ul><ul><li>A functional requirement defines what the system must do- inputs and expected outputs </li></ul></ul><ul><ul><li>A non-functional requirement describes how well the system performs, its usability, performance, reliability, etc </li></ul></ul><ul><li>External: Driven by customer needs, describe what the system must do, but not how it should be implemented </li></ul><ul><li>Internal: Driven by company, could be determined by processes, resources, engineering standards etc </li></ul>
    11. 11. <ul><li>What is Requirements based testing? </li></ul><ul><ul><li>An analytical strategy where test cases are derived from defined requirements </li></ul></ul><ul><ul><li>Complimented by design docs, use cases, data base schemas, etc </li></ul></ul><ul><li>Validation vs. verification of requirements </li></ul><ul><ul><li>Verification: Does software meet requirements? </li></ul></ul><ul><ul><li>Validation: Are requirements correct? Is the need being fulfilled? </li></ul></ul><ul><li>Testers play an important role in requirement review & validation </li></ul>Requirements Based Testing Overview (Requirements Testing)
    12. 12. Requirements Based Testing Mechanics: Test Design <ul><li>Test design should start early in the cycle </li></ul><ul><ul><li>Provides additional requirements validation </li></ul></ul><ul><ul><li>Provides enough time to learn the system and develop tests </li></ul></ul><ul><ul><li>The earlier issues are discovered, the cheaper/easier it is to fix 1, 2, 3 </li></ul></ul><ul><li>Test Design </li></ul><ul><ul><li>Tests can directly reflect the requirement </li></ul></ul><ul><ul><li>Test may not explicitly reflect requirement </li></ul></ul><ul><ul><li>Test ‘around’ requirements </li></ul></ul>
    13. 13. Requirements Based Testing Mechanics: Coverage <ul><li>Every requirement must be tested* </li></ul><ul><ul><li>Does not require one to one relationship </li></ul></ul><ul><ul><li>Multiple requirements can be covered in a single test </li></ul></ul><ul><ul><li>Some requirements may require multiple tests </li></ul></ul><ul><ul><li>Track coverage with a requirements-test case matrix </li></ul></ul><ul><ul><li>Prove requirement is met or exceeded, but not to what extent </li></ul></ul><ul><li>*Untestable requirements </li></ul><ul><ul><li>Can’t test requirements stating ‘shall never’, shall always’ </li></ul></ul><ul><ul><li>Requirements that do not evaluate to a boolean, i.e. reqs that cannot be stated using conditional logic if/then/else </li></ul></ul><ul><ul><li>Those we cannot test directly </li></ul></ul>
    14. 14. Requirements Based Testing Mechanics: Techniques <ul><li>How to identify test cases </li></ul><ul><ul><li>Read the requirement and write a test case </li></ul></ul><ul><ul><li>Pull attributes from requirements and develop tests </li></ul></ul><ul><ul><li>Other test design techniques, for example requirements expressed with conditional logic lend themselves to decision tables / cause-effect graphs. 1 </li></ul></ul><ul><ul><li>Consider flow of system, robustness, performance, stability </li></ul></ul>
    15. 15. Requirements Based Testing Mechanics: No requirements Document? <ul><li>What can you do if there is no formal requirements document? </li></ul><ul><li>Use a different strategy </li></ul><ul><li>But, what if you don’t want to use those? </li></ul><ul><li>Look for something similar to requirements </li></ul><ul><ul><li>Models, for example a Call flow for automated phone system </li></ul></ul><ul><ul><li>Design documentation and user guides </li></ul></ul><ul><ul><li>Consider behaviour of existing software. E.g.: building a new GUI, but must behave similarly to the current interface </li></ul></ul><ul><ul><li>GUI - is there a style guide? </li></ul></ul><ul><ul><li>Standards </li></ul></ul><ul><ul><li>Other resources: BA or the actual customer </li></ul></ul>
    16. 16. Requirements Based Testing Mechanics: Heuristics <ul><li>Heuristics: </li></ul><ul><ul><li>Higher priority requirements need more focus </li></ul></ul><ul><ul><li>Make use of design documentation, user manuals </li></ul></ul><ul><ul><li>Customer habits – Can identify areas used the most, hence heavier testing </li></ul></ul><ul><ul><li>Consider historical risks and bugs </li></ul></ul>
    17. 17. Requirements Based Testing: Strengths and Weaknesses <ul><li>Strengths </li></ul><ul><ul><li>Provides definitive understanding of what system must do </li></ul></ul><ul><ul><li>Prioritized requirements guide test case prioritization </li></ul></ul><ul><ul><li>Ensures system meets requirements - thus provides value 1 </li></ul></ul><ul><ul><li>Metrics: Pass/failure rate more reflective of system quality </li></ul></ul><ul><ul><li>Definitive end to testing </li></ul></ul><ul><li>Weaknesses </li></ul><ul><ul><li>If the requirement is wrong, software is wrong even if test passes </li></ul></ul><ul><ul><li>Does not easily accommodate vague or ambiguous requirements 1 </li></ul></ul><ul><ul><li>May not cover non-functional characteristics, which are harder to specify </li></ul></ul><ul><ul><li>Areas of high risk not explicitly identified </li></ul></ul>
    18. 18. Risk Based Testing
    19. 19. Risk Based Testing Overview <ul><li>Testing is generally concerned with finding problems 1 </li></ul><ul><li>A product risk is an issue which may exist in the product under test </li></ul><ul><li>Risk based testing uses product risks to focus test efforts </li></ul>
    20. 20. Risk Based Testing What is Risk? <ul><li>Risk is a function of likelihood and impact </li></ul><ul><li>Many factors affect likelihood, including 1 : </li></ul><ul><ul><li>Complexity </li></ul></ul><ul><ul><li>Change </li></ul></ul><ul><ul><li>Constraints </li></ul></ul><ul><li>Factors affecting impact include 1 : </li></ul><ul><ul><li>Consequence of failure; safety, financial, image, liability </li></ul></ul><ul><ul><li>Civil and criminal liability issues </li></ul></ul><ul><ul><li>Frequency of use </li></ul></ul><ul><ul><li>Availability of workarounds </li></ul></ul><ul><ul><li>Integration & dependencies </li></ul></ul><ul><ul><li>Organisational factors </li></ul></ul>
    21. 21. Risk Based Testing Mechanics: Process Analysis Control Identification
    22. 22. Risk Based Testing Mechanics: Risk Identification <ul><li>Process: </li></ul><ul><ul><li>Brainstorm potential risks </li></ul></ul><ul><li>Key Decisions 1 : </li></ul><ul><ul><li>Level of detail </li></ul></ul><ul><ul><li>Selection of stakeholders </li></ul></ul><ul><li>Heuristics: </li></ul><ul><ul><li>Quality characteristics 2 </li></ul></ul><ul><ul><li>Bug taxonomies 3 </li></ul></ul><ul><ul><li>Requirements </li></ul></ul><ul><ul><li>Lists of functions </li></ul></ul><ul><ul><li>Historical bugs </li></ul></ul>
    23. 23. Risk Based Testing Mechanics: Risk Analysis <ul><li>Process: </li></ul><ul><ul><li>Assess each risk for likelihood and impact </li></ul></ul><ul><ul><li>Determine appropriate control; Mitigate / Transfer / Ignore </li></ul></ul><ul><li>Key Decisions 1 : </li></ul><ul><ul><li>Quantitative vs. qualitative </li></ul></ul><ul><ul><li>Informal vs. formal </li></ul></ul><ul><ul><li>Selection of stakeholders </li></ul></ul><ul><ul><li>Formula for risk priority </li></ul></ul><ul><li>Heuristics: </li></ul><ul><ul><li>Complexity measures </li></ul></ul><ul><ul><li>Historical defect densities </li></ul></ul><ul><ul><li>Support costs </li></ul></ul>
    24. 24. Risk Based Testing Mechanics: Risk Control <ul><li>What to test: </li></ul><ul><ul><li>Determine test scope based on risks to “mitigate” </li></ul></ul><ul><li>How aggressively to test: </li></ul><ul><ul><li>Determine intensity based on risk priority </li></ul></ul><ul><li>What order to test in: </li></ul><ul><ul><li>Sequence test design and execution based on risk priority </li></ul></ul><ul><li>What types of bugs to test for: </li></ul><ul><ul><li>Design tests based on the types of risks identified </li></ul></ul><ul><li>When to stop: </li></ul><ul><ul><li>Execute tests, reporting mitigated vs. residual risk </li></ul></ul><ul><ul><li>Stop when stakeholders are comfortable with balance of mitigated vs. residual risk </li></ul></ul>
    25. 25. Risk Based Testing: Strengths & Weaknesses <ul><li>Strengths </li></ul><ul><ul><li>Actively engages stakeholders in testing process </li></ul></ul><ul><ul><li>Informs entire test process from planning, through design, to release decision </li></ul></ul><ul><ul><li>Can function in the absence of test basis (requirements, design, models etc) </li></ul></ul><ul><li>Weaknesses </li></ul><ul><ul><li>Requires active stakeholder involvement </li></ul></ul><ul><ul><li>Risk identification and analysis is limited by the knowledge and imagination of the participants </li></ul></ul><ul><ul><li>Tends to rely on “up front” analysis and may be slow to react to change </li></ul></ul>
    26. 26. Dynamic Strategies
    27. 27. Dynamic Test Strategies: Overview <ul><li>There are a variety of dynamic approaches: </li></ul><ul><ul><li>Error Guessing 1 </li></ul></ul><ul><ul><li>Defect taxonomies 2 </li></ul></ul><ul><ul><li>Attacks 3 </li></ul></ul><ul><ul><li>Exploratory 4 </li></ul></ul><ul><li>Here we will focus on Exploratory testing, which can be used to blend all of the above </li></ul>
    28. 28. Dynamic Test Strategies: Exploratory Testing <ul><li>Exploratory testing is simultaneous learning, test design, and test execution 1 </li></ul><ul><li>All testing is exploratory to some degree 2 : </li></ul><ul><li>Exploratory testing is NOT ad hoc testing, it can be managed and directed through: </li></ul><ul><ul><li>Session Based Test Management 3 </li></ul></ul><ul><ul><li>Tours 4 </li></ul></ul>More Exploratory More Scripted
    29. 29. Dynamic Test Strategies: Mechanics <ul><li>What to test: </li></ul><ul><ul><li>Targeting heuristics 1 </li></ul></ul><ul><li>How aggressively to test: </li></ul><ul><ul><li>Intensity heuristics 2 </li></ul></ul><ul><li>What order to test in: </li></ul><ul><ul><li>Early coverage heuristics 3 </li></ul></ul><ul><li>What types of bugs to test for: </li></ul><ul><ul><li>Attacks 4 </li></ul></ul><ul><ul><li>Bug Taxonomies 5 </li></ul></ul><ul><ul><li>Test heuristics 6 </li></ul></ul><ul><li>When to stop: </li></ul><ul><ul><li>Stopping heuristics 7 </li></ul></ul>
    30. 30. Dynamic Testing: Strengths & Weaknesses <ul><li>Strengths </li></ul><ul><ul><li>High yield in terms of finding important bugs fast; excellent under time pressure </li></ul></ul><ul><ul><li>Maximizes time spent hands on testing, at the expense of planning and documentation </li></ul></ul><ul><ul><li>Can be executed with little or no preparation </li></ul></ul><ul><ul><li>Can react rapidly to changes or emerging risks </li></ul></ul><ul><ul><li>Can function in the absence of test basis (requirements, design, models etc) </li></ul></ul><ul><li>Weaknesses </li></ul><ul><ul><li>Requires experienced testers </li></ul></ul><ul><ul><li>Light in terms of documentation and coverage </li></ul></ul><ul><ul><li>Low repeatability </li></ul></ul>
    31. 31. Blended Strategies
    32. 32. Blended Strategies: Overview <ul><li>Every test strategy has its strengths and weaknesses </li></ul><ul><li>Context is the key determinant of strategy selection </li></ul><ul><li>Blending strategies enable you to mitigate the weakness of one with the strength of another </li></ul><ul><li>For example… </li></ul>
    33. 33. Blended Strategies: Example Risk Automation Exploratory
    34. 34. Blended Strategies: Example <ul><li>Risk is the motivating force behind our testing. It helps determine: </li></ul><ul><ul><li>What to test </li></ul></ul><ul><ul><li>When to test </li></ul></ul><ul><ul><li>How aggressively to test </li></ul></ul><ul><ul><li>When to stop testing </li></ul></ul>Risk Automation Exploratory
    35. 35. Blended Strategies: Example <ul><li>Automated regression testing: </li></ul><ul><ul><li>Helps provide rapid feedback on changes </li></ul></ul><ul><ul><li>Is a safety net which lets us change more, and later </li></ul></ul><ul><ul><li>Frees our testers from repetitive regression testing and allows them to perform… </li></ul></ul>Risk Exploratory Automation
    36. 36. Blended Strategies: Example <ul><li>Exploratory testing: </li></ul><ul><ul><li>A creative yet structured investigation into potential issues </li></ul></ul><ul><ul><li>Sufficiently dynamic to respond to changing requirements and newly emerging risks </li></ul></ul>Risk Automation Exploratory

    ×