TEA Presentation V 0.3


Published on

TEA - Test Escape Analysis

  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

TEA Presentation V 0.3

  1. 1. Ian McDonaldTest Escape Analysis - Presentation © 2010 Ian McDonald 1
  2. 2. Cost of Defects Fixing  It can take one person 10 minutes to find a defect in requirements, 10 minutes to fix the document. Cost £10 per defect.  Finding the same defect in Functional Test and then Fixing. Cost £211 per defect.  Finding the same single defect in System Integration Testing. Cost £587 per defect.  Cost following delivery £? + £Reputation. 2
  3. 3. Add Success to Testing  There are usually quick wins, however do we review the success of our testing? 3
  4. 4. Quick Wins  How many defects could have been avoided by changing the review practices and acceptance of Requirements into Development?  Measure by Audit of 1 product: 1. Audit Defect Logs 2. Audit Customer Service Reports. 3. Audit Requirements and their Review. 4
  5. 5. RCA vs TEA  Root Cause Analysis (RCA) allows us to learn, rectify and avoid future defect injection. Providing we use the data.  Test Escape Analysis (TEA) in contrast allows us to become more efficient at testing and avoid future defects from escaping detection. Provided we use the data. 5
  6. 6. TEA Applied to Defect History  Applying TEA to defect history can help to analyse where resources and changes in procedure should be applied for maximum return on investment.  Normally one would focus upon:  a single typical representative project.  High and Critical defects, to reduce sample size.  TEA can be integrated into the defect tracking system. 6
  7. 7. Purpose of TEA Root Cause Analysis is aimed at the code programming – why was the defect introduced.  Analogy why did the tightrope walker fall TEA in contrast is about why a defect was not detected.  Analogy why did we not catch the falling tightrope walker. We do TEA to learn how to detect more defects and how to detect them sooner and smarter. The ultimate goal is to drive down the number of defects leaking to the customer by finding those defects ourselves first. 7
  8. 8. Benefits  The benefits of early detection are:  Reduced costs.  Improved reputation.  Reduced loading on engineers.  Improved output from testing team. 8
  9. 9. Trends With TEA we are interested in trends.  Is there a pattern in the defects that are being let through?  Is there a type of test that could help catch more defects. We target TEA at a sample of defects. This sample includes:  Defects raised for example by the Customer Which are either  Critical or  High The knowledge that we gain from TEA is applied to future Test Plans:  Trend Reports in product TEA - produced by the QA Engineers.  Product Teams are expected to produce their own reports on their individual trends and apply lessons learned to their Test Plans. 9
  10. 10. Resources TEA Meetings are linked to Company Targets  Are to occur at least monthly for each team.  Expectations  For this activity teams usually need a developer and tester to attend the monthly meeting.  In practice a TEA meeting usually takes an hour per month.  In looking at a TEA report one may initially spend around 15 minutes per record, however it is possible to do these in just 5 minutes each.  Remember we are looking at a sample – not all. 10
  11. 11. Managing TEA There are 2 ways that TEA meetings run. Either:  Hold a meeting at which the TEA reports are completed.  Complete the reports before the meeting and at the meeting these are audited. The TEA meetings ensure that there is a consistent approach and it also acts as a mentoring exercise in supporting the rollout of TEA. 11
  12. 12. TEA Assumptions In carrying out TEA there is an assumption that:  The review team have some minimal familiarity with the code impacted by the defect.  The review team are familiar with testing techniques. This can be obtained by either:  Reading BS7925-2;  Laying on an internal course 2 or 3 days;  Attending an ISEB/IQTB Test Foundation Course. 12
  13. 13. Management of TEA and RCA  Root Cause Analysis if employed improves development through the application of lessons learned. RCA is usually managed through the defect tracking tool employed.  TEA in contrast improves testing. However it to can be managed through the defect life cycle. 13
  14. 14. Summary of TEA Data Fields The following Fields are used within TEA:  TEA Status  Reason Introduced  Development Phase  Code Designed From  Test Environment  Test Tools Required  Test Technique  Test Type  Free Text Notes Field All fields are mandatory, but there is a “not known” option available. 14
  15. 15. Reason Introduced A shared field with the main Defect and RCA. The record can be changed from any of the reports. Fields include:  Code Missing  Code not fully built or tested Stage Defect Introduced:  Coding Error “Earliest stage in the processes  Design Incorrect that the defect could have been  Design Missing or Incomplete prevented”.  Design Unclear Reason Introduced:  Environment Unavailable “Provide finer detail as to the initial root cause of the defect”.  Initial Fix Incomplete  Initial Fix Incorrect  Requirement Incorrect  Requirement Missing or Incomplete  Requirement Unclear  Standards Not Followed  Typing Mistake  Other (free text) 15
  16. 16. Development Phase Requirements Review, Design Review, What was the appropriate appraisal process (testing or reviews) that Code Review, would have detected this defect earlier? Unit Test, Component Test, Component Integration Test, System Test 16
  17. 17. Code Designed From API Spec, What was the source used, Detail Design, from which the code with the defect was created? Requirement, Standards, Functional Specification, Developer Led Requirements, Other (Selecting Other allows a free text field to be used). 17
  18. 18. Test Environment Sufficient, Customer Device, External Server, System Component, Stub, HRP (Hardware Reference Platform) Feature, OS / HRP Config, Platform, What other Test Environment would have helped to detect the bug earlier? (other than External Device, what was planned or available) Live Environment, Other (Selecting Other allows a free text field to be used). 18
  19. 19. Test Tools Required What other Test tools would have helped to Sufficient, detect the bug earlier? (other than what was planned or available) Test Execute Framework Extension, Test Driver Extension, Code Analyser, Protocol Tester, Other (Selecting Other allows a free text field to be used). 19
  20. 20. Test Technique Boundary Analysis (See BS7925-2) Cause Effect Graphing (See BS7925-2) Classification Tree Equivalence Partitioning (See BS7925-2) Out Of Memory Random Testing Risked Based Targeted Exploratory Testing State Transition Testing (See BS7925-2) Static Analysis (Reviews and the use of Static Analysis Tools) 20
  21. 21. Test Type  API Validation  Test In Real Environment  Binary Compatibility testing  Test on Emulator  Compliance Testing  Dependency Testing  Test Tools – Code Coverage  Design Verification  Test Tools –Static Binary  Error Guessing Compatibility  Exploratory Testing  Test Tools – Codenomicon  Functional Testing  Test Tools – Copyright  Interoperability Testing  Test Tools – Coverity  Intrusive Testing  Performance Testing  Test Tools – Lint  Platform Verification Testing  Test Tools – Win Runner  Product Configuration Testing  Test Tools – Load Runner  RAM / ROM Testing  Test Tools – Other  Recovery (Robustness) Testing  Open Source Check  Requirements Validation  Security Vulnerability Testing  Static Analysis  Static Testing  Negative Testing 21
  22. 22. Evidence from TEA TEA can be useful for highlighting unforeseen problems. We might have predicted that inadequate negative tests are being written (and this is supported by TEA in this example) but there is also much evidence to suggest that we need to write more positive test cases. 450 Test Techniques - Test Technique that would find the Defect 418 400 350 Nos. of Defects 300 244 250 200 150 96 93 100 61 56 53 46 44 37 35 50 26 17 16 15 7 5 5 2 0 0 Re st ua s Eq ery lity Re r if a t a rt cy fo e r) st n e ce ga est BC ns dA M pe ity S e ls pe e gr is M view St a nc Co s sio e A c l Te nc Ve o O Re lys en Pe lia n vP bi ra De cu r eT Pl (o th v Ne e T To O In epta co ra rm eT nd n rm t iv ui e iv ic p m at an sit rfo c ro un at St Po te Bo 22