Your SlideShare is downloading. ×
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Th14   Balanced Testing   Presentation
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Th14 Balanced Testing Presentation

131

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
131
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Martin luther king . Don’t pretent to be as big as he is. Also don’t want to perform an act/speech about the power of testers .. The swedish ladies did better in there fun session about supertesters, than I could ever do. But I have a dream …. About testers considered as equal as other project members. I was wonder … why ….. The added value of testing is still discussed within projects. The introduction of risk based testing improved the value of testing, but we still can not answer the question about what depth of testing is the most efficience depth …. We often stop testing when we are out of budget or time … but did we achieve the best quality level? So the question is not whether we should test of not (we already convinced the project/organisation of the need for testing) … but what is the optimum balance / most optimum value of testing.
  • - In the top list of quality gurus next to Deming and Shewhart (Plan do check act cycle), Philip Crosby (Quality is free) - Pareto principle: by doing the job 20%, 80% of the advance of the entire job can be created. Or 80% of the problems are produced by 20% of the causes - Six sigma: performance improvement methodology. Toolbox. Six sigma levels represents 3,4 defects per million opportunities (Five sigma 233 defects) - Lean approach: produce what is needed and avoid wastes. 8 wastes: (overproduction, waiting, transport, extra processing, inventory, motion, defects, waste of human potential)
  • Histogram. Lower specification Limit, Higher specification limit. 68.27 percent of any normal population lies between + or – 1 sigma of the average. Sigma is the standard deviation. 95,45 between 2 sigma …. Etc. Define: scope, critical to quality, cost of poor quality, process map Measure: Identify process variables, current measurement system Analyse: Identiy causes of variation, conduct hypothesis testing Improve: identify possible solutions, verify solutions Control: Implement and sustain
  • Reactive QA maturity (far left block) On this level mostly corrective measures are present. Failures are found in the production-environment after going life. Next to the direct costs of correcting the failures the costs of marketing to correct the damage/decreased image are also high. The total cost of quality is much too high and can be reduced by 30-50 percent. The organisation/project only reacts on failures that occur. 2. Passive QA maturity (corrective and detective measures at a late stage) The focus within the level lies on finding the failures on the right side of the V-model during dynamic testing (often during the acceptance test). Testing is present but often only in a unstructured and ineffective way. Almost no preventive measures are present. 3. (Over)active QA maturity (far right block) Lots of preventive and detective measures are taken. Lots of test types are present and less defects are found after going life..... But testing and QA are too expensive. Finding a defect will take more time than correcting them. The damage occurred by the remaining defects is almost null. So: Too much detective and preventive measures in relation to the corrective measures. The total costs of quality can be reduced by reducing the number of preventive and detective measures (or by defining more efficient measures). 4. Balanced QA maturity The level in which the right balance between the quality measures is reached and proven via the collected metrics.
  • High lead time to train new testers.
  • Transcript

    • 1. Th14 - How Much Testing is Enough? Achieving the Optimum Balance 2 december 2010
    • 2. Introduction
      • ISEB Practitioner
      • Lean Six sigma green belt
      • Over 12 years experience within testing area
    • 3. Agenda
      • Introduction
      • Test improvement based on ECL*
          • Identify current position
          • Define goal: Next position
          • Define and implement measures
      * Economic Conformance Level
    • 4. Agenda
      • Introduction
      • Test improvement based on ECL
        • Identify current position
        • Define goal: Next position
        • Define and implement measures
    • 5. The question is: How to convince the project/organization of the added value of testing?
    • 6. Inspired by Joseph Juran
      • Lived from 1904 - 2008
      • One of the Quality gurus
      • Specialized in managing quality
      • Pioneer of Six sigma
      • The Lean approach
      • Founder of the pareto principle
    • 7. Six sigma LSL USL
    • 8. Lean approach – avoid wastes Waste Examples of translation to testing Overproduction Good enough testing Waiting Minimize gaps between cause and detect of a problem Transport Deployment of software / Effective Test data and infrastructure management Extra processing Avoid extra testruns and “ Unconscious” redundancy of test Inventory Avoid to much test capacity / Clear and effective reporting Motion Monitoring the total process Defects Prevent instead of detect or correct / Standardize and Simplify human potential Effective use of human capabilities
    • 9. Jurans curve Conformance
    • 10. Agenda
      • Introduction
      • Test improvement based on ECL
        • Identify current position
        • Define goal: Next position
        • Define and implement measures
    • 11. TMM 1. Initial 2. Managed
      • Test policy and strategy
      • Test planning
      • Test monitoring and control
      • Test design and execution
      • Test environment
      3. Integration
      • Test organisation
      • Test training program
      • Test life cycle and integration
      • Non-functional testing
      • Peer reviews
      4. Management and measurement
      • Test management
      • Software Quality evaluation
      • Advanced peer reviews
      5. Optimisation
      • Defect prevention
      • Test process optimisation
      • Quality control
      Correct Detect Prevent
    • 12. Six sigma approach
      • Define -> Improvement goals & cost of poor quality
      • Measure -> current process & establish metrics
      • Analyse -> problems and causes within process & hypothetic testing
      • Improve -> by identifying and piloting solutions & verify solution
      • Control -> by standardization and monitoring
    • 13. Reach the balance E. Improve D. Analyse B. Measure F. Control C. Define Detailed A. Define Global
    • 14. Agenda
      • Introduction
      • Traditional test improvement
      • Test improvement based on ECL
        • Identify current position
        • Define goal: Next position
        • Define and implement measures
    • 15. Software quality Maturity levels Corrective costs Preventive and Appraisal costs Total costs on software quality Reactive Passive Balanced (Over)Active
    • 16. Quality measures
      • Corrective
        • Measures taken to correct the failure itself, correct damage and reduce the cost/damage of failures
        • Measurement: Categories the failures on cost:
          • Effort needed to correct the failure +
          • Cost needed to correct the loss of image +
          • Financial damage (downtime, lost of turnover)
      • Appraisal
        • Measures taken to detect the failures, before going live
        • Cost: Effort and other costs needed to find the bugs
        • Benefit: Corrective cost saved on the bugs
      • Preventive
        • Measure taken to prevent failures/bugs
        • Cost: Effort needed to prevent the bugs
        • Benefit: Corrective cost saved on the bugs
    • 17. Calculation ECL: Corrective is Appraisal plus Preventive
      • Corrective:
        • # of issues * average corrective costs
          • Direct costs to correct the issues
          • Costs on non productive employees
          • Financial costs (lost of image, extra letters to be sended, lost of turnover, wrong calculations, downtime of website, etc.)
      • Detective (Appraisal) and Preventive:
        • # of issues prevented/detected * costs per issue
      • To be calculated per TEST TYPE and issue CATEGORY !
    • 18. Example * Defect Detection Percentage Start N<8 AND MAN=TRUE M>6 OR L<65 End Statement coverage (SC) Condition coverage (CC) Decision coverage (DC) Modified Condition/Decision coverage (MCDC) Branche Condition combination coverage (BCCC) # Average DDP* 2 25 – 55 % 2/3 60 – 80 % 3 60 – 80 % 4 80 – 85 % 16 90 – 95 %
    • 19. Agenda
      • Introduction
      • Test improvement based on ECL
          • Identify current position
        • Define goal: Next position
        • Define and implement measures
    • 20. Optimizing the balance Reactive Passive Balanced (Over)Active
    • 21. Optimizing the balance
      • Move towards optimum via
      • Increasing/decreasing test coverage
      • Introduction/reduction of appraisal measures
      • Introduction/reduction of preventive measures:
        • Configuration management
        • Requirement management
        • Change/release management
        • Document review
        • Other measures
    • 22. Optimizing the efficiency Reactive Passive Balanced (Over)Active
    • 23. Optimizing the efficiency
      • Being more efficient in quality management by
      • Detecting bugs in an earlier stage
      • Testautomation
      • Reducing the time needed to detect the bugs by:
        • Selecting more efficient test specification techniques
        • Reduce the number of test deliverables (or content of test deliverables)
        • Increase the re-usability of testware
        • Decrease the time needed to maintain testware.
      • Optimising the cooperation between the testtypes and involved parties
      • Avoiding 8 wastes
    • 24. Real life Example
      • Current position:
      • Corrective about 7,5 FTE (inclusive correction)
      • Preventive/Appraisal: 2 FTE + 1 FTE = 3 FTE
      • Additional problems:
      • No overall process documentation available
      • Large number of ‘Technical’ failures during system test
      • Large number of minor releases
      Passive
    • 25. Optimisation suggestions
      • Some of the suggestions:
        • Formalising the review process (based on evaluations)
        • Introduction of UAT* regression testing (including process descriptions)
        • Formalising Unit testing
        • “ Informalising” System testing
        • Reducing number of releases
        • * UAT = User Acceptance Test
      Pragmatic and tailor made
    • 26. Expected results
      • Higher testcoverage
      • Reduction of number of changes and issues
      • Reduction of number of production failures
      • Reduction of test execution costs
      • Reduction of time needed for breaking in new employees
      Expected: Appraisal/Preventive: + 0,5 FTE = 3,5 FTE Corrective: - 2,0 FTE = 5,5 FTE Difference: - 1,5 FTE
    • 27. Reconsidering the Boehm curve
    • 28. Sharing the experience
      • Company/project culture has to be leading
      • Testing less can also be effective
      • Optimizing is measuring (using metrics)
      • Quality is not only testing
      • Being an added value is being proactive and pragmatic
      • Optimizing is a continuous process
      Prove the added value of testing
    • 29. Questions

    ×