The dynamic interaction of passed and failed   requirements during software testing                Copyrights (c) 2011-201...
Motivation• In my professional experience, I have  repeatedly witnessed systemic issues during  Software Verification that...
Requirements Statuses• We are interested in 2 simple sets of  requirements. All requirements within each  set share one of...
Requirement States Probabilities• Let’s call SFR the probability that a  requirement that has been implemented fails  test...
Requirement States Probabilities• The following probability constraints exist:   0 <= SFR <= 1   0 <= SFXR <=1• We can d...
Transition probability matrixLet M be the transition probability matrix,M=                              To ISR            ...
Two-Step Transition probabilities• The following question is particularly  interesting:  – given the transition probabilit...
Two-Step Transition probabilities• The problem is not as obvious as it seems;  here is is why:  – When a Requirement enter...
Two-Step Transition probabilities                                ISR      P(ISR,ISR,ISR) = (1-SFR).(1-SFR)                ...
Two-Step Transition probabilities• So the probability that a Requirement that  started in ISR is in ISR after two transiti...
Two-Step Transition probabilitiesM2 =                       To ISR                                  To FSR   From ISR (1-S...
Steady State Analysis• We are interested in knowing in the long run,  the probability P1 that any given requirement  will ...
Steady State Analysis- Example 1• if:   P1 = SFXR/(SFR+SFXR) [e1]• Then P1 = 0.5 = 50% when SFR = SFXR.• This implies tha...
Steady State Analysis – Example 2• Let’s look at another illustration:  – P1 is the probability of having in the long run,...
Steady State Analysis – Example 2• From [e1] we can say that:  [e1] -> 0.9 = SFXR/(0.2+SFXR)  [e1] -> 0.9.(0.2+SFXR) = S...
Steady State Analysis – Example 2• If SFXR was equal to its maximum value of  100% then the maximum value of P1 can only  ...
Steady State Analysis – Example 2• The only way to achieve P1 = 90% when SFXR  = 100% is to decrease SFR from 20% so that:...
Transient Analysis• It is possible to graph the values of P1 and P2  over several transitions, this is called transient  a...
Transient Analysis – Example 1• SFR = 10% and SFXR = 50%                Copyrights (c) 2011-2013 Pragmatic                ...
Transient Analysis – Example 2• SFR = 10% and SFXR = 10%                Copyrights (c) 2011-2013 Pragmatic                ...
Transient Analysis – Example 3• SFR = 10% and SFXR = 1%                Copyrights (c) 2011-2013 Pragmatic                 ...
Transient Analysis – Example 4• SFR = 90% and SFXR = 90%                Copyrights (c) 2011-2013 Pragmatic                ...
Transient Analysis – Example 5• SFR = 75% and SFXR = 100%               Copyrights (c) 2011-2013 Pragmatic                ...
For Comments and Questions contact didier@pragmaticohesion.com                      Copyrights (c) 2011-2013 Pragmatic    ...
Upcoming SlideShare
Loading in...5
×

The dynamic interaction of passed and failed requirements during software testing

402

Published on

This presentation describes a simple formula to estimate the average defect level (% of none met requirements) of a software product throughout its production cycles based on the probability of finding an unsuccessfully implemented requirements and the probability that this type of defect gets fixed.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
402
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

The dynamic interaction of passed and failed requirements during software testing

  1. 1. The dynamic interaction of passed and failed requirements during software testing Copyrights (c) 2011-2013 Pragmatic 1 Cohesion Consulting
  2. 2. Motivation• In my professional experience, I have repeatedly witnessed systemic issues during Software Verification that prompted me to develop the model presented in this paper.• It is my hope that the non-intuitive results that the model yields help software development team better understand certain dynamic behavior embedded in the software development process. Copyrights (c) 2011-2013 Pragmatic 2 Cohesion Consulting
  3. 3. Requirements Statuses• We are interested in 2 simple sets of requirements. All requirements within each set share one of the following statuses: – The first one represents requirements that have been implemented i.e. requirements for which the development team has created actual software code. Let’s call this set of requirements ISR. – The second one represents requirements that have failed testing after they were implemented by the development team. Let’s call this second set of requirements FSR. Copyrights (c) 2011-2013 Pragmatic 3 Cohesion Consulting
  4. 4. Requirement States Probabilities• Let’s call SFR the probability that a requirement that has been implemented fails testing. When this failure happens the requirement becomes part of the FSR set otherwise it remains in the ISR set.• Let’s call SFXR the probability that a failed requirement gets fixed. When the fix takes place, the requirement is no longer part of the FSR set but it moves into the ISR set. Copyrights (c) 2011-2013 Pragmatic 4 Cohesion Consulting
  5. 5. Requirement States Probabilities• The following probability constraints exist: 0 <= SFR <= 1 0 <= SFXR <=1• We can derive a probability transition matrix that captures the possible transitions of a requirement from one set to the other: From ISR to ISR, probability = 1-SFR From ISR to FSR, probability = SFR From FSR to ISR, probability = SFXR From FSR to FSR, probability = 1-SFXR Copyrights (c) 2011-2013 Pragmatic 5 Cohesion Consulting
  6. 6. Transition probability matrixLet M be the transition probability matrix,M= To ISR To FSR From ISR 1-SFR SFR From FSR SFXR 1-SFXR Copyrights (c) 2011-2013 Pragmatic 6 Cohesion Consulting
  7. 7. Two-Step Transition probabilities• The following question is particularly interesting: – given the transition probability matrix M and the fact that a requirement has been added to the ISR set, what is the probability that this same requirement will be in the ISR set two transitions in the future? Copyrights (c) 2011-2013 Pragmatic 7 Cohesion Consulting
  8. 8. Two-Step Transition probabilities• The problem is not as obvious as it seems; here is is why: – When a Requirement enters set ISR it has a SFR probability of being moved to set FSR and a 1-SFR probability of staying in ISR. Once in FSR, it has a SFXR probability of moving into ISR and a 1-SFXR probability of staying in FSR. – We can represent the possible paths with the following tree Copyrights (c) 2011-2013 Pragmatic 8 Cohesion Consulting
  9. 9. Two-Step Transition probabilities ISR P(ISR,ISR,ISR) = (1-SFR).(1-SFR) 1-SFR ISR SFR 1-SFR FSR P(ISR,ISR,FSR) = (1-SFR).SFR ISR ISR P(ISR,FSR,ISR) = SFR.SFXR SFR SFXR FSR 1-SFXR FSR P(ISR,FSR,FSR) = SFR.(1-SFXR) Copyrights (c) 2011-2013 Pragmatic 9 Cohesion Consulting
  10. 10. Two-Step Transition probabilities• So the probability that a Requirement that started in ISR is in ISR after two transitions is: P(ISR,ISR,ISR) + P(ISR,FSR,ISR) = (1-SFR).(1-SFR) + SFR.SFRX• All other similar two-step transition probabilities can be calculated by M2 Copyrights (c) 2011-2013 Pragmatic 10 Cohesion Consulting
  11. 11. Two-Step Transition probabilitiesM2 = To ISR To FSR From ISR (1-SFR).(1-SRF)+SFR.SFXR (1-SFR).SFR+SRF.(1-SFXR) From FSR SFXR.(1-SFR)+(1-SFXR).SFXR SFXR.SFR+(1-SFXR).(1-SFXR) Copyrights (c) 2011-2013 Pragmatic 11 Cohesion Consulting
  12. 12. Steady State Analysis• We are interested in knowing in the long run, the probability P1 that any given requirement will be in the ISR set. The probability that such requirement will be in the FSR set is P2=1-P1.• Markov chains can help us solve this problem by calculating the Steady State probabilities of Matrix M: P1 = SFXR/(SFR+SFXR) [e1] P2 =1- SFXR/(SFR+SFXR) [e2] Copyrights (c) 2011-2013 Pragmatic 12 Cohesion Consulting
  13. 13. Steady State Analysis- Example 1• if: P1 = SFXR/(SFR+SFXR) [e1]• Then P1 = 0.5 = 50% when SFR = SFXR.• This implies that even if there is a low probability that a requirement that has been implemented fails testing (SFR); if an equally low fixing rate takes place, given sufficient time, a requirement gets closer and closer to having a 50% chance of being in the set that failed testing! Copyrights (c) 2011-2013 Pragmatic 13 Cohesion Consulting
  14. 14. Steady State Analysis – Example 2• Let’s look at another illustration: – P1 is the probability of having in the long run, a non-failed requirement. Let’s say that we want P1 to be 90%. If we know that an implemented requirement has an SFR=20% probability of failing testing then what should SFXR (the rate at which failed requirements are fixed) be to assure P1=90%? Copyrights (c) 2011-2013 Pragmatic 14 Cohesion Consulting
  15. 15. Steady State Analysis – Example 2• From [e1] we can say that: [e1] -> 0.9 = SFXR/(0.2+SFXR) [e1] -> 0.9.(0.2+SFXR) = SFXR [e1] -> 0.18 + 0.9.SFXR = SFXR [e1] -> 0.18/(1-0.9) = SFXR [e1] -> SFXR = 1.8 > 1 which is impossible! Copyrights (c) 2011-2013 Pragmatic 15 Cohesion Consulting
  16. 16. Steady State Analysis – Example 2• If SFXR was equal to its maximum value of 100% then the maximum value of P1 can only be 1/(1+0.2) = 83.3%=P1• This result shows that there is a clear upper bound to the probability that in the long run a requirement will not fail testing given the value of SFR. SFR is the probability that a given requirement that has just been implemented fails testing. Copyrights (c) 2011-2013 Pragmatic 16 Cohesion Consulting
  17. 17. Steady State Analysis – Example 2• The only way to achieve P1 = 90% when SFXR = 100% is to decrease SFR from 20% so that: 0.9 = 1/(SFR+1) SFR + 1 = 1/0.9 SFR = 1/0.9 – 1 SFR = 11.11%• This lower value of SFR implies a better quality of software code that properly implements more requirements. Copyrights (c) 2011-2013 Pragmatic 17 Cohesion Consulting
  18. 18. Transient Analysis• It is possible to graph the values of P1 and P2 over several transitions, this is called transient analysis.• Transient analysis can show how fast the steady states values for P1 and P2 are reached.• Transient analysis can also show instability in P1 and P2 values over several transitions due to higher values of SFR and SFRX (see examples 4 and 5 in subsequent slides). Copyrights (c) 2011-2013 Pragmatic 18 Cohesion Consulting
  19. 19. Transient Analysis – Example 1• SFR = 10% and SFXR = 50% Copyrights (c) 2011-2013 Pragmatic 19 Cohesion Consulting
  20. 20. Transient Analysis – Example 2• SFR = 10% and SFXR = 10% Copyrights (c) 2011-2013 Pragmatic 20 Cohesion Consulting
  21. 21. Transient Analysis – Example 3• SFR = 10% and SFXR = 1% Copyrights (c) 2011-2013 Pragmatic 21 Cohesion Consulting
  22. 22. Transient Analysis – Example 4• SFR = 90% and SFXR = 90% Copyrights (c) 2011-2013 Pragmatic 22 Cohesion Consulting
  23. 23. Transient Analysis – Example 5• SFR = 75% and SFXR = 100% Copyrights (c) 2011-2013 Pragmatic 23 Cohesion Consulting
  24. 24. For Comments and Questions contact didier@pragmaticohesion.com Copyrights (c) 2011-2013 Pragmatic 24 Cohesion Consulting
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×