View stunning SlideShares in full-screen with the new iOS app!Introducing SlideShare for AndroidExplore all your favorite topics in the SlideShare appGet the SlideShare app to Save for Later — even offline
View stunning SlideShares in full-screen with the new Android app!View stunning SlideShares in full-screen with the new iOS app!
Building the Model A. Define Phase The Define Phase primarily involves establishing the projectIn Design for Six Sigma (DfSS), the research is organized accord- definition and acknowledging the fundamental needs of the re-ing to DMADV phases: Define (D), Measure (M), Analyze (A), De- search. A typical software production process which applies the V-sign (D) and Verify (V). DfSS seeks to sequence proper tools and Model software development life cycle is used as the basis for thistechniques to design in more value during new product devel- research. Throughout the process, the testing team is involved inopment, while creating and using higher quality data for key de- all review sessions for each phase, starting from planning untilvelopment decisions. Achievement of DfSS is observed when the the end of the system integration testing phase. The test lab isproducts or services developed meet customer needs rather than involved in reviewing the planning document, the requirementcompeting alternatives. In general, DfSS-DMADV phases are de- analysis document, the design document, the test planning doc-scribed as below: ument and the test cases (see Figure 1). The software production process is governed by project management, quality manage-• Define - identify the project goals and customer (internal and external) requirements ment, configuration and change management, integral and sup- port as well as process improvement initiatives, in compliance to• Measure - determine customer needs and specifications; CMMI®. From the figure, the area of study is the functional or sys- benchmark competitors and industry tem test phase, where defects are going to be predicted. There-• Analyze – study and evaluate the process options to meet fore, only potential factors/predictors in the phases prior to the customer needs system testing phase are considered and investigated.• Design – detailing the process to meet customer needs• Verify – confirm and prove the design performance and abil- ity to meet customer needsFigure 1: Typical Software Production ProcessTwo (2) schematic diagrams are produced, which are a high level are defect containment in the test phase, customer satisfaction,schematic diagram (Figure 2) and a detailed schematic diagram quality of the process being imposed to produce the software and(Figure 3). The high level schematic diagram deals with estab- project management. There are two aspects involved related tolishing the Big Y or business target, little Ys, vital Xs and the goal these little Ys: potential number of defects before the test phasestatement against the business scorecard. In this research, Big which is the research interest, and the number of defects afterY is to produce software with zero-known post-release defects, completing the test phase.while for little Ys, elements that contribute to achieving the Big Ywww.testingexperience.com The Magazine for Professional Testers 53
Big Y Zero-Known Post Release Defects Little ys Defect Contain- Customer Project Quality of process ment in Test Phase satisfaction Management Vital Xs Potential # of Level of People Timeline defect before test satisfaction capability Allocation # of defect after Process Rescue test Effectiveness AllocationFigure 2: Schematic DiagramFor the detailed schematic diagram (or detailed Y-X tree), possible test, and these are summarized in a Y to X tree diagram as shownfactors that contribute to the test defect prediction are defined in the figure below. The highlighted factors are selected for fur-from the Vital X, which is the potential number of defects before ther analysis. Test Defect Prediction Software Historical Knowledge Test Process Errors Fault ProjectComplexity Defect Require- Developer Test Case De- Require- Require- Defect Project Project ment Pages Knowledge sign Coverage ment Error ment Fault Severity Domain Thread Design Tester Targeted Total Design Defect Type/ Design Error Component Pages Knowledge Test Cases Fault Category Programming Test Defect Language CUT Error CUT Fault Application Automation Validity Test Case Execu- Test Plan Integration Total PR (Defects) Code Size tion Productivity Error Fault Raised Total Effort in Test Test Cases Test Case Design Phase Error Fault Total Effort in Phases Prior to System Test Factors to considerFigure 3: Detailed Y-X Tree Diagram54 The Magazine for Professional Testers www.testingexperience.com
B. Measure Phase known results of PASS and FAIL are identified. Then three testA Measurement System Analysis (MSA) is conducted to validate engineers are selected to execute the test cases at random. Thisthat the processes of discovering defects are repeatable and re- is repeated three times for every engineer. The outcome is pre-producible, thus eliminating human errors. Ten test cases with sented in Figure 4 below: Tester 1 Tester 2 Tester 3 TCD ID TC Result 1 2 3 1 2 3 1 2 3 TC1 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS TC2 FAIL FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS TC3 FAIL FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS TC4 PASS FAIL FAIL FAIL FAIL FAIL FAIL PASS PASS PASS TC5 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS TC6 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS TC7 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS TC8 FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL FAIL TC9 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS TC10 PASS PASS PASS PASS PASS PASS PASS PASS PASS PASS Figure 4: Data for AnalysisThe data is then used to run the MSA, where the result is com- As the MSA is passed, the operational definition is prepared con-pared against the Kappa value. Since the overall result of all ap- sisting of type of data to be gathered, measurement to be used,praisers against standard is above 70% (as required by Kappa’s responsibilities, and mechanism to obtain those data so that thevalue), the MSA is considered as a PASS as presented in Figure 5 data can be used for next DfSS phase.below: Overall Result for MSA is PASS since Kappa value is greater than 0.7 or 70%Figure 5: Overall Result of MSAC. Analyze Phase analysis via manual stepwise regression analysis, i.e. the predic- tors are added and removed during the regression until a strongThe data gathered from the Measure Phase is used to perform statistical result is obtained. The figure involves the P-value ofa first round of regression analysis using the data shown in Fig- each predictor against the defects, the R-squared (R-Sq.) value andure 6, which was collected from thirteen projects. The predictors the R-squared adjusted (R-Sq. (adj.)) value. These figures demon-used include requirement error, design error, Code and Unit Test strated the goodness of the equation and how well it can be used(CUT) error, Kilo Lines of Code (KLOC) size, targeted total test cases for predicting the defects. The result of the regression analysis isto be executed, test plan error, test cases error, automation per- presented in Figure 7.centage, test effort in days, and test execution productivity perstaff day. The regression is done against the functional defects asthe target. MINITAB software is used to perform the regression56 The Magazine for Professional Testers www.testingexperience.com
Project Req. Er- Design CUT KLOC Total Test Plan Test Case Automa- Test Test Ex- Func- Name ror Error Error Test Error Error tion % Effort ecution tional Cases Produc- Defects tivity Project A 5 22 12 28.8 224 0 34 0 6.38 45.8000 19 Project B 0 0 1 6.8 17 0 6 0 9.36 17.0000 1 Project C 9 10 14 5.4 24 4 6 0 29.16 5.8333 4 Project D 7 12 2 1.1 25 4 9 0 13.17 7.0000 0 Project E 11 29 3 1.2 28 4 12 0 14.26 3.4000 3 Project F 0 2 7 6.8 66 1 7 0 32.64 31.0000 16 Project G 3 25 11 4 149 5 0 0 7.15 74.5000 3 Project H 4 9 2 0.2 24 4 0 0 18.78 7.6667 0 Project I 7 0 1 1.8 16 1 3 0 9.29 2.6818 1 Project J 1 7 2 2.1 20 1 4 0 6.73 1.9450 0 Project K 17 0 3 1.4 13 1 4 0 8.44 6.5000 1 Project L 3 0 0 1.3 20 1 7 0 14.18 9.7500 1 Project M 2 3 16 2.5 7 1 6 0 8.44 1.7500 0Figure 6: Data for Regression Analysis Figure 7: Regression Analysis Result Project Req. Design CUT KLOC Req. Design Total Test Total Test Func- All Name Error Error Error Page Page Test Cases Effort Design tional Defects Cases Error Effort Defects Project A 5 22 12 28.8 81 121 224 34 16.79 15.20 19 19 Project B 0 0 1 6.8 171 14 17 6 45.69 40.91 1 1 Project C 9 10 14 5.4 23 42 24 6 13.44 13.44 4 4 Project D 7 12 2 1.1 23 42 25 9 4.90 9.90 0 0 Project E 11 29 3 1.2 23 54 28 12 4.72 4.59 3 3 Project F 0 2 7 6.8 20 70 88 7 32.69 16.00 16 27 Project G 3 25 11 4 38 131 149 0 64.00 53.30 3 3 Project H 4 9 2 0.2 26 26 24 0 5.63 5.63 0 0 Project I 17 0 3 1.4 15 28 13 4 9.13 7.88 1 1 Project J 61 34 24 36 57 156 306 16 89.42 76.16 25 28 Project K 32 16 19 12.3 162 384 142 0 7.00 7.00 12 12 Project L 0 2 3 3.8 35 33 40 3 8.86 8.86 6 6 Project M 15 18 10 26.1 88 211 151 22 30.99 28.61 39 57 Project N 0 4 0 24.2 102 11 157 0 41.13 28.13 20 33Figure 8: New Set of Data for Regressionwww.testingexperience.com The Magazine for Professional Testers 57
D. Design PhaseBased on the result, it has been demonstrated that for this roundof regression, possible predictors are design error, targeted total Further analysis is conducted during the Design Phase due to thetest cases to be executed, test plan error, and test effort in days, need to generate a prediction model that both is practical andin which the P-value for each predictor is less than 0.05. As overall makes sense from the viewpoint of software practitioners usingmodel equation, this model portrays strong characteristics of a logical predictors, to filter the metrics to contain only valid data,good model via high percentage of R-Sq. and R-Sq. (adjusted) val- and to reduce the model to have only coefficients that have a logi-ues: 96.7% and 95.0% respectively. From the regression, this equa- cal correlation to the defect. As a result, a new set of data is usedtion is selected for study in the next phase: to refine the model as shown in Figure 8. Using the new set of data, new regression results are presented Defect = - 3.04 + 0.220 Design Error + 0.0624 Targeted Total in Figure 9. Test Cases - 2.30 Test Plan Error + 0.477 Test Efforts Figure 9: New Regression ResultFrom these latest results, it can be demonstrated that significant Based on the the verification result, it is clearly shown that thepredictors for predicting defects are requirement error, CUT er- model is fit for use and can be implemented to predict test de-ror, KLOC, requirement page, design page, targeted total test cas- fects for the software product. This is justified by the predictedes to be executed, and test effort in days from the phases prior to defects number, which falls within 95% Prediction Interval (PI). Assystem test. The P-value for each factor is less than 0.05, while the the test defect prediction model equation is finalized, the nextR-Sq. and R-Sq. (adjusted) values are 98.9% and 97.6% respective- consideration is to emphasize on the control plan with regard toly, which results in a stronger prediction equation. The selected its implementation in the process, i.e. when the actual numberequation is as below: of defects found is lower or greater than the prediction. Figure 11 below summarizes the control plan: Functional Defects (Y) = 4.00 - 0.204 Requirement Error - 0.631 CUT error + 1.90 KLOC - 0.140 Requirement Page + 0.125 Design Actual Defects < Predicted Actual Defects > Predicted Page - 0.169 Total Test Cases + 0.221 Test Effort Defects Defects Perform thorough testing Re-visit the errors captured during ad-hoc test during requirement, designE. Verify Phase and CUT phaseThe selected model in the Design Phase is now verified against Perform additional test Re-visit the errors capturednew projects that have yet to go to the System Test phase. The strategy that relates to the during test case designactual defects found after the System Test has been completed discovery of more functional defectsare compared against the predicted defects to ensure that actualdefects fall between 95% prediction intervals (PI) of the model as Re-visit the model and thepresented in the last column in Figure 10. factors used to define the model No Predicted Actual 95% CI 95% PI Figure 11: Action Plan for Test Defect Prediction Model Functional Functional (min, max) (min, max) Defects Defects 1. 182 187 (155, 209) (154, 209) 2. 6 1 (0, 2) (0, 14) 3. 1 1 (0, 3) (0, 6)Figure 10: Verification Result for the Prediction Model58 The Magazine for Professional Testers www.testingexperience.com