Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Agenda• Background and Motivation• qEstimation Analysis   – Test Size Estimation (Test Case Point Analysis)   – Test Effor...
Background• Software estimation   – Process of determining the cost, time, staff, and other related     attributes of soft...
Motivation• Testing accounts for up to 50% of project effort [1]• Current problems   – estimates are done for the whole pr...
Agenda• Background and Motivation• qEstimation Analysis   – Test Size Estimation (Test Case Point Analysis)   – Test Effor...
qEstimation Principles• Size reflects the mass and complexity of each test cycle  of a testing project• Test case’s comple...
qEstimation ProcessEstimate size and effort of different test cycles of a same project: [Test Cycle i]                  Co...
Count Size of Test Cycle• Size of a test cycle is the total of TCPs of all test cases to be  executed in that test cycle• ...
Count Size of Test Cycle (cont’d)• Checkpoints   – Checkpoint is the condition in which the tester verifies     whether th...
Count Size of Test Cycle (cont’d)• Test Setup or Precondition   – Test setup specifies the condition to execute the test c...
Count Size of Test Cycle (cont’d)• Test Data  – Test Data is used to execute the test case        • It can be generated at...
Count Size of Test Cycle (cont’d)• Adjust TCPs based on Type of Test   – This is an OPTIONAL step   – Adjustment is based ...
Count Size of Test Cycle (cont’d)• Type of Test                                    (*) based on our survey                ...
Estimate Effort of Test Cycle• Overview   – Two estimation methods      • Based on Test Velocity/Productivity      • Regre...
Estimate Effort of Test Cycle (cont’d)• Estimate Effort based on Test Velocity    Effort(person-hour) = Size(TCP) / Test V...
Estimate Effort of Test Cycle (cont’d)• Estimate effort using Linear Regression Analysis   – Find out the equation of effo...
Calibrate the qEstimation Model• Calibration: a process adjusting parameters for a  model using historical data or experie...
Agenda• Background and Motivation• qEstimation Analysis   – Test Size Estimation (Test Case Point Analysis)   – Test Effor...
Note #1: Complexity Categorization• Complexity Categorization: of Test Data and  Precondition   – Select “Medium” (Remembe...
Note #2: Fast Test Design• Lack time for detailing test cases while estimation   Use fast test design techniques   – Requ...
Note #3: Development v.s Maintenance• Development v.s Maintenance release  Trace TCP  count for new feature release and m...
Note #4: Historical Data• Historical data is important   – Effort spent on new release v.s maintenance release   – Formali...
Note #5: Tracking Actual EffortConvention:[Client] – [Project] – [Testing] – [Project Name – Development/Maintenance – Pha...
Note #6: Monitoring Testing MetricsWe have the size in hand, we can easily measure below metricsand use it as quality moni...
Agenda• Background and Motivation• qEstimation Analysis   – Test Size Estimation (Test Case Point Analysis)   – Test Effor...
Conclusion• qEstimation tool is an agile approach to estimating size  and effort of test cycle   – Estimate Size in TCP by...
qEstimation Process[Test Cycle i]                 Count TCPs                   Estimate    Test Case                      ...
Thank You
References•   [1] Y. Yang, Q. Li, M. Li, Q. Wang, An empirical analysis on distribution patterns of software    maintenanc...
Test Estimation using Test Case Point Analysis method
Upcoming SlideShare
Loading in …5
×
Upcoming SlideShare
Test Estimation Techniques
Next
Download to read offline and view in fullscreen.

12

Share

Download to read offline

Test Estimation using Test Case Point Analysis method

Download to read offline

The 3rd KMS Technology' public seminar

Related Books

Free with a 30 day trial from Scribd

See all

Test Estimation using Test Case Point Analysis method

  1. 1. Agenda• Background and Motivation• qEstimation Analysis – Test Size Estimation (Test Case Point Analysis) – Test Effort Estimation• qEstimation in Practice• Conclusion 2
  2. 2. Background• Software estimation – Process of determining the cost, time, staff, and other related attributes of software projects, often before work is performed• Estimation is important for the success or failure of software projects – Making investment decisions – Project planning – Stakeholder/Client negotiation• Estimation Methods – Source Lines of Code (SLOC) – Function Points – Use Case Points/Story Points – Expert Judgment (Experience Based) 3
  3. 3. Motivation• Testing accounts for up to 50% of project effort [1]• Current problems – estimates are done for the whole project rather than testing specific – lack of reliable methods designed for estimating size and effort of software testing• Currently, there are needs of: – accurately estimating effort of testing activities • How many days is needed for test design, test execution, etc.? – measuring size and productivity of testing activities • Your QA team is working less productive! – measuring effectiveness and efficiency of software testing • Why testing last release left so many bugs to Production?• Our aim – To introduce a method for estimating the size of testing activities – To discuss methods to estimate testing effort using this size measure – To introduce a simple toolkit for this estimation process 4
  4. 4. Agenda• Background and Motivation• qEstimation Analysis – Test Size Estimation (Test Case Point Analysis) – Test Effort Estimation• qEstimation in Practice• Conclusion 5
  5. 5. qEstimation Principles• Size reflects the mass and complexity of each test cycle of a testing project• Test case’s complexity is based on – Number of checkpoints – Complexity of test setup or precondition – Complexity of test data• Test Case Point (TCP) is used as size unit – representing the size of the most simple test case• Calibration or model refinement is key to estimating effort – calibration based on feedback from different cycles within project or of similar projects• Focusing on independent testing (V & V) 6
  6. 6. qEstimation ProcessEstimate size and effort of different test cycles of a same project: [Test Cycle i] Count TCPs Estimate Test Case Counted Estimated of all Test Testing Test Case Size Effort Effort Cases Update Parameters Historical Data Calibrate Estimation Historical Data of this Project Model Test Cycle Size Actual Effort by Effort Activity …. …. …. …. Test cycle i …. …. …. …. …. …. …. 7
  7. 7. Count Size of Test Cycle• Size of a test cycle is the total of TCPs of all test cases to be executed in that test cycle• Steps: Count Checkpoints Adjust based on Test Case Determine Set Up Unadjusted Test Type TCPs Complexity TCPs (optional) Determine Test Data Complexity 8
  8. 8. Count Size of Test Cycle (cont’d)• Checkpoints – Checkpoint is the condition in which the tester verifies whether the result produced by the target function matches the expected criterion – One test case consists of one or many checkpoints One checkpoint is counted as one TCP 9
  9. 9. Count Size of Test Cycle (cont’d)• Test Setup or Precondition – Test setup specifies the condition to execute the test case • Include setup steps to prepare environment for testing • Mainly affect the cost to execute the test case • May be related to data prepared for the test case – Four levels of Test Setup complexity • Each is assigned a number of TCPs Number Complexity Description of TCP(*) Level 0 None • The set up is not applicable or important to execute the test case • Or, the set up is just reused from the previous test case to continue the current test case 1 Low • The condition for executing the test case is available with some simple modifications required • Or, some simple set-up steps are needed 3 Medium • Some explicit preparation is needed to execute the test case • Or, The condition for executing is available with some additional modifications required • Or, some additional set-up steps are needed 5 High • Heavy hardware and/or software configurations are needed to execute the test case (*) based on our survey of 18 senior QA engineers. You can adjust according to your project’s experience. 10
  10. 10. Count Size of Test Cycle (cont’d)• Test Data – Test Data is used to execute the test case • It can be generated at the test case execution time, sourced from previous tests, or generated by test scripts • Test data is test case specific, or general to a group of test cases – Four levels of Test Data complexity • Each is assigned a number of TCPs Number of TCP Complexity Description (*) Level 0 None • No test data preparation is needed 1 Low • Simple test data is needed and can be created during the test case execution time • Or, the test case uses a slightly modified version of existing test data and requires little or no effort to modify the test data 3 Medium • Test data is deliberately prepared in advance with extra effort to ensure its completeness, comprehensiveness, and consistency 6 High • Test data is prepared in advance with considerable effort to ensure its completeness, comprehensiveness, and consistency • This could include using support tools to generate data and a database to store and manage test data • Scripts may be required to generate test data (*) based on our survey of 18 senior QA engineers. You can adjust according to your project’s experience.
  11. 11. Count Size of Test Cycle (cont’d)• Adjust TCPs based on Type of Test – This is an OPTIONAL step – Adjustment is based on types of test cases• Test Case Point counted till this point is considered Unadjusted Test Case Point (UTCP) n UTCP = ∑TCPi i=1• UTCP is adjusted by considering types of test case – Each type of test case is assigned a weight – Adjusted Test Case Point (ATCP): n ATCP = ∑UTCPi * Wi i=1 • UTCPi - the number of UTCP counted for the test case ith. • Wi - the weight of the test case ith, taking into account its test type 12
  12. 12. Count Size of Test Cycle (cont’d)• Type of Test (*) based on our survey of 18 senior QA engineers. You can adjust according to your project’s experience. 13
  13. 13. Estimate Effort of Test Cycle• Overview – Two estimation methods • Based on Test Velocity/Productivity • Regression Analysis of Size and Effort of completed test cycles – Effort distributed by activity • Test Planning • Test Analysis and Design Each of these activities may be performed multiple times • Test Execution • Test Tracking and Reporting 14
  14. 14. Estimate Effort of Test Cycle (cont’d)• Estimate Effort based on Test Velocity Effort(person-hour) = Size(TCP) / Test Velocity(TCP per person-hour) – Test Velocity is measured as TCP/person-hour • Test Velocity is dependent on project • It should be calculated based on data from completed test cycles of the similar projects. 15
  15. 15. Estimate Effort of Test Cycle (cont’d)• Estimate effort using Linear Regression Analysis – Find out the equation of effort and size using similar completed test cycles of a project 100 90 80 Equation of 70 Size and Effort y = 0.072x + 1.640 Effort (PM) 60 50 40 The data analysis tool like 30 Excel can be used to find 20 out the equation 10 0 0 100 200 300 400 500 600 700 800 900 1000 Adjusted TCP 16
  16. 16. Calibrate the qEstimation Model• Calibration: a process adjusting parameters for a model using historical data or experiences• With qEstimation tool, you can calibrate: (1) TCP assigned to each complexity level of Test Setup (2) TCP assigned to each complexity level of Test Data (3) Test Velocity/Productivity (4) Effort distribution (5) Weights of test case types• Process can be done with the help of tools Tool Demo 17
  17. 17. Agenda• Background and Motivation• qEstimation Analysis – Test Size Estimation (Test Case Point Analysis) – Test Effort Estimation• qEstimation in Practice• Conclusion 18
  18. 18. Note #1: Complexity Categorization• Complexity Categorization: of Test Data and Precondition – Select “Medium” (Remember bell curve in probability theory) 19
  19. 19. Note #2: Fast Test Design• Lack time for detailing test cases while estimation  Use fast test design techniques – Requirement  Test scenario  High level test case – Select Medium for unknown Test Data and Precondition. Again, refer to bell curve chart 20
  20. 20. Note #3: Development v.s Maintenance• Development v.s Maintenance release  Trace TCP count for new feature release and maintenance release project separately – New feature: requirement analysis, test case development, lot of bugs… – Maintenance: Effort mostly is bug validation and regression test execution, little effort is for test development and bug reporting 21
  21. 21. Note #4: Historical Data• Historical data is important – Effort spent on new release v.s maintenance release – Formalize timesheet reporting mechanisms 22
  22. 22. Note #5: Tracking Actual EffortConvention:[Client] – [Project] – [Testing] – [Project Name – Development/Maintenance – Phaseof Testing – Any text describe the activity]• Development: Project develops new features, major changes in application …• Maintenance: Project that mainly fixes bugs, develops small features …. 1. Export effort from OrangeHRM using TimesheetReport tool 2. Run VBScript to cut effort into Testing Phases (TP, TAD, TE, DTR) 23
  23. 23. Note #6: Monitoring Testing MetricsWe have the size in hand, we can easily measure below metricsand use it as quality monitoring indicators:• Productivity: – Definition: How productive the team is – Formula: TCPs/Actual Effort (in Hrs)• Defect Rate/Defect Density: – Definition: Number of bugs per an unit of work (here is TCP) – Formula: No of bugs/Total TCPs• Defect Escape Rate: – Definition: The bugs escape to next round of testing or to Production environment – (Traditional view) Formula: No of Escaped Bugs / Total Bugs – (Different view) Formula: No of Escaped Bugs / Total TCPs 24
  24. 24. Agenda• Background and Motivation• qEstimation Analysis – Test Size Estimation (Test Case Point Analysis) – Test Effort Estimation• qEstimation in Practice• Conclusion 25
  25. 25. Conclusion• qEstimation tool is an agile approach to estimating size and effort of test cycle – Estimate Size in TCP by measuring the complexity of test cases – Estimate Effort using Test Velocity or Regression – An Excel toolkit to simplify the approach• Advantages and experiences learned – Easy to implement – Reflecting real complexity of test cases – Independent with the level of details of test cases – Useful for estimating testing effort• Limitations and future improvements – A new approach – introduced and applied almost a year now – Need more empirical validations 26
  26. 26. qEstimation Process[Test Cycle i] Count TCPs Estimate Test Case Counted Estimated of all Test Testing Test Case Size Effort Effort Cases Update Parameters Historical Data Calibrate Estimation Historical Data of this Project Model Test Cycle Size Actual Effort by Effort Activity …. …. …. …. Test cycle i …. …. …. …. …. …. …. 27
  27. 27. Thank You
  28. 28. References• [1] Y. Yang, Q. Li, M. Li, Q. Wang, An empirical analysis on distribution patterns of software maintenance effort, International Conference on Software Maintenance, 2008, pp. 456-459• [2] N. Patel, M. Govindrajan, S. Maharana, S. Ramdas, “Test Case Point Analysis”, Cognizant Technology Solutions, White Paper, 2001• [3] QASymphony: www.qasymphony.com
  • mariovarella

    Feb. 4, 2020
  • DaveEdeh

    Sep. 15, 2018
  • yadvendra7669

    Jan. 9, 2018
  • DanielPedroche1

    Mar. 21, 2017
  • VishalVashishta

    Feb. 20, 2017
  • FranciscoMartin20

    Aug. 3, 2016
  • RajNishanth

    May. 4, 2016
  • CharlesNelaturi

    Apr. 27, 2016
  • JayantGoel

    Jul. 31, 2015
  • prkelker

    Jul. 14, 2014
  • KuzmenkoIvan

    Feb. 20, 2014
  • gsquillace

    Nov. 28, 2013

The 3rd KMS Technology' public seminar

Views

Total views

37,835

On Slideshare

0

From embeds

0

Number of embeds

24

Actions

Downloads

850

Shares

0

Comments

0

Likes

12

×