Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Test estimations


Published on

Software Test Estimation

Published in: Software
  • Can you please send to me "Test Estimations" PPT which are not able to download...Thanks
    Are you sure you want to  Yes  No
    Your message goes here

Test estimations

  1. 1. TEST ESTIMATIONS Richa Goel
  2. 2. Session 1 Test Estimation
  3. 3. What is testing? 3 Testing is a process aimed at:  Finding defects in a controlled manner  Detecting the level of quality of the test object  Demonstrating the gap between specifications and the actual product  Demonstrating that the end product functions as called for in the requirements
  4. 4. Structured Software Testing  Testing everything is impractical  Funding limitations  Limited time and resources  Diminishing value after a certain point  Isn’t there a more effective way?  Yes! Structured software testing: a risk- based, quality-centric approach to testing
  5. 5. What is an Estimate?  A tentative evaluation or rough calculation.  A preliminary calculation of the efforts of a project  A judgment based upon one's impressions; opinion Software estimation is the act of predicting the “duration” and “cost” of a project  “Estimation is the process of finding an estimate, or approximation, which is a value that is usable for some purpose even if input data may be incomplete, uncertain, or unstable.”
  6. 6. An estimate should be…  Realistic  Actionable  Flexible
  7. 7. Importance of Estimate  Software overrun case studies by various practitioners throws insight into the project, their estimated cost & schedules and the final status of project being cancelled  Effective software estimation is one of the most difficult software development activities and one of the most important
  8. 8. Why Estimate Software?  30% of projects never complete  Average project exceeds cost by 90%; Schedule by 120%  15% of large projects never deliver anything  Only 30-50% of projects are successful
  9. 9. Bad estimation can lead to poor distribution of work.
  10. 10. Why are we bad at estimating?  Complexity of the system  Infrequency – How often do we do the “same thing”  Vs. manufacturing or construction  Under estimation bias  Software is “easy”  We deal with Goals not estimates  Must be done by June
  11. 11. Why estimations fail?  Practitioners have cited various reasons: › Too many unplanned tasks or change requests › Too many defects results in release delays › Producing project documents add to the costs › Accurate estimate may be replaced by optimistic estimates to win bids › Experience of estimator and project executors matters › Non-availability of right skills and resources
  12. 12. Test Estimation  Test Estimation is the estimation of the testing size, testing effort, testing cost and testing schedule for a specified software testing project in a specified environment using defined methods, tools and techniques.  Test estimation should be realistic and accurate.
  13. 13. Why estimate testing?  To endeavour to quantify:  Timetable  Resourcing  Budget  Environments
  14. 14. Why is estimating so hard?  So many variables…  Product - complexity, availability, stability  Test environment - availability, stability  Issues - volume, severities, time to fix, regression  Personnel - suitability, experience  Test stages - analysis, design, development, execution  Evolving project – re/descoping, priority shifts
  15. 15. Sticking to the estimation is very important to build good reputation with the client.
  16. 16. Test Project Estimation The three W’s  Why?  Why the focus now on Test Projects?  How?  How is this different from normal projects?  What?  What are the important parameters to consider?
  17. 17. Factors helping in estimating  an estimate of the number of faults that are likely to be found during testing;  the percentage of nested faults (faults that can only be found after another one has been fixed); and  the percentage of faults fixed incorrectly (together with the time to wait for each release  not all iterations will use all of the tests
  18. 18. Think…  You have to attend a job interview.  The interview is at 10 a.m.  How will you estimate the time you should wake up?
  19. 19. Think…  You estimate the time it would take you to reach the interview venue, say 1 hour. You add some time e.g. 30 minutes for delays like traffic snarls.  You estimate some time, say 30 minutes for collecting your documents and some time, say 30 minutes for dressing up.  This means that you would need to wake up no later than 7:30 a.m. that morning to reach your interview venue in time.
  20. 20. Reasons that affect estimation  Test estimates can be affected by many factors: timing pressures people factors geographic distribution of the test team
  21. 21. Estimating testing is no different  No different than estimating other activities such as software design or programming.  We must break down the activities into well defined tasks that, in turn, can be estimated.  There are a number of methods that can be employed to estimate the testing effort required.  Whatever method is used, test estimation will always be required in advance and must be reviewed throughout the project once further information is obtained.
  22. 22. How testing is different?  Similar but not same!  Estimation techniques can be similar  Nature of Effort estimation is different  Models can be customized for Test Projects  Overheads and factors are different  Test Projects have to consider these  Analysis of test types  Factors for estimation are based on Development environment and test environment  Demands on Skill, expertise and automation  Technology and infrastructure requirements  Domain knowledge
  23. 23. Test estimation is different  It is not an independent activity –  testing is dependent on development for delivering the software to an agreed date and to an agreed quality standard. Should the quality of the software not be as good as we expected then we will spend more time reporting an greater number of faults and retesting them.  Last minute changes in product just before testing  Environment should be stable
  24. 24. What is important?  Estimation methods  Can we derive from software estimation techniques  Framework  Is there a framework that can be built?  Knowledge and know-how?  Past Experience and know-how is crucial  Can we learn from the past?  Intelligence (metrics and data) from Projects is very important  Team  Teams play a crucial role  They are the backbone of the execution  Who else can predict accurately but them  Estimation is not a individual effort  It is a team effort!
  25. 25. Influencing factors  High Complexity  Stakeholders are many  External Dependencies  Lacking Skills in Domain, Technology  Have to build new tools/frameworks  Experimenting…  Requirement of Hardware/Other infrastructure  Test Data  Legal contractual obligations
  26. 26. Underestimating / Over- estimating Underestimating a project leads to:  Under staffing  Under scoping the quality assurance effort  Setting too short a schedule Which in turn leads to:  Staff burnout  Low quality  Loss of credibility as deadlines is missed Over-estimating can be almost as bad:  The project will take as long as estimated even if the project was overestimated  Loss of credibility as deadlines is reached before the target
  27. 27. Why Now? What we need now?  Established Estimation Techniques  Build large and skilled teams  Remove the fad of ‘Testing’ as a non-interesting job!  Build frameworks and process methods specific to testing  Use automation effectively  Leverage the large talent pool  Incubate talent with the relevant skills
  28. 28. Some tangibles in the whole process!  Very less documentation available  No industry standard exists  Experience of the individual  Maturity of client  Maturity of organization  Different Domains  IT industry – continuously evolving › Mainframe world, Personal Computing, Structured programming, Object oriented programming, super- computing, what is next ?? › Desktop applications, Client-Server, Web based, mobile, Embedded, what is next ??
  29. 29. Characteristics of Good Estimation Ideally a software estimation method should: Be based on structured methodologies Be used throughout all project development phases Be adaptable to the kind of work you will be doing in the future
  30. 30. Tips to Estimate Accurately  Think of Some Buffer Time  Consider the Bug Cycle  Availability of All the Resources for Estimated Period  Can We Do Parallel Testing?  Think of Your Past Experience to Make Judgments!
  31. 31. Tips to Estimate Accurately  Estimations Can Go Wrong – So re-visit the estimations frequently in initial stages before you commit it.  Consider the Scope of Project  Are You Going to Perform Load Testing?  Do You Know Your Team?
  32. 32. Ponder Upon…  You are appearing in an examination. The duration of the examination is 3 hours and you have to answer 30 questions.  You average the time available for answering one question while leaving out certain time for revision at the end.  You look at the questions.  Some questions are easy for you but some are not.  How will you estimate?You reserve less time than average for answering the simple questions and more time than average for the difficult ones.
  33. 33. Factors of Estimating  Market opportunity  Who the competition is?  What is the opportunity now and in the future?  Contractual terms  Are there penalty clauses?  Is it phase wise delivery?  Volatility of requirements  How standard is the requirement?  What could be the change in requirements over time?
  34. 34. Factors of Estimating  Past Experience  Have we had a better or bitter experience?  Have we delivered something similar in the past?  Organizational strengths  Management support  Strength and expertise of the team  Skill set  Talent pool of engineers and the expectations of the project  In house training capability  Ability to learn quickly
  35. 35. Factors of Estimating  Ability to ramp up  Can we ramp up our team in case we win the project?  Option and flexibility to outsource?  Technology  Complexity  Environment – Onsite/Offsite  Virtual Testing  Organizational strengths  Management support  Strength and expertise of the team
  36. 36. Factors of Estimating  Skill set  Talent pool of engineers and the expectations of the project  In house training capability  Ability to learn quickly  Ability to ramp up  Can we ramp up our team in case we win the project?  Option and flexibility to outsource?  Technology  Complexity  Environment – Onsite/Offsite  Virtual Testing
  37. 37. Principles of estimation Principle of Estimation for testing projects shall be  Based on  Software requirements  Previous projects  Metrics  Estimation shall  Never forget the past  Be recorded  Be supported by tools  Be always verified  Consider automation needs  Consider people skills
  38. 38. Principles of estimation Principle of Estimation for testing projects shall be  # of test cases/scenarios,  # test cases per scenario,  # of builds (for regression)
  39. 39. Example  It is the beginning of another day at work. Your manager has given you 20 test cases to execute today.  In addition, you need to complete the annual self- appraisal form.  You estimate that it would take you 1 hour to complete your appraisal form.  Out of 8 hours of your work day, you have 7 hours remaining.  You reckon that you need to execute a test case every 21 minutes (7 hours X 60 minutes / 20 test cases).
  40. 40. Expenses in Test Estimation  Manpower cost  Cost of infrastructure  Operational costs  Special software  Communication  Travel  Training  Overheads  Shared facilities  Infrastructure overheads  Cost of money  Cost of Electricity/welfare/overtime etc
  41. 41. Keynotes  Testing Size – the amount (quantity) of testing that needs to be carried out. Some times this may not be estimated especially in Embedded Testing (that is, testing is embedded in the software development activity itself) and in cases where it is not necessary  Testing Effort – the amount of effort in either person days or person hours necessary for conducting the tests  Testing Cost – the expenses necessary for testing, including the expense towards human effort
  42. 42. Keynotes  Testing Schedule – the duration in calendar days or months that is necessary for conducting the tests  Estimation: An approximate calculation of quantity or degree or worth of some activity or an object of value  Test Estimation: An Act or process of estimation applied to a testing task
  43. 43. Lets estimate!!
  44. 44. Case Study  Let us suppose customer needs an e- commerce website.  We have two ways to implement this project:  one approach is to develop the complete application  second approach is to develop this application with the help of some product like BizTalk server.
  45. 45. Estimated Total no of Development efforts (Man Days) 200 Estimated test efforts as per the industry standard which is taken as 40% of total dev efforts ( Man Days) 80 Estimated Total no of Development efforts (Man Days) 100 Estimated test efforts as per the industry standard which is taken as 40% of total dev efforts ( Man Days) 40 In Second Scenario: Develop the application which is based on BizTalk In First Scenario: development
  46. 46.  In Second scenario, we are saving lot of development efforts using the BizTalk server but in the testing effort estimation we are putting fewer efforts which is not the right decision in such kind of applications in real life experience.  We should not calculate the testing efforts as per the industry standard as 40 % or 50% of the development efforts.  We need to increase this factor of testing effort up to 90% to 100% of Dev effort.  In first scenario taking the industry standard for test effort estimation (40 % of Dev efforts) is correct and we will be able to meet our actual efforts with some contingency factor there.
  47. 47. Sample exercise 1  For a new project, you are the test lead. You have 7 testers under you.  The project has 10 modules.  Prepare a list of factors which will require for the estimation for various test activities.
  48. 48. Estimates are Never Perfect Beware ...
  49. 49. Test Estimation Process
  50. 50. Some methods  Guessing (Finger in the Air – F.I.A. approach):  Not a good method to base our final estimate on.  Not advisable to rely solely on this method.  Can be reasonably accurate depending on past project experience and the expertise of the estimator.  Past project knowledge:  To base our estimates on previous, similar projects is a reasonable thing to do.  Only effective if we have recorded such data from previous projects.  It can be easily challenged.  Work Breakdown Structures (WBS):  Identify tasks that make up the test activities and estimate each one in turn.
  51. 51. Project Estimation Process
  52. 52. The Estimation Process 1. Estimate the size of the product. a) Create a detailed Work Break Down Structure. This directly impacts the accuracy of the estimate. The most serious handicap is the inability to clearly visualize the steps involved in the Project. b) The work Break down structure will include the size and complexity of each software module that can be expressed as number of Test cases, Test Scenarios, or any other unit of measure c) The Work Break down structure should include tasks other than testing such as Software Configuration Management, Documentation, Communication, User Interaction, Implementation, Knowledge Transition, Support tasks(if any) and so on d) Clearly indicate or eliminate any gray areas (vague/unclear specifications etc.)
  53. 53. The Estimation Process 2. Estimate the effort in person-hours. The Result of various tasks involved in step 1 is an effort estimate in person hours. The effort of various Project tasks expressed in person-hours is also influenced by various factors such as: a) Experience/Capability of the Team members b) Technical resources c) Familiarity with the Development Tools and Technology Platform 3. Estimate the schedule in calendar months The Project Planners work closely with the Technical Leads, Project Manager and other stakeholders and create a Project schedule. Tight Schedules may impact the Cost needed to develop the Application. 4. Estimate the project cost in dollars (or other currency) Based on the above information the project effort is expressed in dollars or any other currency.
  54. 54. Simple estimation method  Per day cost of Tester = x  Days spent on preparing test plan, test cases and actual testing = y  Cost of per day usage of testing tools (if any) = a  Days testing tools used for the purpose = b  Overhead Cost = c  Effort estimation for testing activity =  (x * y) + (a * b) + c
  55. 55. Lets do an exercise…  For a new project, you have 10 testers and per day cost for the same is $40 per tester. The project is having only manual testing and requires atleast 4 days in all for preparing test artifacts. For execution, they require 6 days effort of all testers. The overhead cost is calculated as $120. Prepare test cost estimation for the same.
  56. 56. Lets see few methods of estimations: 57
  57. 57. Some methods 58  Guessing (Finger in the Air – F.I.A. approach):  Not a good method to base our final estimate on.  Not advisable to rely solely on this method.  Can be reasonably accurate depending on past project experience and the expertise of the estimator.  Past project knowledge:  To base our estimates on previous, similar projects is a reasonable thing to do.  Only effective if we have recorded such data from previous projects.  It can be easily challenged.  Work Breakdown Structures (WBS):  Identify tasks that make up the test activities and estimate each one in turn.
  58. 58. Finger in Air (FIA) Method 59  This technique is purely guesswork and based on the some sort of experience.  The method is very common, but since it is based on your gut feeling, its uncertainty contingency is probably around 200% or even higher.  It involves the use of sample data to calculate a single value which is to serve as a "best guess" or "best estimate“.
  59. 59. Example 60  PM : "Can you give me an estimate of the time necessary to develop feature xyz?"  Test Lead: "One month"  PM : "That's far too long, we've got only one week!"  Test Lead : "I need at least three"  PM: "I can give you two at most"  Test Lead : "Deal!"
  60. 60. Ad-hoc Method 61  The test efforts are based on tentative timeframe.  The timeline set by managerial or by client without any guess/experience. Alternatively, it is done until the budgeted finances run out.  This is very common practice in extremely immature organizations and has error margins of over 100% at times.
  61. 61. Experience Based: Analogies & experts62  It will always be based on some sort of experience and a number of (unconscious) assumptions.  Estimate, test, check, then revise and update your project schedule.  You already tested similar application in previous project.  Inputs are taken from Subject Matter experts who know the application (as well as testing) very well.
  62. 62. Experience Based: Analogies & experts63  Metrics collected from previous tests.  You already tested similar application in previous project.  Inputs are taken from Subject Matter experts who know the application (as well as testing) very well.
  63. 63. Estimation by analogy 64  The cost of a project is computed by comparing the project to a similar project in the same application domain  Advantages: May be accurate if project data available and people/tools the same  Disadvantages: Impossible if no comparable project has been tackled. Needs systematically maintained cost database
  64. 64. Experience Based: Analogies & experts65  For example:  “This looks very much like the system I tested in my previous job. That took us three months, and we were four people. This is slightly smaller and we are five people—so I guess this will take two months to complete.”
  65. 65. Work Breakdown Structure (WBS)66  It is created by breaking down the test project into small pieces.  Modules are divided into sub-modules. Sub modules are further divided into functionalities and functionalities are divided in sub- functionalities.  Review all the requirements from Requirement Document to make sure they are added in WBS.  Now you figure out the number of tasks your team needs to complete.  Estimate the duration of each task.
  66. 66. 67
  67. 67. Typical Generic Schedule for Waterfall Model Lifecycle68 Phase M1 M2 M3 M4 M5 M6 … TOT Rqmts 25% 25% 25% 15% 10% 100% P Des 15% 30% 20% … 100% D Des … 100% C&UT … 100% Integ. … 100% REVIEWS SRR PDR Process Schedule
  68. 68. Example 69
  69. 69. Delphi Method70
  70. 70. Delphi Technique 71  A simple technique that has proved remarkably resilient even in highly complex situations.  In the Delphi Method is based on surveys and basically collects the information from participants who are experts.  You must appoint an estimation group as appropriate. This can be stake-holders and/or experts in the tasks to estimate.  Most often the participants give their initial estimates based on experience and/or they are experts in a specific area.
  71. 71. Delphi Technique 72  The initial estimates may also be obtained using some of the other estimation techniques to make them even more trustworthy.  Functionalities and each task is allocated to each team member. Then team member gives estimate that they will take this much hours to complete the task.
  72. 72. Delphi Technique - Steps 73 The steps in this estimation process are: 1. Each member of the group gives an estimate. 2. The group is informed about the average and distribution of the estimates. 3. Those giving estimates in the lower quartile and in the upper quartile are asked to tell the rest of the group why their estimates were as they were.
  73. 73. Delphi Technique - Steps 74 4. The group estimates again—this time taking the previous result and the provided arguments for the “extreme” estimates into account. 5. This may continue two, three, four, or more times until the variation in the estimates is sufficiently small
  74. 74. Delphi Technique 75  In Delphi Method, work breakdown structure is decomposed for each task and is distributed to a team comprising of 3-7 members for re- estimating the task.  The final estimate is the result of the summarized estimates based on the team consensus.  This method speaks more on experience rather than any statistical formula.
  75. 75. Use Case method76
  76. 76. Use – Case Point Method 77  Based on the Use Cases  Simple Formula:  Calculate the unadjusted actor weights and unadjusted use case weights to determine the software testing estimation.
  77. 77. UCP – 4 important components78 1. Unadjusted Actor Weight [AW] 2. Unadjusted Use Case Weight [UCW] 3. Technical complexity Factors [TCF] 4. Environmental complexity Factors [ECF]
  78. 78. Unadjusted Actor Weight 79  The total Actor weight can be calculated by categorizing all the actors as simple, medium and complex and then adding up all the factors. Actor Classification Type of Actor Weight Simple External system that must interact with the system using a well-defined API 1 Average External system that must interact with the system using standard communication protocols (e.g. TCP/IP, FTP, HTTP, database) 2 Complex Human actor using a GUI application interface 3 UAW = (Total No. of Simple actors x 1) + (Total No. Average actors x 2) + (Total No. Complex actors x 3)
  79. 79. Unadjusted Use Case Weights 80  Similar to Actor weights.  All the use cases need to be categorized as simple, medium and complex. Use Case Classification No. of Transactions Weight Simple 1 to 3 transactions 5 Average 4 to 7 transactions 10 Complex 8 or more transactions 15 UUCW = (Total No. of Simple Use Cases x 5) + (Total No. Average Use Case x 10) + (Total No. Complex Use Cases x 15)
  80. 80. Unadjusted Use Case Points (UUCP)81 UUCP = Unadjusted Actor Weight [UAW] + Unadjusted Use Case Weight [UUCW]
  81. 81. Technical Complexity Factor [TCF]82  Technical factors are evaluated by the development team and assigned a value between zero and five.  A weight of 0 indicates the factor is irrelevant and the value 5 means that the factor has the most impact.  TCF = 0.6 + (TF/100)
  82. 82. Technical Factor 83 Facto r Description Weight T1 Distributed system 2.0 T2 Response time/performance objectives 1.0 T3 End-user efficiency 1.0 T4 Internal processing complexity 1.0 T5 Code reusability 1.0 T6 Easy to install 0.5 T7 Easy to use 0.5 T8 Portability to other platforms 2.0 T9 System maintenance 1.0 T10 Concurrent/parallel processing 1.0 T11 Security features 1.0 T12 Access for third parties 1.0
  83. 83. Environmental Complexity Factor [ECF]:84  A rating of 0 means the environmental factor is irrelevant for this project; 3 is average; 5 mean it has strong influence.  Environmental complexity factor is calculated as below – ECF = 1.4 + (-0.03 x EF)
  84. 84. Environment Complexity Factor85 Factor Description Weight E1 Familiarity with development process used 1.5 E2 Application experience 0.5 E3 Object-oriented experience of team 1.0 E4 Lead analyst capability 0.5 E5 Motivation of the team 1.0 E6 Stability of requirements 2.0 E7 Part-time staff -1.0 E8 Difficult programming language -1.0
  85. 85. Calculate Final Use Case Points (UCP):86 UCP = UUCP*TCF*ECF
  86. 86. Example 87 UUCW = (Total No. of Simple Use Cases x 5) + (Total No. Average Use Cases x 10) + (Total No. Complex Use Cases x 15) UUCW = (2 x 5) + (3 x 10) + (4 x 15) = 100 UAW = (Total No. of Simple Actors x 1) + (Total No. Average Actors x 2) + (Total No. Complex Actors x 3) UAW = (1 x 1) + (0 x 2) + (4 x 3) = 13 Hence, UUCP=100 +13 = 113
  87. 87. Example: TCF 88 Factor Description Weight Assigned Value Weight x Assigned Value T1 Distributed system 2.0 5 10 T2 Response time/performance objectives 1.0 5 5 T3 End-user efficiency 1.0 3 3 T4 Internal processing complexity 1.0 2 2 T5 Code reusability 1.0 3 3 T6 Easy to install 0.5 1 0.5 T7 Easy to use 0.5 5 2.5 T8 Portability to other platforms 2.0 2 4 T9 System maintenance 1.0 2 2 T10 Concurrent/parallel processing 1.0 3 3 T11 Security features 1.0 5 5 T12 Access for third parties 1.0 1 1 T13 End user training 1.0 1 1 Total (TF): 42 TCF = 0.6 + (TF/100) TCF = 0.6 + (42/100) TCF = 1.02
  88. 88. Example: ECF 89 Factor Description Weight Assigned Value Weight x Assigned Value E1 Familiarity with development process used 1.5 3 4.5 E2 Application experience 0.5 3 1.5 E3 Object- oriented experience of team 1.0 2 2 E4 Lead analyst capability 0.5 5 2.5 E5 Motivation of the team 1.0 2 2 E6 Stability of requirements 2.0 1 2 E7 Part-time staff -1.0 0 0 E8 Difficult programming language -1.0 4 -4 ECF = 1.4 + (-0.03 x EF) ECF = 1.4 + (-0.03 * 10.5) ECF = 1.085
  89. 89. Example: UCP 90  UCP = (UUCW + UAW) x TCF x ECF  UCP = (100 + 13) x 1.02 x 1.085 = 125.06  Estimated Effort = UCP x Hours per UCP  Estimated Effort = 125.06 x 8  Estimated Effort = approx 1000 Hours = approx 42 man-days
  90. 90. Benefits - UCP 91  This scientific method gives more accurate and precise results over any traditional method available for effort estimation.  This method is very well suited for bidding projects as most of the time use case is the only information available at the beginning of a project.
  91. 91. Test Case Enumeration based estimation92  An estimation method which starts with the identification of all the test cases to be executed.  An estimate of the expected effort for testing each test case is calculated, using a Beta distribution formula. (Best Case + Worst Case + (4 * Normal Case)) __________________________________ ____ 6
  92. 92. Test Case Enumeration based estimation93 Steps are: 1. Enumerate the test cases – list down all the test cases 2. Estimate testing effort required for each test case – use person hours or person days – consistently 3. Use Best Case, Normal Case and Worst Case scenarios for estimating effort needed for each test case 4. Compute Expected Effort for each case using Beta Distribution  Sum up the –  Best-Case times to obtain Optimistic estimate  Worst-Case times to obtain Pessimistic estimate  Normal-Case times to obtain normal-case estimate
  93. 93. Example 94
  94. 94. Task (Activity) based estimation 95  This method looks at the project from the standpoint of tasks to be performed in executing the project.  Any project is executed in phases. Phases in a testing project could be –  1. Project Initiation  2. Project Planning  3. Test Planning  4. Test Case Design  5. Set up Test Environment  6. Conduct Testing  a. Integration Testing  b. System Testing  c. Load Testing  7. Log and report test results  8. Regression Testing  9. Prepare Test Report  10. Project closure
  95. 95. Task (Activity) based estimation 96 The following are the steps in Task based effort Estimation: 1. Assign durations for each of the Tasks – either in person hours or person days –consistently 2. Use three time estimates – Best Case, Worst Case and Normal Case for each of the tasks 3. Compute the expected time using formula [Best Case + Worst Case + (4* Normal Case)]/6 4. Make adjustments for project complexity, familiarity with the platform, skill of the testers, tools usage. 5. Sum up the total effort estimate of the project
  96. 96. Example 97
  97. 97. Software Size based estimation 98  A common tool for quantifying software size is function point allocation.  To use this technique the function points should be multiplied by a productivity figure, which represents the number of function points one employee can work through per hour. The outcome would be the effort estimate
  98. 98. Software size based estimation 99  Let us say that it takes 2 person hours to test software of size one Function Point – using this norm we can arrive at the amount of effort required for the testing project based on the size of software to be tested.  Suppose that the size of software to be tested is 1000 Function points, then, using the norm of 2 person hours per function point, we have 2000 person hours for testing the software of size 1000 Function Points.
  99. 99. Software Size based estimation  Different norms are decided for each category to get the correct size of the software. 100
  100. 100. Software Size based estimation 101  For example:  Complex modules – 3  Medium modules – 10  Simple modules – 20  Company ABC, can have the following norms:  Complex module X 20 + Medium module X 10 + Simple module X 5  The size will be 260 function points.  Norm: It took 2 hour to write the test cases and 30 minutes to execute these test cases.  So, for Test case writing will take 520 hours and Test Execution will take 130 hours.
  101. 101. How will you know you will be able to complete Testing as per the estimate ? Getting Trapped?
  102. 102. Trap # 1 - Model of Testing  A simplified model of Testing is assumed.  Testing is an open ended search for problems – decision to stop testing is often decided by the stakeholders depending upon the parameters that are out side the purview of testing  Actual Testing model is complex hence estimates tend to go wrong  Testing is assumed to be complete when planned test cases are executed  Dependencies on environment, availability of Stable build to test and test data are typically ignored or down played
  103. 103. Trap # 2 – Testers do not decide the ship date Project Manager or the customer picks the date – Why rob him the opportunity to do so?  Testers provide service to PM to decide about state of the product  Testers do not have the ability to add or remove the features  Testers do not have the ability (in most of the cases) to extend the Ship date beyond what is planned by the PM  This creates bounded time space for testers to operate on Thus all you have is to test till PM says stop – Will estimates matter then?
  104. 104. Trap # 3 - Unknown Future We’re too optimistic, with short memories that mask the painful overruns from previous projects – Karl E. Wiegers Things change during the course of the project in unimaginable ways Each test cycle presents its own challenges and issues There might be be too many or too less bugs discovered in any cycle There might be developer delays that eats up test time There is might be pressure to skip tests and constantly change test sets Development might lag behind bug find rate There might be requirements changes hence development delays There might be failures in Build verification tests There might be significant number of “non reproducible defects”
  105. 105. Trap # 4 - Too many Variables Giving an Estimate would mean a commitment – which you are not sure about fulfilling – that is a trap  Test scope – that keeps changing depending upon available tim for testing  Set of features available in a given build  Time available for testing for any given cycle  Test environment availability  Stability of build for testing  Unknown number of bugs discovered any given cycle  Additional cycles of Testing for every new bug or regression bu
  106. 106. If the Plan is Not Feasible  DO examine assumptions and data  initial cost estimates are often very conservative  Do examine risk/cost tradeoffs to see if you can accept a higher risk  Do make a list of barriers that must be removed in order to make the estimate fit the constraints
  107. 107. “The quickest way to make a project uneconomical is by doubling the resources needed and using the cover story that you need to prevent failures.” Adams, The Dilbert Principle If the Plan is Not Feasible  DO NOT “cave in” & lower everything to meet a target cost or schedule
  108. 108. The Negotiation Process We MUST have the lowest bid!!! We will, boss!!! Management will try to trim the budget by sending an army of low- ranking, clueless budget analysts to interview you and ask “insightful” questions. Adams, The Dilbert Principle
  109. 109. Good History Data Good historical data are essential for accurate estimating.
  110. 110. How Can We do Test Estimations? Test team Product Exp. Test Capability … Quality Test Specifications Test Size / Complexity … Test Platform Test Environment Test Tools … Test Estimates Test Estimation Model
  111. 111. Effort & Cost Estimation  Effort estimates can be made through historical experience or through models derived from historical experience  Models are often helpful to estimate effort  All are based on historical experience  All should be calibrated to your specific experience  And validated to be sure that they are right for your application  Most have additional parameters that allow you to “fine tune” to your specifics
  112. 112. Negotiating Tip . . .  The more facts you have, the better off you are during negotiation  Get decision makers to review your estimate  Sometimes they don’t bother  Be well prepared to explain it
  113. 113. They are things that many people know, but few people admit. The Dirty Little Secrets of Test Estimation
  114. 114. Dirty Little Secrets Secrets you knew but was never wanted to accept…  Secret 1: Your Estimate is Probably Wrong  Secret 2: Your Estimate Needs to be Wrong – In the Right Direction  Secret 3: The Recipient of the Estimate Already Has a Number in Mind  Secret 4: Your Customer Assumes Your Estimate is Too High  Secret 5: Your Estimates are Incomplete
  115. 115. #1: Your Estimate is Probably Wrong A) As a tester:  Your estimates often depend on other estimates, such as the time needed for development, people available, etc.  These numbers are often inaccurate, so that means yours will be also.
  116. 116. #1: Your Estimate is Probably Wrong B) The Project Scope Will Probably Change:  Scope change happens for a variety of reasons, some foreseen and some unforeseen.  Our assumptions get challenged and often proven wrong.
  117. 117. #1: Your Estimate is Probably Wrong C) You Don't Know What You Don't Know:  won't find that out until it's too late.  One of the biggest project risks much more than estimates. There's not much you can do about this one either.  Realize that you don't know everything and what you do know will change.
  118. 118. #1: Your Estimate is Probably Wrong  People will fudge numbers to make themselves look better.  Deliverables will be late. In fact, they will be later than you think they will. Some things will be early, but more often than not, your work will be delayed by delays experienced by others.
  119. 119. #2: Your Estimate Needs to be Wrong  You only know how close your estimate is when the work is done.  When you truly understand your estimate is just that – an estimate, you can be free to adjust it.  The “reserve” : the safety net having extra time, money and/or people be needed.  Far “high” than real estimates.
  120. 120. #3: Recipient of the Estimate Already Has a Number in Mind  There's a lot of feeling involved and not much logic at times.  Understand that your customer has an expectation already in mind. If your estimate goes too far above or below the customer's early expectation, silent alarms go off in their head.  The client may feel that because of the lower estimate you must have been leaving something out.
  121. 121. #4: Your Customer Assumes Your Estimate is Too High  Let's say that you do your best job of estimation with no reserves included.  You present these to your management or customer and they take one look and immediately reduce it by one-half because they believe you must have inflated it.  The assumption is that everyone who estimates has excess in the numbers.  It forces people to increase their estimates because they know management will assume they have been inflated.
  122. 122. #5: Your Estimates are Incomplete  Most people estimate time and cost. How many people do you know that also estimate the quality of the product?  High quality will probably cost more and may take longer than lower quality – at least in the short term. In the long run, high quality costs less because of lower maintenance.  In software development and testing, we often fail to consider the quality levels in a given estimate.
  123. 123. Conclusion  As you go forth and estimate, keep these little secrets in mind and you will have less stress over your estimates.  However, you can understand that the dynamics exist and be prepared to keep expectations reasonable.
  124. 124. In Other Words  Historical Data usually gives a RANGE of values, not a precise estimate Based on past experience, this project should take from 100 to 125 staff months
  125. 125. But We Don’t Have Until April 30th!  Suppose you propose your realistic, actionable schedule to management, and they say, “Make it shorter! We don’t have that long!” Now what? Reluctantly accept a date from management? That’s not a good plan. How about some other options?
  126. 126. But We Don’t Have Until April 30th!  Relax your entry criteria for testing a bit.  No "one phase must end before the next begins." But suppose you waive the “feature complete” entry criteria for system testing and accept an “almost complete” release.  That’s good; but people must understand that this overlap can increase risks to system quality.  Add Staff.  Suppose you cut test execution in half?
  127. 127. But We Don’t Have Until April 30th!  Reduce the extent of testing across the board.  Identify the least risky of the highest-risk areas, and adopt a balanced (broad, not deep) test approach.  If you’re going to take risks with quality, take those risks with your eyes open.  To do so, carefully consider the comparative levels of risk associated with reducing the testing effort.
  128. 128. Three Risk Management Approaches  Use multiple estimating methods  They will help improve the bounds on your estimate  Each new method will identify issues that other methods overlook  Set aside a “reserve” amount based on the level of risk in your estimate  Plan to update your estimates  You will have better information in the future
  129. 129. A Manager’s Checklist for Validating Software Cost and Schedule Estimates Are the objectives of the estimate clear and correct? o The objectives of the estimate are stated in writing. o The life cycle to which the estimate applies is clearly defined. o The tasks and activities included in (and excluded from) the estimate are clearly identified
  130. 130. A Manager’s Checklist for Validating Software Cost and Schedule Estimates Has the task been appropriately sized?  A structured process has been used to estimate and describe the size of the software product.  The size estimate was checked by relating it to measured sizes of other software products or components.  The size estimating process was checked against measured sizes of similar completed products.
  131. 131. A Manager’s Checklist for Validating Software Cost and Schedule Estimates Have the factors that affect the estimate been identified and explained?  Assumptions have been identified and explained.  A structured process such as a template or format has been used to ensure that key factors have not been overlooked.  Uncertainties in parameter values have been identified and quantified.  A risk analysis has been performed, and risks that affect cost or schedule have been identified and documented.
  132. 132. A Manager’s Checklist for Validating Software Cost and Schedule Estimates Has the situation changed since the estimate was last prepared?  The estimate has not been invalidated by recent events, changing requirements, or management action (or inaction).  The estimate is being used as the basis for assigning resources, deploying schedules, and making commitments.  The estimate is the current baseline for project tracking and oversight.
  133. 133. Some Opportunities to Consider  Plan to re-estimate after important milestones  Prioritize requirements and promise to deliver the top ones by the deadline  Incremental deliveries  Put a high cost on requirements changes  Look at each “adjustment factor” in Cocomo as an opportunity  Get training for everyone
  134. 134. Summary  Use existing models to suit your test project requirements  Estimations could be imprecise – but it’s ok!  Wrong Estimations should be controllable!  Re correct your estimate, if you have made a mistake – but do it early  Prepare for changes and challenges on a continuous basis  „Start Early and Stay focused
  135. 135. Summary  Allow team participation wherever possible  Collaborate, retrospect and co-operate  Experiment only on a small scale  Build contingency and risk plans – but keep them trimmed  Gain Confidence of management with methodologies and processes  Put your team first and not yourself – it is better in the long run
  136. 136. Future Forward  Learn from the past  Put metrics to best use  Build a estimation model for testing projects  Use existing techniques to interpolate for testing projects  Process Maturity  Build a model for quality of testing  Build reusable assets  Leverage existing maturity models of the organization  People skill and talent pool  Incubate new talent  Identify methods and techniques
  137. 137. Thank You!!