PROMISE 2011: "An Analysis of Trends in Productivity and Cost Drivers over Years"


Published on

"An Analysis of Trends in Productivity and Cost Drivers over Years"
Vu Nguyen, Liguo Huang and Barry Boehm.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • The model generated is more tolerant with the data
  • PROMISE 2011: "An Analysis of Trends in Productivity and Cost Drivers over Years"

    1. 1. An Analysis of Trends in Productivity and Cost Drivers over Years Vu Nguyen 1 , LiGuo Huang 2 , Barry Boehm 3 [email_address] ,, 1 University of Science, National University, Vietnam 2 Southern Methodist University 3 University of Southern California
    2. 2. Outline <ul><li>Introduction </li></ul><ul><li>Research Method </li></ul><ul><li>Results </li></ul><ul><li>Conclusions </li></ul><ul><li>Future Work </li></ul>
    3. 3. Motivation <ul><li>Software practices and technologies have evolved significantly over the last decades, e.g., </li></ul><ul><ul><li>Process: sequential to concurrent, traditional plan-driven to agile and rapid development </li></ul></ul><ul><ul><li>Tool and language: from basic code editors and debuggers to sophisticated development environments, from first to fourth generation languages </li></ul></ul><ul><ul><li>Storage and processing capabilities no longer a major concern for software project </li></ul></ul><ul><li>It is important to understand how such advancements affect the productivity of software projects over the years </li></ul>
    4. 4. Objectives <ul><li>Analysis of productivity trends </li></ul><ul><ul><li>Productivity trends of completed projects over the last 40 years </li></ul></ul><ul><ul><li>COCOMO cost drivers contributing to the productivity trends </li></ul></ul><ul><li>Analysis of productivity influencing factors </li></ul><ul><ul><li>Trends of COCOMO cost drivers over the period </li></ul></ul><ul><li>Proposing a measure namely Difficulty to explain productivity trends and variability </li></ul>
    5. 5. COCOMO <ul><li>Co st Co nstructive Mo del (COCOMO) is a model for estimating software cost </li></ul><ul><li>COCOMO originally published in 1981 </li></ul><ul><ul><li>Calibrated using 63 completed projects </li></ul></ul><ul><ul><li>Uses software size and 15 cost drivers as input </li></ul></ul><ul><li>COCOMO II published in 2000 </li></ul><ul><ul><li>Reflects changes in technologies and practices </li></ul></ul><ul><ul><li>Uses 22 cost drivers plus software size </li></ul></ul><ul><ul><li>Introduces 5 scale factors </li></ul></ul><ul><ul><li>Calibrated using 161 completed projects </li></ul></ul>
    6. 6. COCOMO (cont’d) <ul><li>COCOMO II estimates project effort as </li></ul><ul><li>Where, </li></ul><ul><ul><li>PM : is estimated effort in person-month </li></ul></ul><ul><ul><li>Size : size of software project </li></ul></ul><ul><ul><li>A : a constant </li></ul></ul><ul><ul><li>SF i : each of 5 scale factors </li></ul></ul><ul><ul><li>EM i : each of 17 effort multipliers </li></ul></ul><ul><ul><li>β 0 , β 1 : constants </li></ul></ul>
    7. 7. Calibrating COCOMO <ul><li>Process of fitting data to the model to adjust its parameters and constants </li></ul>Initial rating scales for cost drivers Sample data Delphi survey of 8 experts (Expert-judgment estimates) Model Calibration New rating scales for cost drivers and constants <ul><li>Calibration Techniques: </li></ul><ul><li>Ordinary Least Squares Regression (OLS) </li></ul><ul><li>Bayesian Analysis </li></ul>E.g., 341 data points for this study
    8. 8. Outline <ul><li>Introduction </li></ul><ul><li>Research Method </li></ul><ul><li>Results </li></ul><ul><li>Conclusions </li></ul><ul><li>Future Work </li></ul>
    9. 9. Procedure <ul><li>Our analysis procedure consists of </li></ul><ul><li>Review data set of 341 projects </li></ul><ul><li>Compute and evaluate productivity trends </li></ul><ul><li>Calibrate COCOMO using 341 projects to obtain rating scales for cost drivers </li></ul><ul><li>Define a project Difficulty measure using the rating scales from step #3 </li></ul><ul><li>Compute and evaluate project Difficulty trends </li></ul><ul><li>Evaluate trends of cost drivers </li></ul>
    10. 10. Data Set <ul><li>Consisting of 341 projects completed between 1970 and 2009 </li></ul><ul><li>Collected from 25 organizations in 4 countries, US (251 projects), Brazil (16), Vietnam (70) and Thailand (4) </li></ul><ul><li>207 new development and 134 maintenance/enhancement projects </li></ul><ul><li>Attributes of each data point: </li></ul>Attribute Description Effort Actual project effort in person-month (PM) Size Size in KSLOC Cost Drivers Twenty two COCOMO cost drivers and scale factors Completion Year The year in which the project is completed Project Type Either New Development or Maintenance Country Country from which the project was collected
    11. 11. Data Set (cont’d) <ul><li>Number of projects completed in each year </li></ul>
    12. 12. Calculating Productivity <ul><li>We defined productivity as </li></ul><ul><li>PM - project actual effort in person-month </li></ul><ul><li>KSLOC - software size measured in thousand source lines of code </li></ul><ul><li>This productivity is not adjusted by the productivity impact factors </li></ul><ul><li>We referred this as raw productivity </li></ul>
    13. 13. Defining a Difficulty Measure <ul><li>Difficulty is proposed to measure the overall impact of the COCOMO cost drivers on productivity </li></ul><ul><li>Difficulty is derived from the COCOMO effort equation </li></ul><ul><li>Difficulty is defined as </li></ul><ul><li>Where, </li></ul><ul><li> EM i and SF j are COCOMO’s cost drivers (effort multipliers and scale factors) </li></ul><ul><li>β 0 , β 1 are constants </li></ul><ul><li>Rating scales of EM and SF are obtained from calibrating the COCOMO model using 341 projects </li></ul><ul><li>Difficulty is computed for each project </li></ul>
    14. 14. Difficulty Measure <ul><li>Basic characteristics </li></ul><ul><ul><li>Indicates how difficult it is for a project to build the software in terms of effort required </li></ul></ul><ul><ul><li>Reflects the overall impact of the COCOMO cost drivers on the effort of a 100KLOC project </li></ul></ul><ul><ul><li>Is always > 0 </li></ul></ul><ul><ul><li>If Difficulty = 1.0, the cost drivers overall do not affect the effort required for the project </li></ul></ul><ul><ul><li>If Difficulty > 1.0, the overall impact of the cost drivers is to increase the effort </li></ul></ul><ul><ul><li>If Difficulty < 1.0, the overall impact of the cost drivers is to decrease the effort </li></ul></ul>
    15. 15. Evaluation Method <ul><li>Grouping projects based on their completion years </li></ul><ul><li>Trends evaluated by testing the significance of correlations between productivity, cost drivers and completion year </li></ul><ul><li>Kendall’s rank correlation used to test the trends </li></ul><ul><li>Mann-Whitney U test used to validate the difference between two groups </li></ul><ul><li>Level of significance used: 0.05 </li></ul><ul><li>Statistical tests performed using 341 data points </li></ul>
    16. 16. Outline <ul><li>Introduction </li></ul><ul><li>Research Method </li></ul><ul><li>Results </li></ul><ul><li>Conclusions </li></ul><ul><li>Future Work </li></ul>
    17. 17. Productivity Trends <ul><li>Overall trend: productivity increases over the years (Kendall’s correlation between productivity and completion year: p-value = 0 ) </li></ul>Productivity (KSLOC/PM) Project Completion Year No. of projects
    18. 18. Productivity Trends (cont’d) <ul><li>Productivity trends in new development and maintenance projects </li></ul>New Development Projects Maintenance Projects Overall trend: productivity increases over the years ( p-value = 1.8*10 -5 ) Overall trend: productivity increases over the years ( p-value = 0.01 < 0.05 ) Productivity (KSLOC/PM) Productivity (KSLOC/PM)
    19. 19. Difficulty Trends <ul><li>How does Difficulty explain the productivity trends? </li></ul>Project Completion Year Project Difficulty <ul><li>Overall trend: Difficulty of the projects decreases over the years in the opposite direction with productivity (Kendall’s correlation between Difficulty and completion year: p-value = 0 ) </li></ul>
    20. 20. Productivity and Difficulty of New Development and Maintenance Projects <ul><li>New development projects were less productive because they were more difficult </li></ul>Productivity (KSLOC/PM) Difficulty Productivity of New Dev. and Maintenance Projects Difficulty of New Dev. and Maintenance Projects
    21. 21. Trends in Cost Drivers <ul><li>Trends tested using Kendall’s rank correlation between each of the cost drivers and completion year </li></ul><ul><li>13 of 22 cost drivers found to have statistically significant correlations with the completion year (Kendall’s p-value < 0.05 ) </li></ul>Significantly Correlated Cost driver Kendall’s p-value TOOL Use of Software Tools 2.20E-16 PMAT Process Maturity 1.22E-13 STOR Main Storage Constraint 1.31E-11 TIME Execution Time Constraint 6.62E-10 PLEX Platform Experience 1.98E-05 PVOL Platform Volatility 2.04E-05 APEX Applications Experience 4.88E-05 LTEX Language and Tool Experience 2.84E-04 DATA Database Size 1.81E-03 RELY Required Software Reliability 1.42E-02 CPLX Product Complexity 1.58E-02 PREC Precedentedness of Application 2.13E-02 ACAP Analyst Capability 4.87E-02
    22. 22. Trends in Cost Drivers (cont’d) <ul><li>Platform and process factors Use of Tools , Process Capability , Main Storage Constraint , and Execution Time Constraint have significantly improved over the years </li></ul><ul><li>Applications Experience and Analyst Capability were seen to decrease </li></ul><ul><li>Platforms Experience tends to increase over the years (developers on recent projects are more experienced with platforms) </li></ul><ul><li>Product Complexity tends to decrease </li></ul>
    23. 23. Trends in Cost Drivers (cont’d) <ul><li>Use of Software Tools (TOOL) factor </li></ul>TOOL Rating Scales Project Completion Year Very Low Low Nominal High Very High TOOL Rating Levels
    24. 24. Outline <ul><li>Introduction </li></ul><ul><li>Research Method </li></ul><ul><li>Results </li></ul><ul><li>Conclusions </li></ul><ul><li>Future Work </li></ul>
    25. 25. Conclusions <ul><li>Trends of productivity and cost drivers analyzed using 341 projects completed between 1970 and 2009 </li></ul><ul><li>Productivity found to increase over the years </li></ul><ul><li>Characterizing productivity trends using one or a few factors is insufficient. Thus, we proposed Difficulty to combine factors </li></ul><ul><li>Productivity trends explained by the Difficulty measure which is based on the COCOMO’s form and cost drivers </li></ul><ul><li>Ratings of 13 of 22 cost drivers found to have clear trends over the years. Hence, they affect the productivity trends </li></ul>
    26. 26. Limitations and Future Work <ul><li>Some limitations </li></ul><ul><ul><li>Small data set, considering many software products developed over the years </li></ul></ul><ul><ul><li>Analyzed only 22 COCOMO cost drivers. It’s possible that other factors are more influential </li></ul></ul><ul><li>Future work </li></ul><ul><ul><li>Further analysis and evaluation of the Difficulty measure </li></ul></ul><ul><ul><li>Investigating how relative impact of cost drivers differ across projects </li></ul></ul>
    27. 27. Thank You Q&A