Software Management Metrics Herman P. Schultz


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Software Management Metrics Herman P. Schultz

  1. 1. Software Management Metrics Herman P. Schultz 1988 EEL6887: Software Engineering Chi-Hwa J Marcos 3/29/2006
  2. 2. Reference Sources <ul><li>Herman P. Schultz, “Software Management Metrics”, Hanscom AFB, MA 1988 </li></ul><ul><li>David L. Hallowell, “Six Sigma Software Metrics”, iSixSigma LLC, </li></ul><ul><li>Karl E. Wiegers, “A Software Metrics Primer”, Software Development, July 1999, </li></ul>
  3. 3. Overview <ul><li>Introduction </li></ul><ul><li>Coverage </li></ul><ul><li>Reporting </li></ul><ul><li>Analysis </li></ul><ul><ul><li>Correlation </li></ul></ul><ul><ul><li>Extrapolation </li></ul></ul><ul><li>Software Size Metric </li></ul><ul><li>Software Personnel Metric </li></ul><ul><li>Software Volatility Metric </li></ul><ul><li>Computer Resource Utilization Metric </li></ul><ul><li>Schedule Progress Metric </li></ul><ul><li>Metric Tool Examples </li></ul><ul><li>Summary </li></ul><ul><li>Conclusion </li></ul>
  4. 4. Introduction <ul><li>“ metric -- A quantitative measure of the degree to which a system, component, or process possesses a given attribute.” [IEEE Std 610.12-1990] </li></ul><ul><li>This report was the result of approximately 3 years of experience from government and industry use and analysis of metrics. The use of metrics data can detect potential problems in a software project while time still permits for resolution discovery. These potential problems may impact cost and schedule. </li></ul>
  5. 5. Coverage <ul><li>Metrics should cover all phases of software development. </li></ul><ul><li>Metrics can cover some development phases more than once. </li></ul><ul><li>Multiple coverage provides better visibility into each development phases. </li></ul><ul><li>Multiple phases allows consistency checks of metrics. </li></ul><ul><li>Metrics address two aspects of software development: </li></ul><ul><ul><li>Progress metrics tracks deviation between plan and actual progress. </li></ul></ul><ul><ul><li>Planning metrics affects software development progress. </li></ul></ul>
  6. 6. Reporting <ul><li>Recommends pre-Program Management Review (PMR) screening </li></ul><ul><li>For example: </li></ul><ul><ul><li>Deliver metrics to government at least one week prior to PMR. </li></ul></ul><ul><ul><li>Discuss metrics during Technical Interchange Meeting (TIM). </li></ul></ul><ul><ul><li>Discuss TIM results with System Program Office (SPO) to separate issues between PMR and TIMs. </li></ul></ul><ul><li>Metrics presented at PMR provides management with visibility into potential cost and schedule impact problems. </li></ul>
  7. 7. Analysis <ul><li>Metrics provides a means of evaluating the software plan credibility. </li></ul><ul><li>Metrics identify trends. </li></ul><ul><li>Two analysis methods: </li></ul><ul><ul><li>Correlation </li></ul></ul><ul><ul><ul><li>Identifying strong relationships between reported metrics. </li></ul></ul></ul><ul><ul><ul><li>Look for inconsistencies within a group of related metrics. </li></ul></ul></ul><ul><ul><li>Extrapolation </li></ul></ul><ul><ul><ul><li>Identify trends. </li></ul></ul></ul><ul><ul><ul><li>Shows potential impact on schedules. </li></ul></ul></ul>
  8. 8. Correlation Example
  9. 9. Extrapolation Example
  10. 10. Extrapolation Example
  11. 11. Extrapolation Example
  12. 12. Extrapolation Example
  13. 13. Extrapolation Example
  14. 14. Software Size Metric <ul><li>Purpose </li></ul><ul><ul><li>Track magnitude changes in software development effort. </li></ul></ul><ul><ul><li>SLOC </li></ul></ul><ul><li>Behavior - A lack of understanding and appreciation of requirements can cause increase or decrease in SLOC </li></ul><ul><ul><li>Increase SLOC </li></ul></ul><ul><ul><ul><li>Better understanding of requirements. </li></ul></ul></ul><ul><ul><ul><li>Better understanding of design implication and complexity. </li></ul></ul></ul><ul><ul><ul><li>Optimistic original estimate </li></ul></ul></ul><ul><ul><li>Decrease SLOC </li></ul></ul><ul><ul><ul><li>Overestimate at beginning of program. </li></ul></ul></ul><ul><li>Data Inputs </li></ul><ul><ul><li>Estimated new SLOC </li></ul></ul><ul><ul><li>Estimated Reused SLOC </li></ul></ul><ul><ul><li>Estimated modified SLOC </li></ul></ul><ul><ul><li>Estimated total SLOC </li></ul></ul>
  15. 15. Software Size Metric Cont. <ul><li>Tailoring Ideas </li></ul><ul><ul><li>Delete SLOC types not applicable. </li></ul></ul><ul><ul><li>Separate data reporting for each coding language used. </li></ul></ul><ul><ul><li>Required separate reporting for each processor and/or CSCI. </li></ul></ul><ul><ul><li>Report object code size. </li></ul></ul><ul><li>Interpretation Notes </li></ul><ul><ul><li>Should not vary from previous reporting period by more than 5%. </li></ul></ul><ul><ul><li>SLOC vary by more than 5% form previous reporting period. </li></ul></ul><ul><ul><ul><li>Software Developer provides detail explanation. </li></ul></ul></ul><ul><ul><ul><li>Related discussion regarding cost and schedule improvements. </li></ul></ul></ul><ul><ul><li>Total SLOC does not linearly relate to effort. </li></ul></ul><ul><ul><li>New SLOC requires different effort than reuse or modified. </li></ul></ul>
  16. 16. Software Size Metric Example
  17. 17. Software Personnel Metric <ul><li>Purpose </li></ul><ul><ul><li>Tracks planned staffing to maintain level and maintain sufficient staffing to complete task on schedule. </li></ul></ul><ul><li>Behavior </li></ul><ul><ul><li>Too few experience personnel will experience difficulties. </li></ul></ul><ul><ul><li>Bringing many personnel at later phases will experience difficulties. </li></ul></ul><ul><ul><li>Normal shape of total staffing profile: </li></ul></ul><ul><ul><ul><li>Grow through design phases. </li></ul></ul></ul><ul><ul><ul><li>Peak through coding and testing phases. </li></ul></ul></ul><ul><ul><ul><li>Gradually tapered off through integration phases. </li></ul></ul></ul><ul><ul><li>Normal shape of experience staff profile: </li></ul></ul><ul><ul><ul><li>High during initial stages of the project. </li></ul></ul></ul><ul><ul><ul><li>Dip slightly during CSU development. </li></ul></ul></ul><ul><ul><ul><li>Grow during testing. </li></ul></ul></ul>
  18. 18. Software Personnel Metric Cont. <ul><li>Data Input </li></ul><ul><ul><li>Initial </li></ul></ul><ul><ul><ul><li>Planned total personnel level for each month of the contract. </li></ul></ul></ul><ul><ul><ul><li>Planned experience personnel level for each month of the contract. </li></ul></ul></ul><ul><ul><ul><li>Expected attrition rate. </li></ul></ul></ul><ul><ul><li>Each reporting period </li></ul></ul><ul><ul><ul><li>Total personnel. </li></ul></ul></ul><ul><ul><ul><li>Experience personnel. </li></ul></ul></ul><ul><ul><ul><li>Unplanned personnel losses. </li></ul></ul></ul><ul><li>Tailoring Ideas </li></ul><ul><ul><li>Report staffing separately for each development task. </li></ul></ul><ul><ul><li>Report staffing separately for special development skills needed. </li></ul></ul><ul><ul><li>Report staffing separately for each development organization. </li></ul></ul>
  19. 19. Software Personnel Metric Cont. <ul><li>Interpretation Notes </li></ul><ul><ul><li>Understaffing result in schedule slippage. </li></ul></ul><ul><ul><li>Adding staff too late will seldom improve schedule and often cause more delays. </li></ul></ul><ul><ul><li>High experience personnel turnover rate will cause schedule delays. </li></ul></ul><ul><ul><li>Initial staffing level should be at least 25% of average staffing level. </li></ul></ul>
  20. 20. Software Personnel Metric Example
  21. 21. Software Volatility Metric <ul><li>Purpose </li></ul><ul><ul><li>Tracks changes in requirements. </li></ul></ul><ul><ul><li>Tracks developers understanding of the requirements. </li></ul></ul><ul><ul><ul><li>Software Action Items (SAI). </li></ul></ul></ul><ul><li>Behavior </li></ul><ul><ul><li>More changes in requirements during requirement analysis and preliminary design phases. </li></ul></ul><ul><ul><li>Changes after CDR may have significant impact on schedule. </li></ul></ul><ul><ul><li>SAIs expected to rise at each review then tapered off exponentially. </li></ul></ul><ul><ul><ul><li>Clear and complete specification produce less level of rise at each review. </li></ul></ul></ul><ul><ul><ul><li>Good communications among developers and customers will have higher rate of decay. </li></ul></ul></ul>
  22. 22. Software Volatility Metric Cont. <ul><li>Data Inputs </li></ul><ul><ul><li>Current total number of requirements. </li></ul></ul><ul><ul><li>Cumulative number of requirements changes (addition, deletion and modification). </li></ul></ul><ul><ul><li>Number of new SAIs </li></ul></ul><ul><ul><li>Cumulative number of open SAIs </li></ul></ul><ul><li>Tailoring Ideas </li></ul><ul><ul><li>Track longevity of SAIs. </li></ul></ul><ul><ul><li>Track open SAIs by priority. </li></ul></ul><ul><li>Interpretation Notes </li></ul><ul><ul><li>Requirements volatility between CDR and TRR could cause significant schedule impact. </li></ul></ul><ul><ul><li>SAIs open for more than 60 days should be closely examined. </li></ul></ul>
  23. 23. Software Volatility Metric Example
  24. 24. Software Volatility Metric Example
  25. 25. Computer Resource Utilization Metric <ul><li>Purpose </li></ul><ul><ul><li>Tracks changes in estimated/actual computer utilization in target machine resources. </li></ul></ul><ul><ul><ul><li>CPU, Memory and IO </li></ul></ul></ul><ul><li>Behavior </li></ul><ul><ul><li>Most system experience upward creep in resource utilization. </li></ul></ul><ul><ul><li>Large system typically reserve 50% of resource for growth. </li></ul></ul><ul><ul><li>Dependencies among resource results in parallel movements of resources. </li></ul></ul>
  26. 26. Computer Resource Utilization Metric Cont. <ul><li>Data Inputs </li></ul><ul><ul><li>Initial </li></ul></ul><ul><ul><ul><li>Planned spare for each resource. </li></ul></ul></ul><ul><ul><li>Each reporting period </li></ul></ul><ul><ul><ul><li>Estimated/actual percentage of CPU utilization. </li></ul></ul></ul><ul><ul><ul><li>Estimated/actual percentage of memory utilization. </li></ul></ul></ul><ul><ul><ul><li>Estimated/actual percentage of I/O channel utilization. </li></ul></ul></ul>
  27. 27. Computer Resource Utilization Metric Cont. <ul><li>Tailoring Ideas </li></ul><ul><ul><li>Report combined utilization in a multi resource architecture that uses a load-leveling operating system. </li></ul></ul><ul><ul><li>Report utilization separately in a multi resource architecture that has dedicated functions. </li></ul></ul><ul><ul><li>Report average and worst case utilization. </li></ul></ul><ul><ul><li>Report separately for development and target Processors. </li></ul></ul><ul><ul><li>Consider memory addressing limit of the architecture when establishing utilization limits. </li></ul></ul><ul><li>Interpretation Notes </li></ul><ul><ul><li>Performance deteriorates rapidly when utilization exceeds 70 percent for real time applications. </li></ul></ul><ul><ul><li>Planned for resource expansion. </li></ul></ul><ul><ul><li>Necessary optimization cause by resource usage approaching limit will increase cost and schedule. </li></ul></ul>
  28. 28. Computer Resource Utilization Metric Example
  29. 29. Schedule Progress Metric <ul><li>Purpose </li></ul><ul><ul><li>Tracks delivery of software work packages defined in the Work Break Down Schedule (WBS) against scheduled delivery. </li></ul></ul><ul><ul><li>Estimated Schedule (Months) = </li></ul></ul><ul><ul><li>Program Schedule (months) / (BCWP / BCWS) </li></ul></ul><ul><ul><li>BCWP - budgeted cost of work performed </li></ul></ul><ul><ul><li>BCWS - budgeted cost of work scheduled </li></ul></ul><ul><li>Behavior </li></ul><ul><ul><li>Tend to initially fall behind due to insufficient time allocated to the design process. </li></ul></ul><ul><ul><li>Likely to fall behind during testing due to inadequate test planning and testing at the CSU and CSC levels </li></ul></ul>
  30. 30. Schedule Progress Metric Cont. <ul><li>Data Inputs </li></ul><ul><ul><li>Initial </li></ul></ul><ul><ul><ul><li>Number of months in program schedule. </li></ul></ul></ul><ul><ul><li>Each reporting period </li></ul></ul><ul><ul><ul><li>BCWP for software. </li></ul></ul></ul><ul><ul><ul><li>BCWS for software. </li></ul></ul></ul><ul><ul><ul><li>Number of months in program schedule if revised. </li></ul></ul></ul><ul><li>Tailoring Ideas </li></ul><ul><ul><li>Track progress separately for each CSCI. </li></ul></ul><ul><li>Interpretation Notes </li></ul><ul><ul><li>Can be extrapolated to identify trends </li></ul></ul><ul><ul><li>If trend is up, it indicates a worsening condition. </li></ul></ul><ul><ul><li>If trend is down, productivity is under control and improving. </li></ul></ul>
  31. 31. Schedule Progress Metric Example
  32. 32. Imagix Metric Tool
  33. 33. Slim-Metric
  34. 34. Slim-Metric Cont.
  35. 35. Essential Metric
  36. 36. SEER SEM
  37. 37. Summary <ul><li>Metrics is a valuable management tool allowing management to exercise control during each phases of a software development process. </li></ul><ul><li>Metrics provide control by giving different views or visibility into each of the phases of a development process. </li></ul><ul><li>Metric analysis can identify trends which may have impact on cost and schedule. Early detection of trends allow for effective recovery planning. </li></ul>
  38. 38. Conclusions <ul><li>Metric Tools are used by both project managers and software developers. Project managers are more interested in planning and progress metrics. Software developers mainly focus on software specific metrics such as defects, cyclomatic complexity, SLOC, etc. </li></ul><ul><li>Metric Tools </li></ul><ul><ul><li>Essential Metric - </li></ul></ul><ul><ul><li>Imagix - </li></ul></ul><ul><ul><li>McCabeIQ - </li></ul></ul><ul><ul><li>SEER-SEM - </li></ul></ul><ul><ul><li>Semantic Design - </li></ul></ul><ul><ul><li>Slim Metric - </li></ul></ul><ul><ul><li>(Site with various metrics tools) - </li></ul></ul>
  39. 39. Conclusions Cont. <ul><li>At CMM Level 2 basic management control is installed and software costs, schedules and functionality are tracked. Therefore a limited set metric gathering occurs at CMM Level 2. But it is not until CMM Level 4, where process are measure and quality quantified, that a complete set of metrics is gathered and trends identified. </li></ul>
  40. 40. Acronyms <ul><li>SRR - System Requirement Review SDR - System Design Review SSR - System Specification Review PDR - Preliminary Design Review CDR - Critical Design Reviews TRR - Test Readiness Review </li></ul><ul><li> (Descriptions for each review) </li></ul>