Measurement and Estimating
Models
for Software Maintenance
Workshop (COCOMO Forum)
November 2011

Approved for Public Rele...
Workshop Goals
• Three goals for workshop:
– Expose members of the cost estimating
community to recent Army software
maint...
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next
steps
• WB...
Background
• The changing defense environment has
placed a renewed emphasis on the
performance of U.S. Army software
maint...
Goals of Army Maintenance Study
• Goal - provide the Army with objective
decision information to accurately
estimate, budg...
Information Requirements
• Accurate estimates of software
maintenance resources mainly
for weapons systems from a:
– Produ...
Information Requirements
• Consistent description of
software maintenance tasks
– Software Maintenance
– Software Sustaini...
Study Approach
• Direct interface with DoD and Industry
software support organizations
performing software maintenance tas...
Study Approach
• Develop approaches and mechanisms
to capture cost data and context
information architectures and data
sto...
Army Study Approach
• Investigate maintenance
–
–
–
–
–

What are the tasks?
Who does them?
What are the costs?
How they a...
Expected Outputs
• Context-driven software maintenance
performance model - configurable to product,
organization, and ente...
Performance Meta-Modeling
No. of Releases
Done in Parallel

Complexity

Domain

Who – Contractor
Or Government?

Funding
A...
Summary of Study Findings
• Over 250 projects
surveyed
- Eight Army and AF
Centers visited
- Over 100 interviews
- Industr...
Army Study Findings
• Maintenance is done differently by
contractors than government shops
– Contractors develop to requir...
Current Work Distribution
Notes
• About seventy percent
of their work involves:
- Maintenance
- Sustaining Engineering
- I...
Testing is Primary Maintenance Activity
• As much as 55-70%
of the technical
work done during
maintenance
supports retesti...
Not All Of The Work Is Funded
• Estimates formulated • Resulting budgets
based on effort
force maintenance
needed to make
...
Development System Not Ready for
Maintenance
• Transition requirements often
waived, avoided or delayed
- Facilities, tact...
On-Going Tasks
• Stakeholder
- Questionnaire/instrume
Program

• Data collection

nts
- Maintenance cost and
quality datab...
Going Forward
• Characterize software
maintenance, sustaining
engineering, and operational
support
– Understand commonalit...
Collaborative Working Group
• Develop working group with
government and industry to explore
the identified issues and prov...
In Summary
• We invite you to
participate in our
joint efforts
• We ask you to help
us populate our
software
maintenance c...
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next
steps
• WB...
WBS Revision
• In order to understand the factors
that impact maintenance cost, we
are developing a performance
meta-model...
WBS Revision
Contractor
•Typically mimic software development
contracts
•Often require delivery to government
sites that h...
WBS Revision
Software Maintenance:
1.1 Software maintenance
1.1.1
1.1.2
1.1.3
1.1.4
1.1.5
1.1.6
1.1.7
1.1.8

Release requi...
WBS Revision
1.3 Independent test and
verification
1.3.1
1.3.2
1.3.3
1.3.4

Test planning
Test preparation
Test conduct
In...
WBS Revision
Software Sustaining Engineering
1.2 Sustaining engineering
1.2.1
1.2.2
1.2.3
1.2.4

Analysis and studies
Emer...
WBS Revision
Software Support Infrastructure &
Facilities
1.4 Product support
1.4.1 Configuration management
1.4.2 Quality...
WBS Revision
1.11 Parts
1.12 Spares
1.13 Licenses

Program/Project Management
1.10 Management
1.10.1 Release management
1....
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next
steps
• WB...
Influence Factors
• Business Factors
– Extent of policy coverage - governance
– Low level technical & business decision
au...
Influence Factors
• Complexity factors
– Legacy software architectures
– Legacy software technologies
– Backfit security r...
Influence Factors
• Resource and task alignment factors
– Policies, budgets, resources, tasks and
outputs
– Autonomous per...
Influence Factors
• Execution factors
– Event-driven requirements and
reprioritizations
– Short-term mission driven execut...
Size Drivers
Enterprise

Release
•No. of change
requests
•No. of repairs
– Defects by priority
and type
– No. of patches

...
Maintenance Effort Multipliers
• People
– Analyst capability
– Programmer
capability
– Personnel continuity
– Applications...
Effort Multiplers
• Project

• Process

– Degree of
precedentedness
– Development
flexibility
– Architecture/risk
resoluti...
Effort Multipliers
• Because of time limitations, we
will not look at size and effort
multipliers for:
– Software sustaini...
Delphi Survey
• Goal – Determine what factors that
you believe have greatest impact on
software maintenance projects
• Sco...
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next
steps
• WB...
Roundtable Discussion
• What did you think
were the three
biggest influence
factors?
• What do you believe
are the three d...
Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next
steps
• WB...
In Summary
• We have summarized the
results of our study
• We have reviewed the
maintenance WBS that we
have developed
• W...
Upcoming SlideShare
Loading in …5
×

MULTIMEDIA Cocomo forum version5

380 views

Published on

Published in: Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
380
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
4
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Do not put titles on the red banner. Only put presentation name/date as it is indicated here.
  • MULTIMEDIA Cocomo forum version5

    1. 1. Measurement and Estimating Models for Software Maintenance Workshop (COCOMO Forum) November 2011 Approved for Public Release, Distribution A
    2. 2. Workshop Goals • Three goals for workshop: – Expose members of the cost estimating community to recent Army software maintenance study findings and recommendations – Gather feedback from the community on our software maintenance WBS and initial influence factor analysis – Build consensus on what the most important factors are via Delphi survey • Build a community of interest in maintenance that supports development of new measures and models for improved software maintenance budgeting, estimation Approved for Public Release, Distribution A
    3. 3. Workshop Agenda • Introductions • Summary of study findings and recommendations • Study current activities/next steps • WBS review • Influence factor review • Delphi survey • Roundtable discussions • Summary and break Approved for Public Release, Distribution A
    4. 4. Background • The changing defense environment has placed a renewed emphasis on the performance of U.S. Army software maintenance, sustaining engineering, and operational support efforts • Accurate and objective cost estimates are required to ensure that sufficient resources are available to execute the work required to keep systems operational and mission capable • To develop accurate estimates, the Army has been working collaboratively with the Air Force and the Navy to collect and analyze past cost performanceApproved for PublicandDistribution A data Release, build a software
    5. 5. Goals of Army Maintenance Study • Goal - provide the Army with objective decision information to accurately estimate, budget, and allocate the software maintenance, sustaining engineering, and operational support resources (collectively referred to as software maintenance resources) required to meet evolving mission and service affordability requirements – Three years of effort to date focusing on the work done and issues experienced by Army and Air Force life cycle support centers • Findings to-date seem universal across the service weapons systems community Approved for Public Release, Distribution A
    6. 6. Information Requirements • Accurate estimates of software maintenance resources mainly for weapons systems from a: – Product Perspective – Organizational Perspective – Enterprise Perspective – Project and Release Perspective • Objective portrayal of the dynamic PDSS/PPSS maintenance environment – Consistent with work being performedApproved for Public Release, Distribution A
    7. 7. Information Requirements • Consistent description of software maintenance tasks – Software Maintenance – Software Sustaining Engineering – Software Support Infrastructure – Facilities – Program/Project Management • Quantitative understanding of the key performance factors that influence software maintenance resource estimation Approved for Public Release, Distribution A
    8. 8. Study Approach • Direct interface with DoD and Industry software support organizations performing software maintenance tasks under contract and at DOD Life Cycle Support Centers: – Task identification - definition - allocations – Historical cost data collection - budget submissions and actuals – Organizational context data - performance factors • Stakeholder collaboration - protected sharing of data, information, findings: – Air Force, Army, Navy, DoD Agencies and potentially allies Approved for Public and Professional Groups – Industry, Academia Release, Distribution A
    9. 9. Study Approach • Develop approaches and mechanisms to capture cost data and context information architectures and data stores • Perform data modeling - analysis: – Parametric cost model calibration - CER development – Performance “meta” model - factor relationships – Policy and decision information analysis • Make recommendations for improvements and increased affordability – Develop software with maintenance in mind – Smooth the transition from development to Approved for Public Release, Distribution A
    10. 10. Army Study Approach • Investigate maintenance – – – – – What are the tasks? Who does them? What are the costs? How they are estimated? What impacts future costs? • Understand – – – – Current costs and risks Current estimating practice Current budgeting approach What changes are needed Approved for Public Release, Distribution A
    11. 11. Expected Outputs • Context-driven software maintenance performance model - configurable to product, organization, and enterprise activities and scope • Calibrated cost models for maintenance and cost estimating relationships for ACAT 1A systems – Multiple task categories – Domain specific with validated cost relationsh ips • Software maintenance information architecture - common service database • Changes to policy and budgetary guidance needed to support systems transitions and workload growth • Software maintenance reporting requirements like forthe SRDR A Approved Public Release, Distribution
    12. 12. Performance Meta-Modeling No. of Releases Done in Parallel Complexity Domain Who – Contractor Or Government? Funding Approach Constraints and Influence Factors Color of Money Work Model •Maintenance workload •Sustaining engineering tasks •Infrastructure tasks •Program/Project Mgmt tasks Approved for Public Release, Distribution A Project Factors -Process -People -Product
    13. 13. Summary of Study Findings • Over 250 projects surveyed - Eight Army and AF Centers visited - Over 100 interviews - Industry consulted • Findings - Maintenance centers do more than just updates and repairs • Products - Reports, papers, briefings, etc. - Web site - Initial maintenance cost and quality database • Distribution of work much different than expected • Testing is the major maintenance activity • Transition and transfer is done poorly • Estimates and budgets don’t cover all the work - Sustaining engineering Product field & user support Regression testing • Efficiencies are Approved for Public Release, Distribution A
    14. 14. Army Study Findings • Maintenance is done differently by contractors than government shops – Contractors develop to requirements, government supports testing and field support • As a minimum, four software releases are being prepared by maintenance shops in parallel during the calendar year – Development release - Fielded release – To be fielded release release - Requirements • Work for software maintenance differs from development – more Approved for Public Release, Distribution A
    15. 15. Current Work Distribution Notes • About seventy percent of their work involves: - Maintenance - Sustaining Engineering - Independent V&V • The other thirty percent is devoted to other tasks: - Acquisition management - Software development • Maintenance staff includes both government and inhouse contractor personnel Approved for Public Release, Distribution A
    16. 16. Testing is Primary Maintenance Activity • As much as 55-70% of the technical work done during maintenance supports retesting and qualifying the system 10% 20% 55% • Testing is much harder when developers fail to transition and turnover the needed set of regression tests • Support tasks are performed to maintain system Approved for Public Release, Distribution A integrity and
    17. 17. Not All Of The Work Is Funded • Estimates formulated • Resulting budgets based on effort force maintenance needed to make staff to play backlog updates and repairs reduction games • Other activities like • Shops make the sustaining updates and repairs engineering and that they can with testing not fully resources allocated covered • Cost models & cost • Unfunded mandates estimating like Info Assurance relationships (CERs) not adequately used perpetuate covered status quo • Small projects done • Shortfalls in funding on LOE basis need to be corrected • Licenses may Approved for Public Release, Distribution A need
    18. 18. Development System Not Ready for Maintenance • Transition requirements often waived, avoided or delayed - Facilities, tactical equipment and tools often not available when needed - Ownership rights to tools and special test equipment often an issue - Development SIL seldom transitioned for maintenance • Many aspects of “technical debt” are not addressed • Contractor often the only resource available to maintain system Approved for Public Release, Distribution A
    19. 19. On-Going Tasks • Stakeholder - Questionnaire/instrume Program • Data collection nts - Maintenance cost and quality database - Data administration, protection and management • Architecture development - Information needs - Data modeling • Analysis - Gap analysis - Indicators “Working one-onone” - Web site - Case studies - Outreach ▪ Collaborators ▪ Conferences ▪ Presentations - Publications - Working groups • Project management - Status and Approved for Public Release, Distribution A
    20. 20. Going Forward • Characterize software maintenance, sustaining engineering, and operational support – Understand commonalities and differences among services, domains, programs, and maintenance organizations – Clarify the differences between perceptions and realities • Coordinate the efforts of the Services and industry to collect relevant information – Understand thePublic Release, Distribution A Approved for characteristics of post
    21. 21. Collaborative Working Group • Develop working group with government and industry to explore the identified issues and provide recommendations – Requires collecting data to support the findings – Data collection must be done hands-on to reduce noise and increase confidence in results • Develop viable software maintenance cost estimation methods, models, and practices • Work with academia and the DAU to improve the education of the workforce on the Release, Distribution A realities of software Approved for Public
    22. 22. In Summary • We invite you to participate in our joint efforts • We ask you to help us populate our software maintenance cost database • We are interested in any success stories you would like to share Approved for Public Release, Distribution A
    23. 23. Workshop Agenda • Introductions • Summary of study findings and recommendations • Study current activities/next steps • WBS review • Influence factor review • Delphi survey • Roundtable discussions • Summary and break Approved for Public Release, Distribution A
    24. 24. WBS Revision • In order to understand the factors that impact maintenance cost, we are developing a performance meta-model • When discussing the meta-model, we will summarize tasks around four major activities: – Software maintenance – Software sustaining engineering – Software support infrastructure & facilities – Program/project management Approved for Public Release, Distribution A
    25. 25. WBS Revision Contractor •Typically mimic software development contracts •Often require delivery to government sites that handle distribution, certify the software, perform field support and perform test and evaluation Government •Perform maintenance with real operational equipment and boots on the ground •Maintenance is only part of the tasks they perform (often fix hardware, do acquisition support, Release, Distribution A etc.) Approved for Public
    26. 26. WBS Revision Software Maintenance: 1.1 Software maintenance 1.1.1 1.1.2 1.1.3 1.1.4 1.1.5 1.1.6 1.1.7 1.1.8 Release requirements Release plannin g Architecture analysis Hardware defect repair Software defect repair Hardware enhancements Software enhancements Release integration & test 1.1.9 Release qualification & delive ry Approved for Public Release, Distribution A
    27. 27. WBS Revision 1.3 Independent test and verification 1.3.1 1.3.2 1.3.3 1.3.4 Test planning Test preparation Test conduct Independent analysis & verification 1.3.5 Certifications 1.5 Information assurance 1.5.1 Protection services 1.5.2 DIACAP 1.5.3 IAVA Approved for Public Release, Distribution A
    28. 28. WBS Revision Software Sustaining Engineering 1.2 Sustaining engineering 1.2.1 1.2.2 1.2.3 1.2.4 Analysis and studies Emergency repairs User training External support 1.6 Acquisition support 1.7 Operational support 1.9 Field support Approved for Public Release, Distribution A
    29. 29. WBS Revision Software Support Infrastructure & Facilities 1.4 Product support 1.4.1 Configuration management 1.4.2 Quality assurance 1.4.3 Process management (peer reviews) 1.4.4 Supplier management 1.4.5 Security 1.8 Facility support 1.8.1 Maintenance facility sustainment 1.8.2 SIL sustainment 1.8.3 Equipment sustainment 1.8.4 Specialized test equipment and tools 1.8.5 Network operations and Approved for Public Release, Distribution A
    30. 30. WBS Revision 1.11 Parts 1.12 Spares 1.13 Licenses Program/Project Management 1.10 Management 1.10.1 Release management 1.10.2 Sustaining engineering management 1.10.3 Risk management 1.10.4 Measurement analysis 1.14 Contractual capabilities set FY (XX/XX) 1.15 Contractual system mission capability Approved for Public Release, Distribution A
    31. 31. Workshop Agenda • Introductions • Summary of study findings and recommendations • Study current activities/next steps • WBS review • Influence factor review • Delphi survey • Roundtable discussions • Summary and break Approved for Public Release, Distribution A
    32. 32. Influence Factors • Business Factors – Extent of policy coverage - governance – Low level technical & business decision autonomy – Diverse organizational task and activity portfolios – Program and domain characteristics – Product and data rights – Source and color of money – Resourcing business models – Estimation/budgeting approaches – Information system capabilities Approved for Public Release, Distribution A
    33. 33. Influence Factors • Complexity factors – Legacy software architectures – Legacy software technologies – Backfit security requirements – Backfit safety and other certification requirements – System of system integration requirements Approved for Public Release, Distribution A
    34. 34. Influence Factors • Resource and task alignment factors – Policies, budgets, resources, tasks and outputs – Autonomous personnel and funding decisions – Plan versus execution – Alignment of task and funding models – Management reserves (reduced allocations) – Top-level expectations versus realities – Overhead versus direct funded functions – Amount of “technical debt” Approved for Public Release, Distribution A
    35. 35. Influence Factors • Execution factors – Event-driven requirements and reprioritizations – Short-term mission driven execution schedules – Multiple customers – direct user involvement – Multiple funding streams – No. of releases being done in parallel (using same resources) – Backlog at start of release – Uncertainty of planning parameters – Organizational capability – flexibility Approved for Public Release, Distribution A
    36. 36. Size Drivers Enterprise Release •No. of change requests •No. of repairs – Defects by priority and type – No. of patches •Backlog (technical debt) – Defect by priority, age and type •No. of COTS packages updated •No. of programs/ projects competing for resources •No. of releases being supported by program/project •Stability of releases over time as measured by change rate •Stability of core funding Approved for Public Release, Distribution A
    37. 37. Maintenance Effort Multipliers • People – Analyst capability – Programmer capability – Personnel continuity – Applications experience – Platform experience – Language and tool experience • Product – Platform/domain type – Application type – Language type – Requirements volatility (change) – Product complexity – Data base size – Required reliability – Degree of reuse – Documentation match to needs – Execution time constrain – Main storage constrain – Platform volatility Approved for Public Release, Distribution A
    38. 38. Effort Multiplers • Project • Process – Degree of precedentedness – Development flexibility – Architecture/risk resolution – Team cohesion – Use of software tools – Multi-site capability – Required development schedule – Resource dedication – Acquisition method – Development method – Development standard – Use of modern programming practices – Process maturity – Process volatility (change) • Other Approved for Public Release, Distribution A
    39. 39. Effort Multipliers • Because of time limitations, we will not look at size and effort multipliers for: – Software sustaining engineering – Software support infrastructure and facilities – Program/project management • We will address these at next year either by email or at another conference like PSM Approved for Public Release, Distribution A
    40. 40. Delphi Survey • Goal – Determine what factors that you believe have greatest impact on software maintenance projects • Scope: – ID tasks which your maintenance shop performs – ID factors to which their effort is most sensitive – Schedule not considered because it is fixed in maintenance – Size of project considered because many maintenance projects are small • Can view influence factors as constraints imposed on either Approved for Public Release, Distribution A
    41. 41. Workshop Agenda • Introductions • Summary of study findings and recommendations • Study current activities/next steps • WBS review • Influence factor review • Delphi survey • Roundtable discussions • Summary and break Approved for Public Release, Distribution A
    42. 42. Roundtable Discussion • What did you think were the three biggest influence factors? • What do you believe are the three drivers that the maintenance effort is most sensitive to? Approved for Public Release, Distribution A
    43. 43. Workshop Agenda • Introductions • Summary of study findings and recommendations • Study current activities/next steps • WBS review • Influence factor review • Delphi survey • Roundtable discussions • Summary and break Approved for Public Release, Distribution A
    44. 44. In Summary • We have summarized the results of our study • We have reviewed the maintenance WBS that we have developed • We have conducted a Delphi to identify the influence factors and cost drivers important to estimating • We will summarize findings and present them on Friday • If you wish to be on our distribution, let us know Approved for Public Release, Distribution A This is why Don lives in Prescott, AZ In

    ×