MULTIMEDIA Cocomo forum version5

Measurement and Estimating
Models
for Software Maintenance
Workshop (COCOMO Forum)
November 2011

Approved for Public Release, Distribution A

Workshop Goals
• Three goals for workshop:
– Expose members of the cost estimating
community to recent Army software
maintenance study findings and
recommendations
– Gather feedback from the community on our
software maintenance WBS and initial
influence factor analysis
– Build consensus on what the most important
factors are via Delphi survey

• Build a community of interest in
maintenance that supports
development of new measures and
models for improved software
maintenance budgeting, estimation

Workshop Agenda
• Introductions
• Summary of study findings and
recommendations
• Study current activities/next
steps
• WBS review
• Influence factor review
• Delphi survey
• Roundtable discussions
• Summary and break

Background
• The changing defense environment has
placed a renewed emphasis on the
performance of U.S. Army software
maintenance, sustaining engineering,
and operational support efforts
• Accurate and objective cost estimates
are required to ensure that sufficient
resources are available to execute the
work required to keep systems
operational and mission capable
• To develop accurate estimates, the
Army has been working collaboratively
with the Air Force and the Navy to
collect and analyze past cost
performanceApproved for PublicandDistribution A
data Release, build a software

Goals of Army Maintenance Study
• Goal - provide the Army with objective
decision information to accurately
estimate, budget, and allocate the
software maintenance, sustaining
engineering, and operational support
resources (collectively referred to as
software maintenance resources)
required to meet evolving mission and
service affordability requirements
– Three years of effort to date focusing on the
work done and issues experienced by Army
and Air Force life cycle support centers

• Findings to-date seem universal across
the service weapons systems
community

Information Requirements
• Accurate estimates of software
maintenance resources mainly
for weapons systems from a:
– Product Perspective
– Organizational Perspective
– Enterprise Perspective
– Project and Release Perspective

• Objective portrayal of the
dynamic PDSS/PPSS maintenance
environment
– Consistent with work being
performedApproved for Public Release, Distribution A

Information Requirements
• Consistent description of
software maintenance tasks
– Software Maintenance
– Software Sustaining Engineering
– Software Support Infrastructure –
Facilities
– Program/Project Management

• Quantitative understanding of the
key performance factors that
influence software maintenance
resource estimation

Study Approach
• Direct interface with DoD and Industry
software support organizations
performing software maintenance tasks
under contract and at DOD Life Cycle
Support Centers:
– Task identification - definition - allocations
– Historical cost data collection - budget
submissions and actuals
– Organizational context data - performance
factors

• Stakeholder collaboration - protected
sharing of data, information, findings:
– Air Force, Army, Navy, DoD Agencies and
potentially allies
Approved for Public and Professional Groups
– Industry, Academia Release, Distribution A

Study Approach
• Develop approaches and mechanisms
to capture cost data and context
information architectures and data
stores
• Perform data modeling - analysis:
– Parametric cost model calibration - CER
development
– Performance “meta” model - factor
relationships
– Policy and decision information analysis

• Make recommendations for
improvements and increased
affordability
– Develop software with maintenance in mind
– Smooth the transition from development to

Army Study Approach
• Investigate maintenance
–
–
–
–
–

What are the tasks?
Who does them?
What are the costs?
How they are estimated?
What impacts future costs?

• Understand
–
–
–
–

Current costs and risks
Current estimating practice
Current budgeting approach
What changes are needed

Expected Outputs
• Context-driven software maintenance
performance model - configurable to product,
organization, and enterprise activities and
scope
• Calibrated cost models for maintenance and
cost estimating relationships for ACAT 1A
systems
– Multiple task categories
– Domain specific with validated cost relationsh ips

• Software maintenance information
architecture - common service database
• Changes to policy and budgetary guidance
needed to support systems transitions and
workload growth
• Software maintenance reporting
requirements like forthe SRDR A
Approved
Public Release, Distribution

Performance Meta-Modeling
No. of Releases
Done in Parallel

Complexity

Domain

Who – Contractor
Or Government?

Funding
Approach

Constraints and Influence Factors
Color
of
Money

Work Model
•Maintenance workload
•Sustaining engineering tasks
•Infrastructure tasks
•Program/Project Mgmt tasks


Project
Factors
-Process
-People
-Product

Summary of Study Findings
• Over 250 projects
surveyed
- Eight Army and AF
Centers visited
- Over 100 interviews
- Industry consulted

• Findings
- Maintenance centers
do more than just
updates and repairs

• Products
- Reports, papers,
briefings, etc.
- Web site
- Initial maintenance
cost and quality
database

• Distribution of work
much different than
expected
• Testing is the major
maintenance activity
• Transition and
transfer is done
poorly
• Estimates and
budgets don’t cover
all the work
-

Sustaining engineering
Product field & user
support
Regression testing

• Efficiencies are


Army Study Findings
• Maintenance is done differently by
contractors than government shops
– Contractors develop to requirements,
government supports testing and field
support

• As a minimum, four software
releases are being prepared by
maintenance shops in parallel
during the calendar year
– Development release

- Fielded release

– To be fielded release
release

- Requirements

• Work for software maintenance
differs from development – more

Current Work Distribution
Notes
• About seventy percent
of their work involves:
- Maintenance
- Sustaining Engineering
- Independent V&V

• The other thirty percent
is devoted to other
tasks:
- Acquisition
management
- Software development

• Maintenance staff
includes both
government and inhouse contractor
personnel

Testing is Primary Maintenance Activity
• As much as 55-70%
of the technical
work done during
maintenance
supports retesting
and qualifying the
system

10%

20%

55%

• Testing is much
harder when
developers fail to
transition and
turnover the needed
set of regression
tests
• Support tasks are
performed to
maintain system
integrity and

Not All Of The Work Is Funded
• Estimates formulated • Resulting budgets
based on effort
force maintenance
needed to make
staff to play backlog
updates and repairs
reduction games
• Other activities like
• Shops make the
sustaining
updates and repairs
engineering and
that they can with
testing not fully
resources allocated
covered
• Cost models & cost
• Unfunded mandates
estimating
like Info Assurance
relationships (CERs)
not adequately
used perpetuate
covered
status quo
• Small projects done
• Shortfalls in funding
on LOE basis
need to be corrected
• Licenses may Approved for Public Release, Distribution A
need

Development System Not Ready for
Maintenance
• Transition requirements often
waived, avoided or delayed
- Facilities, tactical equipment and tools
often not available when needed
- Ownership rights to tools and special
test equipment often an issue
- Development SIL seldom transitioned for
maintenance

• Many aspects of “technical debt”
are not addressed
• Contractor often the only
resource available to maintain
system

On-Going Tasks
• Stakeholder
- Questionnaire/instrume
Program

• Data collection

nts
- Maintenance cost and
quality database
- Data administration,
protection and
management

• Architecture
development
- Information needs
- Data modeling

• Analysis
- Gap analysis
- Indicators

“Working one-onone”
- Web site
- Case studies
- Outreach
▪ Collaborators
▪ Conferences
▪ Presentations

- Publications
- Working groups

• Project
management
- Status and


Going Forward
• Characterize software
maintenance, sustaining
engineering, and operational
support
– Understand commonalities and
differences among services, domains,
programs, and maintenance
organizations
– Clarify the differences between
perceptions and realities

• Coordinate the efforts of the
Services and industry to collect
relevant information
– Understand thePublic Release, Distribution A
Approved for characteristics of post

Collaborative Working Group
• Develop working group with
government and industry to explore
the identified issues and provide
recommendations
– Requires collecting data to support the
findings
– Data collection must be done hands-on to
reduce noise and increase confidence in
results

• Develop viable software maintenance
cost estimation methods, models, and
practices
• Work with academia and the DAU to
improve the education of the
workforce on the Release, Distribution A
realities of software
Approved for Public

In Summary
• We invite you to
participate in our
joint efforts
• We ask you to help
us populate our
software
maintenance cost
database
• We are interested in
any success stories
you would like to
share

WBS Revision
• In order to understand the factors
that impact maintenance cost, we
are developing a performance
meta-model
• When discussing the meta-model,
we will summarize tasks around
four major activities:
– Software maintenance
– Software sustaining engineering
– Software support infrastructure &
facilities
– Program/project management

WBS Revision
Contractor
•Typically mimic software development
contracts
•Often require delivery to government
sites that handle distribution, certify the
software, perform field support and
perform test and evaluation

Government
•Perform maintenance with real
operational equipment and boots on the
ground
•Maintenance is only part of the tasks
they perform (often fix hardware, do
acquisition support, Release, Distribution A
etc.)
Approved for Public

WBS Revision
Software Maintenance:
1.1 Software maintenance
1.1.1
1.1.2
1.1.3
1.1.4
1.1.5
1.1.6
1.1.7
1.1.8

Release requirements
Release plannin g
Architecture analysis
Hardware defect repair
Software defect repair
Hardware enhancements
Software enhancements
Release integration & test

1.1.9 Release qualification & delive ry

WBS Revision
1.3 Independent test and
verification
1.3.1
1.3.2
1.3.3
1.3.4

Test planning
Test preparation
Test conduct
Independent analysis &
verification
1.3.5 Certifications

1.5 Information assurance
1.5.1 Protection services
1.5.2 DIACAP
1.5.3 IAVA

WBS Revision
Software Sustaining Engineering
1.2 Sustaining engineering
1.2.1
1.2.2
1.2.3
1.2.4

Analysis and studies
Emergency repairs
User training
External support

1.6 Acquisition support
1.7 Operational support
1.9 Field support


WBS Revision
Software Support Infrastructure &
Facilities
1.4 Product support
1.4.1 Configuration management
1.4.2 Quality assurance
1.4.3 Process management (peer reviews)
1.4.4 Supplier management
1.4.5 Security
1.8 Facility support
1.8.1 Maintenance facility sustainment
1.8.2 SIL sustainment
1.8.3 Equipment sustainment
1.8.4 Specialized test equipment and tools
1.8.5 Network operations and

WBS Revision
1.11 Parts
1.12 Spares
1.13 Licenses

Program/Project Management
1.10 Management
1.10.1 Release management
1.10.2 Sustaining engineering
management
1.10.3 Risk management
1.10.4 Measurement analysis
1.14 Contractual capabilities set FY
(XX/XX)
1.15 Contractual system mission
capability

Influence Factors
• Business Factors
– Extent of policy coverage - governance
– Low level technical & business decision
autonomy
– Diverse organizational task and activity
portfolios
– Program and domain characteristics
– Product and data rights
– Source and color of money
– Resourcing business models
– Estimation/budgeting approaches
– Information system capabilities

Influence Factors
• Complexity factors
– Legacy software architectures
– Legacy software technologies
– Backfit security requirements
– Backfit safety and other certification
requirements
– System of system integration
requirements


Influence Factors
• Resource and task alignment factors
– Policies, budgets, resources, tasks and
outputs
– Autonomous personnel and funding
decisions
– Plan versus execution
– Alignment of task and funding models
– Management reserves (reduced
allocations)
– Top-level expectations versus realities
– Overhead versus direct funded functions
– Amount of “technical debt”

Influence Factors
• Execution factors
– Event-driven requirements and
reprioritizations
– Short-term mission driven execution
schedules
– Multiple customers – direct user
involvement
– Multiple funding streams
– No. of releases being done in parallel
(using same resources)
– Backlog at start of release
– Uncertainty of planning parameters
– Organizational capability – flexibility Approved for Public Release, Distribution A

Size Drivers
Enterprise

Release
•No. of change
requests
•No. of repairs
– Defects by priority
and type
– No. of patches

•Backlog (technical
debt)
– Defect by priority,
age and type

•No. of COTS
packages updated

•No. of programs/
projects competing
for resources
•No. of releases
being supported by
program/project
•Stability of releases
over time as
measured by change
rate
•Stability of core
funding


Maintenance Effort Multipliers
• People
– Analyst capability
– Programmer
capability
– Personnel continuity
– Applications
experience
– Platform experience
– Language and tool
experience

• Product
– Platform/domain type
– Application type

– Language type
– Requirements
volatility (change)
– Product complexity
– Data base size
– Required reliability
– Degree of reuse
– Documentation
match to needs
– Execution time
constrain
– Main storage
constrain
– Platform volatility


Effort Multiplers
• Project

• Process

– Degree of
precedentedness
– Development
flexibility
– Architecture/risk
resolution
– Team cohesion
– Use of software tools
– Multi-site capability
– Required
development
schedule
– Resource dedication

– Acquisition method
– Development
method
– Development
standard
– Use of modern
programming
practices
– Process maturity
– Process volatility
(change)

• Other


Effort Multipliers
• Because of time limitations, we
will not look at size and effort
multipliers for:
– Software sustaining engineering
– Software support infrastructure and
facilities
– Program/project management

• We will address these at next
year either by email or at another
conference like PSM

Delphi Survey
• Goal – Determine what factors that
you believe have greatest impact on
software maintenance projects
• Scope:
– ID tasks which your maintenance shop
performs
– ID factors to which their effort is most
sensitive
– Schedule not considered because it is
fixed in maintenance
– Size of project considered because many
maintenance projects are small

• Can view influence factors as
constraints imposed on either

Roundtable Discussion
• What did you think
were the three
biggest influence
factors?
• What do you believe
are the three drivers
that the
maintenance effort
is most sensitive to?


In Summary
• We have summarized the
results of our study
• We have reviewed the
maintenance WBS that we
have developed
• We have conducted a
Delphi to identify the
influence factors and cost
drivers important to
estimating
• We will summarize findings
and present them on Friday
• If you wish to be on our
distribution, let us know

This is why Don lives
in Prescott, AZ
In

MULTIMEDIA Cocomo forum version5

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to MULTIMEDIA Cocomo forum version5

Similar to MULTIMEDIA Cocomo forum version5 (20)

More from philipsinter

More from philipsinter (10)

Recently uploaded

Recently uploaded (20)

MULTIMEDIA Cocomo forum version5

Editor's Notes