The Software Resource Data Report provides DoD cost analysts with a primary source of comprehensive, objective, and model-independent sizing, effort and schedule data on large DoD software programs. The SRDR does not rely on rigid, standardized data fields for collecting software effort and sizing information (a feature advocated by Government, industry, and academia during the design of the SRDR). This flexibility lessens the reporting burden placed on industry and reduces ‘data synthesis’ errors. Consequently, this flexibility creates the need for developing and using sound techniques for analyzing and interpreting, and ultimately normalizing data from multiple DoD weapon system contractors, each whom use different terms and definitions for software sizing and effort. This need for sound techniques to normalize software data from multiple developers will become more critical as the amount of data available to DoD analysts from SRDRs grows over the next decade. Analysts must properly analyze, interpret, and, ultimately, normalize the data before drawing any ‘industry-average’ conclusions such as average productivity. This presentation updates our investigation for the Deputy Assistant Secretary of the Army- Cost and Economics on the methods and techniques for building a normalized software database. We used SRDRs, supplemented with data separately collected for the Army. We believe these techniques discussed in this presentation can serve as a basis of best practices for analyzing SRDR data (and other data collected separately) in the future. Our presentation discusses (1) the analytical approach, (2) problems we encountered with the data (3) results of our analysis (4) a summarized and sanitized view of the normalized data and (5) recommendations for building a sound software industry-average database.

1. Initial Results Building a Normalized
Software Database using SRDRS
2009 ISPA/SCEA
Professional Development and Training Workshop
St. Louis, MO
2 – 5 June 2009
Michael Gallo
Paul Hardin
Elizabeth Koza
Robert Bailey
Sponsor: Deputy Assistant Secretary of the Army for Cost & Economics
p p y y y

2. Background
• The Army is collecting software data Army s
Army’s Desired Uses of the Data
to build a database
• Productivity factors
• Two primary collection sources
• Parametric estimating equations
– SRDRs
– Army internal collection • Calibration of commercial
• Current research is focused on software cost models
weapon system software, not AIS
p y • Sizing estimates
programs • Visualizing trends
• Research approach • Sanity checks
– Minimize contractor’s effort to
report the data (i.e. maximize use of
artifacts used internally by the
contractors)
– Avoid the use of subjective data fields
j
Slide 2
Presented at 2009 ISPA/SCEA Workshop

3. Selected SRDR Data Element Summary
Section 1 - General Context Section II – Product Description
• System/Element Name • Functional Description
• Report as-of date
p • Software Development
Characterization
Ch t i ti
• Authorizing Vehicle • Application Type
• Development Organization – Primary and Secondary Programming
language
• Software Process Maturity – Percentage of Overall Product Size
• Precedents – Development Process
– Upgrade or New Development?
• SRDR Data Dictionary – SW Development Method
Filename
Fil • Non-Developmental Software
• Comments – COTS/GOTS Applications Used
– Integration Effort (Optional)
• Staffing
– Peak Staff
– Peak Staff Date
– Hours per Staff-Month
• Personnel Experience by Domain
• Comments
Slide 3
Presented at 2009 ISPA/SCEA Workshop

4. Selected SRDR Data Element Summary
Section III – Product Size • Effort (staff-hours)
• Requirements Counts • Effort must be partitioned into a
– Total Software Requirements set of activities
– N Software R i
New S f Requirements • For each SW activity reported
the contractor must provide:
– Total External Interface – WBS Element reference
Requirements – Start Month
– New External Interface – End Month
d o t
Requirements
R i – Prime Contractor hours
– Requirements Volatility – All Other Sub-ctr hours
• Total Delivered Code Count
– New Code
– Reused With Modifications
– Reused Without Modifications
– Carryover Code
– Auto-generated C d
A d Code
– Sub-contractor Code
– Counting Convention
• Comments
+ SRDR Data Dictionary
Slide 4
Presented at 2009 ISPA/SCEA Workshop

5. The Problem
• Data is collected from a variety of contractors and
product mission areas and typically used and
reflected as ‘industry averages’
• Contractors use different definitions for reporting
size, effort, and schedule data
ff
– There is no universal standard or mandate for software
accounting and metric data
– Data sources such as the SRDR permit contractors to
tailor the report to the contractor’s internal accounting
and metrics systems
• Before the data can be used across
projects/developers it must be made comparable
j /d l i b d bl
Slide 5
Presented at 2009 ISPA/SCEA Workshop

6. Approach For Normalizing SW Data
1. Review d quality
1 R data l
2. Identify both prevalent sizing metric in the data and
prevalent sizing categories used
3. Identify prevalent software activities included in reported
‘chunks’ of effort, especially activities that are reported
discretely.
4. Formulate a series of estimating equations that estimate
constituent activities buried within reported ‘chunks’;
chunks ;
Review context for important variables that can be used to
explain differences in the data; derive coefficients
5. Apply estimating equations relatively to estimate missing
pieces and to break apart chunks into discrete activities
Slide 6
Presented at 2009 ISPA/SCEA Workshop

7. Review Data Quality
• Understand the SW product
• Raw data points were • Mission, function and complexity of the software
filtered to remove data •
•
Platform and operating environment of the software
Understand what programming languages were used
deemed inadequate • Understand the development project
U d d h d l
– Characterization of the development work
– Missing or incomprehensible – Understand how the software product is put
definitions together
• How it’s integrated
g
– No size or effort reported • How much was built with reused components
• How much was auto-generated
– No language reported • Understand who developed it
– No counting convention – Primes
– Subs
S b
reported • Understand what’s in the data reported
– Reported only Total SLOC – Scope of effort reported (what’s
included/excluded)
– Foreign Military Sales (FMS)
g y ( ) – Understand the units of sizing and rules for sizing
g g
categories
• Many of the data points • Understand other attributes that might drive
cost/schedule/quality
removed came from a
3rd party
Slide 7
Presented at 2009 ISPA/SCEA Workshop

8. Identify Prevalent Sizing Metric and Categories
• What are the basic sizing units of 200,000
measure in the dataset? 180,000
New + .00 * Modified + .00 * Unmodified (i.e. New SLOC)
New + 1.00 * Modified + .10 * Unmodified
– What is the prevalent counting 160,000
160 000
New + 1.00 * Modified + 1.00 * Unmodified (i.e. Total SLOC)
convention? Total SW Development
Hours = 4E-30*ESLOC
5.9339
140,000
– What are the programming
velopment Hours
120,000
languages used?
Total SW Development
0.496
Hours = 216.5*ESLOC
100,000 Total SW Development
1.1955
• What sizing categories are used?
Total SW Dev
Hours = 0 0307*ESLOC
H 0.0307*ESLOC
80,000
– Varying definitions are used 60,000
– Additionally, Auto-Generated, 40,000
Carryover, Deleted, COTS 20,000
• ESLOC weights have a significant 0
0 100 200 300 400 500 600
effect on the normalized data and
KESLOC
will ultimately influence Each represents one project, but with a
different computation of that project’s ESLOC.
– Computed productivity
(ESLOC/hr)
– Observed effort vs. Size trend
Slide 8
Presented at 2009 ISPA/SCEA Workshop

9. Identify Prevalent SW Activities in the Data
1 Detailed Design + Coding + Unit Testing Sys Req + Req Def + Integration + FQT SW PM + SQE + CM Meetings
2 DIT CTS CPTO LIM MIS CFIN REQ
Q SBVT Demo Q
QA CM
SW Requirements Analysis SW Coding and Unit Testing SW Developmental T&E Other
Systems Engineering Software Engineering Integration & Test Database Mgmt & Direct Spt Functions Subcontractor Effort
3
SW Requirements SW Coding and Unit SW Integration and System Integration
System Eng Requirements Analysis and Arch SW Design Management, Support, and Labs
Analysis and Arch Testing Test and Test
SW Architecture and SW Coding and Unit Systems and System
4 SW Requirements Analysis Independent Test Group Metrics, SCM, Documentation & Other Dept Support Efforts
Detailed Design Testing Test
5 Requirements Design, Code, Test System Integration SW CM SW Qual SW Mgmt
6 Ada (Staff-months) Ada83 (Staff-months) Jovial (Staff-months) Jovial/ASM (Staff-months)
SW Requirements Preliminary Design Detailed Design Code & Unit Test Mega Level Test (MLT) Element Level Test (ELT)
7
Detailed Design + Code & Unit Test + Mega Level Test + Element Level Test + PIV Defects
8 Design and Document Review Interface Design Documents Coding Code Reviews Integration and Test Other Documentation
SW Architecture and SW Coding and Unit Software Qualification
SW Requirements Analysis SW Integration and System/SW Integration SW Developmental T&E Other
Detailed Design Testing Testing
9 Detailed Design Iterations, SW/SW Integration SW/HW Integration Support to Systems, SDP,
Requirements Analysis High Level Design, PDR Code & Unit Test SCM, Sys Admin, VDD SQE
CDR & OCSD Testing Testing Test, ER & ILS Management
System Requirements Definition Implementation SW/HW Test System Integration System Verification CM, QA,
CM QA PM
Source
SW Coding and Unit SW Architecture and SW Int. and System/SW SW Qualification
10 SW Requirements Analysis SW Development Test and Evaluation CM, SW Safety, SW Process Improvement
Testing Detailed Design Int. Testing
SW Coding and Unit SW Architecture and SW Int. and System/SW
11 SW Requirements Analysis SW Qualification Testing Mgmt, SW CM, SW Process, System Admin Spt, Subcontract Mgmt Spt
Testing Detailed Design Int.
SW Coding and Unit
12 SW Requirements Analysis SW Architecture and Detailed Design SW Int. and System/SW Int. SW Qualification Testing
Testing
13 Systems Engineering Activities SW Engineering Activities SW Integration Activities
SW Requirements Analysis
q y SW Design/Code/Test & Integration
g g SW Quality Assurance
Q y SW Configuration Mgmt
g g SW Environment Support
pp
SW Requirements Analysis SW Design/Code Test and Integration Other mgmt and spt (build related)
SW Test and
14 SW Requirements Analysis SW Design/Code Help Desk and User Spt Other mgmt and spt (build related)
Integration
SW Coding and Unit SW Integration and System/Software
SW Eng Requirements Analysis SW Architecture and Detailed Design SW Development Test and Evaluation CM, SW QA, and Dev Environment
Testing Integration
SW Coding and Unit SW Architecture and
15 SW Requirements Analysis SW Integration and System/SW Integration SW DT&E SW Environmental Dev: SW Mgmt, CM, SW QM, Environmental Dev
Testing Detailed Design
SW Architecture and SW Integration and
16 SW Requirements Analysis SW Coding SW Unit Testing SW Qualification Testing Build Specific Systems Development
Detailed Design Test
17 SW Requirements Analysis SW Design SW Implementation and Unit Testing SW Verification QA, SCM, SW Environment
18 SW Requirements Analysis SW Architecture and Detailed Design SW Coding and Unit Testing SW Integration and System/SW Integration
SW Architecture & Detailed SW Coding & Unit
19 SW Requirements Analysis SW Integration & System/SW Integration SW Developmental Test & Evaluation Spiral Level Leadership Other Direct Hours
Design Testing
20 SW Requirements + Preliminary Design + Detailed Design + Code & Unit Test + SW I&T
21 Requirements Analysis Model Development Design C&UT Integration & Test FQT PM Data QA DevEnv CM & QA
Each is a discretely reported ‘chunk’ of software activity.
Activities in blue font reflect data from an SRDR source
Slide 9
Presented at 2009 ISPA/SCEA Workshop

10. Identify SW Activities in the Data
1 Detailed Design + Coding + Unit Testing Sys Req + Req Def + Integration + FQT SW PM + SQE + CM Meetings
2 DIT CTS CPTO LIM MIS CFIN REQ SBVT Demo QA CM
SW Requirements Analysis SW Coding and Unit Testing SW Developmental T&E Other
Systems Engineering Software Engineering Integration & Test Database Mgmt & Direct Spt Functions Subcontractor Effort
3
SW Requirements SW Coding and Unit SW Integration and System Integration
System Eng Requirements Analysis and Arch SW Design Management, Support, and Labs
Analysis and Arch Testing Test and Test
SW Architecture and SW Coding and Unit Systems and System
4 SW Requirements Analysis Independent Test Group Metrics, SCM, Documentation & Other Dept Support Efforts
Detailed Design Testing Test
5 Requirements Design, Code, Test System Integration SW CM SW Qual SW Mgmt
6 Ada (Staff-months) Ada83 (Staff-months) Jovial (Staff-months) Jovial/ASM (Staff-months)
SW Requirements Preliminary Design Detailed Design Code & Unit Test Mega Level Test (MLT) Element Level Test (ELT)
7
Detailed Design + Code & Unit Test + Mega Level Test + Element Level Test + PIV Defects
8 Design and Document Review Interface Design Documents Coding Code Reviews Integration and Test Other Documentation
SW Architecture and SW Coding and Unit Software Qualification
SW Requirements Analysis SW Integration and System/SW Integration SW Developmental T&E Other
Detailed Design Testing Testing
9 Detailed Design Iterations, SW/SW Integration SW/HW Integration Support to Systems, SDP,
Requirements Analysis High Level Design, PDR Code & Unit Test SCM, Sys Admin, VDD SQE
CDR & OCSD Testing Testing Test, ER & ILS Management
System Requirements Definition Implementation SW/HW Test System Integration System Verification CM, QA, PM
Source
e
SW Coding and Unit SW Architecture and SW Int and System/SW SW Qualification
Int.
10 SW Requirements Analysis SW Development Test and Evaluation CM, SW Safety, SW Process Improvement
Testing Detailed Design Int. Testing
SW Coding and Unit SW Architecture and SW Int. and System/SW
11 SW Requirements Analysis SW Qualification Testing Mgmt, SW CM, SW Process, System Admin Spt, Subcontract Mgmt Spt
Testing Detailed Design Int.
SW Coding and Unit
12 SW Requirements Analysis SW Architecture and Detailed Design SW Int. and System/SW Int. SW Qualification Testing
Testing
13 Systems Engineering Activities SW Engineering Activities SW Integration Activities
SW Requirements Analysis SW Design/Code/Test & Integration SW Quality Assurance SW Configuration Mgmt SW Environment Support
SW Requirements Analysis SW Design/Code Test and Integration Other mgmt and spt (build related)
SW Test and
14 SW Requirements Analysis SW Design/Code Help Desk and User Spt Other mgmt and spt (build related)
Integration
SW Coding and Unit SW Integration and System/Software
SW Eng Requirements Analysis SW Architecture and Detailed Design SW Development Test and Evaluation CM, SW QA, and Dev Environment
Testing Integration
SW Coding and Unit SW Architecture and
15 SW Requirements Analysis SW Integration and System/SW Integration SW DT&E SW Environmental Dev: SW Mgmt, CM, SW QM, Environmental Dev
Testing Detailed Design
SW Architecture and SW Integration and
16 SW Requirements Analysis SW Coding SW Unit Testing SW Qualification Testing Build Specific Systems Development
Detailed Design Test
17 SW Requirements Analysis SW Design SW Implementation and Unit Testing SW Verification QA, SCM, SW Environment
18 SW Requirements Analysis SW Architecture and Detailed Design SW Coding and Unit Testing SW Integration and System/SW Integration
SW Architecture & Detailed
A hit t D t il d SW Coding & Unit
C di U it
19 SW Requirements Analysis SW Integration & System/SW Integration SW Developmental Test & Evaluation Spiral Level Leadership Other Direct Hours
Design Testing
20 SW Requirements + Preliminary Design + Detailed Design + Code & Unit Test + SW I&T
21 Requirements Analysis Model Development Design C&UT Integration & Test FQT PM Data QA DevEnv CM & QA
Blue font = SRDR Data Source
Slide 10
Presented at 2009 ISPA/SCEA Workshop

11. Graphical View of the ‘Chunks’
nment
System Requirements
s
nt
Development Environ
Process Improvemen
User Documentation
Independent Testing
Acceptance Testing
Integration Testing
Quality Assurance
Formal Integration
Code & Unit Test
Detailed Design
System Testing
System Design
Requirements
Project Plans
Initial Design
Architecture
Safety
IV&V
CM
PM
2 2 2 2 1 1 2 2 2 2 3 3 3,4 3
2 2 2 2 1 1 2 2 2 2 3 3 3,4 3
2 2 2 2 1 1 2 2 2 2 3 3 3,4 3
4 1 1 1 1 1 2 6 5 8 3 3 7
7 1 1 1 1,4 1 2 9 8 11 3,5,6 3,5,6 10
7 1 1 1 1,4 1 2 9 8 11 3,5,6 3,5,6 10
7 1 1 1 1,4 1 2 9 8 11 3,5,6 3,5,6 10
1 2 2 2 3 4 4 6 6 6 5
1,2,3,
1 1 2 3 4 5 6 4,5,6
1 1 1 1 1
9,11,
11,13, 7,13, 7,13, 12,14,1 8,13,
1 2,3 3 4 5 5 6 14,15 14,15 14,15 5 15 5 10,12
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 2 2 2 3 4 4 4 5 6 6 6 6
1 2 2 2 3 4 4 5 7 7 7 7 7 7 6
1 2 2 2 3 4 4 5 5 6 6 6 6 6 6
1 2 2 2 3 4 4 5 5 6 6 6 6 6 6
1 2 2 2 3 4 4 5
1 1 1 2 2 2 3 1 2 1
1 1 2 2 3 3 4 5 6 7 7 7 7
7 1 2 2 2 3,4 5 5 6
1 2 2 2 3 4 4 4 4 5 5 4 5
1 1 1 2 2 2 2 3 3 3 3 4 4 4
1 1 1 2 2 2 2 3 3 3 3 4 4 4
1 2 2 2 3 4 4 4
1 1 2 2 2 3 3 4 6 6 5
1 1 1 2 2 2 3 3 4 5 5 6 6 6
1 1 1 2 2 2 3 3 4 5 5 6,7 6,7 6
= Discretely reported activity = Activity is included in a ‘chunk’ total = Activity was not reported
Slide 11
Presented at 2009 ISPA/SCEA Workshop

12. Some Approaches to Normalize Effort
• Filter to include projects with the same
set of activities
– Need many data points Changes in Distribution of Effort
– Or filter to maximize the # of activities
100%
(results in a small # of projects) 90% PM, CM, QA, Data
– Or Filter to maximize the # of projects
80%
(results in a small # of activities)
70%
SW and System
I&T
60%
• Use a factor to fill in missing activities
% of Total Effort
E
50%
– Can’t derive a factor in isolation to 40% Code & Unit Test
add/remove activities because: 30%
Design
– Factor can change as size increases 20%
– Need unique factors for each unique 10%
System and SW
Requirements
‘chunk’ 0%
10,000 50,000 100,000 500,000 1,000,000 5,000,000 10,000,000
• Derive non-linear equation in isolation to
Development Size (Equivalent New HOL)
add/remove activities Source: VERA
– Basically,
Basically filtering on activities reported discretely
(greens)
– Multiple projects must have reported activity
discretely
• Would result in dropping data that
contain actuals that are included in chunks
(yellows)
Slide 12
Presented at 2009 ISPA/SCEA Workshop

13. Our Analytical Approach
• Derive an effort estimating relationship (EER) for each software development activity Use the
activity.
EERs to break apart the chunks (yellows) & estimate the missing activities (reds)
• Relies on both the discretely reported activities (greens) and information buried in the chunks.
nvironment
System Requirements
ement
ation
sting
sting
ing
Quality Assurance
on
st
Formal Integratio
Process Improve
User Documenta
Independent Tes
Development En
Code & Unit Tes
Acceptance Tes
Integration Testi
Detailed Design
System Testing
System Design
Requirements
Project Plans
Initial Design
Architecture
Safety
IV&V
CM
PM
2 2 2 2 1 1 2 2 2 2 3 3 3,4 3
2 2 2 2 1 1 2 2 2 2 3 3 3,4 3
2 2 2 2 1 1 2 2 2 2 3 3 ,
3,4 3
4 1 1 1 1 1 2 6 5 8 3 3 7
7 1 1 1 1,4 1 2 9 8 11 3,5,6 3,5,6 10
7 1 1 1 1,4 1 2 9 8 11 3,5,6 3,5,6 10
7 1 1 1 1,4 1 2 9 8 11 3,5,6 3,5,6 10
1 2 2 2 3 4 4 6 6 6 5
1,2,3,
1 1 2 3 4 5 6 4,5,6
1 1 1 1 1
, ,
9,11,
11,13, 7,13, 7,13, 12,14,1 8,13,
1 2,3 3 4 5 5 6 14,15 14,15 14,15 5 15 5 10,12
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 1 1 1 1
1 2 2 2 3 4 4 4 5 6 6 6 6
1 2 2 2 3 4 4 5 7 7 7 7 7 7 6
1 2 2 2 3 4 4 5 5 6 6 6 6 6 6
1 2 2 2 3 4 4 5 5 6 6 6 6 6 6
1 2 2 2 3 4 4 5
1 1 1 2 2 2 3 1 2 1
1 1 2 2 3 3 4 5 6 7 7 7 7
7 1 2 2 2 3,4 5 5 6
1 2 2 2 3 4 4 4 4 5 5 4 5
1 1 1 2 2 2 2 3 3 3 3 4 4 4
1 1 1 2 2 2 2 3 3 3 3 4 4 4
1 2 2 2 3 4 4 4
1 1 2 2 2 3 3 4 6 6 5
1 1 1 2 2 2 3 3 4 5 5 6 6 6
1 1 1 2 2 2 3 3 4 5 5 6,7 6,7 6
Slide 13
Presented at 2009 ISPA/SCEA Workshop

14. General Formulation of Approach
• Each ‘chunk’ consists of a set of estimated activities
chunk ET = Total Reported Project Effort
ECH =
ˆ (∑ E )
ˆ
i
ET = Total Estimated Project Effort
ˆ
E = Estimated Chunk Effort
ˆ
• Total estimated project effort consists of a set of estimated chunks
CH
Ei = Estimated effort of activity i
ˆ
ET =
ˆ (∑ E )
ˆ
CH
• Each effort estimating relationship (EER) provides an estimate for one activity
Ei = Ai [ki (ESLOCi )bi * (EAF )
ˆ * (EAF2 ) * (EAF3 ) * (EAF4 )
Platform Dev_Type CMM SystemEffe
ct
1 ]
where Challenges Encountered
• Not enough data points. 211
points
Ai = 1 if activityi is includedin total effort (ET ); else Ai = 0
unknowns, but only 168 d.o.f. in
ESLOCi = EquivalentNew Lines of Code for Activityi the data
Platform = 1 if platformis Air; else Platform = 0 • ‘Partial’ programs drove analysis
Developer Type = 1 if single developeror subcontrac else DeveloperT = 0
tor; ype to illogical results
CMM = CMM Levelof Developer- Avg CMM Levelof Database • Required significant tailoring
SystemEffe = 1 if systemis part of larger developmen project;else SystemEffe = 0
ct t ct – Reduced SLOC
categories
• ESLOC is specified uniquely for each activity – Common ESLOC
weights
m
ESLOC = New * CF + ∑ Not _ New * CF * w
i i j i ij
j =1 – CCommon D O S for
D.O.S. f
where CFi = Counting Convention Factor groups of activities
w i = Equivalent new fractional cost
j = SLOC categories that are considered " Not New"
• Use constrained optimization to solve for coefficients that minimize residuals of estimated
effort at both the project level and chunk level
Slide 14
Presented at 2009 ISPA/SCEA Workshop

15. Initial Results
Effort = A * ESLOCB*CPLATFORM*DDEV-TYPE*E(CMM-4 23)
• Eff (CMM-4.23)
1 ESLOC
SW Activity A B C D E SW Activity
New Mod Reused Carryover Autogen
Requirements 0.00470 1.1100 1.6 0.89 0.91 Requirements 1.0000 0.5624 0.2200 0.0420 0.0400
Architecture 0.00110
0 00110 1.1733
1 1733 1.6
16 0.89
0 89 0.91
0 91 Architecture 1.0000
1 0000 0.5624
0 5624 0.2200
0 2200 0.0420
0 0420 0.0400
0 0400
Initial Design 0.00190 1.1733 1.6 0.89 0.91 Initial Design 1.0000 0.5624 0.2200 0.0420 0.0400
Detailed Design 0.00450 1.1733 1.6 0.89 0.91 Detailed Design 1.0000 0.5624 0.2200 0.0420 0.0400
Code & Unit Test 0.03000 1.0893 1.6 0.89 0.91 Code & Unit Test 1.0000 0.5624 0.2200 0.0420 0.0400
Formal Integration 0.00400 1.2393 1.6 0.89 0.91 Formal Integration 1.0000 0.5624 0.2200 0.0420 0.0400
Integration Testing 0.01540 1.2393 1.6 0.89 0.91 Integration Testing 1.0000 0.5624 0.2200 0.0420 0.0400
System Testing 0.00450 1.2393 1.6 0.89 0.91 System Testing 1.0000 0.5624 0.2200 0.0420 0.0400
Acceptance Testing 0.00370 1.2393 1.6 0.89 0.91 Acceptance Testing 1.0000 0.5624 0.2200 0.0420 0.0400
Configuration Mgt 0.00540 1.1116 1.6 0.89 0.91 Configuration Mgt 1.0000 0.5624 0.2200 0.0420 0.0400
Project Plans 0.00100 1.1116 1.6 0.89 0.91 Project Plans 1.0000 0.5624 0.2200 0.0420 0.0400
Program Mgt 0.00390 1.1116 1.6 0.89 0.91 Program Mgt 1.0000 0.5624 0.2200 0.0420 0.0400
Q
Quality Assurance
y 0.00800 1.1116 1.6 0.89 0.91 Q
Quality Assurance
y 1.0000 0.5624 0.2200 0.0420 0.0400
1
Included in Normalized Database SLOC = Logical Lines of Code
Average Percent
PRED(20)
Error
Data Pts Data Pts Data Pts Data Pts
Included Excluded Included Excluded
Project
j 4% 200% 80% 20%
Chunk 169% 344% 26% 9%
Slide 15
Presented at 2009 ISPA/SCEA Workshop

16. Initial Results (Cont’d)
700,000 Full Development (Included in Solution)
600,000 Partial Development (Exlcluded from Solution)
Std, Nor Total Proje Effort
Aggregate EER Estimate Trend
500,000
ect
400,000
300,000
Data points are normalized to
rm
common set of activities and
f d
200,000
standardized to remove effects of
Air Platform, CMM, and Multi-
100,000
vendor development.
0
0 100,000 200,000 300,000 400,000 500,000
Standardized Avg ESLOC
Slide 16
Presented at 2009 ISPA/SCEA Workshop

17. Application of EERs
Breaking Apart Chunks (“Yellows”) Estimating Missing Activities (“Reds”)
EER i j EER i
Factori = Factori =
∑ EER i j ∑ EER i
where where
EER ij = EER Estimate of activity i in chunk j EER i = EER Estimate of activity i
∑ EER i j
= Estimate of all activities that are included in chunk j ∑ EER i = Estimate of all activities that are included in the actual project effort
E i = Factor i * E CH j
ˆ E i = Factori * E Project
ˆ
where E CH j = Actual effort of chunk j where E Project = Actual Project Effort
• E ti ti equations are used relatively t
Estimating ti d elati el to
add/remove activities from normalized effort
Slide 17
Presented at 2009 ISPA/SCEA Workshop