This document discusses software measurement and selecting useful measures. It provides examples of measures that NASA software managers have found useful, including measures tracking progress, functionality, quality, requirements, staffing and more. The document emphasizes selecting measures that help manage projects by answering questions and providing early warnings of potential issues.

1. Are Your Software Measures
Useful?
Or
Are You Getting Your Money’s
Worth From Your Measures?
Date: 02/09/10 Used with Permission Slide 1

2. Agenda
• Why Measure?
• How Should Measures
# Activities
Be Selected?
• Are There Measurement
Requirements?
• Which Measures Are
Useful?
Date: 02/09/10 Slide 2

3. Why Measure? (1 of 2)
• For the benefit of your current project
– Use objective measurement data to plan, track, and correct
project
– Verify that technical/quality requirements have been met
• For the benefit of your future projects (and the rest of your
organization, too!)
– Help create basis for planning future projects
– Create a baseline of “typical” or “expected” performance
– Provide a subset of your measures to your organization
Date: 02/09/10 Slide 3

4. Why Measure? (2 of 2)
• Because it’s required!
– Comply with NPR 7150.2 measurement requirements
– Tools can help meet these requirements
• Selection and evaluation of new technology
– Develop a rationale for adopting/refining technologies
– Assess the impacts of techniques
• Process Improvement
– Determine the strengths and weaknesses of the current
process and product
– Assess impacts of improvements
Date: 02/09/10 Slide 4

5. Why Measure?
Measures Help You Manage!
Measure Analyze Report
Point
I’m on plan
Counts
Where am I? or
Staffing What should I’m varying
Schedule I do now? from plan
or
Risk
I’m off plan
Defects and need
help!!!!
Changes
Date: 02/09/10 Slide 5

6. How to Select Measures
• Preferred methods for guiding measurement selection:
– The Goal-Question-Metric (GQM) method or
– The Goal-Question-Indicator-Metric (GQIM) method
• First, decide what your measurement “goals” are:
– What information are you trying to learn?
– What decisions will require additional information?
• Next, what “questions” will provide the information to meet your goal
– Each goal may require several questions
• Then decide what types of charts or “indicators” might show the answers to
the questions
– Data at points in time, trends, stoplight charts ….
• Now ---What “metrics” or measures are required for those charts?
Date: 02/09/10 Slide 6

7. An Example: Selecting Measures for
Progress Tracking (1 of 3)
• Goals or Objectives
– Keep cost and schedule within acceptable bounds
– Monitor effort by process activity to assure correct staffing
• Possible Questions:
– Will the project finish on schedule?
– Can the project stay within its allotted budget without
reducing requirements?
– Is there enough staffing in the right areas?
• Possible Indicators:
– Would a stoplight chart tell me what I need?
– If I see a problem, do I have enough other information to fix
it?
Date: 02/09/10 Slide 7

8. An Example: Selecting Measures for
Progress Tracking (2 of 3)
• Typical measures – Planned and actual data for …
– Event dates
– Progress tracking points
– Budget and staff effort
– Effort for each process activity
• Selection rules
– Require at least one measure of schedule progress and one measure
of cost or effort
– Need process measures for some CMMI process areas
– Start small – know how you are going to use the measures
Date: 02/09/10 Slide 8

9. An Example: Selecting Measures for
Software Progress Tracking (3 of 3)
Budget and Schedule How will I know if I’m falling behind?
Objectives
Planned and Actual Staffing
Baselined: Oct-04 ABC Monthly Planned vs Actual/Projected Staffing-Project Funded Only Actuals As Of: Oct-05
30.00
25.00
How will I know if I have enough
Staff-Months per Month (FTEs)
20.00
resources? The right ones?
15.00
10.00
Scheduled Activities 5.00
0.00
Nov- Dec- Jan- Feb- Mar- Apr- May- Jun- Aug- Sep- Oct- Nov- Dec- Jan- Feb- Mar- Apr-
Jul-05
04 04 05 05 05 05 05 05 05 05 05 05 05 06 06 06 06
Planned / Mo 1.60 1.80 1.60 1.85 3.55 4.40 9.05 13.55 23.25 23.25 23.40 24.85 24.85 24.85 24.85 24.85 24.85 24.85
Projected / Mo 1.60 1.80 1.60 2.40 3.80 4.20 6.50 10.55 17.10 18.10 16.85 16.15 24.85 24.85 24.85 24.85 24.85 24.85
Actual / Mo 1.60 1.80 1.60 2.40 3.80 4.20 6.50 10.55 17.10 18.10 16.85 16.15
Planned and Actual Progress Points
SDO Build 1 Test As Of Nov-15-2005
Point Counting Status
1800
1600
1400
1200
Plan
1000 Actual
Points
Baseline
800 Upper
Lower
600
400
200
0
6/29/2004
7/29/2004
8/29/2004
9/29/2004
10/29/2004
11/29/2004
12/29/2004
1/29/2005
2/28/2005
3/29/2005
4/29/2005
5/29/2005
6/29/2005
7/29/2005
8/29/2005
9/29/2005
Week Ending
Date: 02/09/10 Slide 9

10. Measurement Requirements
(As Per NPR 7150.2)
• Required measurement areas for all software projects
– Software Progress Tracking
– Software Functionality
– Software Quality
– Software Requirements Volatility
– Software Characteristics
• Additional NPR requirements for Class A and B projects
– Process monitoring as required for CMMI Generic Practice 2.8
– Data specified for Software Inspection/Peer Review Report
– Data collected “on a CSCI basis”
Date: 02/09/10 Slide 10

11. Examples of Required Measures
(1 of 2)
Software Progress Tracking
Software Functionality
Date: 02/09/10 Slide 11

12. Examples of Required Measures
(2 of 2)
Software Volitility
Software Quality
Date: 02/09/10 Slide 12

13. Charts for Discussion
So----
What Did the Software Managers Submit as
“Useful” Charts?
Date: 02/09/10 Slide 13

14. Progress Tracking:
Development Progress
Analysis: Progress has fallen below threshold due to 1) SS1: More effort than planned to address DR/ERs 2) SS2: Lead developer delayed due to office
move and change of development platform and 3) SS3 and SS$: No development staff available
Impact: 1) SS1 will defer some of their testing until release 2.5 2) SS2 will not implement all defect fixes as planned and 3) SS3 and SS4 will not be
included in the April release as planned (Not required project deliverables)
Corrective Action: 1) Reschedule the non-implemented tasks and 2) Work with component leads to redefine the work for SS3 and SS4.
Date: 02/09/10 Slide 14

15. Progress Tracking:
18 Month Staffing Chart
Analysis: Lost one of SS1 developers but received additional staffing this month. Staffing is still less than planned on other components
Impact: Additional staffing needs to come up to speed and this may affect the scheduled April release.
Corrective: Let the customers know of the staff changes.
Date: 02/09/10 Slide 15

16. Progress Tracking:
Requirements Implementation Status
This chart was used in an MMR by the software manager of a FSW project.
Baselined L4 requirements have been through MRR and are being used by module developers. These will be verified in
SW I&T.
Date: 02/09/10 Slide 16

17. Progress Tracking:
Earned Value –Schedule Variance
Date: 02/09/10 Slide 17

18. Progress Tracking:
Earned Value –Cost Variance
Date: 02/09/10 Slide 18

19. Status Tracking:
Other Direct Costs (ODC)
Multiple Replans Due To Delayed Funding!
Date: 02/09/10 Slide 19

20. Progress Tracking:
Overall RID Status
Have 103 of 449 RIDs left to be
implemented. (Less than 25% of
RIDs remain unimplemented)
Date: 02/09/10 Slide 20

21. Requirements Volatility
XXX
Date: 02/09/10 Slide 21

22. Quality Example - Defect Data
R2002.1 As of 7-Jan Example Status
PLOG
Opened Closed Active Expected R1 Expected R2
180
160
140
120
PLOG Count
100
80
60
40
20
0
11/8/2001 11/15/2001 11/22/2001 11/29/2001 12/6/2001 12/13/2001 12/20/2001 12/27/2001 1/3/2002
Status: More than expected number of reports, but opening and closing at about the same rate until near
the end.
Impact: None since we are not gaining or losing ground.
Corrective Action: Additional attendees are now in the code walkthroughs and test inspections.
Date: 02/09/10 Slide 22

23. Quality/Progress Tracking:
Software Defect Status
Expect 1386 work hours to repair
defects
- 25% increase from November
- V&A and CA activities only
- 8.2 hours on average to complete
(Based on institutional)
- 169 defects are in these states
Days open are starting to increase
again
Date: 02/09/10 Slide 23

24. Progress Analysis:
Defect Status by State
This chart was requested by line management to help determine the amount of work to be performed by the line organization.
Defect life-cycle states above the line are to be worked by the implementing organization. States below the line are to be
worked by the project.
DPFR-Development Problem Failure Report; PFR-Problem Failure Report; PRS-Problem Reporting System
Date: 02/09/10 Slide 24

25. Quality Analysis:
Defect Containment
PDR CDR
This metric will be shown at the conclusion of each Formal Review. The
purpose of this metric is to show all of the “out of phase” defects in the
Project Work Products. There typically should be a very low percentage
of out of phase RIDs, if the documents have been adequately reviewed
prior to the formal reviews.
Date: 02/09/10 Slide 25

26. Quality Analysis:
Need Better Code Inspections?
Date: 02/09/10 Slide 26

27. Quality/Progress:
RID Monthly Status
Document
Suspense Date Suspense Date RIDs Awaiting
Total Number of RIDs 12/03/2008 01/21/2010 RIDs Open to Date Concurrence RIDs Recommending Closure RIDs Closed
Requirement Document 60 59 1 0 0 0 59
System Design Document 13 8 5 5 0 0 8
Verification and Validation Plan 14 7 7 7 0 0 7
Date: 02/09/10 Slide 27

28. Functionality:
Deliveries as Promised?
The due dates for
some milestones
have been realigned.
Appear to be on
track to meet the
realigned schedule,
Date: 02/09/10 Slide 28

29. Technology Assessments: Are Inspections
More Effective For Certain Defects?
Date: 02/09/10 Slide 29

30. Summary
• NASA software managers found a variety of measures
useful
• They chose measures to:
– Help them manage their projects---
– Answer their questions----
– Give them early indicators of possible problems
– Improve their processes
– Evaluate potential technologies
• Decide what you want to know and get busy measuring it!
Date: 02/09/10 Slide 30

31. Contact Information
Sally Godfrey
Metrics Subgroup Lead, NASA Software Working Group
NASA/Goddard Space Flight Center
Code 585
Greenbelt, MD 20771
301-286-5706
Sara.H.Godfrey@nasa.gov
Date: 02/09/10 Slide 31

32. Back-Up Slides
Date: 02/09/10 Slide 32

33. Date: 02/09/10 Slide 33

34. Errors Found in System Testing
Need More Focus On Code Logic?
Date: 02/09/10 Slide 34

35. Process Document Peer
Review Metric
Burn-down chart for the Requirements/Design
Section Peer Review Comments. All 59 comments
were ready to verify as of 6/26/09.
Date: 02/09/10 Slide 35

36. Requirements Stability Example
Requirements Stability
800
700
Number of Requirements
600
500
400
300
200
100
0
Oct-04 Nov-04 Dec-04 Jan-05 Feb-05 Mar-05 Apr-05 May-05 Jun-05 Jul-05 Aug-05 Sep-05
Oct 04 Nov 04 Dec 04 Jan 04 Feb 04 Mar 04 Apr 04 May 04 Jun 04 Jul 04 Aug 04 Sep 04
Changes 0 27 53 37 8 16 0 0
Additions 0 78 127 78 22 0 0 0
Deletions 0 0 0 53 0 0 0 0
Baseline 475 475 553 680 705 727 727 727 727 727 727 727
TBDs 125 125 66 10 5 5 5 5
Requirements Volatility Status:
Status: Requirements have begun to stabilize since the SRR in February but 5 TBDs remain.
• Requirements have begun to stabilize since the SRR in February. We are continuing to work with the
Impact: One TBD could cause significant impact.
CorrectiveProject to resolve the remaining 5the Project to resolve all remaining TBDs before the PDR incause significant
Action: We are working closely with TBDs before the PDR in August, one of which could August. Status is
impact.
reported weekly.
Date: 02/09/10 Slide 36