In an IT context companies struggling to increase profits and often view IT as a necessary evil: one that consumes resources rather contributes to the bottom line. These organizations often don’t see value in data collection analysis or benchmarking either. However, IT can be a significant contributor when IT decisions are made after measuring and estimating both cost and return. IT data collection analysis and benchmarking continue to improve the cost of IT systems and help make decisions regarding where to spend money to stop the bleeding. As such, repeatable processes for estimating cost, schedule and risk will be addressed along with the “iron triangle” of software. The Iron Triangle looks at issues of cost, schedule, scope and quality and helps determine what must give when client increases scope, reduces schedule or reduces budget. Additionally this presentation will address the risk adjusted Total Cost of Ownership and return an IT investment along with its consistency with long-range investment and business strategy of an organization measured against risk and key technical and performance parameters and technical debt.Finally, since this presentation will address the overriding business concerns: how much value does this software contribute to the business and is the best place to spend the money. (IT Confidence 2013, Rio de Janeiro (Brazil))

1. IT Data Collection, Analysis and
Benchmarking: From Processes and
Benchmarks into Real Business Value
Dan Galorath
galorath@galorath.com
Copyright 2013 Galorath Incorporated

2. Key Points
Measurement
needs to think
like business
leaders…
provide
information
understandable
to management
and other
stakeholders
Measurement
needs to
produce value
not cost in the
eyes of
management
and other
stakeholders
Measurement
results should
include
confidence and
risk so leaders
can understand
and include in
decisions

3. Context
© 2011 Copyright Galorath Incorporated

4. •Cutter Consortium Software Project Survey:
• 62% overran original schedule by more than 50%
• 64% more than 50% over budget
• 70% had critical product quality defects after release
• Standish Group CHAOS Report
• 46% challenged
• 19% failed
• 35% successful
~$875 billion spent on IT
~$300 billion spent on IT projects
~$57 billion wasted annually
ROI of better planning huge
IT Failures Are Pervasive: And Even
Successful Projects May Not Be
Some suggest 80% of systems NEVER produce a positive ROI

5. Human Nature: Humans Are
Optimists
HBR Article explains this Phenomenon:
• Humans seem hardwired to be optimists
• We routinely exaggerate benefits and discount costs
Delusions of Success: How Optimism Undermines
Executives' Decisions (Source: HBR Articles | Dan Lovallo
, Daniel Kahneman | Jul 01, 2003)
5
Solution - Temper with “outside view”:
Past Measurement Results, traditional forecasting,
and statistical parametrics can help
Don’t remove optimism, but balance optimism and
realism

6. Example of Tempering: German Mark V
Tanks (Source: How To Measure Anything)
• Allies estimated production by analyzing serial numbers
• Used the captured tanks as a random sample and predicted
probability of various production levels

7. Some of Dan’s Heroes
• Frederick Taylor: The Principals of Scientific
Management 1901 “Let data and facts do the talking”
• W. Edwards Demming: “In God We Trust… All Others
Bring Data”
• Frederick Brooks: “There is an incremental person when
added to a software project that makes it take longer”
• Ed Yourdon: “Avoiding Death Marches in Software
Projects”
• Steven Covey: “Sharpen the Saw” Focus on improvement

8. Measurement Manifesto For Software
& IT
Measurement Perspective
• We measure to ultimately
produce business value to
the organization
• MEASURED Measurement
should not ultimately be a
cost
• The analysis of
measurements produces
decisions that produce
business value
Management & Stakeholder
Perspective
• Speak so I can understand
• Give me actionable items
• Don’t just give me
problems… Give me
solutions
• Help me make the best
decisions so we can produce
business value
© 2011 Copyright Galorath Incorporated 8
Measurement itself is N ENABLER….
Management decision making, better performance, quality
improvements and better serving stakeholders is

9. Example: Project Cost Alone Is not
The Cost of IT Failure (Source: HBR)
• Case Study: Levi Strauss
• $5m ERP deployment contracted
• Risks seemed small
• Difficulty interfacing with customers systems
• Had to shut down production
• Unable to fill orders for 3 weeks
• $192.5M charge against earnings on a $5m IT project
failure
© 2011 Copyright Galorath Incorporated 9
“IT projects touch so many aspects of organization
they pose a new singular risk”
http://hbr.org/2011/09/why-your-it-project-may-be-riskier-than-you-think/ar/1

10. Exuding Confidence or
Concern With Measurement
© 2009 Copyright Galorath Incorporated 10

11. Frederick Brooks Classic Paper “No
Silver Bullets”
• “There is no single development, in either
technology or management technique, which by
itself promises even one order-of magnitude
improvement within a decade in productivity, in
reliability, in simplicity.”
• -- Fred Brooks, 1986
• i.e. There is no magical cure for the “software crisis”
• Not software measurement
• Not better tools
• Not ---- (fill in the blank… the
current great hope)

12. Size & Productivity
© 2011 Copyright Galorath Incorporated

13. Fundamental Metrics For
Estimation, Planning & Control
• Size
• AKA Volume, Mass
• Units: Source Lines of Code
(SLOC); Function Points
(FP) Use Cases
• New versus rework
• COTS & Packages
• Technology
• AKA Productivity Potential,
Efficiency
• Units: none
• Time
• AKA Duration, Schedule
• Units: Calendar Months,
Calendar Weeks
• Effort
• AKA Work, Labor
• Units: Staff Months, Staff
Hours
• Cost
• AKA Budget, Money
• Units: $, other currencies
• Staffing
• AKA Manpower Loading
• Units: FTE People
• Defects
• AKA Reliability, Quality
• Units: Defect Count

14. Many Size Measures Are Viable
• We know about:
• Lines of Code &
• COSMIC Both provide a method for Sizing Software
• Story Points (different since their values are team
defined) provide the same capability for User Stories or
Features or anything that can sized
• Many others
• Function Points
© 2012 Copyright Galorath Incorporated 14

15. A1-15
Size Is More Than Just Total
Application
NewNew
Effective Effort Units
Lines, Functions, Objects, Use Cases, Etc.Lines, Functions, Objects, Use Cases, Etc.
Pre-existingPre-existing

16. Reuse: Watch Out For Low Cost
Assumptions on “Heritage”
• Reuse or Heritage: applying existing software to a new
mission (or additional innovation in its current mission)
• Effort to reuse software is routinely under estimated
Design
Test
Implementation
Why should we care: Heritage is often underestimated
and causes major schedule / cost overruns

17. Sample Effort Factors for Reuse
© 2013 Copyright Galorath Incorporated 17

18. Software Design For Reuse (Lower
Cost Heritage)
• Designed for reuse
• Reusability requirements during original development
• Significant extra coding and documentation effort
during original development to insure reusability
• Minor to moderate work (mostly retest) required when
reused IF NOT MODIFIED
• Not designed for reuse
• Developed without extra effort to ensure reusability
• Sources include any legacy code not specifically
designed for reuse
• Other projects & programs
• Prior builds of the current program
• Moderate to major rework required
• Redesign
• Reimplementation

19. Rework Causes Software Project
Issues
• Rework: Doing the same work over again because it
was incorrect the first time
• Not prototyping
• Not refactoring (tuning to make software better)
• Between 40% and 50% of the total effort on software
projects is spent on rework.. Barry Boehm
• Initiatives to reduce rework can save significantly
Why should we care: Unplanned rework can drive cost up
dramatically AND reducing rework can be a boon to
projects

20. Customization
COTS or GOTS Hoped For Reuse
• Developmental
Software:
• Functionality
developed
specifically for the
project at hand
• May include
customization of
COTS
• “Glue” Code:
• Code written to bind
COTS to
developmental
software
• Development effort
must be captured
• COTS Software:
• Purchased functionality
• Direct Cost component of COTS
integration
• COTS Cognition:
• Required functionality within the COTS
software that must be understood
• Effort component of COTS integration
Developmental
Software
COTS Software
“COTS Cognition”
Glue
Why should we care: COTS promises are often not accurate
meaning additional development cost & Schedule

21. Packaged Applications Are Costly
To Organizations
• “Commercial application program or
collection of programs developed to meet
the needs of a variety of users, rather than
custom designed for a specific
organization”
• Many are enterprise applications
• Often allows / requires customization
• Examples: SAP; Rational PPM, SEER for
Software; Microsoft Excel, CA Clarity, Oracle
Business Suite
Why should we care: Packages sometimes comprise
solutions for parts of complicated systems and can be
trouble
"One-third [of the budget] has to go to testing. Don’t
ever short change testing. Everyone always underestimates it,
and says it’s the last thing to worry about. Don’t do that!“
- Jim Larson, consultant for communications solutions provider

22. Software Productivity Laws
• LAW 1 – Smaller teams are more efficient: The smaller the team
the higher the productivity of each individual person.
• LAW 2 – SOME schedule compression can be bought: Adding
people to a project, to a point, decreases the time and
increases the cost.
• LAW 3 – Every project has a minimum time: There is an
incremental person that consumes more energy than
he/she produces. Team size beyond this point decreases
productivity and increases time. (aka Brooks’ Law--”Adding
staff to a late software project makes it later.”)
• LAW 4 – Productivity is scalable: Projects of larger software
size can use larger teams without violating LAW 3.
• LAW 5 – Complexity limits staffing: As complexity increases, the
number of people that can effectively work on the project
and the rate at which they can be added decreases.

23. Software Productivity Laws
(continued)
• LAW 6 – Staffing can be optimized: There exists an optimal
staffing function (shape) that is effectively modeled by the
Rayleigh function. Flat (level load) staffing is rarely
optimal.
• LAW 7 – Projects that get behind, stay behind: It is extremely
difficult to bring a project that is behind schedule back on
plan.
• LAW 8 – Work expands to fill the available volume: It is possible to
allow too much time to complete a project (aka Parkinson’s
Law).
• LAW 9 – Better technology yields higher productivity: More capable
teams, better tools, and advanced, stable processes yield
higher productivity.
• LAW 10 – No “silver bullets”: There is no methodology, tool,
or process improvement strategy out there that yields
revolutionary improvements in project efficiency.

24. The Bottom Line
• Effort for reuse and rework must be considered
• Standard definitions of productivity e.g.
• 18 hours per function point…
• Is necessary but not sufficient for detailed estimates
that must be viable
• Use a range
• Identify key cost drivers that impact productivity
• Ensure you know what is IN and NOT IN the hours
© 2013 Copyright Galorath Incorporated 24

25. Relative Sizing Can Provide Quick ROM
Numbers E.G. ISBSG
• War and Peace is 587,287 words
• How long will it take you to read it?
• Using a different reference try…
• Girl with the Dragoon Tattoo =144,000
• The Bourne Identity = 80,000
• Harry Potter and the Philosopher's Stone
= 76,944
© 2012 Copyright Galorath Incorporated 25

26. Data Collection, Normalization, Usage,
Caution
© 2013 Copyright Galorath Incorporated
26

27. Before You Start - Some Basic Rules
1. Know from where the data comes (total, effective,
new, etc?)
2. Know what is included (Phases, People, Processes, …)
3. Know the different sizing schemas and how applied
4. Know what was intentionally left out (and why)
5. Consider the hindsight – if things happen they should
be accounted for in the data collection
6. Data needs to be consistent or normalized

28. Galorath’s Data Review Process
Management
is excluded
from actual
Concept is
excluded
from actual
CM / QA is
excluded
from actual
…management
from the estimate
…concept from the
estimate
…CM / QA from
the estimate
THE ACTUAL You should exclude...
Productivity
Reasonable ?
Is Effort
Accounting By
Phase Probably
Complete?
Summary Effort
Match Accounting
By Phase?
No No No
Adjust For
Missing
Phases
Is There Effort
Accounting By
Phase?
Yes
YesYesYes
No
Do Not
Adjust
Do Not
Adjust
Do Not
Adjust
Is
“Project Activity
Scope” field
Complete?
No
Do Not
Adjust
Yes
Productivity
Reasonable ?
Is Effort
Accounting By
Phase Probably
Complete?
Summary Effort
Match Accounting
By Phase?
No No No
Adjust For
Missing
Phases
Is There Effort
Accounting By
Phase?
Yes
YesYesYes
No
Do Not
Adjust
Do Not
Adjust
Do Not
Adjust
Is
“Project Activity
Scope” field
Complete?
No
Do Not
Adjust
Yes
Normalization
Mapping

29. Data Must Be Used With Caution
• Run sanity checks on data
• 10,000 Function Points can’t be developed in 3 months
• Ongoing issue between statisticians and engineers
• Some Statisticians claim.. “That is what the data says
so it must be right”
• Sometimes even if it is obviously wrong

30. Data Doesn’t Have To Be Perfect To Be
Useful: But Is Has To Be Viable
• 80 Calories per serving
• 2.5 Servings per can
• 4 Ounces, Condensed, 8 Ounces With Water

31. Data Improves Estimates For New
Programs Source: John Vu, Boeing SEPG 1997
.
0 %
140%
-140%
..
.
.
.
..
.
.
.
.
.
.
. .
. . . .
.
. .
.
. .
.
.
. . . .. . . . . .. .
..
. . . .. .
.
. .
. ..
.
.
.
. .. .. ...... . .. . ... . ..
.
.
. .
.
Without Historical Data With Historical Data
Variance between + 20% to - 145% Variance between - 20% to + 20%
(Efforts = Labor Hours)
(Mostly Level 1 & 2) (Level 3)
Over/UnderPercentage
.
(Based on 120 projects in
Boeing Information Systems)
.
. . .
.
.
.
.
..
.
. .
.
.
.
.
. .
.
.
.. .
. . .
. . . . . .. . . . . .. .
..
. . .
. . .
. . . .
. . . .. . . . .
. . . . .
. . . . .. . . . . .
. . . . .. . . . . .
. . . . . .. . . . . . . .
. . .
. . . . .
. . . . .
. . . .
. . . . . .
. . . . . .
. . . . . .
John Vu, Boeing, keynote talk at SEPG ‘97,
“Software Process Improvement Journey (From Level 1 to Level 5)”

32. Just the Top Level Data Isn’t
Sufficient For Estimation
Hours Versus UFPs
1
10
100
1000
10000
100000
1000000
1 10 100 1000 10000 100000
UFPs
Hours

33. Use Historical Data to evaluate your
estimate!

34. The Error of Causal Analysis
Is often encountered in data analysis!
NORMALIZE:
There is no correlation that estimate of size ‘X’ is an
equivalent of historical projects of about ‘X’.
Code and Unit
Test Only
System and SW
Requirements
were done
separately
This point does
not include SW
Testing Effort

35. Fallacy of Silent Evidence
What about what we don’t know?
How confident would you feel if the Silent Evidence was visible?

36. 36
Software Has Additional Characteristics That
Should Be Considered
Track defect
discovery and
removal rates
against expected
rates
Increased defect
reporting rate
shows a
worsening trend
Heath and Status Indicator
shows status and trends from
the previous snapshot
Thresholds are user definable

37. Watch Total System Requirements: IT Systems
Total Ownership Costs; 60+% Can Be Infrastructure &
Services
• Software Development
• Software Maintenance
• IT Infrastructure
• IT Services
• Other
Measurement Results must provide real
analysis

38. Risk and Uncertainty
© 2013 Copyright Galorath Incorporated
38

39. 39
Statistician Drowns in River
with Average Depth of 3 Feet!

40. Software Is A Critical Component of Military &
Aerospace Systems Failures Ariane 5
• Ariane 5 video
• Note: $500 Million payload
• Failure due to reused software from (Ariane 4)
• with incompatible assumptions for exception condition
that was not required
ariane_5_explosion_video.mp4
“the culture within the Ariane program…” only addressed
“random hardware failures… which can quite rationally be
handled by a backup system”
“the view had been taken that software should be considered
correct until it is shown to be at fault”!
Why should we care: Software cost & schedule
should be sufficient for successful missions

41. Minimum
Most Likely:
Maximum
If everything goes
well (optimistic
estimate)
If everything
goes badly
(pessimistic
estimate)
Using Ranges To Address
Uncertainty
pert beta Incremental Likelihood
Pert Beta

42. Adding Reality to Schedules
Example (Source: SEI)
Process Durations
Step Expected
1 30
2 50
3 80
4 50
5 90
6 25
7 35
8 45
9 70
10 25
500
What would you forecast
the schedule duration to
be?

43. Adding Reality to Schedules –
Example - 2
Process Durations
Step Best Expected Worst
1 27 30 75
2 45 50 125
3 72 80 200
4 45 50 125
5 81 90 225
6 23 25 63
7 32 35 88
8 41 45 113
9 63 70 175
10 23 25 63
500
What would you
forecast the schedule
duration to be now?

44. Adding Reality to Schedules –
Example - 3
With 90%
confidence, the
project will be
under 817 days
duration.
The project is almost
guaranteed to miss
the 500 days duration
100% of the time.

45. Adding Reality to Schedules –
Example - 4
With only 50%
confidence, the
project will be under
731 days duration.

46. Typical Growth Range From Initial
Sizing To Delivery
• Functional sizing for estimates should include growth
range to mitigate risk
© 2011 Copyright Galorath Incorporated 46

47. Dealing With the “Problem of
Assumptions”
• Assumptions are essential but…
• Incorrect assumptions can drive an estimate to
uselessness
• Use an assumption verification process
© 2012 Copyright Galorath Incorporated 47
Estimates must have assumptions defined but…
Bad assumptions should not be justification for
bad estimates
ASSUMPTIONS MAY BE RISKS

48. Business Value Is Part of Measurement &
Estimation
© 2011 Copyright Galorath Incorporated

49. Software Projects Must Return
Business Value
“Economics is primarily a science of choice…
software economics should provide methods for analyzing
the choices software projects must make.” Leon Levy
“Base choices on those providing the maximum business
value to the organization” Eli Goldratt
Measurement and its uses such as estimating and defect
analysis help this science of choice
• Some say business value is not our problem
• While others generally need to perform benefit analysis
• We need to build systems that optimize the business
• Make IT part of the solutionIf IT & measurement don’t generate
sufficient profit money will go elsewhere

50. The Gap Between IT &
Management Needs (Source: Fraunhofer)
© 2011 Copyright Galorath Incorporated 50
Financial
Customer Business Process
Innovation / Growth
Vision &
Strategy
Goals
Data Collection
MeasurementMetrics
AnswersQuestions
Goal Attainment
Software Development & Maintenance
6σ
GQM
PSM
CoBIT…
Business/
Organization
Level
IT
Governance
IT
Governance
Customer
Focus

51. Software & IT Systems Are
About Business Value
© 2009 Copyright Galorath Incorporated 51

52. Generating the Business Value
Side of the Equation (Benefits)
• The business owns benefit calculations
• IT should participate
• Exception: projects solely improving internal IT
• Beware of subjectivity translating soft benefits
• Use probability and risk
© 20011 Copyright Galorath Incorporated 52

53. Quantify Intangible Software Benefits & Costs
When Possible … But Be Realistic; Examples
• Improved Customer Satisfaction: May yield
Improved Customer Retention
• Higher Quality Product: May yield lower rework and
higher customer satisfaction
• Improved Decision Making: May yield many things
• Improved Win Ratio: Yields additional revenue
• Time to Market: May improve profit/market share
© 2009 Copyright Galorath Incorporated 53

54. An ROI Analysis of A New System
© 2013 Copyright Galorath Incorporated
Cost of capital 8.0%
Initial Investment Year 1 Year 2 Year 3 Year 4 Year 5 Year 6 Year 7
Total
Ownership
Investment $100,000 $100,000
Increase/(dec.) in
revenue ($40,000) $60,000 $110,000 $100,000 $100,000 $150,000 $150,000 $630,000
Increase/(dec.) in
op. exp. $90,000 $70,000 $70,000 $22,000 $24,000 $27,000 $28,000 $331,000
Cash Flow ($100,000) ($130,000) ($10,000) $40,000 $78,000 $76,000 $123,000 $122,000 $199,000
PV of Cash Flow ($100,000) ($120,370) ($8,573) $31,753 $57,332 $51,724 $77,511 $71,186 $60,563
NPV 60,563 $60,563
IRR 13.5% 13.5%
ROI 121% 121.1%

55. The Rule of Measurement
When performance is measured
Performance improves…
When performance is measured and
reported the rate of improvement
accelerates..
Thomas Monson
The work you do is a root cause of
program success