Estimation and planning processes are critical for project success. Poor estimates can lead to cost overruns, schedule delays, and project failures. There are various estimation methods, each with advantages and limitations. Initial estimates are often too optimistic due to cognitive biases, pressure to win contracts, and lack of understanding of complexity and risk. Accurate and realistic estimates require a repeatable process using historical data and parametric modeling to avoid common challenges like underestimating requirements and resources.
Estimation is critical to IT demand management as today's senior IT executives deal with a familiar challenge - how to balance the size of the development team with the company's software wish list. Modern estimation techniques offer critical insight into this challenge. In this presentation, you will learn the ins and outs of estimation and how to effectively utilize estimation to ensure project success.
PMICOS Webinar: Building a Sound Schedule in an Enterprise EnvironmentAcumen
Dr. Dan Patterson presented a one-hour webinar on effective scheduling using metrics analysis. He reviewed some of the common problems found in schedules and the research that backs the claim that, in the end, the schedule drives project success.
Modeling: the holy grail for designing complex systems?xmoneva
presentation given at the ESI Symposium 2009 (Eindhoven, The Netherlands); abstract: http://www.esi.nl/events/esi_symposium_2009/programme/2_1_abstract.html
Everyone has been given a 2 paragraph document listing the "scope of services" for a potential project. The client would like an estimate in 48 hours and there are no more details to help you deliver that required fixed bid contract. At the same time, many teams have also been given (or created) a detailed PRD or backlog document and still had a project budget balloon out of control. In this session I would like to discuss the not only the problems associated with estimation and how to avoid them, but more importantly how we can plan for them, turning our estimation process into not only an art, but a science. Well cover how to sell your estimate internally, and arm you with the methodologies to support your numbers. The problem with software estimation The morale The metrics The reality - an estimation metaphor Avoiding Risk Project entry point of sale At what point of the project lifecycle is your first sale? Risk association with point of sale Products in the front, estimations in the back The Elusive Discovery phase How to estimate a discovery How to sell a discovery How to include discovery in a full fixed bid RFP Planning for Risk Estimation types Gut - An art form Comparables - An art/science Factors/formula - A science Contingency Rating systems Formulas Granularity
Avoid software project horror stories - check the reality value of the estima...Harold van Heeringen
Many large software projects turn into software horror stories, resulting in newspaper headlines and even political issues. Often, the project costs and schedule were estimated unrealistically optimistic, using immature estimation techniques. A relatively simple way to avoid many problems is to perform a reality check on the estimate. This presentation was given on the conference of the International Cost Estimating and Analysis Association (ICEAA2014), June 2014 (Denver, USA)
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Let's dive deeper into the world of ODC! Ricardo Alves (OutSystems) will join us to tell all about the new Data Fabric. After that, Sezen de Bruijn (OutSystems) will get into the details on how to best design a sturdy architecture within ODC.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
3. Delusions of Success: How Optimism
Undermines Executives' Decisions Richard Hartley, HBR)
(Source:
• Problem:
Humans seem hardwired to be
optimists
• Optimism from cognitive biases &
organizational pressures
• Exaggerate talents & degree of control
• Attribute negative consequences to external factors
• Anchoring (relying too heavily on one piece of
information) magnifies optimism
Solution: Temper with “outside view”
• Most pronounced for new initiatives
Supplement traditional forecasting w/ statistical
Parametrics
Don’t remove optimism, but balance optimism & realism
4. Example of Tempering: (Source How To
Measure Anything)
• German Mark V Tanks
• Allies estimated production by analyzing serial numbers
• Used the captured tanks as a random sample and predicted
probability of various production levels
5. An Estimate Defined
• An estimate is the most knowledgeable statement you
can make at a particular point in time regarding:
• Effort / Cost
• Schedule
• Staffing
• Risk
• Reliability
• Estimates more precise with progress
• A WELL FORMED ESTIMATE IS A
DISTRIBUTION
5
6. Estimation Methods 1 of 2
Model
Description Advantages Limitations
Category
No Basis or substantiation
Guessing Quick Can obtain any No Process
Off the cuff estimates
(Common) answer desired Usually Wrong
Generally optimistic
Truly similar projects must
Analogy Compare project with past Estimates are based on
exist.
(Common) similar projects. actual experience.
Less optimistic
Experts tend to be biased;
Expert Little or no historical knowledge level is sometimes
Consult with one or more
Judgment data is needed; good for questionable; may not be
experts.
(Common) new or unique projects. consistent.
Generally optimistic
A hierarchical Need valid requirements.
Provides an estimate
decomposition of the Difficult to track architecture;
Top Down linked to requirements
system into progressively engineering bias may lead to
Estimation and allows common
smaller components is used underestimation.
(Common) libraries to size lower
to estimate the size of a Generally optimistic (miss
level components.
software component. non-WBS items)
6
7. Estimation Methods 2 of 2
Model Category Description Advantages Limitations
Uses expert judgment
to determine how Easy to get under
Design To Cost Little or no engineering basis.
much functionality can stakeholder
(sometimes) Optimistic
be provided for given number
budget.
Simple relationships may not tell the
Equation with one or
whole story
Simple CER’s more unknowns that Some basis in
Historical data may not tell the whole
(common) provides cost / data
story
schedule estimate
Less optimistic
Models are
usually fast and Models can be inaccurate if not
easy to use, and properly calibrated and validated;
Perform overall
useful early in a historical data may not be relevant
Comprehensive estimate using design
program; they are to new programs; optimism in
Parametric Models parameters and
also objective parameters may lead to
(common) mathematical
and repeatable. underestimation.
algorithms.
Easy tradeoffs More or less optimistic depending
can provide on parameters
better plans
Why should we care: Each method has challenges and
we should be familiar with each
7
9. Poor Estimates Effect on Projects
• Inaccurate estimates can reduce project success:
• Poor implementations
• Critical processes don’t scale
• Emergency staffing
• Cost overruns caused by underestimating project needs
• Scope creep
• Forever changing project goals
• Frustration
• Customer dissatisfaction
• Cost overruns and missed schedules
• Project Failures
• Important project business decisions made early with
minimum knowledge & maximum uncertainty
Why should we care: Poor estimates and plans are
a root cause of program failure
9
10. Initial Cost Estimation Problems (Software
Program Managers Network)
• Many programs that have been evaluated tend to initially estimate using a
very optimistic method.
• Low bid to win contract
• Naïve level of effort estimation
• Human optimism (HBR & Rich Hartley USAF Undersec Acquisition)
• Often wrong since they are not based on a thorough analysis of requirements. The formula
Cost = Size x Complexity/Productivity
• Has three unknowns upfront: size, complexity, and productivity.
• Methods & estimating tools determine size given system requirements
• Organizational databases of productivity on comparable size and
complexity projects can be used
• Or other bottom-up estimation techniques
“In any event, all initial cost estimates should
be considered as potential high risk and
should be reviewed at each program review”
SPMN
11. Software Is A Critical Component of Military &
Aerospace Systems Failures Ariane 5
• Ariane 5 video
• Note: $500 Million payload
• Failure due to reused software from (Ariane 4)
• with incompatible assumptions for exception
condition that was not required
“the culture within the Ariane program…” only addressed
“random hardware failures… which can quite rationally be
handled by a backup system”
“the view had been taken that software should be considered
correct until it is shown to be at fault”!
Why should we care: Software cost & schedule
should be sufficient for successful missions
12. Software Is A Key Risk Item In
Weapons Systems
• Navy Mobile User Objective Satellite Communication
System delays to the Joint Tactical Radio
System, a set of software-defined radios causes
advanced MUOS capabilities to drastically underused…
GAO
• GAO identified 42 programs at risk for cost & schedule
1. military requirements changes
2. software development challenges
3. workforce issues
• National Institute of Standards and Technology (NIST)
• Software defects cost nearly $60 Billion Annually
• 80% of development costs involve identifying and
correcting defects
Software, not Hardware or technology readiness levels
were called out
13. Death March Projects Are Likely To Fail
Why should we care: If you have a project on
a death march failure is highly probably
14. Balancing Resources & Schedule Is
A Science
For a given Size, Complexity and Technology
Minimum Time Work Expands
To Fill Time
To Complete (Effort Increases
(Effort Increases
due to lack of pressure )
to Reduce Schedule)
Effort Increase
Minimum Time due to Longer
Effort Months
Schedule
Optimal Effort
(Lower Effort
for Longer
Schedule)
Calendar Time
15. Why Total Cost Grew for Two Space Programs
David Graham, NASA
Development Growth Causes
Requirements
Generation & Translation
8% Budget/Funding
11% 25%
Cost Estimation
Underestimation of Risk
11%
11% Schedule Slips (Govt &
Contractor)
9%
Price Increases
25%
Other
Quantitative Framework
5 “TheSuccess Triangle of Cost, Schedule, and Performance: A Blueprint for Development of Large-Scale
Systems in an Increasingly Complex Environment” - (Booz|Allen|Hamilton, 2003)
16. Use Earned Value To Quantify Progress
Versus Effort (Oversimplified)
• Main EVM concern is what has been accomplished in a
given time and budget, versus what was planned for the
same time and budget
• Project generally deemed healthy if what has been
accomplished is what was planned, or more
• Project deemed unhealthy if accomplishment lags
expectations
• Definition: Earned value = budgeted value for the work
accomplished (what you got for what it cost you)
Healthy Unhealthy
$ Budget $ Budget
EV
EV
Time = Now Time = Now
16
18. Understand Project Risks Include Them In Planning
Decisions (Example SEER-SEM Outputs)
Probability
Schedule Probability Probability
Effort Probability
Example Application 1 Example Application 1
99% 99%
90% 90%
80% 80%
70% 70%
60% 60%
50% 50%
40% 40%
30% 30%
20% 20%
10% 10%
1% 1%
0 4 8 12 16 20 0 1800 3600 5400 7200 9000
Time (calendar months) Effort (person-hours)
Defects Probability
Probability Example Application 1
99%
90%
80%
70%
60%
50%
40%
30%
20%
10%
1%
0 12 24 36 48 60
Defects (count)
18
19. Considering Maintenance During Planning Can
Yield More Successful Long Term Results
Staff Vs Maintenance Rigor
3500
staff hours per month
3000
2500 develop
2000 Rigor vhi+
1500 Rigor nom
1000
Rigor vlo
500
0
1 7 13 19 25 31 37 43 49 55 61 67 73 79 85
Time
19
20. Data Doesn’t Have To Be Perfect To Be
Useful: But Is Has To Be Viable
• 80 Calories per serving
• 2.5 Servings per can
• 4 Ounces, Condensed, 8 Ounces With Water
22. The Error of Causal Analysis
Creating a False Association
• Correlation does not imply causation
• Just because two data points may sit side by side
doesn’t mean they are the same or will have the same
outcome
• Casual analysis is a recognized error in medicine
Tumor Can Cause
Headache Perhaps ???
Headache doesn’t mean a
tumor
22
23. Use Historical Measurement to
evaluate your estimate!
It’s easy to dig deeper and deeper to justify an estimate!
23
24. Data Beware Apples and Oranges:
Phase : All activities may not be included.
System Concept Integration
missing missing
Phases
System Concepts System Req & Design
System Req Analysis Preliminary Design
Detailed Design Code / Unit Testing
Software Test System Integration / OT&E
24
26. Basic Cost Estimating Process (Source CEBOK)
WBS • Work Breakdown Structure
(WBS) Development
Baseline
• Program/System Baseline
Data
Collection
Development
Data
Analysis
Methodology
Validation
Reports
26
28. 10 Step System Estimation Process
2011
1. Establish
Estimate Scope 10. Track Project
Throughout
Development
2. Establish Technical 9. Document Estimates
Baseline, Ground and Lessons
Rules, Assumptions Learned
8. Generate a
Project Plan
4. Refine Technical
Baseline Into
Estimable Components 6. Validate Business
Case Costs &
Benefits (go / no
go)
4. Collect data /
estimation inputs 6. Quantify Risks
and Risk Analysis
5. Estimate Baseline Cost,
Schedule, Affordability Value
28
29. Estimation Organizational Maturity V1.7
Level Informal or no
Manual effort
estimating
0
estimating without a
process
Level Direct Task
Spreadsheets
Ad Hoc
1
Estimation Process
Formal Simple model
Level Sizing
(e.g.
Direct
Task
(Size *
Productivity)
or informal
Some
measureme Informal
2 Process
nt &
function Estimation SEER Use
analysis
points)
Level Formal
Robust
Parametric Estimate vs.
Formalized
Multiple
Rigorous
measurement
Parametric
planning &
Risk Repeatable
3
Sizing estimation actual capture Estimate Management process
& analysis Control
(SEER) Process
Level Formal sizing
Repeatable
Robust
parametric
Rigorous
measurement
Parametric
estimation Risk
Process
improvement
4
process estimating with tracking Management via lessons
& analysis
(SEER) & control learned
Level Formal sizing
Repeatable
Robust
parametric
Rigorous
measurement
Parametric
estimation Risk
Continuous
process
5
process estimating with tracking Management
& analysis improvement
(SEER) & control
Why should we care? Maturity is related to estimate
viability… With better estimation process, projects
more likely to be successful in execution
30. Estimation Should Be Key Part of Process
(Source Q Redman, APMP Just Say No)
Phase -1 0 1 2 3 4
DDE
ROM
ROM
Formal Bid
Gate 4
Scope & Accuracy
ROM
15-20 people
ROM 4 weeks
(Bid Stds+
History)
EARLY ESTIMATING Modified
3-5 people, 3 - 5 days Budgetary
Top Down, parametric model Estimate
based price estimating Draft RFP/Gate 3
Vs. 6-8 people, 3
Current state: 90 people, 6wks weeks
(Bid Stds + History
)
Market Opportunity Acquisition
Us
Creation/ Customer Procurement
e
Assessment/ Planning/ POM Draft RFP RFP
Decision Plans Initiation
“What If’s” and Plus Ups
30
31. GAO Publication: Characteristics of credible cost
estimates and a reliable process for creating them
• This chapter discusses a 1972 GAO report on cost estimating
• We reported that cost estimates were understated and causing unexpected cost growth
• Many of the factors causing this problem are still relevant today
• We also discuss a 12 step process for producing high quality cost estimates
31
32. GAO Publication: Why cost estimates are required
for government programs and challenges developing
credible results
• Introduces why cost estimates are required for government programs
• Developing annual budgets, supporting management decisions about
which program to fund, and evaluating resource requirements at key
decision points
• Discusses various challenges associated with developing credible
results
32
33. Ask These Questions When
Identifying Estimate Scope
• Challenged projects
• Would you still go forward if you knew
• Schedule would be significantly longer?
• Cost would be dramatically higher?
• Probably: but perhaps more insight could identify
mitigation
• Plan functionality differently
• Certainly you could notify stakeholders of real costs
• Ensure staffing is appropriate for the constraints
• Failed Projects
• Would you start a project you knew was unaffordable?
Or if schedule was completely unrealistic?
• If knowing up-front could you do something about it?
• Often better to kill project before it begins than waste
resources & let the organization down 33
34. Lesson Learned: Estimate Must
Quantify Risk & Uncertainty
Firm Fixed
Price?
Feel lucky?
What is
likely to
happen
Understand the risk before you commit!
34
34
36. Cost Estimate Qualities
(Adapted from CEBOK)
• The characteristics of high quality cost
estimates are:
• Accurate (Viable Within Range)
• Comprehensive
• Replicable and Auditable
• Traceable
• Credible
• Timely
36
Unit I - Module 1
37. System Description (Parametrics Can
Estimate More, Earlier) Adapted from CEBOK
“If you can’t tell me what it is, I
can’t tell you what it costs.”
-Mike Jeffers
“If you can tell me the range of
what it might be I can tell you the
range of cost, schedule &
probability”
-Dan Galorath
37
38. Types of Cost Estimates (Adapted From
CEBOK)
• Life Cycle Cost Estimate (LCCE)
• Independent Cost Estimate (ICE) 1
• Budget Estimate
• Rough Order of Magnitude (ROM)
• Estimate At Completion (EAC)
• Independent Cost Assessment (ICA)
• Analysis of Alternatives (AoA)
• Economic Analysis (EA)
38
Unit I - Module 1
39. Aircraft Example: Estimating
Techniques Adapted from CEBOK)
• Where applicable, use primary methods
• Analogy
• Model X100 series jet engines have only been used on one other
plane, but weight is 100% higher on this model; estimate 2x other
model
• Simple Parametric Param etric Estim ate
• As shown, need to estimate 2 lb 12
brake rotors, should be roughly $4M 10
from regression curve 8
Cost
6
• Commercial Parametric 4
2
• Key characteristics range are 0
0 2 4 6 8 10
Param eter (ie. w eight)
• 30-50k lines of code and 600-800
• kilos engine & 7 PC boards
• Engineering Build-Up
• Know Air Conditioning (AC) system costs on plane because received
quotes from HVAC vendor for all duct work and AC blower off the
39
shelf
40. Manual Estimates: Human Reasons For
Error (Metrics Can Help)
• Manual Task estimates yield SIGNIFICANT error
• Desire for “credibility” motivates overestimate
behavior (80% probability?)
• So must spend all the time to be “reliable”
• Better approach: force 50% probability & have “buffer”
for overruns
• Technical pride sometimes causes underestimates
40
41. Lesson Learned: The Risk In Risk
Analysis
"It's tough to make predictions, especially about the future." -- Yogi
Berra.
41
42. Cost Readiness Levels at Low TRLs and/or
Less Than Firm Requirements
• If the project has critical items at
less than TRL 4…
• Like asking Edison in 1876 “How
much longer for the light bulb”
• “Hard to say”, he would have said
as he had you thrown out
• Note that this is not the same as
asking, in 1879, once he had found
a workable carbon filament, “How
TRL
9 TRL9: Actual system “flight proven” thorough successful mission operations
much will a production version of
TRL
8 TRL8: Actual system completed and “flight qualified” through test and demonstration
the light bulb cost to develop and TRL
7 TRL7: System prototype demonstration in a space environment
produce Tom?” TRL
6 TRL6: System/subsystem model or prototype demonstration in a relevant environment
TRL
• This would have been a TRL 4 5 TRL5: Component and/or breadboard validation in relevant environment
TRL
question 4 TRL4: Component and/or breadboard validation in laboratory environment
TRL
• Tom’s cost estimators could have 3
TRL
TRL3: Analytical and experimental critical function and/or characteristic proof-of-concept
begun to model this 2
TRL
TRL2: Technology concept and/or application formulated
TRL1: Basic principles observed and reported
•
1
So if 1<TRL<3 CRL < 4
• Likewise, if requirements are not
firm, CRL < 4 42
43. Table 1: CRL Rating Prior to Availability of
Probabilistic Risk Analysis
Basic CRL9 5 5.5 6 6.5 7 7.5 8 8.5 9
“Complexity*” CRL Description
of the CRL8 4.5 5 5.5 6 6.5 7 7.5 8 8.5
to go work
at the time CRL7 End of project actual cost
of the
4 4.5 5 5.5 6 6.5 7 7.5 8 9
estimate
CRL6 3.5 4 4.5 5 5.5 6 6.5 7 7.5 Cost fit for very firm
8 engineering decisions and
very firm budget
commitments (+/-5%)
CRL5 3 3.5 4 4.5 5 5.5 6 6.5 7
Cost fit for firm
CRL4 2.5 3 3.5 4 4.5 5 5.5 6 6.5 7 engineering decisions and
firm budget commitments
(+/-15%)
CRL3 2 2.5 3 3.5 4 4.5 5 5.5 6
Cost fit for PDR
CRL2 6 engineering decisions and
1.5 2 2.5 3 3.5 4 4.5 5 5.5 PDR budget use
(+/-25%)
CRL1 1 1.5 2 2.5 3 3.5 4 4.5 5
Cost fit for preliminary
5 engineering decisions and
Extrem Very Very Very Very Very Very Very Very preliminary budget use (+/-
el 35%)
Minimal
Cost fit for very
4 preliminary engineering
decisions and very
*Complexity considerations include human rating, launch Adequacy of “Estimating Methods”, preliminary budget use (+/-
system requirements, planetary destination, operational vs
experimental requirements, materials complexity, use of
experience of the estimators, quality of 45%)
deployables, parts count, challenging thermal CARD, availability of analogous data and
requirements, high data rates, electronic parts cost tools, time allowed for estimate,
class, stabilization requirements, power generation
independence of estimate, number of 43
type, propellant choice, propulsion requirements and many
other factors. Programmatic complexity includes team cross checks performed
size, team experience, schedule and many other factors.
44. Table 2: CRL Rating After Availability of Probabilistic Risk Analysis
Use S Curve inter-quartile cost range to translate to a CRL rating
–Calculate ratio of 75th percentile cost to 25th percentile cost; then lookup ratio on chart to read CRL
Lookup
25th Percentile Median Cost 75th Percentile Cost Ratio of 75th Read
Cost Percentile Cost to 25th CRL
Percentile Cost Description
End of project actual cost
100 100 100 1.00 9
Cost fit for very firm
95 100 105 1.11 8 engineering decisions and very
firm budget commitments
(+/-5%)
Cost fit for firm engineering
85 100 115 1.35 7 decisions and firm budget
commitments (+/-15%)
Cost fit for PDR engineering
75 100 125 1.67 6 decisions and PDR budget use
(+/-25%)
Cost fit for preliminary
65 100 135 2.08 5 engineering decisions and
preliminary budget use (+/-35%)
Cost fit for very preliminary
55 100 145 2.64 4 engineering decisions and very
44
preliminary budget use (+/-45%)
45. Estimation Best Practices
• Decide Why You Want An Estimate
• Map Estimation Goals To Estimate Process Maturity &
Develop Plan To Achieve The Maturity
• Have A Documented, Repeatable Estimation Process
• Make The Estimating Process As Simple As Possible;
But No Simpler
• Be Proactive: The Process Is Important, The Tools Go
Along With The Process
• Get Buy-in From Program Managers
• Hold People Accountable: Center Of Excellence Can
Prepare Estimate But Program Managers Must Own
Them
• Tie The Estimate To The Plan
45
46. Estimation Best Practices 2
• Evaluate Total Ownership Cost; Not Just Development
• Estimate A Range And Pick A Point For The Plan
• Re-estimate The Program When It Changes
• Avoid Death Marches: Programs With Unachievable
Schedules Are Likely To Fail And Drain Morale
• Keep A History: Start An Enterprise Database NOW…
• Business Case: Evaluate ROI In Addition To Costs
• Convert Expert Spreadsheets Into A Common
Language
46
47. Estimation Best Practices 3
• Track Progress Vs. Estimate Throughout The Life Cycle
• Estimate Schedule As Well As Effort (Cost) For
Complete Picture
• Tie The Business Case Into The Estimating Process
• Attack Non-productive Rework As Part Of The Process
47
48. Estimation Best Practices 4
• Have clear definitions:
• What does “complete” mean
• What activities are included and excluded (E.g.
development only or total ownership; help desk
included or excluded, etc.)
• Which labor categories are included and excluded in the
estimate (e.g. are managers included? Help desk? Etc.)
• Don’t ignore IT infrastructure and IT services costs
• Tracking defect sources can go along with the
process
48
49. Project Management Challenges
Relate To Estimation Planning
• “No good deed will go unpunished.” Unreasonable
expectations on the next project are supported by
heroic behavior on the previous project
• The most important business decisions about a
software project are made at the time of minimum
knowledge and maximum uncertainty.
• Adding and/or changing means more time and/or
more effort
• When a project is in trouble ask for more time rather
than more people. Deferring functionality common
approach to asking for more time
• Increasing concurrency is usually not a solution (e.g.
designing several releases concurrently)
A1-49
50. 7 Characteristics of Dysfunctional
Software Projects (Source: Mike Evans, et al.)
• Based on 350 projects:
• Failure to Apply Essential Project Management
Practices
• Unwarranted Optimism and Unrealistic
Management Expectations
• Failure to Implement Effective Software
Processes
• Premature Victory Declarations
• Lack of Program Management Leadership
• Untimely Decision-Making
• Lack of Proactive Risk Management
50
Presentation Abstract: Estimation, planning and control processes can make the difference between project success and project failure. Unfortunately, estimation is often considered unimportant by the engineering community, who may just want to make the best, fastest, etc. without considering affordability. This paper covers estimation best practices, estimation process maturity, and examples of estimation, planning and control done right. Estimation process maturity -- what it is, how to apply it to your programs and how to seek the appropriate level for your organization -- will be discussed, along with the Return on Investment of such practices.Presentation Synopsis: Estimation, planning and control processes decide project success. This paper covers estimation best practices, process maturity, and examples. Estimation process maturity -- definintion, application, and how to seek the appropriate level for your organization -- will be discussed, along with the Return on Investment of such practices.
Estimation, planning and control processes can make the difference between project success and project failure. Unfortunately, estimation is often considered unimportant by the engineering community, who may just want to make the best, fastest, etc. without considering affordability. This paper covers estimation best practices, estimation process maturity, and examples of estimation, planning and control done right. Estimation process maturity -- what it is, how to apply it to your programs and how to seek the appropriate level for your organization -- will be discussed, along with the Return on Investment of such practices.