Estimating Project Management and Systems Engineering Costs in a Changing NASA Environment<br />Presenters:<br />Steve Shinn<br />Larry Wolfarth<br />Co-Authors:<br />Meagan Hahn<br />Sally Whitley<br />2011 NASA Project Management Challenge<br />02/08/2011<br />Used with permission<br />
Agenda<br />Definitions<br />Trends in Project Management (PM) and Systems Engineering (SE)<br />Trends in PM/SE Costs<br />How Well Do Our Current Cost Methods Capture Trends in PM/SE Costs?<br />Can We Develop More Robust PM/SE Cost Estimating Relationships?<br />For the Near Term: What Do We Do?<br />
Project Management in the Space Context<br />NASA NPR 7120.5D defines project management (PM) as:<br /><ul><li>The business and administrative planning, organizing, directing, coordinating, analyzing, controlling, and approval processes used to accomplish overall project objectives, which are not associated with specific hardware or software elements.
[PM] includes project reviews and documentation, non-project owned facilities, and project reserves.
[PM] excludes costs associated with technical planning and management and costs associated with delivering specific engineering, hardware, and software products.</li></ul>Project management can be traced back to core ideas developed by Frederick Taylor in the 1880s. <br />Management research, development, and career advancement have been around for decades, with newer theories and tools replacing older ones. <br />The introduction of Earned Value Management (EVM) is one more recent example of an innovation in project management.<br />
Systems Engineering in the Space Context<br />NASA NPR 7120.5D defines project-level systems engineering (SE) as:<br />The technical and management efforts of directing and controlling an integrated engineering effort for the project.<br />[SE] includes the efforts to define the project’s space flight vehicle(s) and ground system, conducting trade studies, the integrated planning and control of the technical program efforts of design engineering, software engineering, specialty engineering, system architecture development and integrated test planning, system requirements writing, configuration control, technical oversight, control and monitoring of the technical program, and risk management activities.<br />Documentation products include requirements documents, interface control documents, a Risk Management Plan, and a master verification and validation plan.<br />
Conceptual Linkage of SE and PM<br />Systems Engineering provides the “bridge” between a project’s technical and management elements:<br />“Systems engineering is an inherent part of project management―that part that is concerned with guiding the engineering effort itself―setting its objectives, guiding its execution, evaluating its results, and prescribing necessary corrective actions to keep it on course” <br />−Alexander Kossiakoff &William Sweet, Systems Engineering: Principles and Practice<br />
SE/PM Connection in the Modern Aerospace Context<br />The disciplines of SE and PM are tied together now more than ever before.<br />Example: 2007 Revision D of NASA NPR 7120.5, an engineering document, prescribes the following:<br />Processes for overall management, risk management, as well as engineering of NASA space missions <br />A range of products for each discipline<br />New processes and products + increased technical complexity =<br />Greater rigor<br />Additional processes for analysis and reporting<br />More resources<br />Time<br />
The “Growth” of Modern PM (1 of 2)<br />Since its first edition in 1996, the Project Management Institute’s Project Management Body of Knowledge (PMBOK®) has increased in size and detail with each release:<br />
The “Growth” of Modern PM (2 of 2)<br />The last two editions of the Defense Acquisition Guidebook (DAG) have increased its coverage of both PM and SE:<br />
What Drives the Modern Expansion of Systems Engineering<br />Modern SE concepts and capabilities are a response to three distinct trends in system development: <br />Increasing complexity – As systems become more complex and inclusive of advanced technologies, new SE approaches have been needed to provide a holistic system view.<br />Increasing scale – As systems become larger or based on systems of systems, trade-offs have become a necessary component of any major project.<br />Specialization – Specialists are assigned to build certain components of the larger system. Therefore, a new discipline focused on management of these component interfaces is needed to ensure compatibility and interoperability among components and overall system performance.<br />* Alexander Kossiakoff and William Sweet, Systems Engineering: Principles and Practice, New York: Wiley<br />
Increasing Complexity of Space Robotic Missions: Discovery Cost Cap Trend<br />Trend in U.S. space robotic missions over time is to accomplish more than prior missions.<br />Accordingly, the NASA cost cap for Discovery and other competed missions has increased faster than inflation.<br />At the same time, the number and scope of science objectives has increased significantly from the 1990s missions to those being launched in the 21st century.<br />NEAR: First mission to orbit & land on an asteroid<br />DAWN: First mission to orbit an asteroid, then travel to a 2nd asteroid (requires ion propulsion)<br />
Contribution of Systems Engineering to Project Success<br /><ul><li>A recent INCOSE study indicates the contribution of SE activity to project success:
When SE effort is less than 8% of total project cost, actual project cost can exceed planned costs by as much as 100%.
Conversely, for the few examples where SE effort exceeds 16% of total project cost, cost overruns are minimal.
The conclusion is that failure to commit sufficient SE effort up-front can lead to project cost overruns.</li></li></ul><li>PM/SE Resources in the Modern Space Context<br />From in-house research on trends in space mission costs, we’ve observed that the management and engineering budgets of the 1990s are no longer sufficient. Data from external sources confirms this is a common trend.<br />Our management now wants to know:<br />What is the appropriate balance between funds for actual science and funds for management and control functions? <br />Are more dollars being directed to PM and SE activities now than would have been directed for comparable missions in the 1990s?<br />If so, what is the amount and rate of increased management costs in order to account for them in future missions? How are the increases affecting overall project performance and cost?<br />In short: <br />What is our trend in PM/SE “cost to launch”? –and-<br />How much PM/SE effort is sufficient to ensure mission success?<br />
The Trend: Integration of Cost Tools Across the Project Life-Cycle <br />“newer” initiatives<br />
PM/SE Cost Estimating Challenge<br />Cost analysts rely on historical data to estimate future requirements for financial resources.<br />To answer the questions posed by our management, we examined post-1995 cost histories of:<br />NASA-funded APL space missions<br />NASA-funded missions led by NASA centers and JPL<br />Costs of remote sensing instruments developed for NASA space missions<br />We considered both actual and projected costs as well as cost trends over time.<br />
APL Trends in PM/SE Costs for Robotic Space Missions Since 1995<br />PM/SE costs for APL robotic space missions launched since 1995 have increased in absolute dollars and as a portion of mission cost. The trend is clearer for SE costs, as shown below.<br />Trends in APL PM/SE costs as a percentage of spacecraft cost for NASA-funded APL missions launched from 1995 to 2002<br />Trends in APL PM/SE costs as a percentage of spacecraft cost for NASA-funded APL missions launched from 2002 forward, including planned and conceptual<br />
PM Cost as Percentage of Spacecraft Cost, APL NASA Robotic Space Missions, 1996−2018 <br /><ul><li> APL’s data shows the trend over time is increasing expenditures for program management relative to spacecraft cost.
Roughly half of the variation in PM can be explained by launch year.
The increase in PM as a percentage of spacecraft costs over time is statistically significant:
95% confidence interval around the slope =[0.0006,0.0059].</li></ul>R2 = 0.1585<br />Source: CADRe Database<br />
SE Cost as Percentage of Spacecraft Cost, NASA Robotic Space Missions, 1996−2018 (CADRe data)<br /><ul><li>Parallel with APL’s results CADRe SE data shows a stronger relationship with launch year than that for PM.
About 40% of the variation in relative SE cost can be explained by launch year.
95% confidence interval around the slope = [0.0029,0.0072].</li></ul>R2 = 0.4080<br />Source: CADRe Database<br />
Trends in Instrument Management and SE Costs, 1988−2009 <br />The trends in instrument management and SE costs since 2002 seem to counter the trend of increasing management & engineering costs for space missions overall…<br />…unless one notes that the delivered costs of space instruments launched since 2002 have in fact declined! <br />Data Source: NASA Instrument Cost Model Dataset (JPL)<br />
Drivers of Recent and Future PM/SE Cost Growth in APL Space Missions (1 of 2)<br />APL’s Space Department was certified in 2009 as an AS-9100 vendor. This standard dictates:<br />Quality improvement<br />Variation control or management for key product characteristics<br />Production and service provisions unique to the aerospace industry, such as part accountability, foreign object detection, production documentation, part identification, and part traceability<br />Process/tooling change control and management<br />Supply chain quality control, which covers the control of purchasing and acceptance processes<br />Design and development control<br />Product configuration control management<br />Product quality, reliability, and safety control<br />Continual improvement<br />Requirement to respond to these new standards ensures that PM/SE costs will continue to increase at least in the near term.<br />
Drivers of Recent and Future PM/SE Cost Growth for APL Space Missions (2 of 2)<br />Other Drivers<br />More technically challenging missions<br />NPR 7120.5D<br />Earned value management<br />Safety requirements<br />New SE documents<br />Other drivers suggest that PM/SE costs will continue to increase near term.<br />
How Do We Predict PM/SE Costs for Space Missions?<br />Most spacecraft cost models estimate PM/SE cost as a function of prime mission equipment costs.* <br />The cost estimating relationships (CERs) generally look like this. <br />In summary, most models predict increases in PM/SE costs only when hardware costs increase.<br />None provide “adjustments” for launch year.<br />DD SE = a× (HW DDTE Costb)<br />where <br /><ul><li>DD SE = SE design and development cost in millions of FY06 dollars
a is a coefficient that depends on the selected analogy
HW DDTE Cost is the cost of design, development, test, and evaluation of spacecraft hardware in millions of FY06 dollars
b is a CER-specific value</li></ul>* A notable exception is The Aerospace Corporation’s SSCM (Small Spacecraft Cost Model), which predicts PM/SE costs for planetary missions, many of which must be completed in time to fit very tight launch windows, as a function of schedule duration.<br />An example CER from NAFCOM 07<br />
Problems with Using Mission/Hardware Costs as Predictors of PM/SE Costs<br />Many cost and budget estimators rely on cost-to-cost factors for initial estimates of non-hardware program costs.<br />Reliance on [predicted] hardware cost as the single predictor variable of PM/SE costs creates several problems for cost analysts and planners:<br />Ignores the fact that similarly priced missions and hardware can vary widely in their non-hardware implementation costs because of unforeseen technical and programmatic uncertainties.<br />When the initial prediction of hardware costs are low, the consequence is that final PM/SE costs will be underestimated (see next slide).<br />
SE/PM Estimates at Completion (EACs) By Program Milestone, Recent APL Missions<br />APL’s Share of PM/SE EAC as a Percentage of Spacecraft Bus Development EAC―PDR, CDR, and Launch Milestones<br />For APL’s three most recently launched robotic space missions:<br /><ul><li>The PM cost/ spacecraft cost ratio has been relatively stable from PDR through launch.
After CDR, however, growth in SE costs has tended to outstrip growth in spacecraft costs.*
With the exception of MESSENGER, the SE cost increases exceed those experienced by spacecraft bus development.</li></ul>*In other words, SE cost estimates based on spacecraft bus cost predictions at PDR would have been underestimated.<br />
PM/SE CERs for Space Missions<br />At least one spacecraft cost model owner has expressed an interest in developing multivariate CERs to predict PM/SE costs for space missions. <br />NASA CADRe objective is to provide cost data on all NASA missions according to a consistent, clearly defined work breakdown (WBS) structure.<br />CADRe data are being collected at major milestones.<br />When sufficient data are available, PM/SE CERs can be developed.<br />Also possible: comparison of PM/SE EACs across program milestones.<br />
Findings (1 of 2)<br />Management of space missions requires better means to estimate PM/SE effort and costs from project start.<br />Both PM and SE contribute to project success.<br />External and internal pressures for new PM/SE processes and products are resulting in higher costs over time.<br />Current cost estimation methods rely on predicted hardware costs to predict PM/SE costs.<br />Problems with this approach include resource underestimation and no ability to adjust estimates for complexity or increasing PM/SE requirements.<br />
Findings (2 of 2)<br />Cost data are becoming available to develop more robust, multivariate CERs.<br />Example provided for instrument PM/SE cost prediction.<br />May be too early to show effect of recent demands for additional PM/SE products on PM/SE effort.<br />Statistical evidence suggests that what drives PM and SE effort may be slightly different:<br />PM effort correlates more to the overall size and complexity of the mission—e.g., total cost, number of instruments, number of organizational participants.<br />SE effort correlates more to technical complexity as measured by, e.g., spacecraft cost.<br />In APL’s experience, in addition to dealing with technical and schedule issues that result in overruns within the spacecraft subsystem, SE has also been more profoundly and quantitatively impacted by increased requirements from sponsors and internal quality assurance improvements<br />SE is being affected over time more directly and concretely than the management discipline, the major difference being attributed to rapidly increasing requirements and documentation<br />This idea gains credence in light of relatively stable management costs.<br />
Estimating PM/SE Cost in an Evolving Environment<br />To account for the changing management/engineering environment, we currently estimate PM/SE costs of our space missions as follows:<br />Start with a parametric estimate from whatever hardware estimating model is used<br />Apply in-house cost-to-cost factors based on a moving average of last two or three space missions<br />Smoothes out effects of implementation costs<br />Adjusts for exceptions, anomalies<br />Request senior managers and lead engineers generate bottom-up labor profile estimates along with bases of estimates (provides a “sanity check” against model-derived results)<br />Compare the results to identify inconsistencies and to confirm the reasonableness of the labor profiles<br />
PM/SE Resources in a Dynamic, Increasingly Complex Environment<br />To the question posed by our management, “How much is enough?”, our answers are:<br />More resources than 10 years ago.<br />More (SE) resources than might be predicted by the hardware budget.<br />Resources more like those of the most recent successfully completed mission, adjusted for<br />Sufficient resources to meet requirements for all deliverables and products<br />Resources needed to implement new processes<br />Levels of resources consistent with the technical, organizational, schedule complexity of the mission<br />Hopefully, as EVM and other processes are institutionalized, the process of answering the question can become more mechanical.<br />On-going data collection and analysis will yield more precise and verifiable methods for estimating and managing SE/PM effort and anticipating future trends. <br />