This document discusses the challenges of partnering on major research platforms and facilities. It notes that the high costs and complexity of such projects have driven increased partnering between U.S. agencies and with international entities. However, ensuring alignment between partner processes and practices can be challenging. The document analyzes the practices of three science agencies - DOE, NASA, and NSF - to identify similarities and differences in their approaches to developing and managing large science projects. Understanding these comparative practices is important for facilitating effective interagency and international cooperation on major research infrastructure initiatives.

1. Challenges in partnering on major research
platforms and facilities
William L. Miller, Ph.D.
AAAS Science and Technology Policy Fellow
National Science Foundation
Used with permission

2. Scope
• High cost and complexity of major projects
have drive partnering between U.S. agencies
and with international entities.
 How do partner processes and practices
align? (Start with agencies)
 What are the challenges and best practices
for partnering?
PMC2010: Partnering challenges on large projects W. Miller 2

3. Science platforms and facilities: a highly varied class…
PMC2010: Partnering challenges on large projects W. Miller 3

4. Big science projects engage many stakeholders…
National
Priorities
Science Technology
Enterprise Enterprise
Large Science
Infrastructure
Interagency International
Partnerships Cooperation
 lots of interest in
process, performance and outcomes…
PMC2010: Partnering challenges on large projects W. Miller 4

5. Comparison of project practices at DOE, NASA and NSF
Organizations and portfolios studied:
DOE Office of Science
“Science User Facilities”
NASA Science Missions
Directorate (SMD)
“Robotic Science Missions”
NSF Science & Engineering
Directorates, OPP
“Large Facilities” (MREFCs)
• Reviewed key agency guidelines, external reports & analyses…
• “Looked under the hood”:
~45 stakeholder interviews; site visits; direct observations…
W. Miller, PRECONSTRUCTION PLANNING FOR LARGE SCIENCE INFRASTRUCTURE PROJECTS: A comparative
analysis of practices and challenges at DOE, NASA and NSF, November 3, 2009
PMC2010: Partnering challenges on large projects W. Miller 5

6. Comparative Approach
• Framework for agency comparison along many dimensions
Organizational structures
Development processes
Oversight and decision/approval
Funding and acquisition
etc….
• Identify partnering issues along these and other
dimensions…
PMC2010: Partnering challenges on large projects W. Miller 6

7. Comparative anatomy of agency science structures
$33.8B DOE $17.8B NASA $6.5B NSF
NSB
Secretary Administrator Director
Administration
Management Budget, Finance
Chief
Support Office & Award Mgmt
Engineer
OECM LFO
$4.8B UnderSec $4.5B Science Missions Directorates $5.3B
Ofc of Science Directorate & Offices
Programs
Program Offices OPA Divisions Divisions
Divisions Programs SRBs Programs Panels
FPDs 42 U.S.C.§1873b
Projects
Laboratories $2.0B Centers Awardees (“shall not operate
laboratories”)
Projects Projects Projects
“Project assurance” bodies FPD Federal Project Director LFO Large Facilities Office
Provides policy OECM Office of Engineering and Construction Mgmt NSB National Science Board
OPA Office of Project Assessment S&E Science and engineering
Provides independent review SRB Standing Review Board Budgets: FY 2009 plans (no ARRA), from FY2010 Requests
PMC2010: Partnering challenges on large projects W. Miller 7

8. Development process framework  look at elements
Ongoing community input, horizon planning,
reprioritization, and portfolio adjustment
Early Early
acquisitions operations
Conceptualization Preconstruction Planning*
Final Construction Operations D
& Prioritization Conceptual Preliminary design
Development
Begin major Project Terminate
Investment Closeout Facility
• Plan Management/governance plans, WBS  assemble project team
• Design Goals, requirements  iterative design  bring to readiness
• Invest R&D, necessary technologies  bring to readiness
• Estimate Effort, cost, schedule, reserves, risks  refine to believability
• Govern Progress oversight and decision-making
*Also called definition, formulation, front-end planning…
PMC2010: Partnering challenges on large projects W. Miller 8

9. Development processes: alignment and terminology
Initiation Definition Execution
Pre-conceptual Conceptual Preliminary Trans/Closeout
Final Design Construction Operations
Planning Design Design
DOE-SC CD-0 CD-1 CD-2 CD-3 CD-4
Ref: DOE O 413.3A IPR** CDR EIR FDR ORR/RA
CDR Conceptual Design Review PDR IPR/EIR**
EIR External Independent Review (OECM) Critical Decision (CD) approvals
FDR Final Design Review CD-0 Approve mission need
IPR Independent Project Review (SC) CD-1 Approve Alternatives selection & cost range
ORR Operations Readiness Review CD-2 Approve Performance baseline
PDR Preliminary Design Review CD-3 Approve Construction start
RA Readiness Assessment CD-4 Approve Operations start **CD-0 IPR and CD-3 EIR for >$750M projects
Formulation Implementation
Pre-Phase A Phase A Phase B Phase C Phase D Phase E
Assembly,
Concept Concept & Prelim Design & Final Design &
Integ & Test, Operations
Studies Tech Devel Tech Completion Fabrication* Launch
NASA-SMD KDP-A KDP-B KDP-C KDP-D KDP-E
Ref: NASA NPR 7120.5D MCR MDR PDR CDR SIR FRR CERR
SRR LRR
CDR Critical Design Review ORR Operational Readiness Review ORR PLAR
CERR Critical Events Readiness Review PDR Preliminary Design Review Key Decision Point (KDP) approvals
FRR Flight Readiness Review PLAR Post-Launch Assessment Review KDP-C Approve Implementation *elongated to visually align NASA, NSF & DOE
LRR Launch Readiness Review SDR System Definition Review
MCR Mission Concept Review SIR System Integration Review
equivalent events. Sequences on this chart do
MDR Mission Definition Review SRR System Requirements Review not represent typical or relative phase durations.
Readiness NSB Approved
Horizon planning and Preliminary
Final Design Construction Operations
Conceptual Design Design
NSF
Ref: NSF 0738 CDR PDR FDR Ops
CDR Conceptual Design Review Approvals
FDR Final Design Review Post-CDR Approve advance to Readiness
PDR Preliminary Design Review Post-PDR Approve submission to Nat. Science Board (NSB) Review Decision
Post-FDR Congress appropriates MREFC funds
W.MillerOperations Review & Sponsor Processes
Ops
PMC2010:Big Projectschallenges on large projects
- Partnering Post-Ops Approve Operations start
ProjSci Oct 22, 2009
W. Miller 9
WLM Rev2.05052009

10. Oversight: two modes of assessment
Preconstruction
Conception Construction Operations D
Planning
Initiate Ready Finished
 Qualitative
assessment
(Reviews at major Review &
milestones) Approve
 Quantitative Tracking 0.40
• EVM required by OMB for
0.20
CV
0.00
major projects
-0.20
• Industry standards -0.40
SV
• Encourages baselining -0.60
• Fuller assessment of
outcomes
PMC2010: Partnering challenges on large projects W. Miller 10

11. Review-based oversight and decision/approval
Independent Internal decision Sign Off
Review support (e.g. Implement Project)
PDR Energy Systems Acquisition CD-2
DOE by OPA Advisory Board (ESAAB) Acquisition Exec (AE)
“Lehman Review”
(DepSec, US Sci, SC Dir/AD)
• OPA briefs AE, calls ESAAB
(also: EIR by OECM) Approve Performance Baseline
• Federal Project Director
(Enter Final Design Stage)
PDR Program Mgmt Council KDP-C
NASA by Standing Review Decision Authority (DA)
Board • Program Manager (AA for Cat 1, otherwise MDAA)
• Center Mgmt Council
Approve Implementation
• Technical Authority (Enter Phase C – Final Design & Fab)
• Project Manager
PDR NSF Director NSB Approval
NSF by Review Panel for inclusion in a future
MREFC* Panel
budget in MREFC*
• CFO, DDLFP construction account
• Directorate/Division
• Program Officer
PMC2010: Partnering challenges on large projects * Major Research Equipment and Facilities Construction W. Miller 11

12. Baseline establishment – with focus on cost
During construction,
projects performance
Cost
assessed relative to Cost
baseline… “Baseline” “Outcome”
(what you were promised) (what you got)
Cost estimation toward the baseline:
DOE • “Bottom-up" Independent Cost Estimate (ICE) or Independent Cost
Review as part of External Independent Review for CD-2.
NASA • Project’s bottom-up Life-Cycle Cost Estimate (LCCE).
• ICE prepared and owned by the Standing Review Board (SRB),
(generally parametric, using same definitions as LCCE)
 Project must reconcile ICE with LCCE at Preliminary Design Review.
NSF • Bottom-up cost estimate in proposal.
• NSF may acquire an independent top-down estimate for comparison.
PMC2010: Partnering challenges on large projects W. Miller 12

13. Funding
Initiation Definition Execution
Pre-conceptual Conceptual Preliminary Construction Trans/Closeout
Final Design
Planning Design Design Operations
CD-0 CD-1 CD-2 CD-3 CD-4
DOE-SC LIC Prelim Engr & Design (PED) LIC Construction
Program Funds Program Ops Funds
Major Item of Equipment (MIE)
Two modes for intramural projects: Line Item Construction (LIC) or
• Separate funds support research Major Item of Equipment (MIE, no major construction)
Formulation Implementation
Pre-Phase A Phase A Phase B Phase C Phase D Phase E
Assembly,
Concept Concept & Prelim Design & Final Design &
Integ & Test, Operations
Studies Tech Devel Tech Completion Fabrication* Launch
KDP-A KDP-B KDP-C KDP-D KDP-E
NASA-SMD Program Funds, project line-item MO&DA
Mission Operations
and Data Analysis
• Separate funds support research (Research and Analysis, R&A)
Readiness NSB Approved
Horizon planning and Preliminary
Final Design Construction Operations
Conceptual Design Design
NSF Program Funds (Research & Related Activity, R&RA) MREFC R&RA
Major Research Equipment
and Facilities Construction
• Same funds support research (Research & Related Activities, R&RA) Requires separate appropriation
PMC2010: Partnering challenges on large projects W. Miller 13

14. Scope
 How do partner processes and practices
align? (Start with agencies)
 What are the challenges and best practices
for partnering?  Try to “systematize”
Sources:
• Interviews with ~45 stakeholders for study
• “Lessons learned” and other reports
PMC2010: Partnering challenges on large projects W. Miller 14

15. Some recent cases of major partnered projects
NASA DOE NSF International
Ulysses ▲ ESA
Cassini ▲ ESA, ASI
Tropical Rainfall Measuring Mission (TRMM) ▲ Japan/JAXA
Global Precipitation Measurement (GPM) ▲ Japan/JAXA
Fermi (operating) ▲ ▲
CERN
LHC (operating) ▲ ▲
members
International Ocean Drilling Program (IODP) ▲ Japan/Mext
Europe,
Atacama Large Millimeter Array ▲
Japan, Asia
PMC2010: Partnering challenges on large projects W. Miller 15

16. Examples of interagency Joint Oversight Groups (JOG)
• DOE-NASA: FERMI, Joint Dark Energy Mission
• NSF-DOE: U.S. Large Hadron Collider (LHC), Deep
Underground Science and Engineering Laboratory
(DUSEL, in planning)
PMC2010: Partnering challenges on large projects W. Miller 16

17. Partnering  Levels of impedance (mis)matches
Capabilities
& Practices
Strategic
valuation
Technical
domains
Management
practices
 Better planning could involve assessment of COMPLEMENTARITY
across capabilities, practices, lessons learned, etc…)
-Source: R. Staffin, 14Feb2006, FY06 presentation to HEPAP, www.er.doe.gov/hep/files/pdfs/HEPAPFeb142005Staffin.pdf
PMC2010: Partnering challenges on large projects W. Miller 17

18. Challenges: Strategic Management Technical
• Valuation. How highly does each partner prioritize the
project – and how much is it willing to spend based on that
priority? What does it expect as an outcome (science, jobs,
prestige) and when? Where does it fit in the partner's national
plan?
• Goals and roles. Are the overall goals and specific
objectives shared? Does each partner want a leadership or
participatory role? How committed are the commitments?
• Approval and allocation. Which hoops does each partner
have to jump through to obtain funding and approval? What
are the pressures? What is the funding source and cycle?
How long does prioritization and approval typically take?
PMC2010: Partnering challenges on large projects W. Miller 18

19. Challenges: Strategic Management Technical
• Role of project management. Is project management valued? Are
managers well chosen, given sufficient authority? Who do they report to?
• Project definition and baselining. How are requirements, risks and
baseline elements defined, documented and managed? Change control?
How much emphasis placed on system I&T and operations planning?
• Budgeting. Can be large differences in costing – e.g. labor cost inclusion
in project budgets. Can funds be used early, carried-over? Contribution
approach (in kind, etc.)? How are contingency and reserve handled?
• Oversight. Partners may not perform reviews with the same rigor or
frequency. Identified problems may be addressed with less urgency.
What systems do partners use for tracking and quantifying performance?
• Cultural differences. Sense of urgency to obtain desired outcome?
Consequences of (and solutions adopted for) cost overruns, late
development, poor performance, outright failure?
PMC2010: Partnering challenges on large projects W. Miller 19

20. Challenges: Strategic Management Technical
• Complementarity. Synergies among partners in technical
know-how relative to requirements of the project? Is at least
one partner strong in each area? What are the gaps?
• Design process. What approaches are employed by each
partner? How are software and IT systems developed?
• Systems engineering. To what degree is systems
engineering valued by each partner? Is each partner familiar
with establishing interfaces? With I&T planning?
• HR. How do partners obtain their skill contributions – via
contractors, in-house staff, exchanges of personnel from
member academic institutions…? Workforce stablity?
PMC2010: Partnering challenges on large projects W. Miller 20

21. Synchronizing “On-ramps”: when does a project start?
Preconstruction Planning
Conceptualization
Conceptual Preliminary Final
?
“Mission-driven” – projects determined via strategically-defined goals and priorities
DOE • Projects identified in a SC 20-year prioritized facility plan1
• Establish “mission need”, feasibility at CD-0  Definition phase
NASA • Strategic and Science Plans2 based on Decadal Surveys, roadmaps.
Two flavors:
• Strategic missions are internally initiated (KDP-A) and managed
• “PI-led” missions are competed in Phase A  selected  Phase B
“Community-driven” – projects “bubble up” from the scientific disciplines
NSF • Peer-reviewed unsolicited proposals, workshops, studies, etc.
• Evolved concepts may be brought to development
1. Facilities for the Future of Science, A twenty year outlook, DOE/SC-0078, Dec 2003; and Four Years Later: an Interim
Report on Facilities for the Future…, Aug 2007. 2. NASA Strategic Plan, 2006; and NASA Science Plan 2007–2016
PMC2010: Partnering challenges on large projects W. Miller 21

22. Reported best practices – responses to challenges
• Early engagement. Agree on clear goals, timeframe, and effective
membership rules and governance structure. Having a dominant partner
may work best…
• Communication. Identify the appropriate interfaces at all levels. Maintain
open, frequent and honest communication.
• Complementarity. Learn partner enterprises and realities. Identify
technical domains required and respective partner(s) capabilities.
• Strong project management. Devise clear process for project
management and oversight. Aim for integrated a single project
management team, in place before funding begins and with effective
budget authority (not just a coordinator).
• Budgeting. Aim to adopt standard costing and budgeting techniques.
PMC2010: Partnering challenges on large projects W. Miller 22

23. Acronyms
CD Critical Decision (DOE) OECM Office of Engineering and Construction
Management (DOE)
CDR Conceptual Design Review (DOE, NSF)
Critical Design Review (NASA) OPA Office of Program Assessment (DOE/SC)
DDLFP Deputy Director for Large Facility Projects, PED Preliminary Engineering and Design
head of LFO (NSF) funding account (DOE)
EIR External Independent Review (DOE PDR Preliminary Design Review
OECM)
PNAR Preliminary Non-Advocate Review (MDR,
IPAO Independent Program Assessment & NASA)
Oversight Office (NASA)
R&RA Research and Related Activities funding
KDP Key Decision Point (NASA) account (NSF)
LFO Large Facilities Office (NSF) R&A Research and Analysis funding account
(NASA)
MO&DA Mission Operations and Data Analysis
funding account (NASA) SC Office of Science (DOE)
MREFC Major Research Equipment and Facilities SRB Standing Review Board (NASA)
Construction funding account (NSF)
SMD Science Missions Directorate (NASA)
NAR Non-Advocate Review (PDR, NASA)
PMC2010: Partnering challenges on large projects W. Miller 23