Proceedings of the 2013 Industrial and Systems Engineering Res.docx

Proceedings of the 2013 Industrial and Systems Engineering
Research Conference
A. Krishnamurthy and W.K.V. Chan, eds.
Effect of the Analysis of Alternatives on the DoD Acquisition
System
Eugene Rex L. Jalao; Danielle Worger; Teresa Wu, PhD
Arizona State University
Tempe, AZ, 85281
J. Robert Wirthlin, PhD; John M. Colombi, PhD
The Air Force Institute of Technology
Wright-Patterson AFB, OH 45430
Abstract
The Enterprise Requirements and Acquisition Model (ERAM) is
a discrete event simulation that models the major
tasks and decisions within the DoD acquisition system. A

majority of DoD acquisition projects are being completed
behind schedule and over budget. ERAM suggests process
improvements can have salutary effects. Hence,
enhancements in improving the end-to-end acquisition process
would be worthwhile. Until 2008, the Analysis of
Alternatives (AoA) process is a mandatory task for acquisition
category (ACAT) level 1 projects. As such expected
program completion time for ACAT 2 and ACAT3 categories is
shorter. Since 2008, the AoA became a required
procedure for all programs. However, to the best of our
knowledge, the impact of requiring all programs to complete
an AoA has not yet been studied in literature. This research
addresses this gap with two main contributions. First,
this research seeks to quantify the amount of delay on total
completion time when the AoA is required for all ACAT
programs. Secondly, the sensitivity of the processing time and
variability of the AoA process is simulated and its
effect is studied on total program completion time. Viable
policies and intervention strategies are then inferred from
these contributions to further improve acquisition program
completion time.
Keywords
DoD, Acquisition, Simulation, Analysis of Alternatives

1. Introduction
It is a known fact that a large number of Department of Defense
(DoD) projects are being completed behind
schedule and over-budget [1]. A Government Accountability
Office (GAO) report released in 2009 states that for the
DoD’s 2008 portfolio, on average a program faced a 22-month
delay and exceeded the original budget [2].
Generally, total cost growth has been consistent over the past
few decades with a recent assessment by [3] of 1.44 or
44% growth. The current DoD Acquisition system which is
composed of three separate and distinct processes,
including the Joint Capabilities Integration Development
System (JCIDS), the Planning, Programming, Budgeting &
Execution (PPBE) process, and the formal acquisition
development system outlined by the DoD 5000 series of
instructions, does not exist in a static environment. The system
is constantly being adjusted, either through policy
changes or statute [4-6]. Since the acquisition process is a
large, complex, socio-technological system, it is difficult
to determine which processes or factors affect performance
metrics like time, cost, and resource utilization. Hence,
alternative modeling tools to improve the DoD acquisition
process are the subject of current research.

In 2009, a discrete event simulation (DES) model called the
Enterprise Requirements and Acquisition Model
(ERAM) is developed by Wirthlin [7]. This model simulates the
actual acquisition processes of the US DoD using
the Air Force implementation of acquisition processes for
Acquisition Category (ACAT) levels as the basis of the
model. This is done in order to provide further insight and
understanding of the complex system’s behavior.
However, this research did not include new policies set forth by
the DoD, specifically the DoD Instruction 5000.02
that requires all programs to go through an Analysis of
Alternatives (AoA) process [6] before reaching Milestone A.
The AoA is a requirement for all military acquisition programs.
By definition, the AoA is an analytical comparison
of multiple alternatives process that needs to be performed prior
to committing resources to a given acquisition
program [8]. According to the DoD 5000.02 instruction, “The
AoA shall focus on identification and analysis of
2110
Jalao, Worger, Wirthlin, Colombi, Wu
alternatives, measures of effectiveness, cost, schedule, concepts
of operations, and overall risk. The AoA shall assess

the critical technology elements (CTEs) associated with each
proposed materiel solution, including technology
maturity, integration risk, manufacturing feasibility, and, where
necessary, technology maturation and
demonstration needs.” Through this requirement, an implicit
assumption is being made that this step will actually
shorten the overall lifecycle development time for a given
acquisition program and increase the quality of the final
form of the materiel solution. However to the best of our
knowledge this policy effect has not yet been
quantitatively studied in existing literature. Nevertheless, the
AoA theoretically contributes to longer program time
completion of all DoD acquisition projects as it is an additional
task that must be performed during the process.
Hence potential policies could be developed to counter the
effect of requiring the AoAs, e.g. acknowledging better
quality solutions earlier in development, which could be easily
translated to viable policies to further improve not
only the duration but the entire end-to-end DoD acquisition
system.
Against this background, this research addresses these
limitations by performing additional simulation and statistical
analysis on the ERAM model. The primary goal of this research
is to determine the effect on the total acquisition

program completion time by requiring all ACAT programs to go
through the AoA process. Furthermore, the effect
of reducing the variability and the time spent on the AoA
process is studied and potential intervention strategies are
developed. Additionally, this research provides viable policies
and discussion points from the intervention strategies
that could further reduce program completion time. The rest of
this paper is organized as follows. Section 2
provides an overview of the related literature on the AoA, while
Section 3 provides the methodology for the
analysis. Section 4 presents the results of the study while
section 5 provides conclusions and future research options.
2. Review of Literature
2.1 DoD Enterprise Requirements Acquisition Model (ERAM)
ERAM is originally an Arena simulation model which provides
the foundations for research on applying discrete
event system simulation to the DoD acquisition process.
Extensive validation of the model is done by comparing the
performance results of the model to the actual DoD acquisition
data and expert reviews. Initially, 20 intervention
strategies are explored using the ERAM model to assess the
potential of the total completion time reduction of DoD

acquisition projects. If all 20 interventions are implemented, a
20% reduction in the total program time could be
realized. It is found that the most effective interventions to
improve the system are those that reduce the variability
of the processes. Since its publication in 2009, the Arena model
has been translated to ExtendSim and extended by
The Aerospace Corporation’s Developmental Planning and
Architectures Division for use in the Concept
Development Center of the Space and Missile Systems Center at
Los Angeles AFB, CA [9]. Moreover, Montgomery
[10] provided the research for Aerospace to extend the model to
further include ACAT 2/3 programs along with
modeling the Rapid Acquisition process for space programs.
Table 1 summarizes the different versions of the
ERAM model. Please note the earlier version of ERAM (1.0)
does NOT implement AoA on all ACAT projects.
Table 1: ERAM Versions Adapted from [11]
Author Version Number Changes
Wirthlin [7] ERAM 1.0 Baseline Translation from Arena to
ExtendSim
Leach and Searle [9]
ERAM 1.1

Updates by the Aerospace Design Team and Served as new
baseline model
ERAM 1.2 Implemented new DoD 5000.02 policies
ERAM 2.0
Incorporated the global variables that modify acquisition
capabilities
ERAM 2.1 Incorporated the JCIDS review process
Montgomery [10] ERAM 2.2
Added more capabilities for ACAT 2/3 and Rapid Acquisition
Process
Verification and validation of the baseline distributions
included hand modeling, iterations of correction from
feedback of experts in all three branches of acquisition, and
comparison of schedule and budget information from
the DAMIR and SMART databases to distributions of the
schedule time of model-generated data [7].
2.2 Implementation of the Analysis of Alternatives
There are also several articles that illustrate the implementation
of the AoA in the DoD acquisition process.
Cervantes et al. [12] applies a rapid AoA implementation to find
a NATO Special Operations Headquarters (NSHQ)

2111
Air Wing. Georgiadis et al. [8] propose an analytical multiple
criteria decision making methodology to handle the
AoA. The AoA also plays a part in the replacement of the aging
presidential helicopter fleet [13].
Research also explores policies to improve the implementation
of the AoA in the end-to-end DoD acquisition
system. For example, Schank [14] identifies several important
factors for the success of the AoA specifically: (1) the
AoA must have a study plan that considers a wide range of
alternatives and must be flexible in the analysis
methodology (2) oversight committees that manage effect
relationships, (3) trade-off analysis should be conducted
on all alternatives, and (4) good estimation and recognition of
technical, design and production risks is a must. A
white paper from the Training and Doctrine Comman
(TRADOC) Analysis Center (TRAC) addresses two problems
of the implementation of the AoA specifically: (1) by clearly
describing and differentiating the purpose and scope of
each AoA for each milestone decision, (2) by clearly defining
and describing “materiel solution” as it can be used

interchangeably with the “alternatives” term [15]. Stadterman
et al. [16] propose improvements on the AoA in the
Weapon Systems Acquisition Reform Act of 2009. Among these
recommendations are: (1) all concerned parties
within the AoA should build a working relationship, (2) the
AoAs should focus on the decision choices and the
decision space, and have an achievable, affordable and
operationally relevant set of criteria and (3) the Army should
follow a formal analytical process that support the AoA
throughout the acquisition process. Additionally, Ford et al.
[17] improve the implementation of the AoA by incorporating
benefits in the decision methodology through the use
of a system dynamics model of a military operation and
integrating it with a Knowledge Value Added methodology.
Furthermore, in a Government Accountability Office (GAO)
report [18], the GAO recommends that the Joint
Requirements Oversight Council should establish a review
mechanism for AoA earlier in the acquisition process.
According to Roper [19], all alternatives being considered
within the AoA must go through consistent analysis
methodologies and assumptions in order to ensure
comparability. Hence, adopting these quality improvement
suggestions also imply a reduction in the length and/or the
variability in the AoA process. Doing so will allow

AoAs to obtain consistent AoA process times.
Based on this review, the following gaps in literature can be
gleaned:
• Currently, a quantitative study to assess the impact of
requiring AoA implementation in all ACAT
programs is lacking.
• Secondly, the effect of reducing the variability and length of
the AoA process has not yet been addressed as
a potential intervention to further improve the program
acquisition completion time.
3. Simulation Analysis
This research is composed of three distinct phases. The first
phase is a simulation study to determine the effect of the
AoA for ACAT 2 and ACAT 3 programs on the total completion
time. The second phase consists of two simulation
experiments in which the variability and the mean of the AoA
process time are reduced and its effect on total
completion time is determined. Lastly, the third phase is the
translation of results into recommendations and further
research.
3.1 Requiring an AoA on a Percentage of ACAT 2 and 3

Programs
The original ERAM 1.0 model is utilized since it is the baseline
model that did not require an AoA on all programs.
By varying the “ACAT 2 or ACAT 3 funding” process,
specifically the percentage of programs with funding
already available for an AoA, the effect of requiring a certain
percentage of programs to undergo an AoA on the
total program completion time, or the time the program
completes Milestone C (MS-C), is determined. Figure 1
presents the screenshot of the module. This module decides if
the program has enough funding to perform an AoA.
If it is found there is enough funding, the program will have to
undergo an AoA.
ACAT 1 programs are not included in the analysis since AoAs
are required previously. The baseline scenario is set
as such the probability that the program would be required to
undergo an AoA is 1%. This setting is set before the
DoD policy change in 2008 on AoA implementation. Please note
in [7], only ACAT 2 and 3 programs with
sufficient funding are needed to undergo an AoA. Simulation
trials each with 3000 iterations are run separately for
ACAT 2 and ACAT 3 programs. During each trial, the
probability that a program would need to perform an AoA is

increased incrementally, and its corresponding completion time
is tabulated. Each increase is compared to the
baseline trial. A t-test is then performed in order to determine if
there is a statistically significant difference between
the trial results and the baseline trial in terms of the time until
MS-C. Hence, the probabilities are incrementally
increased until a statistical significant difference from the
baseline is obtained at a 95% confidence level.
2112
Figure 1: Arena Screenshot of “ACAT 2 or ACAT 3 funding”
Process
3.2 Sensitivity Analysis on the AoA Process Parameters
The objective of this phase is to identify the impact of
improving the AoA process specifically its variability and its
program completion time. The ExtendSim model is utilized over
the Arena model because the model reflects the
most recent policy changes. The current AoA process is
distributed according to a triangular distribution with a
lower limit of 180, most likely value of 360, and an upper limit

of 720 days (Tria(180,360,720)), and all programs
are assumed to undergo an AoA. These settings constitute the
baseline trial for this phase. The ACAT 1 level
projects are utilized in this phase. The ExtendSim simulation
module being modified in the ERAM model is an
Activity block from the items library called “Analysis of
Alternatives.” Figure 2 presents a screenshot of the
ExtendSim activity.
This phase is composed of two simulation experiments. The first
experiment studies the effect of reducing the
variability of the AoA process time while the second section
deals with reducing the mean of the AoA process time.
For the first experiment, 3000 iterations are performed and
during each trial, and the variability of the time to
perform the AoA is adjusted. The variability is adjusted by
reducing the difference between the mode and the
maximum/minimum by a constant. Hence, the triangular
distribution would then have less variance. The variance of
the triangular distribution is reduced incrementally until a
statistical significant difference is obtained from the
baseline trial. A t-test is then performed in order to compare the
time to milestone C in the trial to the baseline trial.
Furthermore, the second simulation experiment deals with

reducing the AoA process time to determine its effect on
the time ACAT 1 programs reach Milestone C. Three thousand
iterations are again performed and for each trial, the
mean of the AoA program length distribution which is
distributed according to a triangular distribution
(Tria(180,360,720)) is reduced. The ratio between the minimum,
most likely, and maximum of 1:2:4 is maintained
as the parameters are changed. The mean completion time of the
triangular distribution of the AoA is again reduced
until a statistically significant difference from the baseline trial
is obtained. A t-test is again performed in order to
compare the time in the trials to the baseline trial.
3.3 Results and Interpretation
The final phase in this paper compiles the results from the tests
and translates them into recommendations for viable
policy changes. This phase includes identifying which
parameter adjustments improved the performance metrics and
making recommendations on further research, like interactions
among parameters.
2113

Figure 2: ExtendSim Screenshot of “Analysis of Alternatives”
Activity
4. Results and Discussions
Table 2 summarizes the results of the t-tests performed from
section 3.1 for ACAT 2 programs. The table shows a
subset of trials corresponding to 50%, 85%, 87.5%, 90%, 95%
or 99% of programs are required to undergo an AoA.
These settings are selected to show the sensitivity of changing
the percentage of programs required to undergo an
AoA. These are compared to the baseline AoA setting in which
only the funded programs (1%) are required to
undergo an AoA. The null hypothesis for the t-tests is
��:��
��% which corresponds to an insignificant
difference between the baseline and the ��percentage if not
rejected and alternative hypothesis��:��
��% if
there is significant difference.
Table 2: Summary of t-test Results of ACAT 2 Programs
% of Programs required to undergo an AoA
1%

(Baseline)
50% 85% 87.5% 90% 95% 99%
Average Time to
MS-C (Days)
3898.099 4001.104 4080.293 4093.409 4108.58 4115.981
4141.088
Standard Deviation
(Days)
1411.37 1550.684 1625.738 1634.814 1650.243 1655.492
1673.537
T-Value -1.34133 -1.78231 -1.90576 -2.04267 -2.04267 -
2.34049
Conclusion Fail to
Reject H0
Fail to
Reject H0
Fail to
Reject H0
Reject

H0
Reject
H0
Reject
H0
It is evident that when requiring an AoA on at most 87.5% of
the ACAT 2 programs will result in a failure to reject
that the �� percentage is similar to the baseline scenario.
Any percentage less than 87.5% will have no effect on the
average time a program arrives at MS-C. This means that a
majority of the ACAT 2 programs can be subject to the
AoA and no significant increases in the total completion time
can be obtained.
Table 3 summarizes the results of the t-tests performed for
ACAT 3 programs. The table shows a subset of trials
corresponding to 50%, 55%, 57.5%, 60%, 65% or 75% of
programs are required to undergo an AoA. Furthermore,
these settings are selected to show the sensitivity of changing
AoA. These are compared to the baseline in which only the
funded programs (1%) are required to undergo an AoA.

The null hypothesis for the t-tests is ��:��
��% which corresponds to no significant difference between
the
baseline and the �� percentage if not rejected and alternative
hypothesis��:��
��% if there is significant
difference.
2114
1%
(Baseline)
50% 55% 57.5% 60% 65% 75%
Average Time to
MS-C (Days)
3334.926 3470.918 3493.278 3512.661
3524.566 3551.2 3604.434

Standard Deviation
(Days)
1143.19 1314.598 1350.852 1365.653 1372.66 1392.929
1419.016
T-Value -1.63342 -1.86606 -2.08392 -2.2198 -2.51299 -3.1095
Conclusion Fail to
Reject H0
Fail to
Reject H0
Reject
H0
Reject
H0
Reject
H0
Reject
H0
It is evident that when requiring an AoA on at most 55% of the

ACAT 3 programs will result in a failure to reject
that the �� percentage is similar to the baseline scenario. On
the other hand, the AoA contributes to a significant
change in the total program completion time when at least
57.5% of the programs require AoA. In general, it is
evident from tables 2 and 3 that requiring an AoA on all
programs significantly increase the total completion time of
ACAT 2 and 3 programs when programs are required to undergo
an AoA.
Table 4 summarizes the results of the t-tests performed in terms
of reducing the variability of the AoA process time.
Four settings are tested specifically, Tria(270, 360, 540),
Tria(315, 360, 450), Tria(337.5, 360, 405), and Tria(360,
360, 360). These settings are selected to illustrate the
sensitivity of the variability of the AoA processing time from
the baseline scenario to a deterministic scenario (time is fixed
at 360 days) as is done in [7]. These settings are
compared to the baseline AoA process time which is distributed
Tria(180, 360, 720). The null hypothesis for the t-
tests is ��:��
�� which corresponds to no significant difference between
the baseline and the �
��AoA
triangular distribution variance setting if not rejected and

alternative hypothesis ��:��
�� if there is
significant difference.
Table 4: t-test Results for the Variance Reduction of AoA
Process Time
AoA Distribution Settings
Tria(180, 360, 720)
(Baseline)
Tria (270, 360,
540)
Tria (315, 360,
450)
Tria (337.5,
360, 405)
Tria (360, 360,
360)
Average Time to
MS-C (Days)
6903.567

6883.159 6867.149 6865.280
6848.717
Standard Deviation
(Days)
1584.258 1591.61 1591.442 1590.301 1576.526
T-Value 0.2528 0.4512 0.4745 0.6828
Conclusion Fail to Reject
H0
Fail to Reject
H0
Fail to Reject
H0
Fail to Reject
H0
It is evident that reducing the variability of the AoA process
time does not have an effect on the total time ACAT 1
programs reaches milestone C. Hence any improvement on the

AoA process to make it more consistent and standard
would not have a significant effect on the total program
completion time.
On the other hand, table 5 presents the results of reducing the
mean process time of the AoA from section 3.2 Four
settings are tested specifically, Tria(135, 270, 540), Tria(112.5,
225, 450), Tria(101.25, 202.5, 405), and Tria(90,
sensitivity of the length of the AoA processing time from the
baseline scenario to the fastest scenario (Tria(90, 180, 360)).
These settings are compared to the baseline AoA
process time which is distributed Tria(180, 360, 720). The null
hypothesis for the t-tests is ��:��
�� which
corresponds to no significant difference between the baseline
and the ��AoA triangular mean time distribution
setting if not rejected and alternative hypothesis��:��
�
�� if there is significant difference.
Table 5: t-test Results for the Mean Reduction of AoA Process
Time
Tria(180, 360, 720) Tria (135, Tria (112.5, Tria (101.25, Tria

(90, 180,
2115
(Baseline) 270, 540) 225, 450) 202.5, 405) 360)
Average Time to
MS-C (Days)
6903.567 6827.941 6752.989 6732.038 6681.659
Standard Deviation
(Days)
1584.258 1597.607 1578.558 1567.143 1574.53
T-Value 0.9351 1.8732 2.1415 2.7640
H0
Fail to Reject
H0
Reject H0 Reject H0
It is evident that reducing the AoA process time to at the least

Tria(112.5, 225, 450) will result in a failure to reject
This implies that reducing the AoA process time to at most
Tria(101.25, 202.5, 405) contributes to a significant change in
the total program completion time.
5. Conclusions
The analysis of alternatives (AoA) process is a new requirement
that all acquisition category (ACAT) level projects
to be completed by the US government are required to undergo.
Previously, this process is only required on all
ACAT 1 programs and ACAT 2 and 3 programs with sufficient
funding. Hence, program completion is expected to
increase further given that a majority of the programs finish
over budget and with a delayed schedule when AoA is
required on all programs. This concept is validated through
simulation documented in this paper. By requiring all
programs to undergo an AoA, significant completion delays are
observed. Furthermore, this paper provides
quantitative insights on the amount of time that the AoA
contributes to the overall DoD end-to-end process time
through Wirthlin [7]’s discrete event system simulation model.
Through additional simulation analysis, it is inferred that a

AoA without significantly affecting completion time. However,
requiring an AoA on ACAT 3 programs has a
significant effect on program completion time. A frequently
cited reason for not conducting an AoA is that an AoA
causes a significant delay in the program. The simulation results
in this paper should provide strength to the
argument towards providing funds for full AoAs. However, as
there is a percentage of programs where the AoA can
cause significant delays in program completion (87.5% and 55%
for ACAT II and ACAT III respectively), policy
improvements with regards to selection criteria for programs
that must undergo an AoA may be beneficial.
Support for this conclusion can be found in the system
engineering case study for the A-10 Thunderbolt II
performed by [20]. From this case study, it can be seen that the
correct level of focus is needed for an AoA to be
beneficial. During the development of the A-10 three
prototyping studies are performed. These prototyping efforts
are analogous to an AoA. One prototype is a system level test
in the form of a fly-off. The other two are subsystem
prototyping of the guns and ammunition for the aircraft. While
the system level prototyping provided little
meaningful information, the two subsystem prototyping efforts

provide benefits to the DoD in terms of both overall
cost reduction and design improvements. The findings from the
case study in conjunction with the simulation results
in this paper indicate that policy applying tailored criteria for
ACAT 2 and ACAT 3 AoAs for programs that must
undergo an AoA could be beneficial.
Furthermore, findings from this paper show that any variance
reduction improvements or standardizations on the
AoA process would not affect the total program completion time
of ACAT 1 programs. However, any
improvements that reduce the processing time of the AoA, or in
other words making it more efficient and lean
would significantly reduce program completion time. With the
DoD’s requirement to conduct an AoA, an implicit
assumption has been made that the step will shorten the overall
lifecycle development. The simulation’s finding
suggests that from a statistical perspective, the DoD must be
expeditious during the AoA process, or the value
sought from requiring the AoA decreases. Hence, policy
improvements with regards to an AoA should focus on
reducing processing and execution time of the AoA without
sacrificing quality in its output.

The use of simulation in this study places some limitations on
this research as well as provides opportunities for
further research. One limitation is that the simulation takes a
program rather than a portfolio perspective. Previously,
fewer AoAs were performed because of lack of funding. The
need to allocate funding to AoAs for ACAT 2 and
ACAT 3 programs may result in more delays across the DoD
portfolio because funding may be spread too thinly.
This effect cannot be illustrated by the model if it is present.
Further research may be needed to investigate this
2116
possibility. In addition, AoAs were implemented with the
assumption that overall program quality would increase
with the inclusion of an AoA. The simulation does not
incorporate the interactions between the AoA and quality.
Further study can be conducted to investigate these interactions
or add them to the ERAM simulation.
Acknowledgements
This publication is developed under work supported by the
Naval Postgraduate School Acquisition Research
Program Assistance Agreement No. N6227112MPZG201

“Enterprise Requirements and Acquisition Model
(ERAM) Analysis and Extension” awarded by the Naval Supply
Systems Command (NAVSUP) Fleet Logistics
Center San Diego (NAVSUP FLC San Diego).
References
1. Schwartz,M., 2010, "Defense acquisitions: How DoD
acquires weapon systems and recent efforts to reform the
process," DIANE Publishing.
2. Sullivan,M.J., Andrew,C., Bowman,R., Fairbairn,B., Neill,S.,
Oppenheim,J., Schwenn,R., 2009, "Defense
Acquisitions.Measuring the Value of DOD's Weapon Programs
Requires Starting with Realistic Baselines,"
DTIC Document.
3. Arena,M.V., Obaid,Y., Galway L. A., Fox,B., Graser,J.C.,
Sollinger,J.M., Wu,F., W. Carolyn, 2006, "Impossible
Certainty: Cost Risk Analysis for Air Force Systems," Santa
Monica, CA: RAND Corporation.
4. Chairman Joint Chief of Staff (CJCS)., 2012, "CJCS
Instruction 3170.01H," CJCS.
5. US 111th Congress, 2009, "Weapon Systems Acquisition
Reform Act of 2009," Congress: Washington DC.
6. USD(AT&L), 2008, "Department of Defense Instruction
5000.02," Operation of the Defense Acquisition System.

7. Wirthlin,J.R., 2009, "Identifying enterprise leverage points in
Defense Acquisition Program performance," MIT
Dissertation.
8. Georgiadis,D.R., Mazzuchi,T.A., Sarkani,S., 2012, "Using
multi criteria decision making in analysis of
alternatives for selection of enabling technology," Systems
Engineering. doi: 10.1002/sys.21233.
9. Leach,D., Searle,C., 2011, "Department of Defense
Enterprise Requirements and Acquisition Model," Air Force
Institute of Technology Master's Thesis.
10. Montgomery,J., 2012, "Department of Defense Update to the
Enterprise Requirements and Acquisition Model,"
Air Force Institute of Technology Master's Thesis.
11. Houston,D., 2012, "Enterprise Requirements and
Acquisition Model," Unpublished Manuscript.
12. Cervantes,M.A., Enderton,C., Powers,J.S., 2012, "Rapid
Execution of an Analysis of Alternatives for NATO
Special Operations HQ A Smart Defence Approach," Monterey,
California. Naval Postgraduate School.
13. Sullivan,M.J., 2012, "Presidential Helicopter Acquisition:
Effort Delayed as DOD Adopts New Approach to
Balance Requirements, Costs, and Schedule," GAO.

14. Schank,J.F., 2012, "Analysis of Alternatives: Keys to
Success," DTIC Document.
15. TRAC., 2011, "TRADOC Assessment AoA Guidance in DoD
5000 Series," WSARA and In Practice.
Unpublished Manuscript.
16. Stadterman,T.J., Barber,E., Ground,A.P., 2012, "Improving
US Army Analysis of Alternatives to Better Address
the Weapon Systems Acquisition Reform Act of 2009," SSCF
RESEARCH REPORT.
17. Ford,D.N., Housel,T., Dillard,J.T., 2010, "Integrating
System Dynamics Modeling and Knowledge Value Added
for Improved Analysis of Alternatives: A Proof of Concept
Study," DTIC Document.
18. Sullivan,M.J., Schwenn,R.E., Bleicher,N.B.,
Marchesani,S.V., Patton,K.E., Richey,K.A., Russell,A.K.,
Tranquilli,N.A., 2011, "DoD Weapon Systems: Missed Trade-
off Opportunities During Requirements
Reviews," GAO.
19. Roper,M., 2010, "Risk Considerations in Pre-Milestone-A
Cost Analysis," . Office of the Deputy Assistant
Secretary of the Army for Cost and Economics.
20. Jacques,D.R., Strouble,D.D., 2010, "A-10 Thunderbolt II
(Warthog) Systems Engineering Case Study," Air

Force Center for Systems Engineering (AFIT/SY).
2117
Reproduced with permission of the copyright owner. Further
reproduction prohibited without
permission.
TIM7100
Data File 1
Assignment 1
1. Consider the set of all students enrolled in a semester-long
statistics course. Suppose you are interested in learning about
the current grade point averages (GPAs) of this group.
a. What is the population of interest?
b. What is the variable of interest?
c. Suppose you determine the GPA of every member of the
class. Would this represent a population or a sample?
d. Suppose you determine the GPA of 10 members of the class.
Would this represent a population or a sample?
2. A researcher argues that US Postal zip codes are quantitative
data because they measure location, with low numbers in the
east and high numbers in the west. Is she correct? Why or Why
not?
3. Colleges and universities require several items of information
from prospective students. Classify each of the types of data
below as either quantitative or qualitative.
a. High school GPA.
b. High school class rank.
c. Applicant’s score on the SAT.

d. Gender of the applicant.
e. Parents’ income.
f. Age of applicant.
4. Chemical and manufacturing plants often discharged toxic-
waste materials such as DDT prior to its ban in the US in the
1980s into nearby rivers and streams. In the mid-1990s, the US
Army Corps of Engineers conducted a study of fish in the
Tennessee River (in Alabama) and its three tributary creeks:
Flint Creek, Limestone Creek, and Spring Creek. A total of 144
fish were captured and five variables were measured for each
fish:
a. River/creek where the fish was captured
b. Species (channel catfish, largemouth bass, or smallmouth
buffalo fish)
c. Length (centimeters)
d. Weight (grams)
e. DDT concentration (parts per million)
Classify each of the five variables measured as quantitative or
qualitative.
5. Determine if the data described are qualitative or
quantitative.
a. The blood groups of A, B, AB, and O.
b. The white blood cell counts of different people, consisting of
the numbers f white blood cells per microliter of blood.
c. Breaking reaction times (in seconds) as measured as part of a
driver education program.
6. All highway bridges in the United States are inspected
periodically for structural deficiency by the Federal Highway
Administration (FHWA). Data from the FHWA inspections are
compiled into the National Bridge Inventory (NBI). Several of
the nearly 100 variables maintained by the NBI are listed below.
Classify each variable as quantitative or qualitative.
a. Length of maximum span (feet)
b. Number of vehicle lanes

c. Toll bridge (yes or no)
d. Average daily traffic
e. Condition of deck (good, fair, poor)
f. Bypass or detour length (miles)
g. Route type (interstate, U.S., state, county or city)
7. Refer to Problem 6. Using the FHWA inspection ratings,
each of the 470,515 bridges in the United States was
characterized as structurally deficient, functionally obsolete, or
safe. About 26% of the bridges were found to be structurally
deficient while 19% were functionally obsolete.
a. What is the variable of interest to the researchers?
b. Is the data set analyzed a population or a sample?
8. Pollsters regularly conduct opinion polls in the United States
to determine the popularity rating of the current president.
Suppose a poll is to be conducted tomorrow in which 2,000
individuals will be asked whether the president is doing a good
job or a bad job. The 2,000 individuals will be selected by
random-digit telephone dialing to landlines and asked the
questions over the phone.
a. What is the relevant population?
b. What is the variable of interest? Is it quantitative or
qualitative?
c. What is the sample?
d. How likely is the sample to be representative?
TA K I NG T H E SHOW ON T H E ROA D
Some of the workshops JPEO-CBD is conducting
in collaboration with DAU focus on understanding
contracting in order to achieve superior contract
vehicles and contractor performance, while others
seek ways to reduce administrative burden. With the
program now in its second year, the curricula were
recently offered to other PEOs. (SOURCE: JPEO-CBD)

118 Army AL&T Magazine July-September 2017
“Opportunities multiply as they are seized.”
– Sun Tzu
As the Trump administration completes its transition, Better
Buying Power may be replaced
with something di�erent, but the tenets and goals of acquisition
reform will remain largely
the same. All program executive o�ces within the O�ce of the
Assistant Secretary of the
Army for Acquisition, Logistics and Technology (OASA(ALT))
will be seeking ways to
reduce administrative lead time while �elding and maintaining
the best-quality equipment
and services possible within resource constraints.
�e Joint Program Executive O�ce for Chemical and Biological
Defense (JPEO-CBD)
has partnered with Defense Acquisition University (DAU)
Mission Assistance to sponsor
a series of workshops designed speci�cally to improve
acquisition timelines. �e curricula
are intended for pre-milestone A or B program teams of
Acquisition Category (ACAT)
III programs but can be tailored to any e�ort. �ey focus on
understanding contract-
ing and applying that understanding to achieve superior contract
vehicles and contractor
performance. Other workshops allow credentialed DAU
instructors and program teams
to immerse themselves in developing exceptional solicitation
documentation and �nding

ways to reduce the administrative burden. �is is achievable by
eliminating oversight that
is appropriate for ACAT I programs but overkill for the ACAT
III e�orts that make up the
bulk of the JPEO-CBD’s portfolio. �e JPEO-CBD is in the
second year of this program
and recently o�ered the curricula to other PEOs across
ASA(ALT).
by Mr. Jeff Megargel
JPEO - CBD seeks to reduce acquisition lead time through
tailored professional training workshops with DAU.
NO TIME
TO LOSE
A S C . A R M Y . M I L 119
W
O
R
K
F
O
R
C
E
NO MORE ‘CONTRACT OF THE DAY ’

Douglas Bryce, joint program executive o�cer, is convinced
that
program managers (PMs) must have more than a fundamental
knowledge of contracting to in�uence contracting-related deci-
sions that impact their programs. “Far too often, the program
management team throws their input over the wall to the con-
tract team, and 24 months later we have a contract,” Bryce said.
“�is leads to the ‘contract of the day’ approach. �e key is to
use
the right contract type and incentives for the program.”
With this goal in mind, Bryce directed his sta� to reach out to
DAU to create a “Contracting for Program Managers” workshop
that orients newly assigned program management personnel
to the art and science of government contracting. �e topics
include contracting strategies, types of contracts, incentivizing
contractor performance, the Federal Acquisition Regulation
(FAR) and Defense Federal Acquisition Regulation Supplement,
and how they all relate to “DOD Instruction 5000.02, Opera-
tion of the Defense Acquisition System.” �e intent is not to
create contracting experts, but to establish a level of
understand-
ing that facilitates proactive engagement with the contracting
community as the program management team plans acquisition
strategies.
�e JPEO-CBD also has assigned former civil service con-
tracting professionals to each program o�ce. �ey assist in
developing acquisition packages and liaise with their peers in
the supporting contracting activities. �is enables the program
teams to collaborate with contracting subject matter experts
who are also fully vested in program acquisition strategies. �e
result of this collaboration is acquisition packages that require
far less rework between the acquisition and contracting shops,

as well as procurement strategies that are more tailored to a spe-
ci�c requirement versus one size �ts all.
WOR K I NG S T E P BY S T E P
In partnership with DAU Mission Assistance, the JPEO-CBD
has tailored its workforce development
sessions to the various segments of the procurement cycle, such
as the workshop on building a
request for proposal (RFP) and the contracting officer’s
representative (COR) course on awarding
contracts. (SOURCE: JPEO-CBD)
NO TIME TO LOSE
�e JPEO-CBD also has sponsored
several workshops on contract incen-
tives, with a DAU subject matter expert
providing a comprehensive review of
contract incentives and their appropri-
ate use in acquisition programs. After
completing the workshop, participants
should understand the fundamentals of
how and when to incentivize contractor
performance, when cost or �xed-price
incentive contracts are appropriate and,
most importantly, how to discuss con-
tract incentives with the contracting
professionals during the formulation of
acquisition and procurement strategies.
Bryce requires new-start programs to com-
plete a streamlined acquisition strategy

development workshop well before mile-
stone A. �e workshop brings together
program teams, functional sta� and
user community stakeholders to address
major topic areas for development and
potential streamlining of their program
acquisition strategy. �e DAU instruc-
tor tailors the workshop to one program
and encourages optimal participation
from the stakeholder community. �is
always includes the contracting o�cer
and specialist, but also can include bud-
get analysts, legal advisers, small business
advocates and technical specialists who
might only engage for selected topics.
As an example, the workshop conducted
for the Enhanced Maritime Biological
Detection (EMBD) program included
participants from the U.S. Naval Sur-
face Warfare Center Dahlgren Division.
Representing the Navy user community,
they provided insight concerning the
challenges of upgrading a legacy sensor
system on a surface platform, including
compatibility with other shipboard sys-
tems and �elding the systems in line with
deployment schedules.
Over several iterations, the program teams
have universally praised the workshops
for facilitating an immersive environment
where the team can work collaboratively
and develop critical thinking skills and
ideas that are directly relevant to reduc-

ing the administrative burden as they
develop and gain approval for ACAT III
program acquisition strategies. �e lec-
tures cover multiple topics that must be
addressed in acquisition plans, including
risk management, a�ordability, should-
cost and supportability.
Immediately following the lecture, the
teams “murder board” each topic as it
relates to their program: Small teams
address each topic and document any
assumptions, constraints, risk mitiga-
tions and proposed solutions on a big
sheet of butcher block paper. At the end
of the session, each team briefs its �nd-
ings to the workshop as a whole. �e
program team members can tear o� the
page and carry their brainstorming back
to their workspaces to re�ne and include
in formal documentation.
“�e workshop is very helpful in breaking
down the components of the acquisition
strategy into manageable parts,” said
Michele Parrish, EMBD team lead. In
some cases, the workshops have revealed
the need for additional market research
or more detailed analysis of data rights
provisions. In others, the teams have
identi�ed how contracting methodolo-
gies can have a major impact on reducing
documentation requirements.
For instance, using existing multiple-
award contract vehicles often is more

e�cient than creating a new contract
vehicle speci�c to one requirement. �e
assumed duration to complete all the
steps necessary to award the typical
stand-alone single award contract today
is 18 to 24 months. But many program
teams are unaware that they have access
to existing contract vehicles that can
reduce procurement schedules by months.
�ey just need this input early enough to
incorporate it into the acquisition strat-
egy. Workshops can make teams aware of
this bene�t.
AN INTRODUCTION TO
NONTRADITIONAL METHODS
JPEO-CBD’s portfolio contains a number
of programs that are suited for procure-
ment within the commercial marketplace,
including vaccines and specialized tex-
tiles. As a result, the JPEO-CBD has
established an other transaction authority
(OTA) consortium in procuring vaccines
and therapeutic drugs. “Companies that
have never participated in a FAR-based
procurement are now in line to support
multiple Joint Vaccine Acquisition Pro-
gram requirements,” said Gary Wright,
director of the JPEO-CBD Contracting
Management O�ce. �e Joint Project
Manager for Protection is also using an
OTA to work with manufacturers of
specialty fabrics and materials. �e proj-
ect o�ce is seeking advanced chemical
The OTA training

“was perfect to
help expand our
horizons and
develop a full
acquisition tool set.”
A S C . A R M Y . M I L 121
W
O
R
K
F
O
R
C
E
and biological protection ensembles and
design concepts that might be used in
handwear, footwear and respiratory pro-
tection systems.
Companies that produce cutting-edge
technologies might not be willing to
conform to accounting practices or
other regulations that are mandatory
for participation in DOD programs.
For instance, maintaining a compliant
accounting system is extremely expen-

sive, and the revenue that results from a
given DOD program may be immaterial
in a company’s overall income stream.
OTA agreements allow such companies
to provide prototypes for the JPEO-
CBD programs without having to meet
the many regulatory requirements of an
arrangement governed by the FAR.
To implement and sustain the consortium,
the JPEO-CBD created two workshops
that enabled potential program teams to
leverage OTAs. �e training is divided
into an introductory workshop that
allows program teams to test the waters
and an advanced workshop that goes
through the detailed process for estab-
lishing and managing an OTA program.
In January, DAU conducted two basic
workshops at U.S. Special Operations
Command using the JPEO-CBD spon-
sored curriculum. One was tailored to
the contracting community and the
other to the program managers. �e OTA
training “was perfect to help expand our
horizons and develop a full acquisition
tool set,” said Col. John Reim, program
executive o�cer for Special Operations
Forces – Warrior. “I suspect that you
will be hearing more from SOF AT&L
[Special Operations Forces Acquisition,
Technology and Logistics] in the near
future for additional information and les-
sons learned.”

�e JPEO-CBD also o�ers workshops
focused on developing high-quality
solicitation documentation and on train-
ing government personnel to serve on
source selection evaluation boards. �e
RFP development workshop capitalizes
on the work already completed by acqui-
sition teams but blends in best practices as
presented by DAU. �e DAU instructors
have the bene�t of observing program
teams across DOD and can o�er lessons
learned as they lead the team through
re�ning its documentation.
For instance, despite the best e�orts of
contracting and program management
personnel, some solicitations require mul-
tiple amendments following release as a
result of industry feedback and questions
regarding the documentation. Borrow-
ing from industry practice, the Joint
Project Manager for Nuclear, Biological
and Chemical Contamination Avoid-
ance adopted a process in which a senior
contracting expert performs a formal
cross-check of the draft solicitation sec-
tions, including the Statement of Work
(Section C), Instructions to O�erors
(Section L) and Evaluation Criteria (Sec-
tion M). �e emphasis is on ensuring that
statements of work re�ect performance
speci�cations and that instructions to
o�erors and evaluation criteria are opti-
mized so that the government procures
the right solutions for its acquisition

needs.
Using this process, the program team
corrects the draft documentation
before it goes to the contracting activity.
Normally, these major sections of a solici-
tation are prepared by two completely
D OU BL E T E A M I NG TO DE L I V E R S OLU T ION S
The workshops developed by JPEO-CBD and DAU empower
program teams to accelerate
schedules and reduce costs while maintaining high standards of
capability delivery. (SOURCE:
U.S. Army Acquisition Support Center/chipstudio/iStock)
NO TIME TO LOSE
di�erent interests: �e acquisition team generates the statement
of work and the performance speci�cation, but the contracting
o�cer generates sections L and M—often weeks if not months
later. �e workshop seeks to complete all major sections in a
deliberate and fully integrated environment. �e result is less
confusion among o�erors when they prepare proposals; more
realistic cost proposals, as o�erors are less likely to mitigate
risks
through management reserves; and better-performing programs
post-award because the government and the winning o�erors
have a clearer understanding of what the program really needs
to provide the capability to the war�ghter.
After the solicitation is released and before receipt of proposals,
the JPEO-CBD’s source selection evaluation boards conduct a

practice evaluation of the proposals using the actual solicitation
documentation. �e source selection workshop (DAU Course
WSC 005) covers the roles of each board member, drafts
practice
source-selection decision documentation and has the team con-
duct mock debriefs for unsuccessful o�erors.
Finally, the JPEO-CBD and DAU are providing the Acquisition
Program Transition Workshop (DAU course WSM 011), which
brings together the program teams from the government and
the winning o�eror to reach a common understanding of the
government’s expectations and how the contract will be man-
aged. According to Ashton B. Carter, former secretary of
defense,
“�e bene�ts of this workshop include early alignment of gov-
ernment and industry team organizations, publication of road
maps to integrated baseline review and other near-term plan-
ning events, agreement on management of scope and processes,
and resolution of issues including di�erences in interpretation
of
contracts and other documents.”
�roughout the workshop, DAU instructors highlight best prac-
tices they have observed across DOD. In nearly all cases, there
are opportunities to improve the quality of deliverables—such
as monthly cost reports or administrative processes—simply by
demonstrating how a company uses automated tools and skilled
employees to accomplish the same tasks on other contracts.
�ere is opportunity for open dialogue that enables the com-
pany to demonstrate the value-added aspects of its reporting
and to tailor the soft deliverables—e.g., monthly cost reports or
government-furnished property inventories—based upon what
the COR can really use. �is workshop provides a forum in
which the government can meet face to face with its counter-

parts. It is an opportunity to emphasize the need to manage
program risks and establish an e�ective methodology to lever-
age the contractor’s capabilities while meeting the
government’s
expectations.
CONCLUSION
Although Better Buying Power may be replaced with new direc-
tion for achieving acquisition objectives, the basic tenets are
likely to remain the same. �e JPEO-CBD has demonstrated
that PEOs can tailor DAU expertise to achieve material results
at the program level. �e key is to leverage the knowledge resi-
dent at DAU to tailor training for each program team depending
upon where it is in the acquisition cycle, and then conduct the
workshops in an immersive environment where teams can
concentrate on producing quality results in collaboration with
functional sta� and technical experts.
�e JPEO-CBD is planning a full calendar in FY18, including
workshops for all PEOs across ASA(ALT). It is the JPEO-
CBD’s desire to conduct multiple iterations of the training in
the �ve geographic areas where the PEOs are concentrated. �e
Program Executive O�ce for Combat Support and Combat
Service Support has requested a streamlined acquisition strategy
development workshop in the Warren, Michigan, area this sum-
mer. Some of the curricula, including the OTA workshops, are
appropriate for any agency within DOD and beyond.
For more information, contact the author at je�rey.w.megargel.
[email protected], or go to https://www.jpeocbd.osd.mil/ or
https://
www.dau.mil/consulting-services/.
MR. JEFF MEGARGEL is a former Marine Corps contracting
o�cer and vice president with Science Applications
International

Corp. He is currently supporting the JPEO-CBD Contracting
Management O�ce as an employee of Moss Cape LLC. He
holds
an M.S. in contract and acquisition management from the Naval
Postgraduate School and a B.A. in history from �e
Pennsylvania State University. He specializes in helping
program teams develop contracting strategies.
“The workshop is very helpful in
breaking down the components
of the acquisition strategy into
manageable parts.”
A S C . A R M Y . M I L 123
W
O
R
K
F
O
R
C
E
mailto:[email protected]
https://www.jpeocbd.osd.mil/
https://www.dau.mil/consulting-services/
Reproduced with permission of copyright owner.

Further reproduction prohibited without permission.
TA K I NG T H E SHOW ON T H E ROA D
Some of the workshops JPEO-CBD is conducting
in collaboration with DAU focus on understanding
contracting in order to achieve superior contract
vehicles and contractor performance, while others
seek ways to reduce administrative burden. With the
program now in its second year, the curricula were
recently offered to other PEOs. (SOURCE: JPEO-CBD)
“Opportunities multiply as they are seized.”
– Sun Tzu
As the Trump administration completes its transition, Better
Buying Power may be replaced
with something di�erent, but the tenets and goals of acquisition
reform will remain largely
the same. All program executive o�ces within the O�ce of the
Assistant Secretary of the
Army for Acquisition, Logistics and Technology (OASA(ALT))
will be seeking ways to
reduce administrative lead time while �elding and maintaining
the best-quality equipment
and services possible within resource constraints.
�e Joint Program Executive O�ce for Chemical and Biological
Defense (JPEO-CBD)

has partnered with Defense Acquisition University (DAU)
Mission Assistance to sponsor
a series of workshops designed speci�cally to improve
acquisition timelines. �e curricula
are intended for pre-milestone A or B program teams of
Acquisition Category (ACAT)
III programs but can be tailored to any e�ort. �ey focus on
understanding contract-
ing and applying that understanding to achieve superior contract
vehicles and contractor
performance. Other workshops allow credentialed DAU
instructors and program teams
to immerse themselves in developing exceptional solicitation
documentation and �nding
ways to reduce the administrative burden. �is is achievable by
eliminating oversight that
is appropriate for ACAT I programs but overkill for the ACAT
III e�orts that make up the
bulk of the JPEO-CBD’s portfolio. �e JPEO-CBD is in the
second year of this program
and recently o�ered the curricula to other PEOs across
ASA(ALT).
by Mr. Jeff Megargel
JPEO - CBD seeks to reduce acquisition lead time through
tailored professional training workshops with DAU.
NO TIME
TO LOSE
A S C . A R M Y . M I L 119
W
O

R
K
F
O
R
C
E
NO MORE ‘CONTRACT OF THE DAY ’
Douglas Bryce, joint program executive o�cer, is convinced
that
program managers (PMs) must have more than a fundamental
knowledge of contracting to in�uence contracting-related deci-
sions that impact their programs. “Far too often, the program
management team throws their input over the wall to the con-
tract team, and 24 months later we have a contract,” Bryce said.
“�is leads to the ‘contract of the day’ approach. �e key is to
use
the right contract type and incentives for the program.”
With this goal in mind, Bryce directed his sta� to reach out to
DAU to create a “Contracting for Program Managers” workshop
that orients newly assigned program management personnel
to the art and science of government contracting. �e topics
include contracting strategies, types of contracts, incentivizing
contractor performance, the Federal Acquisition Regulation
(FAR) and Defense Federal Acquisition Regulation Supplement,
and how they all relate to “DOD Instruction 5000.02, Opera-
tion of the Defense Acquisition System.” �e intent is not to

create contracting experts, but to establish a level of
understand-
ing that facilitates proactive engagement with the contracting
community as the program management team plans acquisition
strategies.
�e JPEO-CBD also has assigned former civil service con-
tracting professionals to each program o�ce. �ey assist in
developing acquisition packages and liaise with their peers in
the supporting contracting activities. �is enables the program
teams to collaborate with contracting subject matter experts
who are also fully vested in program acquisition strategies. �e
result of this collaboration is acquisition packages that require
far less rework between the acquisition and contracting shops,
as well as procurement strategies that are more tailored to a spe-
ci�c requirement versus one size �ts all.
WOR K I NG S T E P BY S T E P
In partnership with DAU Mission Assistance, the JPEO-CBD
has tailored its workforce development
sessions to the various segments of the procurement cycle, such
as the workshop on building a
request for proposal (RFP) and the contracting officer’s
representative (COR) course on awarding
contracts. (SOURCE: JPEO-CBD)
NO TIME TO LOSE
�e JPEO-CBD also has sponsored
several workshops on contract incen-
tives, with a DAU subject matter expert
providing a comprehensive review of

contract incentives and their appropri-
ate use in acquisition programs. After
completing the workshop, participants
should understand the fundamentals of
how and when to incentivize contractor
performance, when cost or �xed-price
incentive contracts are appropriate and,
most importantly, how to discuss con-
tract incentives with the contracting
professionals during the formulation of
acquisition and procurement strategies.
Bryce requires new-start programs to com-
plete a streamlined acquisition strategy
development workshop well before mile-
stone A. �e workshop brings together
program teams, functional sta� and
user community stakeholders to address
major topic areas for development and
potential streamlining of their program
acquisition strategy. �e DAU instruc-
tor tailors the workshop to one program
and encourages optimal participation
from the stakeholder community. �is
always includes the contracting o�cer
and specialist, but also can include bud-
get analysts, legal advisers, small business
advocates and technical specialists who
might only engage for selected topics.
As an example, the workshop conducted
for the Enhanced Maritime Biological
Detection (EMBD) program included
participants from the U.S. Naval Sur-
face Warfare Center Dahlgren Division.

Representing the Navy user community,
they provided insight concerning the
challenges of upgrading a legacy sensor
system on a surface platform, including
compatibility with other shipboard sys-
tems and �elding the systems in line with
deployment schedules.
Over several iterations, the program teams
have universally praised the workshops
for facilitating an immersive environment
where the team can work collaboratively
and develop critical thinking skills and
ideas that are directly relevant to reduc-
ing the administrative burden as they
develop and gain approval for ACAT III
program acquisition strategies. �e lec-
tures cover multiple topics that must be
addressed in acquisition plans, including
risk management, a�ordability, should-
cost and supportability.
Immediately following the lecture, the
teams “murder board” each topic as it
relates to their program: Small teams
address each topic and document any
assumptions, constraints, risk mitiga-
tions and proposed solutions on a big
sheet of butcher block paper. At the end
of the session, each team briefs its �nd-
ings to the workshop as a whole. �e
program team members can tear o� the
page and carry their brainstorming back
to their workspaces to re�ne and include
in formal documentation.

“�e workshop is very helpful in breaking
down the components of the acquisition
strategy into manageable parts,” said
Michele Parrish, EMBD team lead. In
some cases, the workshops have revealed
the need for additional market research
or more detailed analysis of data rights
provisions. In others, the teams have
identi�ed how contracting methodolo-
gies can have a major impact on reducing
documentation requirements.
For instance, using existing multiple-
award contract vehicles often is more
e�cient than creating a new contract
vehicle speci�c to one requirement. �e
assumed duration to complete all the
steps necessary to award the typical
stand-alone single award contract today
is 18 to 24 months. But many program
teams are unaware that they have access
to existing contract vehicles that can
reduce procurement schedules by months.
�ey just need this input early enough to
incorporate it into the acquisition strat-
egy. Workshops can make teams aware of
this bene�t.
AN INTRODUCTION TO
NONTRADITIONAL METHODS
JPEO-CBD’s portfolio contains a number
of programs that are suited for procure-
ment within the commercial marketplace,
including vaccines and specialized tex-
tiles. As a result, the JPEO-CBD has
established an other transaction authority

(OTA) consortium in procuring vaccines
and therapeutic drugs. “Companies that
have never participated in a FAR-based
procurement are now in line to support
multiple Joint Vaccine Acquisition Pro-
gram requirements,” said Gary Wright,
director of the JPEO-CBD Contracting
Management O�ce. �e Joint Project
Manager for Protection is also using an
OTA to work with manufacturers of
specialty fabrics and materials. �e proj-
ect o�ce is seeking advanced chemical
The OTA training
“was perfect to
help expand our
horizons and
develop a full
acquisition tool set.”
A S C . A R M Y . M I L 121
W
O
R
K
F
O
R
C
E

and biological protection ensembles and
design concepts that might be used in
handwear, footwear and respiratory pro-
tection systems.
Companies that produce cutting-edge
technologies might not be willing to
conform to accounting practices or
other regulations that are mandatory
for participation in DOD programs.
For instance, maintaining a compliant
accounting system is extremely expen-
sive, and the revenue that results from a
given DOD program may be immaterial
in a company’s overall income stream.
OTA agreements allow such companies
to provide prototypes for the JPEO-
CBD programs without having to meet
the many regulatory requirements of an
arrangement governed by the FAR.
To implement and sustain the consortium,
the JPEO-CBD created two workshops
that enabled potential program teams to
leverage OTAs. �e training is divided
into an introductory workshop that
allows program teams to test the waters
and an advanced workshop that goes
through the detailed process for estab-
lishing and managing an OTA program.
In January, DAU conducted two basic
workshops at U.S. Special Operations
Command using the JPEO-CBD spon-

sored curriculum. One was tailored to
the contracting community and the
other to the program managers. �e OTA
training “was perfect to help expand our
horizons and develop a full acquisition
tool set,” said Col. John Reim, program
executive o�cer for Special Operations
Forces – Warrior. “I suspect that you
will be hearing more from SOF AT&L
[Special Operations Forces Acquisition,
Technology and Logistics] in the near
future for additional information and les-
sons learned.”
�e JPEO-CBD also o�ers workshops
focused on developing high-quality
solicitation documentation and on train-
ing government personnel to serve on
source selection evaluation boards. �e
RFP development workshop capitalizes
on the work already completed by acqui-
sition teams but blends in best practices as
presented by DAU. �e DAU instructors
have the bene�t of observing program
teams across DOD and can o�er lessons
learned as they lead the team through
re�ning its documentation.
For instance, despite the best e�orts of
contracting and program management
personnel, some solicitations require mul-
tiple amendments following release as a
result of industry feedback and questions
regarding the documentation. Borrow-
ing from industry practice, the Joint

Project Manager for Nuclear, Biological
and Chemical Contamination Avoid-
ance adopted a process in which a senior
contracting expert performs a formal
cross-check of the draft solicitation sec-
tions, including the Statement of Work
(Section C), Instructions to O�erors
(Section L) and Evaluation Criteria (Sec-
tion M). �e emphasis is on ensuring that
statements of work re�ect performance
speci�cations and that instructions to
o�erors and evaluation criteria are opti-
mized so that the government procures
the right solutions for its acquisition
needs.
Using this process, the program team
corrects the draft documentation
before it goes to the contracting activity.
Normally, these major sections of a solici-
tation are prepared by two completely
D OU BL E T E A M I NG TO DE L I V E R S OLU T ION S
The workshops developed by JPEO-CBD and DAU empower
program teams to accelerate
schedules and reduce costs while maintaining high standards of
capability delivery. (SOURCE:
U.S. Army Acquisition Support Center/chipstudio/iStock)
NO TIME TO LOSE
di�erent interests: �e acquisition team generates the statement

of work and the performance speci�cation, but the contracting
o�cer generates sections L and M—often weeks if not months
later. �e workshop seeks to complete all major sections in a
deliberate and fully integrated environment. �e result is less
confusion among o�erors when they prepare proposals; more
realistic cost proposals, as o�erors are less likely to mitigate
risks
through management reserves; and better-performing programs
post-award because the government and the winning o�erors
have a clearer understanding of what the program really needs
to provide the capability to the war�ghter.
After the solicitation is released and before receipt of proposals,
the JPEO-CBD’s source selection evaluation boards conduct a
practice evaluation of the proposals using the actual solicitation
documentation. �e source selection workshop (DAU Course
WSC 005) covers the roles of each board member, drafts
practice
source-selection decision documentation and has the team con-
duct mock debriefs for unsuccessful o�erors.
Finally, the JPEO-CBD and DAU are providing the Acquisition
Program Transition Workshop (DAU course WSM 011), which
brings together the program teams from the government and
the winning o�eror to reach a common understanding of the
government’s expectations and how the contract will be man-
aged. According to Ashton B. Carter, former secretary of
defense,
“�e bene�ts of this workshop include early alignment of gov-
ernment and industry team organizations, publication of road
maps to integrated baseline review and other near-term plan-
ning events, agreement on management of scope and processes,
and resolution of issues including di�erences in interpretation
of
contracts and other documents.”

�roughout the workshop, DAU instructors highlight best prac-
tices they have observed across DOD. In nearly all cases, there
are opportunities to improve the quality of deliverables—such
as monthly cost reports or administrative processes—simply by
demonstrating how a company uses automated tools and skilled
employees to accomplish the same tasks on other contracts.
�ere is opportunity for open dialogue that enables the com-
pany to demonstrate the value-added aspects of its reporting
and to tailor the soft deliverables—e.g., monthly cost reports or
government-furnished property inventories—based upon what
the COR can really use. �is workshop provides a forum in
which the government can meet face to face with its counter-
parts. It is an opportunity to emphasize the need to manage
program risks and establish an e�ective methodology to lever-
age the contractor’s capabilities while meeting the
government’s
expectations.
CONCLUSION
Although Better Buying Power may be replaced with new direc-
tion for achieving acquisition objectives, the basic tenets are
likely to remain the same. �e JPEO-CBD has demonstrated
that PEOs can tailor DAU expertise to achieve material results
at the program level. �e key is to leverage the knowledge resi-
dent at DAU to tailor training for each program team depending
upon where it is in the acquisition cycle, and then conduct the
workshops in an immersive environment where teams can
concentrate on producing quality results in collaboration with
functional sta� and technical experts.
�e JPEO-CBD is planning a full calendar in FY18, including
workshops for all PEOs across ASA(ALT). It is the JPEO-
CBD’s desire to conduct multiple iterations of the training in
the �ve geographic areas where the PEOs are concentrated. �e

Program Executive O�ce for Combat Support and Combat
Service Support has requested a streamlined acquisition strategy
development workshop in the Warren, Michigan, area this sum-
mer. Some of the curricula, including the OTA workshops, are
appropriate for any agency within DOD and beyond.
For more information, contact the author at je�rey.w.megargel.
[email protected], or go to https://www.jpeocbd.osd.mil/ or
https://
www.dau.mil/consulting-services/.
MR. JEFF MEGARGEL is a former Marine Corps contracting
o�cer and vice president with Science Applications
International
Corp. He is currently supporting the JPEO-CBD Contracting
Management O�ce as an employee of Moss Cape LLC. He
holds
an M.S. in contract and acquisition management from the Naval
Postgraduate School and a B.A. in history from �e
Pennsylvania State University. He specializes in helping
program teams develop contracting strategies.
“The workshop is very helpful in
breaking down the components
of the acquisition strategy into
manageable parts.”
A S C . A R M Y . M I L 123
W
O
R
K
F

O
R
C
E
mailto:[email protected]
https://www.jpeocbd.osd.mil/
Reproduced with permission of copyright owner.
Further reproduction prohibited without permission.
Proceedings of the 2013 Industrial and Systems Engineering
Research Conference
A. Krishnamurthy and W.K.V. Chan, eds.
Effect of the Analysis of Alternatives on the DoD Acquisition
System
Eugene Rex L. Jalao; Danielle Worger; Teresa Wu, PhD
Arizona State University
Tempe, AZ, 85281

J. Robert Wirthlin, PhD; John M. Colombi, PhD
The Air Force Institute of Technology
Wright-Patterson AFB, OH 45430
Abstract
The Enterprise Requirements and Acquisition Model (ERAM) is
a discrete event simulation that models the major
tasks and decisions within the DoD acquisition system. A
majority of DoD acquisition projects are being completed
behind schedule and over budget. ERAM suggests process
improvements can have salutary effects. Hence,
enhancements in improving the end-to-end acquisition process
would be worthwhile. Until 2008, the Analysis of
Alternatives (AoA) process is a mandatory task for acquisition
category (ACAT) level 1 projects. As such expected
program completion time for ACAT 2 and ACAT3 categories is
shorter. Since 2008, the AoA became a required
procedure for all programs. However, to the best of our
knowledge, the impact of requiring all programs to complete
an AoA has not yet been studied in literature. This research
addresses this gap with two main contributions. First,

this research seeks to quantify the amount of delay on total
completion time when the AoA is required for all ACAT
programs. Secondly, the sensitivity of the processing time and
variability of the AoA process is simulated and its
effect is studied on total program completion time. Viable
policies and intervention strategies are then inferred from
these contributions to further improve acquisition program
completion time.
Keywords
DoD, Acquisition, Simulation, Analysis of Alternatives
1. Introduction
It is a known fact that a large number of Department of Defense
(DoD) projects are being completed behind
schedule and over-budget [1]. A Government Accountability
Office (GAO) report released in 2009 states that for the
DoD’s 2008 portfolio, on average a program faced a 22-month
delay and exceeded the original budget [2].
Generally, total cost growth has been consistent over the past
few decades with a recent assessment by [3] of 1.44 or
44% growth. The current DoD Acquisition system which is
composed of three separate and distinct processes,
including the Joint Capabilities Integration Development
System (JCIDS), the Planning, Programming, Budgeting &

Execution (PPBE) process, and the formal acquisition
development system outlined by the DoD 5000 series of
instructions, does not exist in a static environment. The system
is constantly being adjusted, either through policy
changes or statute [4-6]. Since the acquisition process is a
large, complex, socio-technological system, it is difficult
to determine which processes or factors affect performance
metrics like time, cost, and resource utilization. Hence,
alternative modeling tools to improve the DoD acquisition
process are the subject of current research.
In 2009, a discrete event simulation (DES) model called the
Enterprise Requirements and Acquisition Model
(ERAM) is developed by Wirthlin [7]. This model simulates the
actual acquisition processes of the US DoD using
the Air Force implementation of acquisition processes for
Acquisition Category (ACAT) levels as the basis of the
model. This is done in order to provide further insight and
understanding of the complex system’s behavior.
However, this research did not include new policies set forth by
the DoD, specifically the DoD Instruction 5000.02
that requires all programs to go through an Analysis of
Alternatives (AoA) process [6] before reaching Milestone A.
The AoA is a requirement for all military acquisition programs.
By definition, the AoA is an analytical comparison

of multiple alternatives process that needs to be performed prior
to committing resources to a given acquisition
program [8]. According to the DoD 5000.02 instruction, “The
AoA shall focus on identification and analysis of
2110
alternatives, measures of effectiveness, cost, schedule, concepts
of operations, and overall risk. The AoA shall assess
the critical technology elements (CTEs) associated with each
proposed materiel solution, including technology
maturity, integration risk, manufacturing feasibility, and, where
necessary, technology maturation and
demonstration needs.” Through this requirement, an implicit
assumption is being made that this step will actually
shorten the overall lifecycle development time for a given
acquisition program and increase the quality of the final
form of the materiel solution. However to the best of our
knowledge this policy effect has not yet been
quantitatively studied in existing literature. Nevertheless, the
AoA theoretically contributes to longer program time
completion of all DoD acquisition projects as it is an additional
task that must be performed during the process.

Hence potential policies could be developed to counter the
effect of requiring the AoAs, e.g. acknowledging better
quality solutions earlier in development, which could be easily
translated to viable policies to further improve not
only the duration but the entire end-to-end DoD acquisition
system.
Against this background, this research addresses these
limitations by performing additional simulation and statistical
analysis on the ERAM model. The primary goal of this research
is to determine the effect on the total acquisition
program completion time by requiring all ACAT programs to go
through the AoA process. Furthermore, the effect
of reducing the variability and the time spent on the AoA
process is studied and potential intervention strategies are
developed. Additionally, this research provides viable policies
and discussion points from the intervention strategies
that could further reduce program completion time. The rest of
this paper is organized as follows. Section 2
provides an overview of the related literature on the AoA, while
Section 3 provides the methodology for the
analysis. Section 4 presents the results of the study while
section 5 provides conclusions and future research options.

2. Review of Literature
2.1 DoD Enterprise Requirements Acquisition Model (ERAM)
ERAM is originally an Arena simulation model which provides
the foundations for research on applying discrete
event system simulation to the DoD acquisition process.
Extensive validation of the model is done by comparing the
performance results of the model to the actual DoD acquisition
data and expert reviews. Initially, 20 intervention
strategies are explored using the ERAM model to assess the
potential of the total completion time reduction of DoD
acquisition projects. If all 20 interventions are implemented, a
20% reduction in the total program time could be
realized. It is found that the most effective interventions to
improve the system are those that reduce the variability
of the processes. Since its publication in 2009, the Arena model
has been translated to ExtendSim and extended by
The Aerospace Corporation’s Developmental Planning and
Architectures Division for use in the Concept
Development Center of the Space and Missile Systems Center at
Los Angeles AFB, CA [9]. Moreover, Montgomery
[10] provided the research for Aerospace to extend the model to
further include ACAT 2/3 programs along with
modeling the Rapid Acquisition process for space programs.

Table 1 summarizes the different versions of the
ERAM model. Please note the earlier version of ERAM (1.0)
does NOT implement AoA on all ACAT projects.
Table 1: ERAM Versions Adapted from [11]
Author Version Number Changes
Wirthlin [7] ERAM 1.0 Baseline Translation from Arena to
ExtendSim
Leach and Searle [9]
ERAM 1.1
Updates by the Aerospace Design Team and Served as new
baseline model
ERAM 1.2 Implemented new DoD 5000.02 policies
ERAM 2.0
Incorporated the global variables that modify acquisition
capabilities
ERAM 2.1 Incorporated the JCIDS review process
Montgomery [10] ERAM 2.2
Added more capabilities for ACAT 2/3 and Rapid Acquisition
Process
Verification and validation of the baseline distributions

included hand modeling, iterations of correction from
feedback of experts in all three branches of acquisition, and
comparison of schedule and budget information from
the DAMIR and SMART databases to distributions of the
schedule time of model-generated data [7].
2.2 Implementation of the Analysis of Alternatives
There are also several articles that illustrate the implementation
of the AoA in the DoD acquisition process.
Cervantes et al. [12] applies a rapid AoA implementation to find
a NATO Special Operations Headquarters (NSHQ)
2111
Air Wing. Georgiadis et al. [8] propose an analytical multiple
criteria decision making methodology to handle the
AoA. The AoA also plays a part in the replacement of the aging
presidential helicopter fleet [13].
Research also explores policies to improve the implementation
of the AoA in the end-to-end DoD acquisition
system. For example, Schank [14] identifies several important
factors for the success of the AoA specifically: (1) the
AoA must have a study plan that considers a wide range of

alternatives and must be flexible in the analysis
methodology (2) oversight committees that manage effect
relationships, (3) trade-off analysis should be conducted
on all alternatives, and (4) good estimation and recognition of
technical, design and production risks is a must. A
white paper from the Training and Doctrine Comman
(TRADOC) Analysis Center (TRAC) addresses two problems
of the implementation of the AoA specifically: (1) by clearly
describing and differentiating the purpose and scope of
each AoA for each milestone decision, (2) by clearly defining
and describing “materiel solution” as it can be used
interchangeably with the “alternatives” term [15]. Stadterman
et al. [16] propose improvements on the AoA in the
Weapon Systems Acquisition Reform Act of 2009. Among these
recommendations are: (1) all concerned parties
within the AoA should build a working relationship, (2) the
AoAs should focus on the decision choices and the
decision space, and have an achievable, affordable and
operationally relevant set of criteria and (3) the Army should
follow a formal analytical process that support the AoA
throughout the acquisition process. Additionally, Ford et al.
[17] improve the implementation of the AoA by incorporating
benefits in the decision methodology through the use
of a system dynamics model of a military operation and

integrating it with a Knowledge Value Added methodology.
Furthermore, in a Government Accountability Office (GAO)
report [18], the GAO recommends that the Joint
Requirements Oversight Council should establish a review
mechanism for AoA earlier in the acquisition process.
According to Roper [19], all alternatives being considered
within the AoA must go through consistent analysis
methodologies and assumptions in order to ensure
comparability. Hence, adopting these quality improvement
suggestions also imply a reduction in the length and/or the
variability in the AoA process. Doing so will allow
AoAs to obtain consistent AoA process times.
Based on this review, the following gaps in literature can be
gleaned:
• Currently, a quantitative study to assess the impact of
requiring AoA implementation in all ACAT
programs is lacking.
• Secondly, the effect of reducing the variability and length of
the AoA process has not yet been addressed as
a potential intervention to further improve the program
acquisition completion time.
3. Simulation Analysis

This research is composed of three distinct phases. The first
phase is a simulation study to determine the effect of the
AoA for ACAT 2 and ACAT 3 programs on the total completion
time. The second phase consists of two simulation
experiments in which the variability and the mean of the AoA
process time are reduced and its effect on total
completion time is determined. Lastly, the third phase is the
translation of results into recommendations and further
research.
3.1 Requiring an AoA on a Percentage of ACAT 2 and 3
Programs
The original ERAM 1.0 model is utilized since it is the baseline
model that did not require an AoA on all programs.
By varying the “ACAT 2 or ACAT 3 funding” process,
specifically the percentage of programs with funding
already available for an AoA, the effect of requiring a certain
percentage of programs to undergo an AoA on the
total program completion time, or the time the program
completes Milestone C (MS-C), is determined. Figure 1
presents the screenshot of the module. This module decides if
the program has enough funding to perform an AoA.
If it is found there is enough funding, the program will have to
undergo an AoA.

ACAT 1 programs are not included in the analysis since AoAs
are required previously. The baseline scenario is set
as such the probability that the program would be required to
undergo an AoA is 1%. This setting is set before the
DoD policy change in 2008 on AoA implementation. Please note
in [7], only ACAT 2 and 3 programs with
sufficient funding are needed to undergo an AoA. Simulation
trials each with 3000 iterations are run separately for
ACAT 2 and ACAT 3 programs. During each trial, the
probability that a program would need to perform an AoA is
increased incrementally, and its corresponding completion time
is tabulated. Each increase is compared to the
baseline trial. A t-test is then performed in order to determine if
there is a statistically significant difference between
the trial results and the baseline trial in terms of the time until
MS-C. Hence, the probabilities are incrementally
increased until a statistical significant difference from the
baseline is obtained at a 95% confidence level.
2112
Figure 1: Arena Screenshot of “ACAT 2 or ACAT 3 funding”

Process
3.2 Sensitivity Analysis on the AoA Process Parameters
The objective of this phase is to identify the impact of
improving the AoA process specifically its variability and its
program completion time. The ExtendSim model is utilized over
the Arena model because the model reflects the
most recent policy changes. The current AoA process is
distributed according to a triangular distribution with a
lower limit of 180, most likely value of 360, and an upper limit
of 720 days (Tria(180,360,720)), and all programs
are assumed to undergo an AoA. These settings constitute the
baseline trial for this phase. The ACAT 1 level
projects are utilized in this phase. The ExtendSim simulation
module being modified in the ERAM model is an
Activity block from the items library called “Analysis of
Alternatives.” Figure 2 presents a screenshot of the
ExtendSim activity.
This phase is composed of two simulation experiments. The first
experiment studies the effect of reducing the
variability of the AoA process time while the second section
deals with reducing the mean of the AoA process time.

For the first experiment, 3000 iterations are performed and
during each trial, and the variability of the time to
perform the AoA is adjusted. The variability is adjusted by
reducing the difference between the mode and the
maximum/minimum by a constant. Hence, the triangular
distribution would then have less variance. The variance of
the triangular distribution is reduced incrementally until a
statistical significant difference is obtained from the
baseline trial. A t-test is then performed in order to compare the
time to milestone C in the trial to the baseline trial.
Furthermore, the second simulation experiment deals with
reducing the AoA process time to determine its effect on
the time ACAT 1 programs reach Milestone C. Three thousand
iterations are again performed and for each trial, the
mean of the AoA program length distribution which is
distributed according to a triangular distribution
(Tria(180,360,720)) is reduced. The ratio between the minimum,
most likely, and maximum of 1:2:4 is maintained
as the parameters are changed. The mean completion time of the
triangular distribution of the AoA is again reduced
until a statistically significant difference from the baseline trial
is obtained. A t-test is again performed in order to
compare the time in the trials to the baseline trial.

3.3 Results and Interpretation
The final phase in this paper compiles the results from the tests
and translates them into recommendations for viable
policy changes. This phase includes identifying which
parameter adjustments improved the performance metrics and
making recommendations on further research, like interactions
among parameters.
2113
Figure 2: ExtendSim Screenshot of “Analysis of Alternatives”
Activity
4. Results and Discussions
Table 2 summarizes the results of the t-tests performed from
section 3.1 for ACAT 2 programs. The table shows a
subset of trials corresponding to 50%, 85%, 87.5%, 90%, 95%
or 99% of programs are required to undergo an AoA.
These settings are selected to show the sensitivity of changing
AoA. These are compared to the baseline AoA setting in which
only the funded programs (1%) are required to

undergo an AoA. The null hypothesis for the t-tests is
��:��
��% which corresponds to an insignificant
difference between the baseline and the ��percentage if not
rejected and alternative hypothesis��:��
��% if
there is significant difference.
1%
(Baseline)
50% 85% 87.5% 90% 95% 99%
Average Time to
MS-C (Days)
3898.099 4001.104 4080.293 4093.409 4108.58 4115.981
4141.088
Standard Deviation
(Days)
1411.37 1550.684 1625.738 1634.814 1650.243 1655.492
1673.537
T-Value -1.34133 -1.78231 -1.90576 -2.04267 -2.04267 -

2.34049
Conclusion Fail to
Reject H0
Fail to
Reject H0
Fail to
Reject H0
Reject
H0
Reject
H0
Reject
H0
It is evident that when requiring an AoA on at most 87.5% of
the ACAT 2 programs will result in a failure to reject
Any percentage less than 87.5% will have no effect on the
average time a program arrives at MS-C. This means that a

AoA and no significant increases in the total completion time
can be obtained.
Table 3 summarizes the results of the t-tests performed for
ACAT 3 programs. The table shows a subset of trials
corresponding to 50%, 55%, 57.5%, 60%, 65% or 75% of
programs are required to undergo an AoA. Furthermore,
these settings are selected to show the sensitivity of changing
AoA. These are compared to the baseline in which only the
funded programs (1%) are required to undergo an AoA.
The null hypothesis for the t-tests is ��:��
��% which corresponds to no significant difference between
the
baseline and the �� percentage if not rejected and alternative
hypothesis��:��
��% if there is significant
difference.
2114

1%
(Baseline)
50% 55% 57.5% 60% 65% 75%
Average Time to
MS-C (Days)
3334.926 3470.918 3493.278 3512.661
3524.566 3551.2 3604.434
Standard Deviation
(Days)
1143.19 1314.598 1350.852 1365.653 1372.66 1392.929
1419.016
T-Value -1.63342 -1.86606 -2.08392 -2.2198 -2.51299 -3.1095
Conclusion Fail to
Reject H0
Fail to
Reject H0
Reject

H0
Reject
H0
Reject
H0
Reject
H0
It is evident that when requiring an AoA on at most 55% of the
ACAT 3 programs will result in a failure to reject
that the �� percentage is similar to the baseline scenario. On
the other hand, the AoA contributes to a significant
change in the total program completion time when at least
57.5% of the programs require AoA. In general, it is
evident from tables 2 and 3 that requiring an AoA on all
programs significantly increase the total completion time of
ACAT 2 and 3 programs when programs are required to undergo
an AoA.
Table 4 summarizes the results of the t-tests performed in terms
of reducing the variability of the AoA process time.
Four settings are tested specifically, Tria(270, 360, 540),
Tria(315, 360, 450), Tria(337.5, 360, 405), and Tria(360,

sensitivity of the variability of the AoA processing time from
the baseline scenario to a deterministic scenario (time is fixed
at 360 days) as is done in [7]. These settings are
compared to the baseline AoA process time which is distributed
Tria(180, 360, 720). The null hypothesis for the t-
tests is ��:��
�� which corresponds to no significant difference between
the baseline and the �
��AoA
triangular distribution variance setting if not rejected and
alternative hypothesis ��:��
�� if there is
significant difference.
Table 4: t-test Results for the Variance Reduction of AoA
Process Time
Tria(180, 360, 720)
(Baseline)
Tria (270, 360,
540)
Tria (315, 360,

450)
Tria (337.5,
360, 405)
Tria (360, 360,
360)
Average Time to
MS-C (Days)
6903.567
6883.159 6867.149 6865.280
6848.717
Standard Deviation
(Days)
1584.258 1591.61 1591.442 1590.301 1576.526
T-Value 0.2528 0.4512 0.4745 0.6828
H0
Fail to Reject

H0
Fail to Reject
H0
Fail to Reject
H0
It is evident that reducing the variability of the AoA process
time does not have an effect on the total time ACAT 1
programs reaches milestone C. Hence any improvement on the
AoA process to make it more consistent and standard
would not have a significant effect on the total program
completion time.
On the other hand, table 5 presents the results of reducing the
mean process time of the AoA from section 3.2 Four
settings are tested specifically, Tria(135, 270, 540), Tria(112.5,
225, 450), Tria(101.25, 202.5, 405), and Tria(90,
sensitivity of the length of the AoA processing time from the
baseline scenario to the fastest scenario (Tria(90, 180, 360)).
These settings are compared to the baseline AoA
process time which is distributed Tria(180, 360, 720). The null
hypothesis for the t-tests is ��:��

�� which
corresponds to no significant difference between the baseline
and the ��AoA triangular mean time distribution
setting if not rejected and alternative hypothesis��:��
�
�� if there is significant difference.
Table 5: t-test Results for the Mean Reduction of AoA Process
Time
Tria(180, 360, 720) Tria (135, Tria (112.5, Tria (101.25, Tria
(90, 180,
2115
(Baseline) 270, 540) 225, 450) 202.5, 405) 360)
Average Time to
MS-C (Days)
6903.567 6827.941 6752.989 6732.038 6681.659
Standard Deviation
(Days)

1584.258 1597.607 1578.558 1567.143 1574.53
T-Value 0.9351 1.8732 2.1415 2.7640
H0
Fail to Reject
H0
Reject H0 Reject H0
It is evident that reducing the AoA process time to at the least
Tria(112.5, 225, 450) will result in a failure to reject
This implies that reducing the AoA process time to at most
Tria(101.25, 202.5, 405) contributes to a significant change in
the total program completion time.
5. Conclusions
The analysis of alternatives (AoA) process is a new requirement
that all acquisition category (ACAT) level projects
to be completed by the US government are required to undergo.
Previously, this process is only required on all
ACAT 1 programs and ACAT 2 and 3 programs with sufficient
funding. Hence, program completion is expected to
increase further given that a majority of the programs finish

over budget and with a delayed schedule when AoA is
required on all programs. This concept is validated through
simulation documented in this paper. By requiring all
programs to undergo an AoA, significant completion delays are
observed. Furthermore, this paper provides
quantitative insights on the amount of time that the AoA
contributes to the overall DoD end-to-end process time
through Wirthlin [7]’s discrete event system simulation model.
Through additional simulation analysis, it is inferred that a
AoA without significantly affecting completion time. However,
requiring an AoA on ACAT 3 programs has a
significant effect on program completion time. A frequently
cited reason for not conducting an AoA is that an AoA
causes a significant delay in the program. The simulation results
in this paper should provide strength to the
argument towards providing funds for full AoAs. However, as
there is a percentage of programs where the AoA can
cause significant delays in program completion (87.5% and 55%
for ACAT II and ACAT III respectively), policy
improvements with regards to selection criteria for programs
that must undergo an AoA may be beneficial.

Support for this conclusion can be found in the system
engineering case study for the A-10 Thunderbolt II
performed by [20]. From this case study, it can be seen that the
correct level of focus is needed for an AoA to be
beneficial. During the development of the A-10 three
prototyping studies are performed. These prototyping efforts
are analogous to an AoA. One prototype is a system level test
in the form of a fly-off. The other two are subsystem
prototyping of the guns and ammunition for the aircraft. While
the system level prototyping provided little
meaningful information, the two subsystem prototyping efforts
provide benefits to the DoD in terms of both overall
cost reduction and design improvements. The findings from the
case study in conjunction with the simulation results
in this paper indicate that policy applying tailored criteria for
ACAT 2 and ACAT 3 AoAs for programs that must
undergo an AoA could be beneficial.
Furthermore, findings from this paper show that any variance
reduction improvements or standardizations on the
AoA process would not affect the total program completion time
of ACAT 1 programs. However, any
improvements that reduce the processing time of the AoA, or in
other words making it more efficient and lean

would significantly reduce program completion time. With the
DoD’s requirement to conduct an AoA, an implicit
assumption has been made that the step will shorten the overall
lifecycle development. The simulation’s finding
suggests that from a statistical perspective, the DoD must be
expeditious during the AoA process, or the value
sought from requiring the AoA decreases. Hence, policy
improvements with regards to an AoA should focus on
reducing processing and execution time of the AoA without
sacrificing quality in its output.
The use of simulation in this study places some limitations on
this research as well as provides opportunities for
further research. One limitation is that the simulation takes a
program rather than a portfolio perspective. Previously,
fewer AoAs were performed because of lack of funding. The
need to allocate funding to AoAs for ACAT 2 and
ACAT 3 programs may result in more delays across the DoD
portfolio because funding may be spread too thinly.
This effect cannot be illustrated by the model if it is present.
Further research may be needed to investigate this
2116

Proceedings of the 2013 Industrial and Systems Engineering Res.docx

Proceedings of the 2013 Industrial and Systems Engineering Res.docx

Recommended

Recommended

More Related Content

Similar to Proceedings of the 2013 Industrial and Systems Engineering Res.docx

Similar to Proceedings of the 2013 Industrial and Systems Engineering Res.docx (20)

More from stilliegeorgiana

More from stilliegeorgiana (20)

Recently uploaded

Recently uploaded (20)

Proceedings of the 2013 Industrial and Systems Engineering Res.docx