This document summarizes a paper that argues optimization is an outdated paradigm in computer-aided engineering (CAE). Stochastic simulation is proposed as a better alternative that can account for uncertainty and complexity in a more complete way. Optimization focuses too much on numerical details rather than physics. Nature produces robust, "good enough" designs through self-organization and emergence rather than optimization. Stochastic simulation using Monte Carlo techniques can replace optimization by evaluating design performance across many random samples to find robust, high-performing designs rather than strictly optimal ones. This paradigm of stochastic simulation and design improvement is argued to provide a simpler and more effective approach for engineering problems.

1. AIAA-2000-4929
STOCHASTIC MULTIDISCIPLINARY IMPROVEMENT: BEYOND OPTIMIZATION1
Jacek Marczyk, Ph.D., Senior AIAA Member∗
EASi Engineering, Madison Heights, MI 48071, USA
1
Copyright 2000 by J. Marczyk. Published by the American Institute of Aeronautics and Astronautics,
Inc. with permission.
∗
Vice President Advanced Technologies
ABSTRACT
Contrary to popular belief, contemporary optimization practice in CAE is an outdated and limiting
paradigm. Given the performance of today’s computers, the practice of optimization is difficult to justify,
not only from a philosophical, but also practical standpoint. It is argued that optimization should be
substituted by the more complete and simpler stochastic simulation, which apart from enabling a rigorous
treatment of uncertainty and complexity, establishes a firm conceptual platform for the unification and
simplification of CAE. The complexity of today's optimization tools, together with the multitude of
available algorithms, very often diverts the engineer's attention from physics to purely numerical aspects. It
is shown, how in the presence of uncertainty, robustness is a much more valuable feature than optimality
and how stochastic simulation, based on Monte Carlo techniques, can replace all forms of optimization.
Introduction and Background
Contemporary CAE is stagnant and fragmented.
If one takes a step back and looks at what the
CAE market offers, one notices excessive
articulation in almost every direction. There are
tens and tens of tools that replicate each other's
capabilities and features. As Kuhn points out in
his influential book, The Structure of Scientific
Revolutions, such articulations are a
characteristic proper of a state of crisis.
Whenever a discipline of science runs out of
ideas, the first symptom is fragmentation and
fractalization of the same concepts. Small
incremental improvements and refinements, but
no major and significant innovation. One such
overly articulated concept is the popular and
trendy Multi-Disciplinary Optimization, MDO.
What comes after crisis is revolution. Crises
precede revolutions, and CAE is precisely in this
state now. According to Kuhn, every time a
revolution in science takes place, a change of
paradigms is the result.
Contemporary CAE is medieval in its structure
and orthodoxy. Almost "geocentric". The
fundametalizm in today's CAE hinges on a few
dogmatic (untouchable) pillars: determinism,
reductionism, and optimality. Although these
concepts are highly related, the analysis of the
significance of optimization and optimality shall
be the focus of this paper. A little bit of history is
necessary here.
When something is optimal, it is essentially
perfect, impossible to improve, non-plus-ultra.
But why do humans pursue optimality, this state
of grace, so fiercely? The roots of this are to be
found, not surprisingly, in religions. In medieval
Europe, governed by the Inquisition, and based
on the pseudo-science established by Aristoteles,
perfection was the name of the game. Since God
was perfect, humans were obligated to seek
perfection everywhere, in every angle of the
universe. Geometry, the motions of planets,
architecture, were all based on perfect
geometrical figures and constructions (circles,
triangles, pentagons, etc.). Even Copernicus,
when he discovered that our solar system was in
reality heliocentric, did not dare to propose
elliptical orbits, since an ellipse was not a perfect
geometrical figure. It was not divine enough,
1

2. politically incorrect. It was then Kepler who had
the courage to replace circular orbits by the more
vulgar ellipses. Science has suffered greatly due
to the search of perfection. Even Darwin had
difficulties in having his theory of evolution of
the species accepted. Surprisingly, it was not
because of the fact that man, according to
Darwin, descended from the ape, although
already this assumption was on a frontal
collision course with the dogmas of his days, it
was because Darwin's theory, that evolved the
simplest organisms to the human being, was an
open-ended process. It did not stop at the final
and perfect (optimal) result, man. Now, since
man is an image of God, and therefore perfect,
any theory, whether based on evolutionary or
single-shot creation, has to lead to this optimal
design. Since, according to Darwin, evolution
never stops, it does not guarantee a perfect and
happy ending, and is therefore unacceptable.
What Darwin's contemporaries did not accept
was the fact that in his theory there was no
Master Choreographer who would decide what
the final result was to be.
Today in CAE, but also in other branches of
science and life, the situation is strikingly
similar. Western culture, in particular, still relies
on dogmatic and authoritative forms of
education that make people want certainty
instead of knowledge. Uncertainty has been
axiomatized out of CAE, because it is politically
incorrect and because it inhibits the use of many
elaborate, and often elegant theories and tools
that abound, to the delight of the Establishment.
Uncertainty is incompatible with optimality, and
is therefore an uncomfortable thing to deal with.
Optimality and uncertainty can't coexist.
Uncertainty eliminates the possibility of
imposing top-down choreographies that produce
pleasing and a-priori known results. What is
argued, and argumented in this paper, is that the
basic idea behind optimization is wrong and that
the optimization paradigm is responsible for the
fragmentation in CAE. However, as the history
of science teaches us, major advances in any
discipline have taken place when generalizations
and simplifications of paradigms were made.
This paper is about the next paradigm:
simulation.
What Is Wrong With Optimization?
There is something terribly wrong with
optimization. Optimality and optimization don't
exist in nature. Nature only produces designs
that are fit for the function and the mechanism it
chooses is not optimization, it is spontaneous
self-emergence, or self-organization. Let us
consider, as an example of a fantastically
complex and efficient system, the system of
veins and capillaries in a human body. Is the
topology of the vein network the result of an
optimization problem with millions of variables
that nature has solved? Clearly the answer is no.
In fact, every individual has a different one,
although dominant features and patterns may
easily be identified. The details are not
repeatable. How could a similar problem even be
formulated in terms of optimality? What is an
optimal vein network? Clearly, on paper
everything is possible. Is the vein system
perfect? Who cares? As long as it works. This is
the main point. Nature settles for design that are
good enough. Formulating problems in an
optimization context is equivalent to making
them extremely complex, much more than
necessary. There exist very simple tools that can
"solve" extremely complex problems if we are
willing to accept a paradigmatic change. Nature
bases its most powerful mechanisms and
machinery on the recently discovered laws of
complexity and spontaneous self-organization.
Complexity is a phenomenon that arises on the
frontier between chaos and order, in a sort of
phase transition. It is on this frontier that Nature
is most prolific and creative. Life, in fact, has
emerged spontaneously from an auto-catalytic
chemical environment obeying a small set of
very simple laws. Complexity did the rest.
Complexity, the study of which is being
pioneered by the Santa Fe Institute, has been
claimed to be the next major revolution in
science. In CAE, we could see Monte Carlo
simulation as being the "equivalent" to self-
organization.
There are always many ways to formulate the
same problem. The best one is the simplest that
works. However, human nature finds complex
theories more credible, and certainly more
appealing. Why has it taken nearly twenty
centuries to abolish Ptolemy's model of the solar
system based on epicycles? Because it predicted
the motions of the planets correctly! In fact,
philosophers have shown that many theories can
fit the same set of observed data. Most of them
are wrong. For this reason, it sometimes occurs
that baroque and excessively ornamented
numerical procedures, composed of Design Of
2

3. Experiments, response surfaces, genetic
algorithms and so forth, yield results that can be
contrasted with measurements. Let us look at
another example.
In the brain there are 10 to the power 12
neurons. Imagine that each neuron is a "design
variable". Does it make sense to talk of
sensitivities of each single neuron with respect to
some performance descriptor of the brain? Does
there exist one single most important neuron in
the brain? Clearly the answer is no. In complex
systems, it is the connectivity of the components
and their basic properties that establish the
global behavior, not the details of the
components. Therefore, if we want to change, or
improve, the global behavior of a complex
system, what is required is a collective change of
the fundamental properties of the single
components, or a change of topology.
Approaching complex problems on the basis of
sensitivity, or optimization, is a cause that is lost
from the very beginning, although from the point
of view of computer algebra everything is
possible (people call these problems Grand
Challenges, such as the Traveling Salesman
problem with millions of cities). In complex
systems, even minute changes in the properties
of the operational environment can often have
macroscopic implications. The point is therefore
how to steer and manage the intrinsic uncertainty
that all complex systems possess, so that they
can co-exist with their changing environments in
a robust and controllable manner.
Looking for a global optimum in a complex
system is like trying to hit a bullet with a bullet.
Although in theory any problem may be
formulated in this fashion, complex optimization
problems are essentially ill-formulated and
exceptionally fragile. This fragility is apparent
when one changes algorithm, or even starting
point. An important source of debility of the
optimization paradigm is the fact that while
engineers concentrate on modifying the system
parameters, so that some “optimal” performance
is found, they overlook the importance of
boundary conditions and their unexpected
changes and variations. This point is of
fundamental importance. What sense does
optimality make when the boundary conditions
are beyond our control? In Nature fitness comes
from the most advanced form of “robustness”,
namely from adaptivity. This is what
characterizes the most resilient designs, like
humans and complex mammals. These systems
can react and adapt to very large changes in their
environments (boundary conditions). Other
systems only have built-in (static) or pre-
confectioned robustness. This is a less intelligent
form of fitness/survivability, also a kind of
overkill. The most brutal form of overkill is
certainly deterministic optimization based on
safety factors. What this guarantees is a high
degree of fragility and built-in excessive
conservatism. Optimal designs are in fact overly
optimistic and fragile. Changes in boundary
conditions, due to the fact that they are seldom
contemplated in the process of optimization,
result critical in terms of robustness and
survivability. One way, though, to overcome this
problem would be to formulate many
optimization problems, each one with a different
set of boundary conditions, and solve them all, in
an attempt to reach some reasonable
compromise. But then, there exist more elegant
techniques to reach this compromise (read
robustness, or fitness) and one such technique is
stochastic simulation. With simulation we can
move to higher, more advanced forms of design,
namely direct uncertainty management and
design improvement, the latter being a sort of
evolution. Simulation, in fact, enables to obtain
designs that are not optimal, but fit. Robust
designs versus fragile designs. Optimization is
actually just the opposite of robustness.
In contemporary deterministic design, what the
engineer is after are repeatable details. This
necessity is justified by the need to reach
predictability (again, something impossible in
the face of uncertainty). Nature looks at its
problems a bit differently. What is of concern
are patterns and structures, and not numbers.
Examples of patterns are illustrated in Figures 1
and 2, where the axes represent performance
quantities (displacements, frequencies). These
entities, called also response clouds (in
opposition to response surfaces) or fitness
landscapes, may be obtained via Monte Carlo
simulation even for applications as expensive to
run as automotive crash. Abstract creations like
smooth and differentiable surfaces and curves
don’t exist in reality, and serve the purpose of
keeping most of our modern numerical methods
and computer algebra busy. While the study of
curves and surfaces leads to data and
information, the understanding of response
clouds, their patterns and complex structures, is
a source of knowledge and insight. Examining
3

4. Figures 1 and 2, one realizes that being able to
explain such properties as clustering
(bifurcations), non-symmetry, anisotropy,
attractor sets, outliers, etc. is equivalent to the
understanding of the underlying physics.
Clearly, it is more important to understand the
structure and dominant features of response
clouds than imposing some abstract model on
top of them, like a second order polynomial
surface, and, at the same time, destroying the
information they contain.
This obsession of top-down choreography
pervades many branches of science. A good
example is Artificial Intelligence. In the past,
people tried to build-in artificial intelligence into
computer programs in a top-down fashion,
thinking that it was only a matter of getting
enough flops. We all know how this approached
has failed. Intelligence cannot be imposed, it has
to be acquired, it has to emerge naturally. This
logic applies also to CAE. If we impose on a
group of points a second order response surface,
we are already defining what type of physics will
the model be able to yield (unwrap). A simple
example of unwrapping is the following.
Imagine we write a computer program that
simulates the solar system. If all we code is
Newton’s laws of dynamics and gravity, we will
find Kepler’s laws from using the model. We
will have learnt something new. If, on the other
hand, we pre-confection elliptical orbits into the
code, Kepler’s laws will not stem from our
computer program. The important conclusion
that must be drawn from this simple example is
that computer codes should be based on first
principles, and not on complex built-in models
or behavioral mechanisms. Just recall how much
has been found from the innocent set of Lorenz
equations.
It is often argued that when many parameters of
a system are changed in a random and
independent fashion, it is impossible to separate
any parameter influence and impact on the
performance descriptors. The fact is that this is
the true face of reality. Complex systems should
be designed considering collective changes of all
of the variables because this is what happens in
reality. In systems of hundreds and thousands of
variables it is very difficult, not to say practically
impossible, to see single dominant design
variables. Only in the abstract world of response
surfaces is this possible, where even sensitivities
can be computed. The whole point is not to look
at design variables, but at design features. I have
often come across publications (from the DOE
community) in which it is stated that Monte
Carlo techniques have the scope of desperately
scrambling variables in an attempt to find these
sensitivities. This is not the case. Here lies the
whole point. We are not after one-to-one
relationships, we're looking for global patterns,
structure and dominant features (clusters,
bifurcations, etc.). Actually, once a stochastic
look at reality is taken, design improvement
rather than optimization becomes the obvious
design paradigm. Improvement has the target of
positioning oneself on the fitness landscape in a
point where performance is good enough or
acceptable. In complex fitness landscapes, with
hundreds of local minima, there is no such thing
as an optimal location. Essentially, instead of
trying to hit a bullet with a bullet, we prefer to
hit a truck with a shotgun. One way to
accomplish design improvement (see Figure 3) is
to change collectively the design variables, in
particular their mean values, so that the system
globally behaves in a different and acceptable
manner. It is then the control over the standard
deviations of the components that establishes the
quality and robustness of our design. This is
exactly what stochastic simulation-based
improvement is about: steering collectively
design variables so as to achieve acceptable, not
optimal, performance.
How does improvement work in practice? The
method is very simple. A set of N random
samples is generated around the nominal design.
A target location in the performance space is
defined and the Euclidean distance of each
sample to the target is computed. The best one is
chosen as the starting point for the next scramble
of N points. It may be shown that with N=16, the
probability of improving the design (i.e. moving
towards the target performance) is approximately
0.999986. Figure 3 illustrates the concept.
Improvement of this type may be seen as sort of
evolutionary process, where the best design
survives and is chosen as the seed for a new
generation.
One important problem that is overlooked in
optimization practice is that of dependent
variables. It happens very often that the objective
function to be minimized is a function of
performance variables, such as mass, frequency,
displacement, stress, etc. Clearly, in many cases,
these variables are correlated, i.e. are dependent.
4

5. Since these dependencies are impossible to
formulate in a closed form, they cannot be
eliminated. One consequence of this is the over-
specification (a form of over-constraining) of the
problem. This makes the optimization process a
lot more difficult and awkward since it involves
more variables than are really necessary. In fact,
in stochastic improvement, since the
dependencies are naturally computed as a by-
product of Monte Carlo simulation, only the
independent variables are steered. This greatly
simplifies the problem and reduces its dimension
to the natural minimum.
Some will surely argue that optimization does
work. Well, due to the complexity of multi-
dimensional systems and their fantastically
intricate fitness landscapes (objective functions)
there is a very high chance of finding
accidentally a better design. Complexity can be
very generous. Then, once such a design is
found, no one ever cares to prove that the
solution is optimal. In fact, in many cases sub-
optimal solutions are found and are erroneously
accepted as optimal.
Optimization is the wrong thing to do. Let us see
another example of a problem that has a
relatively simple solution if formulated in an
appropriate manner, and which assumes insane
proportions if seen as an optimization of
something. There are approximately 100000
gene types in the human chromosomes. This
yields 2 to the power 100000 possible
combinations (10 to the power 30000).
Neglecting some basic differences between
races, only one combination leads to the human
being! Imagine now how difficult it must be to
find this combination in such a huge search
domain. Not even cosmological time scales
would be sufficient to solve the problem. Nature,
however, did not see homo-sapiens as anything
exceptionally perfect and approached the
problem from the point of view of spontaneous
self-organization, or self-emergence. It is in fact
known from complexity theory that a similar
system will spontaneously settle in 317 attractor
states, where 317 is the square root of 100000.
The fantastic coincidence is that this is the
approximate number of cell types in the human
body. Actually, this rule has been confirmed also
in other organisms; the number of cell types is
the square root of the number of genes.
Complexity and self-organization have offered
an explanation to ontogeny. Cell types are
attractors in an incredibly large design space. If
Nature had solved the problem through the
optimization of some objective function, and
indeed found a solution, life would surely be a
miracle.
Simulation (in the Monte Carlo sense) is the last
(known) frontier for computers. If we can do
simulation, why settle for anything less? Once a
simulation has been performed, we have all the
information about a system that the underlying
model can deliver. The resulting response
clouds will, because of physics, and not due to
some top-down choreography, arrange into
patterns (attractors) and form structures, which
can convey impressive amounts of information
and knowledge. As we have already seen, the
number of these patterns/attractors is generally a
minute fraction of the apparent dimension of the
problem. For this reason, basic macroscopic
information about a system can be obtained with
a very small number of Monte Carlo samples,
generally in the range of tens. Why then go for
Baroque procedures, like the response surface
method, or DOE, which add approximations to
other approximations and which add complexity
to complexity? How many tools for optimization
exist? Tens, if not hundreds. So which one do
we choose? Normally the one that gives the most
pleasing results. DOE, on the other hand, happily
samples multi-dimensional and intricate fitness
landscapes without prior knowledge of their
characteristics and independently of the
underlying physics. Can this possibly capture the
dominant features of a problem? Then, in order
to complete the final picture, and to insulate the
problem from physics, a smooth second order
surface is forced to pass through the arbitrarily
selected points. Clearly, a second-order multi-
dimensional surface can only unwrap very
simplistic and naïve types of behavior, and is
incapable, for example to reflect bifurcations,
clustering, or other interesting phenomena that
abound in physics. At the end of this top-down
choreography, one has mapped physics onto a
numerical flatland where search for extremal
points is, in effect, the only logical thing to do.
Numerical alchemy.
Simulation is a monument to simplicity, but at
the same time it is tremendously powerful. In
essence, simulation provides a means to generate
and explore fitness landscapes. Once we can
appreciate how much subtle complexity can even
small systems hide, one is inevitably led to ask
5

6. the following question: do we always need a
model? Why not leave the fitness landscape as it
is and explore its complexity, instead of
destroying it with some numerical and physic-
less recipe? Knowledge is hidden in the
structure, the patterns and the complexity of
multi-dimensional landscapes and therefore
these features must be preserved. From this
perspective, the Response Surface method is
clearly a destructive technique, which imposes in
a top-down fashion what the model will be able
to deliver (unwrap). The funny thing is that
response surfaces are often used to conduct
Monte Carlo studies where thousands of samples
are computed (in virtue of the fast turn-around
times) in order to enable “accurate” evaluations
of durability or probability of failure! These
studies are either accurately approximate or
approximately accurate.
Science does not have the objective of being
predictive, but to help explain and comprehend.
We think that we are already beyond that stage
and now it's only a matter of optimizing. As
already mentioned, it is possible to prove that
different theories can fit the same data points.
For this reason, different models can be made to
appear as correct (e.g. geocentric and
heliocentric models can predict the same
positions of the planets). This is why it is
possible for so many variants of the same theory
to flourish and make them appear as correct.
Many of us recall the Guyan reduction, which
was used to select the important degrees of
freedom prior to computing the eigen-
frequencies of an elastic system. When powerful
computers and techniques became available, like
the Lanczos method, Guyan reduction became
obsolete. Today, nobody uses the Guyan
reduction technique anymore. With today’s
powerful computers running DOE, in
combination with the response surface method
and with some exotic optimization algorithm, is
like returning to the “Guyan approach”. Why do
we need to stack approximations upon
approximations, allowing the "numerics" to
progressively occult reality, if we can afford to
run a full-scale Monte Carlo simulation? It is
surprising, that people assemble very complex
procedures, never validate them versus
experimental data (on stochastic grounds, of
course), and then speak of accuracy and precise
solutions. In similar systems, it is difficult to
separate the influence and impact of the model
type, the optimization technique, or the starting
point, on the final result. Computers should be
used as a means of peeling away these layers of
approximation, and digging into the physics, not
paying tribute to numerical analysis, domain
decomposition or message passing. We should
seek not numerically optimal but physically
significant solutions.
Conclusions
In opposition to techniques like the response
surface, DOE, and myriads of optimization
algorithms, Monte Carlo simulation engulfs
complexity, uncertainty, and robustness in a
unified and simple manner. The major steps in
science have been possible when generalizations
and unifications took place. We now have
enough CPU to enable one such step in CAE and
HPC. With the possibility of running coarse
grain applications like Monte Carlo simulation,
CAE can transition from the overly complicated
“geocentric” to the simple and elegant
“heliocentric” paradigm. CAE and HPC have
today enough ammunition to help study the real
complexity stemming from physics, not to
complicate things by the use of increasingly
complex and trendy tools. CAE can today reach
“escape-velocity”.
Systems that are optimal can be terribly sensitive
to small changes, especially those in the
operational environment. In the classical design
practice, not enough importance is attributed to
the fact that survivability stems from robustness,
not from optimality. Optimality is actually the
opposite of robustness. It is fragility versus
resilience. It is much more sensible to settle for
solutions that are “only good enough”, but
robust. Trading optimality for robustness is the
mechanism by means of which Nature has been
so successful in creating fit designs with a high
degree of survivability. The trick is to settle for a
design that is good enough because the number
of good-enough solutions is incredibly larger
than the number of optimal solutions for a given
problem.
In the future we should steer computer power
towards first investigating fitness landscapes and
understanding them, and to subsequently seeking
the good-enough solutions. We still have very
little knowledge and yet we pretend to jump
directly into optimization. It is far more
important to get a crude picture of the whole,
6

7. rather than indulge in sophisticated multi-CPU
numerical hairsplitting of some details. Holism
instead of reductionism. Computers have
fantastic potential to help us learn, not to
enforce.
Illustrations
Figure 3. Stochastic design improvement. The
initial model (response cloud) is shifted, using a
simple return mapping technique, towards the
desired target performance represented by the
circle. The response clouds shown here are local
sub-sets of the landscapes shown in Figures 1
and 2. Improvement has the objective of
positioning the response cloud in a desired, or
good-enough, portion of the fitness landscape.
The process takes place in the N-dimensional
performance space of the problem. Its cost is
independent of N.
Figure 1. Example of response clouds generated
by Monte Carlo Simulation. The axes represent
performance quantities, such as frequencies,
displacements, or design variables, such as
thickness, or material property, or combinations
of both Biologists call such patterns fitness
landscapes.
Bibliography
1. Kuhn, T.S., The Structure Of Scientific
Revolutions, The University of Chicago Press,
1996.
2. Marczyk, J., ed., Computational Stochastic
Mechanics in a Meta-Computing Perspective,
International Center for Numerical Methods in
Engineering (CIMNE), Barcelona, December,
1997.
3. Marczyk, J., Principles of Simulation-Based
CAE, FIM Publications, Madrid, 1999.
4. Waldrop, M., Complexity, Simon and
Schuster, Inc., 1992.
Figure2. Fitness landscape showing clustering
phenomena. It is clearly visible that small
variations of the variable on the horizontal axis
around zero, can be associated with large
changes in the other variable, where the system
can jump from one cluster to another with
considerable probability. Response surface
techniques are unable to even approximate
similar patterns.
5. Briggs, J. and Peat, F.D., Seven Life Lessons
Of Chaos, Harper Collins Publishers, New York,
1999.
6. Holland, J.H., Hidden Order, Perseus Books,
1995.
7