This document provides an overview of experimental design, response surface analysis, and optimization. It discusses key terminology like factors, responses, and treatments. Full factorial designs are explained as a way to systematically vary factor levels to estimate main and interaction effects. Response surfaces and metamodels like regression analysis are introduced as tools to approximate simulation responses. Response surface methodology is outlined as a sequential approach combining metamodeling and optimization. Examples help illustrate concepts like estimating effects from a full factorial design. Situations with many factors and terminology in experimental design are also covered at a high level.

1. 1
Experimental
Design, Response
Surface Analysis,
and Optimization

2. 2
Outline
Motivation and Terminology
Difficulties in Solving the Basic
Problem
Examples of Factors and Responses
Types/Examples of Experimental
Design
Full Factorial Designs
Randomness of Effects
Example: Full Factorial Design
Situations with Many Factors
Response Surfaces and Metamodels
Regression Analysis
Response Surface Methodology

3. 3
Motivation and
Terminology
Useful when there are many alternatives
to consider (e.g., numerous capacity levels
of various types, numerous parameters
for a proposed inventory system)
Two basic types of variables: factors and
responses
Factors: input parameters:
-controllable or uncontrollable
-quantitative or qualitative
Responses: outputs from the simulation
model:
-uncertain in nature
Basic problem: find the best levels (or
values of the parameters) in terms of the
responses
Experimental Design can tell you which
alternatives to simulate so that you obtain
the desired information with the least
amount of simulation

4. 4
Multiple Responses
Uncertain Responses
Difficulties in Solving
the Basic Problem

5. 5
Examples of Factors
1. Mean interarrival time (uncontrollable,
quantitative)
2. Mean service time (controllable or
uncontrollable, quantitative)
3. Number of servers (controllable,
quantitative)
4. Queuing discipline (controllable,
qualitative)
5. Reorder point (controllable, quantitative)
6. Mean interdemand time (uncontrollable,
quantitative)
7. Distribution of interdemand time
(uncontrollable, qualitative)

6. 6
1. Mean daily production rate
2. Mean time in the system for patients
3. Mean inventory level
4. Number of customers who wait for more
than 5 minutes
Examples of Responses

7. 7
Types/Examples of
Experimental Designs
Completely Randomized Designs
Randomized Complete Block Designs
Nested Factorial Designs
Split Plot Type Designs
Latin Square Type Designs
Full Factorial Designs
Fractional Factorial Designs

8. 8
2k
Factorial Designs
Suppose that we have k (k > 2)
factors. A 2k
factorial design would
require that two levels be chosen for
each factor, and that n simulation
runs (replications) be made at each
of the 2k
possible factor-level
combinations (design points). For 3
factors, this yields a Design Matrix:

9. 9
Design Matrix for 3 Factors
Factor 1 Factor 2 Factor 3
Design Point Level Level Level
Response
1 - - - O1
2 + - - O2
3 - + - O3
4 + + - O4
5 - - + O5
6 + - + O6
7 - + + O7
8 + + + O8
“+” refers to one level of a factor and “-”
refers to the other level. Normally, for
quantitative factors, the smallest and largest
levels for each factor are chosen.
2k
Factorial Designs

10. 10
2k
Factorial Designs:
Estimating Main Effects
The main effect of factor 1 is the change
in the response variable as a result of the
change in the level of the factor, averaged
over all levels of all of the other factors
If the effect of some factor depends on the
level of another factor, these factors are
said to interact.
The degree of interaction (two-factor
interaction effect) between two factors i
and j is defined as half the difference
between the average effect of factor i when
factor j is at its “+” level ( and all factors
other than i and j are held constant) and
the average effect of i when j is at its “-”
level; for example,
( ) ( ) ( ) ( )e
O O O O O O O O
1
2 1 4 3 6 5 8 7
4
=
− + − + − + −
e
O O O O O O O O
12
4 3 8 7 2 1 6 51
2 2 2
=
− + −
−
− + −



( ) ( ) ( ) ( )

11. 11
Example: Full Factorial (2k
)
Design
Consider a simulation model of reorder
point, reorder quantity inventory system.
The two decision variables, or factors, to
consider are the reorder point (P) and the
reorder quantity (Q) for the inventory
system. The maximum and minimum
allowable values for each are given below:
Suppose that the response variable
output by the model is the long-run
average monthly cost (composed of three
components: holding cost, shortage costs,
and ordering costs) in thousands of
dollars.
Factor Minimum
Value
Maximum
Value
P 20 40
Q 15 50

12. 12
Example: Full Factorial (2k
)
Design
A 22
factorial design matrix with simulation
results (for 10 replications at each design
point) might be given by:
where a factor level of “-” indicates the
minimum possible value for that factor, and
a factor level of “+” indicates the maximum
possible level; for example, design point 2
has P=40 and Q=15.
Design Point Factor Level for
P Q
Response
1
2
3
4
- -
+ -
- +
+ +
135.6
128.2
121.7
131.5

13. 13
The response given is the average cost
over the 10 replications. Now, the main
effects are given by:
The interaction effect (ePQ) is given by:
Therefore, the average effect of increasing
P from 20 to 40 is to increase monthly
cost by 1.2, and the average effect of
increasing Q from 15 to 50 is to decrease
monthly cost by 5.3. Hence, it would be
advisable to set P as low as possible and
set Q as high as possible.
e
O O O O
Q =
− + −
=
− + −
= −
( ) ( ) ( . . ) ( . . )
.3 1 4 2
2
1217 1356 1315 1282
2
53
e
O O O O
P =
− + −
=
− + −
=
( ) ( ) ( . . ) ( . . )
.2 1 4 3
2
1282 1356 1315 1217
2
12
ePQ =
− − +
=
1356 128 2 1217 1315
2
8 6
. . . .
.
Example: Full Factorial (2k
)
Design

14. 14
Also since the interaction effect is
positive, it would seem advisable to set
P and Q at opposite levels. (Of course,
all of the above could be inferred from
a cursory analysis of the responses for
the various design points.) Note also
that the literal interpretation of main
effects assumes no interaction effects
(pages 669 and 670 of Law and Kelton,
1991).
Example: Full Factorial (2k
)
Design

15. 15
Randomness of Effects
Note that the main and interaction
effects computed in the previous
examples are just random variables.
To determine if the effects are
“significant” or real, and not due to
random fluctuations, one could
compute values for the main and
interaction effects 10 times (once for
each replication) and form confidence
intervals for each of the main effects,
and the interactions effect. If the
confidence interval contains 0, then
the effect is not statistically
significant. (Note that statistical
significance does not necessarily
imply practical significance).

16. 16
Situations with Many
Factors
When there are many factors to
consider, full factorial, or even
fractional factorial designs may not be
feasible from a computational
standpoint.
Other types of design (e.g., Plackett-
Burman designs or “supersaturated
designs”) may be appropriate (Mauro,
1986).
Another approach is to reduce the
number of factors to consider via
“factor screening” techniques,
involving, for example, group
screening in which a whole group is
treated as a factor.

17. 17
Response Surfaces and
Metamodels
A response surface is a graph of a
response variable as a function of the
various factors.
A metamodel (literally, model of the
simulation model), is an algebraic
representation of the simulation
model, with the factors as
independent variables and a response
as the dependent variable. It
represents an approximation of the
response surface.
The typical metamodel used in a
simulation application is a regression
model.

18. 18
A metamodel through the use of
response surface methodology can be
used to find optimal values for a set of
factors. It can also be used to answer
“what if” questions. (Experimentation
with a metamodel is typically much
less expensive than using a simulation
model directly).
An experimental design process
assumes a particular metamodel, e.g.,
E C P Q B B P B Q B PQ[ ( , )] = + + +0 1 2 12
Response Surfaces and
Metamodels

19. 19
Basic Concepts of
Regression Analysis
Regression is used to determine the
“best” functional relation among
variables.
Suppose that the functional
relationship is represented as:
E(Y) = f (X1, ..., Xp / B1, ..., BE)
where E(Y) is the expected value of
the response variable Y; the
X1, ..., Xp are factors; and the
B1, ..., BE are function parameters;
e.g.,
E(Y) = B1 + B2 X1 + B3 X2 + B4 X1 X2

20. 20
The observed value for Y, for a
given set of X ’s, is assumed to be a
random variable, given by:
Y = f (X1, ..., Xp/B1, ..., BE) +
Where is a random variable with
mean equal to 0 and variance .
The values for B1,...,BE are obtained
by minimizing the sum of squares
of the deviations.
ε
ε
σE
2
Basic Concepts of
Regression Analysis

21. 21
Response Surface
Methodology
Source: (Fu, 1994)
Response surface methodology
(RSM) involves a combination of
metamodeling (i.e., regression) and
sequential procedures (iterative
optimization).

22. 22
RSM involves two phases:
Fit a linear regression model to some
initial data points in the search space
(through replications of the simulation
model). Estimate a steepest descent
direction from the linear regressions
model, and a step size to find a new
(and better) solution in the search
space. Repeat this process until the
linear regression model becomes
inadequate (indicated by when the
slope of the linear response surface is
approximately 0; i.e., when the
interaction effects become larger than
than the main effects).
Fit a nonlinear quadratic regression
equation to this new area of the search
space. Then find the optimum of this
equation.
Response Surface
Methodology

23. 23
Terminology in
Experimental Design
Source:(Ostle, 1963)
Replication - the repetition of the basic
experiment
Treatment - a specific combination of
several factor levels
Experimental Unit - the unit to which a
single treatment is applied to one
replication of the basic experiment
Experimental Error - the failure of two
identically treated experimental units to
yield identical results
Confounding - the “mixing up” of two or
more factors so that it’s impossible to
separate the effects

24. 24
Randomization - randomly assigning
treatments to experimental units (assures
independent distribution of errors)
Main Effect (of a factor) - a measure of the
change in a response variable to changes
in the level of the factor averaged over all
levels of all the other factors
Interaction is an additional effect (on the
response) due to the combined influence
of two or more factors
Terminology in
Experimental Design