validation of experiment

1. Model Validation
How do you plan your validation?

2. empirical Data
 Data acquired by means of obervation
 Or experiment

3. Empirical Analysis
 Quantitative – what, where, when
 Qualitative - in-depth understanding
why and how

4. Model
 simulation to reproduce behavior of a
system
 Enables prediction (predictive)
 Enables control of a system (perscriptive)

5. Model Validation
 validation is necessary for both predictive
and prescriptive model use.
 Validation exercises
 improve model performance
 Improve insight.

6. Basic Model
Maximize f(X) - g(Z)
s.t. X -DY ≤ 0
GY - Z ≤ 0
AY ≤ b
X,Y,Z ≥ 0
The optimal values, X*, Y*, Z are assumed to correspond to real world
observations X’, Y’, Z’. The model also has shadow prices U*, V*, W*
that correspond to real world values U’, V’, W’.
X is some output measure. Z is some input measures.

7. Approaches to Validation
 Validation by Construction
 Validation by Results
 Validation Experiments

8. Validation
 The goal - test how well a model predicts
 Determine if the prediction is good enough
 comparing predictions to data

9. General Approaches
 use historical events.
 Identify area of model use.
 Places where model does not apply.

10. Validation is Subjective
 Selection of tests is subjective
 Criteria of what is good enough, subjective

11. Validation by Construction
 Constructive synthesis
 Destructive synthesis

12. Limitations
Constructive synthesis must be tested
Destructive synthesis must be test

13. Validation by Results
 1. Gather inputs and outputs
o 2. Use inputs on model
o 3. Gather model outputs
o 4. Compare model outputs with real-world outputs
o 5. Determine validity

14. Validation Experiments
 Feasibility Experiment
 Quantity Experiment
 Price Experiment
 Prediction Experiment
 Change Experiment
 Tracking Experiment
 Partial Tests

15. Prediction Experiment
The prediction experiment is the most common validation by results test.
the ultimate validation experiment in that it tests whether a model can
replicate reality.

16. Change Experiment
To test a model's ability to predict change, one must have
data on two real world situations and the resultant model
solutions.
Then, a comparison is made between the change in the
model solution variables (e.g.,X1 *, X2 *) and the change
observed in the real world solution X1’,X2’.

17. Tracking Experiment
One may wish to track adjustments through time.
comparisons are made between changes in the model
solution and observed changes in the real world
solution

18. Comments
 Validation is important
 A good model is validated by construction
and by results
 Ultimate test of validity is adoption of the
model by decision makers