There are two main statistical techniques for comparing systems: independent sampling and correlated sampling. When comparing two systems, it is necessary to use confidence intervals. There are three possible scenarios when computing confidence intervals depending on if the sampling is independent or correlated. When comparing several designs, the goals may be to estimate each performance measure, compare to a present system, or select the best. The Bonferroni approach can be used to make statements about multiple alternatives while controlling the overall confidence level. Design of experiments tools like factorial designs, screening, and response surface methods can help understand the effect of design alternatives on performance measures.

1. Comparison and Evaluation of Alternative Designs Adapted from J. Banks

2. 2 Statistical techniques used in comparing Systems: Independent Sampling – the systems are responding in negative correlation between each other. This has been observed in some inventory problems. Correlated Sampling or Common Random Numbers – responds in the same direction for each input of random variates (monotonic). This is common for certain simple queuing problems.

3. Comparing 2 Systems: It is necessary to make use of confidence intervals when comparing two systems. The confidence interval should be the difference between two systems.

4. Three possible scenarios when computing for the confidence interval of the differences (90%, 95%, or 99%): 0 0 0

5. When using independent sampling: Independent sampling with equal variances Independent sampling with unequal variances

6. When using correlated sampling: Dedicate a random number stream for a specific purpose. Use as many as needed. Use attributes of an entity to consistently apply same service times, order quantity, etc. (which are dependent on the entity) Use a specific stream for activities with cycles. Examples are changes in shifts. Synchronize if possible. Otherwise, use independent random numbers

8. Comparison of Several Designs Possible Goals of an Analyst: Estimate each performance measure Compare each performance measure to a present system (control) All possible comparison Selection of the best

9. Using Bonferroni Approach for Comparison: When making statements about several alternatives, an analyst would like to be confident that all statements are true simultaneously.

10. This method can be used in three ways: Individual C.I.s of a single system with multiple performance measures. The alpha is simply the product of all alphas used in the comparison. They are assumed to be independent. Comparison to a present system. Construct a 1-alpha confidence interval for each comparison. All possible comparison. Use the equation above. This assumes that correlated sampling was used.

11. Selecting the Best Two objectives: Determining the best of the alternatives Determining how much the best is relative to the rest of the alternatives NOTE: It might be possible that selecting the second best is more practical, less costly, more feasible, but still very insignificantly close to being the best.

12. Understanding the Effect of the Design Alternatives: Use the power of design of experiments Some of the tools under DOE are: Factorial Designs (useful for understanding the effects of the alternatives) Screening , these are fractional factorial and Placket Burman (useful for trimming down the unimportant alternatives) Response Surface , these are Central Composite and Box Benhken (useful for identifying the optimal setup in an alternative)

13. Metamodeling Constructing a relationship between the performance measure, Y, and the design variables, X. Some common relationships are: Simple Linear Regression, Nonlinear relationships, Multiple Linear Regression. To verify whether these relationships are reliable for predicting the effect on the performance measure, it is necessary to test the significance of the regressions (ANOVA is used).