Using Common Random Numbers and Control Variates in Multiple-Comparison Procedures

Abstract

This paper considers the determination of the relative merits of two or more system designs via stochastic simulation experiments by constructing simultaneous interval estimates of certain differences in expected performance. Tukey's all-pairwise-comparisons procedure, Hsu's multiple-comparisons-with-the-best procedure, and Dunnett's multiple-comparisons-with-a-control procedure are standard methods for making such comparisons. We propose refinements for all three procedures through the use of two variance reduction techniques: common random numbers and control variates. We show that the proposed procedures are better than the standard multiple-comparison procedures in the sense that they have a larger probability of containing the true difference and, at the same time, not containing zero when a difference exists.