Multiple Comparison Procedures: The Practical Solution

TLDR

The paper reviews the multiple‑comparison controversy, focusing on consistency across five common procedures. All procedures are inconsistent except the unrestricted LSD, which is recommended as a hypothesis‑generating tool, and a new recommendation for Scheffé’s test is proposed.

Abstract

Abstract A practicing statistician looks at the multiple comparison controversy and related issues through the eyes of the users. The concept of consistency is introduced and discussed in relation to five of the more common multiple comparison procedures. All of the procedures are found to be inconsistent except the simplest procedure, the unrestricted least significant difference (LSD) procedure (or multiple t test). For this and other reasons the unrestricted LSD procedure is recommended for general use, with the proviso that it should be viewed as a hypothesis generator rather than as a method for simultaneous hypothesis generation and testing. The implications for Scheffé's test for general contrasts are also discussed, and a new recommendation is made.