Multiple Comparison Procedures—Cutting the Gordian Knot

Abstract

Multiple comparison procedures (MCPs), or mean separation tests, have been the subject of great controversy since the 1950s. Essentially, these procedures are an attempt at simultaneously formulating and testing pairwise comparison hypotheses using data from a single experiment. An unacceptable operating characteristic of most MCPs is their “inconsistency,” an idea that is illustrated in this article. This characteristic led to the development of a “practical solution” to the MCP problem, which is to “cut the Gordian knot” by abandoning any attempt at simultaneous formulation and testing. Instead, I recommend using the simplest multiple comparison procedure, the unrestricted least significant difference procedure, to (i) formulate new hypotheses at a known “false discovery rate” (in the null case) such as 5%, and (ii) independently test interesting new hypotheses in a second experiment. I also discuss the implications for sample size calculations of the choice of MCP.