EXPERIMENTS WITH FRESHWATER INVERTEBRATE ZOOPLANKTIVORES: QUALITY OF STATISTICAL ANALYSES

Abstract

We examined the statistical analyses of experimental data in 95 papers published during 1966-1990 on the ecology, physiology, and behavior of freshwater invertebrate zooplanktivores. Serious statistical errors were found in 51% of the papers. The frequencies of particular types of errors were as follows: sacrificial pseudoreplication (31%), simple pseudoreplication (11%), temporal pseudoreplication (7%), pseudofactorialism (5%), misuse of one-tailed tests (4%), and failure to use log-transformation in multi-way ANOVA when appropriate (9%). Several of these frequencies are much higher when calculated on the basis of the number of papers where a given type of error is possible. For example, pseudofactorialism occurred in 28% of the papers using multi-way ANOVA, the only type of paper for which this error is possible. We hope that this exhaustive review of a very specific but important area of plankton research will create a positive feedback network among this core group of researchers. This could so improve the use of statistics in future plankton studies that this field could serve as a model for other areas of biology equally plagued by improper statistical analysis. The quality of statistical analyses in the biological sciences is poor, as numerous critical reviews have shown. Indeed, if the frequencies of statistical error reported by reviews of the ecological (Hurlbert, 1984), marine biological (Underwood, 1981), behavioral (Machlis et al., 1985; Kroodsma, 1989), and biomedical (Schor and Karten, 1966; Glantz, 1980; O'Fallon et al., 1978) literature are representative, the majority of papers that employ statistical analyses contain serious statistical errors. The errors are diverse in kind, and their consequences for the validity and strength of conclusions are thus also diverse. In many disciplines one of the commonest types of error is pseudoreplication (sensu Hurlbert, 1984). Its most usual consequence is the underestimation of P values in significance tests. These values are often one to several orders of magnitude lower than the P values that would have been yielded by correct statistical procedures. The acceptance of the conclusions of such papers by trusting editors and readers has been based on false premises. Reanalysis of many specific findings will show them to be statistically unsupported. Such error may not permanently bias our understanding of any general phenomena (e.g., the ability of substances produced by Chaoborus to induce production of protective spines and helmets in certain prey species) that ultimately are studied in multiple experiments by multiple investigators. It does seem likely to slow progress and to lead us on wild goose chases occasionally. The greatest immediate consequence, however, is the tremendous burden it places on conscientious editors, reviewers, thesis advisors, and statisticians. The morass of incorrect statistical analyses in the literature creates a Sisyphean task for them. It provides an abundance of negative models that continually are undoing their instructional efforts. The immediate sources of this problem are many, but they all may derive chiefly from lack of conceptual and terminological clarity in many key areas of statistics. If so, attempts to clarify concepts and terminology through critiques of large numbers of specific examples of statistical malpractice may be the most effective countermeasure. This was the premise of an earlier review (Hurlbert, 1984), the favorable reception of which has reinforced our faith in that approach.