Measurement Schmeasurement: Questionable Measurement Practices and How to Avoid Them

TLDR

Doubts arise from lack of transparency, ignorance, negligence, or misrepresentation of evidence. The paper defines questionable measurement practices, outlines their scope and the role of transparency, and proposes questions to help researchers identify and avoid them. The authors describe QMPs, explain how transparency can mitigate them, and present a set of questions for researchers and consumers to assess measurement practices. The study shows that lack of transparency hampers validity assessment, that QMPs are widespread and threaten cumulative science, and that transparent answers to the proposed questions enhance rigor and enable replication.

Abstract

In this paper, we define questionable measurement practices (QMPs) as decisions researchers make that raise doubts about the validity of the measures, and ultimately the validity of study conclusions. Doubts arise for a host of reasons including a lack of transparency, ignorance, negligence, or misrepresentation of the evidence. We describe the scope of the problem and focus on how transparency is a part of the solution. A lack of measurement transparency makes it impossible to evaluate potential threats to internal, external, statistical conclusion, and construct validity. We demonstrate that psychology is plagued by a measurement schmeasurement attitude: QMPs are common, hide a stunning source of researcher degrees of freedom, pose a serious threat to cumulative psychological science, but are largely ignored. We address these challenges by providing a set of questions that researchers and consumers of scientific research can consider to identify and avoid QMPs. Transparent answers to these measurement questions promote rigorous research, allow for thorough evaluations of a study’s inferences, and are necessary for meaningful replication studies.