Advances in Methods and Practices in Psychological Science · 2023 · 49 citations · 40 references
Social SciencesBehavioral SciencesPsychologyExperimental DesignExperimental PsychologyGroup DynamicsBiasHypothesis TestingSocial JudgmentStatistical AnalysisCross-group GeneralizationGroup Effects
When experimental psychologists make a claim (e.g., “Participants judged X as morally worse than Y”), how many participants are represented? Such claims are often based exclusively on group-level analyses; here, psychologists often fail to report or perhaps even investigate how many participants judged X as morally worse than Y. More troubling, group-level analyses do not necessarily generalize to the person level: “the group-to-person generalizability problem.” We first argue for the necessity of designing experiments that allow investigation of whether claims represent most participants. Second, we report findings that in a survey of researchers (and laypeople), most interpret claims based on group-level effects as being intended to represent most participants in a study. Most believe this ought to be the case if a claim is used to support a general, person-level psychological theory. Third, building on prior approaches, we document claims in the experimental-psychology literature, derived from sets of typical group-level analyses, that describe only a (sometimes tiny) minority of participants. Fourth, we reason through an example from our own research to illustrate this group-to-person generalizability problem. In addition, we demonstrate how claims from sets of simulated group-level effects can emerge without a single participant’s responses matching these patterns. Fifth, we conduct four experiments that rule out several methodology-based noise explanations of the problem. Finally, we propose a set of simple and flexible options to help researchers confront the group-to-person generalizability problem in their own work.
40
Statistical parametric maps in functional imaging: A general linear approach
Karl Friston, Andrew P. Holmes, Keith J. Worsley et al. · Human Brain Mapping · 1994 · 9.8K citations
Handbook of research methods for studying daily life
Choice Reviews Online · 2012 · 2.1K citations
Francis Galton · Nature · 1907 · 1.1K citations · Full text