Lack of group-to-individual generalizability is a threat to human subjects research

TLDR

The study aimed to evaluate how well aggregated social and medical science data reflect individual participants. The authors used intensive repeated‑measures data across many individuals to compare within‑subject and between‑subject bivariate correlation distributions. The study quantified that group data poorly represent individuals, showing up to fourfold greater individual variance and indicating that aggregated conclusions may be imprecise, thereby urging a shift toward idiographic and open science practices.

Abstract

Significance The current study quantified the degree to which group data are able to describe individual participants. We utilized intensive repeated-measures data—data that have been collected many times, across many individuals—to compare the distributions of bivariate correlations calculated within subjects vs. those calculated between subjects. Because the vast majority of social and medical science research aggregates across subjects, we aimed to assess how closely such aggregations reflect their constituent individuals. We provide evidence that conclusions drawn from aggregated data may be worryingly imprecise. Specifically, the variance in individuals is up to four times larger than in groups. These data call for a focus on idiography and open science that may substantially alter best-practice guidelines in the medical and behavioral sciences.