Theoretically Informed Generative Models Can Advance the Psychological and Brain Sciences: Lessons from the Reliability Paradox

Abstract

Theories of individual differences are foundational to psychological and brain sciences, yet they are traditionally developed and tested using superficial summaries of data (e.g., mean response times) that are both (1) disconnected from our otherwise rich conceptual theories of behavior, and (2) contaminated with measurement error. Traditional approaches therefore lack the flexibility required to test increasingly complex theories of behavior. To resolve this theory- description gap, we present the generative modeling approach, which involves formally specifying how behavior is generated within people and how generative processes vary across people. Generative modeling shifts our focus away from estimating descriptive statistical “effects” toward estimating psychologically interpretable parameters, while simultaneously enhancing the reliability and validity of our measures. We demonstrate the utility of generative models in the context of the “reliability paradox”, a phenomenon wherein highly replicable group effects (e.g., Stroop effect) fail to capture individual differences (e.g., low test-retest reliability). Simulations and empirical data from the Implicit Association Test, and Stroop, Flanker, Posner, and Delay Discounting tasks show that generative models yield (1) more theoretically informative parameters, and (2) higher test-retest reliability estimates relative to traditional approaches, illustrating their potential for enhancing theory development.