Credit assignment to state-independent task representations and its relationship with model-based decision making

TLDR

Model‑free learning improves decisions by relying on prior experience while encoding only minimal knowledge of the environment’s structure, and it is usually assumed that such state representations depend on outcome‑relevant features. Here, we challenge this assumption by providing evidence that a putative model‑free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well‑described 2‑step reward decision task where stimulus identity but not spatial‑motor aspects predicted reward. Participants assigned value to spatial‑motor representations that were irrelevant to reward, and these associations influenced behavior across all outcome‑relevant features and task stages, yet their impact was weaker in individuals who employed more goal‑directed (model‑based) strategies, underscoring the need to rethink how model‑free representations are formed and regulated.

Abstract

Model-free learning enables an agent to make better decisions based on prior experience while representing only minimal knowledge about an environment’s structure. It is generally assumed that model-free state representations are based on outcome-relevant features of the environment. Here, we challenge this assumption by providing evidence that a putative model-free system assigns credit to task representations that are irrelevant to an outcome. We examined data from 769 individuals performing a well-described 2-step reward decision task where stimulus identity but not spatial-motor aspects of the task predicted reward. We show that participants assigned value to spatial-motor representations despite it being outcome irrelevant. Strikingly, spatial-motor value associations affected behavior across all outcome-relevant features and stages of the task, consistent with credit assignment to low-level state-independent task representations. Individual difference analyses suggested that the impact of spatial-motor value formation was attenuated for individuals who showed greater deployment of goal-directed (model-based) strategies. Our findings highlight a need for a reconsideration of how model-free representations are formed and regulated according to the structure of the environment.