Neural Prediction Errors Reveal a Risk-Sensitive Reinforcement-Learning Process in the Human Brain

TLDR

Humans and animals are highly sensitive to risk, yet traditional reinforcement‑learning models focus only on mean rewards and ignore variance, despite risk being learned through experience. The study aimed to determine whether neural correlates of human reinforcement learning are sensitive to experienced risk. Using fMRI, the authors examined BOLD signals in anatomically defined nucleus accumbens regions, hypothesizing that these signals reflect reinforcement‑learning quantities. The results show that nucleus accumbens BOLD signals encode reward prediction errors, that learned cue values vary with experienced risk, and that neural fluctuations in risk evaluation correlate with behavioral risk aversion, indicating that risk sensitivity is integral to human learning.

Abstract

Humans and animals are exquisitely, though idiosyncratically, sensitive to risk or variance in the outcomes of their actions. Economic, psychological, and neural aspects of this are well studied when information about risk is provided explicitly. However, we must normally learn about outcomes from experience, through trial and error. Traditional models of such reinforcement learning focus on learning about the mean reward value of cues and ignore higher order moments such as variance. We used fMRI to test whether the neural correlates of human reinforcement learning are sensitive to experienced risk. Our analysis focused on anatomically delineated regions of a priori interest in the nucleus accumbens, where blood oxygenation level-dependent (BOLD) signals have been suggested as correlating with quantities derived from reinforcement learning. We first provide unbiased evidence that the raw BOLD signal in these regions corresponds closely to a reward prediction error. We then derive from this signal the learned values of cues that predict rewards of equal mean but different variance and show that these values are indeed modulated by experienced risk. Moreover, a close neurometric-psychometric coupling exists between the fluctuations of the experience-based evaluations of risky options that we measured neurally and the fluctuations in behavioral risk aversion. This suggests that risk sensitivity is integral to human learning, illuminating economic models of choice, neuroscientific models of affective learning, and the workings of the underlying neural mechanisms.