Frontiers in Neuroscience · 2012 · 216 citations · 53 references
Life SciencesNeurobiologyNeuroscienceBehavioral NeuroscienceNeuropharmacologyBasal GangliaBasal Ganglia PhysiologyDopaminergic SignalingExploration-exploitation TradeoffNeurotransmissionReward PathwaysBasal Ganglia FunctionDopaminergic NeuronsDopaminergic NeurophysiologyDopaminergic PathwaysExploitation Trade-offExploration - Exploitation Trade-offExploration-exploitation Trade-offStriatal Physiology
We continuously face the dilemma of choosing between actions that gather new information or actions that exploit existing knowledge. This "exploration-exploitation" trade-off depends on the environment: stability favors exploiting knowledge to maximize gains; volatility favors exploring new options and discovering new outcomes. Here we set out to reconcile recent evidence for dopamine's involvement in the exploration-exploitation trade-off with the existing evidence for basal ganglia control of action selection, by testing the hypothesis that tonic dopamine in the striatum, the basal ganglia's input nucleus, sets the current exploration-exploitation trade-off. We first advance the idea of interpreting the basal ganglia output as a probability distribution function for action selection. Using computational models of the full basal ganglia circuit, we showed that, under this interpretation, the actions of dopamine within the striatum change the basal ganglia's output to favor the level of exploration or exploitation encoded in the probability distribution. We also found that our models predict striatal dopamine controls the exploration-exploitation trade-off if we instead read-out the probability distribution from the target nuclei of the basal ganglia, where their inhibitory input shapes the cortical input to these nuclei. Finally, by integrating the basal ganglia within a reinforcement learning model, we showed how dopamine's effect on the exploration-exploitation trade-off could be measurable in a forced two-choice task. These simulations also showed how tonic dopamine can appear to affect learning while only directly altering the trade-off. Thus, our models support the hypothesis that changes in tonic dopamine within the striatum can alter the exploration-exploitation trade-off by modulating the output of the basal ganglia.
53
Reinforcement Learning: An Introduction
Richard S. Sutton, Andy Barto · IEEE Transactions on Neural Networks · 1998 · 27.1K citations
Reinforcement Learning: An Introduction
IEEE Transactions on Neural Networks · 2005 · 25.8K citations
Cortical substrates for exploratory decisions in humans
Nathaniel D. Daw, John P. O’Doherty, Peter Dayan et al. · Nature · 2006 · 2.3K citations · Full text
Learning the value of information in an uncertain world
Timothy E.J. Behrens, Mark W. Woolrich, Mark E. Walton et al. · Nature Neuroscience · 2007 · 2.2K citations