Philosophical Transactions of the Royal Society B Biological Sciences · 2014 · 160 citations · 49 references
BusinessBehavioral SciencesDecision SciencesBehavior Change TheoryCognitive PsychologyDecision TheoryBehavioral ChangeBehavioral Decision-makingDecision-making ModelsHabit FormationReinforcement LearningSequential Decision-makingAction PlanningAction SequenceExtinctionGoal-directed Action ControlGoal-directed LearningGoal-directed PlanningHabitual ActionSequence Boundaries
Goal-directed action involves making high-level choices that are implemented using previously acquired action sequences to attain desired goals. Such a hierarchical schema is necessary for goal-directed actions to be scalable to real-life situations, but results in decision-making that is less flexible than when action sequences are unfolded and the decision-maker deliberates step-by-step over the outcome of each individual action. In particular, from this perspective, the offline revaluation of any outcomes that fall within action sequence boundaries will be invisible to the high-level planner resulting in decisions that are insensitive to such changes. Here, within the context of a two-stage decision-making task, we demonstrate that this property can explain the emergence of habits. Next, we show how this hierarchical account explains the insensitivity of over-trained actions to changes in outcome value. Finally, we provide new data that show that, under extended extinction conditions, habitual behaviour can revert to goal-directed control, presumably as a consequence of decomposing action sequences into single actions. This hierarchical view suggests that the development of action sequences and the insensitivity of actions to changes in outcome value are essentially two sides of the same coin, explaining why these two aspects of automatic behaviour involve a shared neural structure.
49
Reinforcement Learning: An Introduction
Richard S. Sutton, Andy Barto · IEEE Transactions on Neural Networks · 1998 · 27.1K citations
Reinforcement Learning: An Introduction
IEEE Transactions on Neural Networks · 2005 · 25.8K citations
Artificial intelligence: a modern approach
Choice Reviews Online · 1995 · 22.2K citations · Full text
Cognitive maps in rats and men.
E. C. Tolman · Psychological Review · 1948 · 6.7K citations