Anticipatory robot control for a partially observable environment using episodic memories

Abstract

This paper explains an episodic-memory-based approach for computing anticipatory robot behavior in a partially observable environment. Inspired by biological findings on the mammalian hippocampus, here, episodic memories retain a sequence of experienced observation, behavior, and reward. Incorporating multiple machine learning methods, this approach attempts to help reducing the computational burden of a partially observable Markov decision process (POMDP) problem. In particular, proposed computational reduction techniques include: (1) abstraction of the state space via temporal difference learning; (2) abstraction of the action space by utilizing motor schemata; (3) narrowing down the state space in terms of goals through instance-based learning; (4) elimination of the value-iteration by assuming a unidirectional-linear-chaining formation of the state space; (5) reduction of the state-estimate computation by exploiting the property of the Poisson distribution; and (6) trimming the history length by imposing a cap on the number of episodes that are computed. Claims (5) and (6) were empirically verified, and it was confirmed that the state estimation can be in fact computed in an 0(n) time (where n is the number of the states), more efficient than a conventional Kalman-filter based approach of 0(n <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> ).