Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning

TLDR

Model‑free reinforcement learning has explored using learned dynamics models to generate imagined data and reduce sample complexity, yet practical methods are constrained by heuristic limitations. The authors propose model‑based value expansion, which limits imagination to a fixed depth to control model uncertainty. This method expands value estimates by simulating trajectories up to a fixed depth using the learned dynamics model. The approach yields more accurate value estimates and lowers sample complexity in continuous control tasks.

Abstract

Recent model-free reinforcement learning algorithms have proposed incorporating learned dynamics models as a source of additional data with the intention of reducing sample complexity. Such methods hold the promise of incorporating imagined data coupled with a notion of model uncertainty to accelerate the learning of continuous control tasks. Unfortunately, they rely on heuristics that limit usage of the dynamics model. We present model-based value expansion, which controls for uncertainty in the model by only allowing imagination to fixed depth. By enabling wider use of learned dynamics models within a model-free reinforcement learning algorithm, we improve value estimation, which, in turn, reduces the sample complexity of learning.