Model-Ensemble Trust-Region Policy Optimization

TLDR

Model-free reinforcement learning has achieved success in many tasks, yet its high sample complexity limits real-world deployment, while model-based approaches can reduce data needs but often require careful tuning and work only in simple domains. The authors investigate why vanilla model-based RL with deep neural networks tends to exploit poorly learned regions, leading to instability, and propose using an ensemble of models to capture uncertainty and regularize training. They implement this by maintaining a model ensemble and employing likelihood‑ratio derivatives instead of backpropagation through time to stabilize learning. The resulting Model‑Ensemble Trust‑Region Policy Optimization (ME‑TRPO) markedly lowers sample complexity compared to model‑free deep RL on challenging continuous‑control benchmarks.

Abstract

Model-free reinforcement learning (RL) methods are succeeding in a growing number of tasks, aided by recent advances in deep learning. However, they tend to suffer from high sample complexity, which hinders their use in real-world domains. Alternatively, model-based reinforcement learning promises to reduce sample complexity, but tends to require careful tuning and to date have succeeded mainly in restrictive domains where simple models are sufficient for learning. In this paper, we analyze the behavior of vanilla model-based reinforcement learning methods when deep neural networks are used to learn both the model and the policy, and show that the learned policy tends to exploit regions where insufficient data is available for the model to be learned, causing instability in training. To overcome this issue, we propose to use an ensemble of models to maintain the model uncertainty and regularize the learning process. We further show that the use of likelihood ratio derivatives yields much more stable learning than backpropagation through time. Altogether, our approach Model-Ensemble Trust-Region Policy Optimization (ME-TRPO) significantly reduces the sample complexity compared to model-free deep RL methods on challenging continuous control benchmark tasks.