Parameterized penalties in the dual representation of Markov decision processes

Abstract

Duality in Markov decision processes (MDPs) has been studied recently by several researchers with the goal to derive dual bounds on the value function. In this paper we propose the idea of using parameterized penalty functions in the dual representation of MDPs, which allows us to integrate different types of penalty functions and guarantees a tighter dual bound with more penalties used. To complement and diversify the existing linear penalties developed in the literature, we also introduce a new class of nonlinear penalties that can be used for a broad class of problems and are also easy to implement in practice. Based on this new class of penalties, our framework of parameterized penalties is a promising method to produce tighter dual bounds than existing duality-based methods. We compare the performance of the dual bounds induced by different penalties on a numerical example, demonstrating the effectiveness of our method.