An Equivalence between Continuous and Discrete Time Markov Decision Processes.

Abstract

A continuous time Markov decision process with bounded sojourn parameters (the parameters of the exponential sojourn times in the states) is shown to be equivalent to a simpler discrete time Markov decision process. This is for both the discounted and average cost criteria on an infinite horizon. This result follows from (1) an equivalence of certain functionals of equivalent Markov processes, and (2) the property that a continuous time jump Markov process with bounded sojourn parameters is equal in distribution to a similar Markov process whose sojourn parameters are all the same. (Author)