Operations Research · 1973 · 1.5K citations · 5 references
EngineeringMathematicsSystems EngineeringOptimizationProbability TheoryStochastic ProcessesFinite-state MachinesMaintenance OptimizationMarkov ChainsMarkov ProcessesOptimal ControlStochastic ControlConvex FunctionsDiscrete-time Markov ChainsDiscrete-time Markov ProcessesFinite Horizon AnalysisFinite Horizon ControlFinite-horizon Decision MakingFinite-horizon OptimizationFinite-state Markov Chains
The formulation is illustrated by a simple machine‑maintenance example, and other specific application areas are also discussed. This paper formulates the optimal control problem for a class of mathematical models in which the system to be controlled is characterized by a finite‑state discrete‑time Markov process and outlines an algorithm to compute the optimal control policy and payoff function for any finite horizon. Because the internal finite‑state Markov process is not directly observable, the controller uses probabilistically related observable outputs, which the authors exploit to compute the optimal policy. The optimal payoff function is a piecewise‑linear, convex function of the current state probabilities when only a finite number of control intervals remain, as illustrated by a numerical machine‑maintenance example.
This paper formulates the optimal control problem for a class of mathematical models in which the system to be controlled is characterized by a finite-state discrete-time Markov process. The states of this internal process are not directly observable by the controller; rather, he has available a set of observable outputs that are only probabilistically related to the internal state of the system. The formulation is illustrated by a simple machine-maintenance example, and other specific application areas are also discussed. The paper demonstrates that, if there are only a finite number of control intervals remaining, then the optimal payoff function is a piecewise-linear, convex function of the current state probabilities of the internal Markov process. In addition, an algorithm for utilizing this property to calculate the optimal control policy and payoff function for any finite horizon is outlined. These results are illustrated by a numerical example for the machine-maintenance problem.
5
A Simple Model of Search for a Moving Target
Stephen M. Pollock · Operations Research · 1970 · 101 citations
Engineering, Computer Science, Markov Decision Processes +16