Reinforcement Learning for Linear-Convex Models with Jumps via Stability Analysis of Feedback Controls

Abstract

.We study finite-time horizon continuous-time linear-convex reinforcement learning problems in an episodic setting. In this problem, the unknown linear jump-diffusion process is controlled subject to nonsmooth convex costs. We show that the associated linear-convex control problems admit Lipschitz continuous optimal feedback controls and further prove the Lipschitz stability of the feedback controls, i.e., the performance gap between applying feedback controls for an incorrect model and for the true model depends Lipschitz-continuously on the magnitude of perturbations in the model coefficients; the proof relies on a stability analysis of the associated forward-backward stochastic differential equation. We then propose a novel least-squares algorithm which achieves a regret of the order \(O(\sqrt{N \ln N})\) on linear-convex learning problems with jumps, where \(N\) is the number of learning episodes; the analysis leverages the Lipschitz stability of feedback controls and concentration properties of sub-Weibull random variables. Numerical experiment confirms the convergence and the robustness of the proposed algorithm.Keywordscontinuous-time reinforcement learninglinear-convexjump-diffusionLipschitz stabilityleast-squares estimationsub-Weibull random variableMSC codes93E3562G3593E2468Q32