Reinforcement Learning Control of Double-Layer Markov Jump Systems With PDT-Switched Transition Probabilities

Abstract

This paper proposes a data-based reinforcement Q-learning control method to efficiently solve the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> control problem for discrete-time double-layer Markov jump systems (DLMJSs). The transition probabilities are considered to be piecewise-homogeneous and governed by a persistent dwell-time (PDT) switching scheme. First, a model-based stability and performance optimization criterion for DLMJSs is derived. On this basis, a novel reinforcement Q-learning method in the form of linear matrix inequality is presented, such that optimal disturbance attenuation level and <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> control policy can be calculated from the collected data. At the core of the approach is a two-step learning design associated with the Q-function. The first step is to estimate the kernel matrix by solving the minimization problem. The second step is to find an improved control policy based on the estimated kernel matrix. Furthermore, the rigorous analysis of the stability of closed-loop DLMJSs and the convergence of the designed algorithm are presented, ensuring that closed-loop systems achieve mean-square exponential stability with an optimal disturbance attenuation level. Finally, the applicability and effectiveness of the proposed method in engineering applications are verified by an armature controlled DC motor system model. Note to Practitioners—Due to the complexity of engineering environment, the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> control of DLMJSs has garnered extensive attention. Nowadays, DLMJSs find widespread application in various domains including tidal turbine systems, robot manipulators, and smart grids. It should be noted that existing <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> control methods for DLMJSs typically necessitate precise system dynamics and assume a prescribed disturbance attenuation level. However, in practical applications, these assumptions are difficult to satisfy. In view of these facts, we employ the reinforcement Q-learning control method to solve the <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> control problem of unknown DLMJSs with PDT-switched transition probabilities. Moreover, the designed control method can achieve both the optimal disturbance attenuation level and the optimal <inline-formula xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink"> <tex-math notation="LaTeX">$H_{\infty }$ </tex-math></inline-formula> control policy of the studied systems, providing a feasible strategy for improving control performance in industrial applications.