Reinforcement Learning and Feedback Control: Using Natural Decision Methods to Design Optimal Adaptive Controllers

TLDR

Adaptive control and optimal control are distinct philosophies; optimal controllers rely on solving Hamilton–Jacobi–Bellman equations with full system knowledge, which is difficult for nonlinear systems, whereas adaptive controllers learn online from data but are not typically optimal. The study aims to apply reinforcement learning principles to design feedback controllers that merge adaptive and optimal control for discrete‑ and continuous‑time systems. The authors employ reinforcement learning and indirect adaptive control, using system identification to model the system and then solving optimal design equations. The authors find that adaptive controllers can meet specific inverse optimality conditions.

Abstract

This article describes the use of principles of reinforcement learning to design feedback controllers for discrete- and continuous-time dynamical systems that combine features of adaptive control and optimal control. Adaptive control [1], [2] and optimal control [3] represent different philosophies for designing feedback controllers. Optimal controllers are normally designed of ine by solving Hamilton JacobiBellman (HJB) equations, for example, the Riccati equation, using complete knowledge of the system dynamics. Determining optimal control policies for nonlinear systems requires the offline solution of nonlinear HJB equations, which are often difficult or impossible to solve. By contrast, adaptive controllers learn online to control unknown systems using data measured in real time along the system trajectories. Adaptive controllers are not usually designed to be optimal in the sense of minimizing user-prescribed performance functions. Indirect adaptive controllers use system identification techniques to first identify the system parameters and then use the obtained model to solve optimal design equations [1]. Adaptive controllers may satisfy certain inverse optimality conditions [4].