Online learning algorithm for Stackelberg games in problems with hierarchy

Abstract

This paper presents an online adaptive optimal control algorithm based on policy iteration reinforcement learning techniques to solve the continuous-time Stackelberg games with infinite horizon for linear systems. This adaptive optimal control method finds in real-time approximations of the optimal value and the Stackelberg-equilibrium solution, while also guaranteeing closed-loop stability. The optimal-adaptive algorithm is implemented as a separate actor/critic parametric network approximator structure for every player, and involves simultaneous continuous-time adaptation of the actor/critic networks. Novel tuning algorithms are given for the actor/critic networks. The convergence to the closed-loop Stackelberg equilibrium is proven and stability of the system is also guaranteed. A simulation example shows the effectiveness of the new online algorithm.