In this paper, we investigate a model-free optimal control design that minimizes an infinite horizon average expected quadratic cost of states and control actions subject to a probabilistic risk or chance constraint using input-output data. In particular, we consider linear time-invariant systems and design an optimal controller within the class of linear state feedback controls. Two different policy gradient (PG) based algorithms, natural policy gradient (NPG) and Gauss-Newton policy gradient (GNPG) are developed and compared to deep deterministic policy gradient (DDPG), the optimal risk-neutral linear-quadratic regulator (LQR), chance constrained LQR, and a scenario-based model predictive control (MPC). The convergence properties and the accuracy of all the algorithms are compared numerically. We also establish analytical convergence properties of the NPG algorithm under the known model scenario, while convergence analysis for the unknown model scenario is part of our ongoing work.

Abstract:Flow  -learning has recently been introduced to integrate learning from expert demonstrations into an actor-critic structure. Central to this innovation is the ``the one-step policy'' network, which is optimized through a  -function that is regularized with the behavioral cloning from expert trajectories, allowing learning more expressive policies using flow-based generative models. In this paper, we studied the convergence property and stabilizablity of the one-step policy during learning for linear quadratic problems under the offline settings. Our theoretical results are based on a new formulation of the one-step policy loss based on the average expected cost, and regularized with the behavioral cloning loss. Such a formulation allows us to tap into existing strong theoretical results from the policy gradient theorem to study the convergence properties of the one-step policy. We verify our theoretical finding with simulation results on a linearized inverted pendulum.