Reinforcement learning based computational adaptive optimal control and system identification for linear systems

Abstract The duality of estimation and control problems is a well known fact in control theory literature. Simultaneous parameter estimation while maintaining closed loop stability is a very difficult proposition and more so for unstable systems, even for linear systems. This typically motivates system identification to be performed only in offline experiments. Clearly, there is a need for a higher level abstraction for a control and identification scheme which acts in stages and prioritizes various aspects of the problem at each of these stages. The stage abstraction for the controller design in this paper is inspired by human intuition towards dealing with control and identification simultaneously and hence named “Intuitive Control Framework”. The first phase prioritizes stabilization of the system only. The controller moves onto the next phase after the unknown system is stabilized. The subsequent stages during this phase involve optimization with different performance metrics through adaptive learning. After enough information for identification is acquired, the control schemes developed for various optimal metrics are used to estimate the unknown parameters in the final phase. This narrative for selective prioritization of objectives and a higher level abstraction for control schemes is illustrated for a continuous linear time invariant state space realization with state feedback. Numerous real-world applications can benefit from this online system identification routine inspired by the human cognitive process. This offers a seamless integration of control and identification with a higher level of priorities. Such a framework is presented with explicit formulations for certain classes of dynamic systems, and evaluated with computer simulations as well as experimental results. An unstable multi-input multi-output linear system is used as an example to illustrate the approach.