Optimal CPS Command Dispatch Based on Hierarchically Correlated Equilibrium Reinforcement Learning

A hierarchical correlated Q-learning(HCEQ)approach is presented to solve the dynamic optimization of generation command dispatch(GCD)for automatic generation control(AGC).In order to decrease the dimensions of GCD,the AGC units are classified into different clusters according to their time delay during frequency control.Compared with single-agent reinforcement learning,the HCEQ method introduces the solution of equilibrium objective function,which effectively improves the optimization speed.The generating error,hydropower capacity margin and AGC regulating cost are turned into the Markov decision process reward function via the linearly weighted aggregate algorithm.The application of the hierarchical correlated Qlearning algorithm in China southern power grid(CSG)model shows that the method proposed is capable of reducing the converging time in the pre-learning process and the AGC regulating cost while improving the control performance of AGC systems in a complicated environment of random perturbation.