Towards Feature Selection In Actor-Critic Algorithms

Abstract : Choosing features for the critic in actor-critic algorithms with function approximation is known to be a challenge. Too few critic features can lead to degeneracy of the actor gradient, and too many features may lead to slower convergence of the learner. In this paper, the authors show that a well-studied class of actor policies satisfy the known requirements for convergence when the actor features are selected carefully. They demonstrate that two popular representations for value methods -- the barycentric interpolators and the graph Laplacian proto-value functions -- can be used to represent the actor so as to satisfy these conditions. A consequence of this work is a generalization of the proto-value function methods to the continuous action actor-critic domain. Finally, they analyze the performance of this approach using a simulation of a torque-limited inverted pendulum.