A novel feature sparsification method for kernel-based approximate policy iteration

In this paper, we present a novel feature sparsification approach for a class of kernel-based approximate policy iteration algorithms called KLSPI. We firstly introduce the relative approximation error in the sparsification process based on the approximate linear dependence (ALD) analysis. The relative approximation error is used as the criterion for selecting the kernel-based features. An improved KLSPI algorithm is also proposed by integrating the new sparsification method with KLSPI. Experimental results on the Inverted Pendulum problem demonstrate that the proposed sparsification method can obtain a smaller size of kernel dictionary than the previous ALD method. Furthermore, by using the more representative samples as the kernel dictionary, the precision of value function approximation has been increased. The improved KLSPI algorithm can also achieve better learning efficiency and policy quality than the original one. The feasibility and validity of the new method are proven.