Efficient estimation of dynamically optimal learning rate using higher order derivatives

Efficient estimation of the dynamically optimal learning rate is a critical problem in backpropagation learning. In this paper, a higher-order method for efficiently estimating the dynamically optimal learning rate is established, which explores the first four derivative information gathered from an extended feedforward propagation procedure. The near-optimal learning rate for each iteration is obtained with a moderate increase in computational and storage burden which remains the same scale as the standard backpropagation algorithm. Extensive computer simulations provided in this paper indicate that the present higher-order method can provide rapid convergence and very significant gains in running time savings.