Batch gradient method with smoothing L1/2 regularization for training of feedforward neural networks

The aim of this paper is to develop a novel method to prune feedforward neural networks by introducing an regularization term into the error function. This procedure forces weights to become smaller during the training and can eventually removed after the training. The usual regularization term involves absolute values and is not differentiable at the origin, which typically causes oscillation of the gradient of the error function during the training. A key point of this paper is to modify the usual regularization term by smoothing it at the origin. This approach offers the following three advantages: First, it removes the oscillation of the gradient value. Secondly, it gives better pruning, namely the final weights to be removed are smaller than those produced through the usual regularization. Thirdly, it makes it possible to prove the convergence of the training. Supporting numerical examples are also provided.