Deterministic Convergence Analysis via Smoothing Group Lasso Regularization and Adaptive Momentum for Sigma-Pi-Sigma Neural Network

Abstract In this paper, we propose a sparse and accelerated method for Sigma-Pi-Sigma neural network training based on smoothing group lasso regularization and adaptive momentum. It is shown that group sparsity can more efficiently sparsity the network structure at the group level, and the adaptive momentum term can speed up the learning convergence during the iteration process. Also, another important contribution lies in the analysis of theoretical results. However, the group lasso regularization is not differentiable at the origin. This leads to oscillations observed in numerical experiments and poses a challenge to theoretical analysis. So, we overcome those problems by smoothing techniques. Under suitable assumptions, we strictly proved the monotonicity, and weak and strong convergence theorems of the new algorithm. Finally, the numerical experiments are presented to support our theoretical findings.