Nonlinear Kaczmarz algorithms and their convergence

Abstract This paper proposes a class of randomized Kaczmarz algorithms for obtaining isolated solutions of large-scale well-posed or overdetermined nonlinear systems of equations. This type of algorithm improves the classic Newton method. Each iteration only needs to calculate one row of the Jacobian instead of the entire matrix, which greatly reduces the amount of calculation and storage. Therefore, these algorithms are called matrix-free algorithms. According to the different probability selection patterns of choosing a row of the Jacobian matrix, the nonlinear Kaczmarz (NK) algorithm, the nonlinear randomized Kaczmarz (NRK) algorithm and the nonlinear uniformly randomized Kaczmarz (NURK) algorithm are proposed. In addition, the NURK algorithm is similar to the stochastic gradient descent (SGD) algorithm in nonlinear optimization problems. The only difference is the choice of step size. In the case of noise-free data, theoretical analysis and the results of numerical based on the classical tangential cone conditions show that the algorithms proposed in this paper are superior to the SGD algorithm in terms of iterations and calculation time.