Two-Point Step Size Gradient Method for Solving a Deep Learning Problem

This paper is devoted to an analysis of the rate of deep belief learning by multilayer neural networks. In designing neural networks, many authors have applied the mean field approximation (MFA) to establish that the state of neurons in hidden layers is active. To study the convergence of the MFAs, we transform the original problem to a minimization one. The object of investigation is the Barzilai–Borwein method for solving the obtained optimization problem. The essence of the two-point step size gradient method is its variable steplength. The appropriate steplength depends on the objective functional. Original steplengths are obtained and compared with the classical steplength. Sufficient conditions for existence and uniqueness of the weak solution are established. A rigorous proof of the convergence theorem is presented. Various tests with different kinds of weight matrices are discussed.