Generalization error of deep neural networks: Role of classification margin and data structure

Understanding the generalization properties of deep learning models is critical for their successful usage in many applications, especially in the regimes where the number of training samples is limited. We study the generalization properties of deep neural networks (DNNs) via the Jacobian matrix of the network. Our analysis is general to arbitrary network structures, types of non-linearities and pooling operations. We show that bounding the spectral norm of the Jacobian matrix in the network reduces the generalization error. In addition, we tie this error to the invariance in the data and the network. Experiments on the MNIST and ImageNet datasets support these findings. This short paper summarizes our generalization error theorems for DNNs and for general invariant classifiers [1], [2].