Training Low Bitwidth Model with Weight Normalization for Convolutional Neural Networks

Convolutional Neural Networks (CNNs) is now widely utilized in computer vision applications, including image classification, object detection and segmentation. However, high memory complexity and computation intensive have limited the deployment on low power embedded devices. We propose a method to train convolutional neural networks with low bitwidth by performing weight normalization. By normalization, the distribution of the weight can be narrowed, which enables the low bitwidth network to achieve a good trade-off between range and precision. Moreover, adding a scaling factor to the weight solves the problem of inadequate expressiveness at low bits, which further improves the performance of classification. The experiments on various datasets show that our method can achieve comparable prediction accuracy as that of full-precision models. To emphasize, the proposed scheme can quantize the network of AlexNet to 3-bit fixed point on ImageNet, and the accuracy of top-1 drop only by 1%.