Search-and-Train: Two-Stage Model Compression and Acceleration

Convolutional neural networks have achieved great success in many fields. However, the practical application of convolutional neural networks is hindered due to the high consumption of memory and computational. In this paper, we propose a two-stage method for model compression and acceleration. More specifically, the training process mainly includes the search stage and train stage, the approach is abbreviated as ST. In the search stage, we first search and remove the unnecessary parts of a large pre-trained network (named supernet) by certain evaluation criteria to get a pruned network. Then the weights in the pruned network are initialized to get a small network (called a subnet). During the training stage, the supernet is untrainable, and the subnet will be trained under the supervision of the supernet. The knowledge extracted from the supernet will be transmitted to the subnet, then the subnet will be able to learn from the dataset and the knowledge at the same time. We have proved the effectiveness of our method through implement extensive experiments on several state-of-the-art CNN models (including VGGNet, ResNet, and DenseNet). The subnet only with 1/10 parameters and 1/2 calculations achieves more significant performance than the supernet.