Optimizing CNN Model Inference on CPUs.

The popularity of Convolutional Neural Network (CNN) models and the ubiquity of CPUs imply that better performance of CNN model inference on CPUs can deliver significant gain to a large number of users. To improve the performance of CNN inference on CPUs, current approaches that treat the model as a graph mostly rely on the use of high-performance libraries such as Intel MKL-DNN and some basic graph-level optimizations, which is restrictive and misses the opportunity to optimize the end-to-end inference pipeline as a whole. This paper presents a more comprehensive approach of CNN model inference on CPUs that employs a full-stack and systematic scheme of optimizations. The proposed solution optimizes the operations as templates, which enables further improvement of the performance via operation- and graph-level joint optimization. Experiments show that the proposed solution achieves up to 3.45x lower latency for CNN model inference than the current state-of-the-art implementations on various kinds of popular CPUs.