A compression-driven training framework for embedded deep neural networks

Deep neural networks (DNNs) are brain-inspired machine learning methods designed to recognize key patterns from raw data. State-of-the-art DNNs, even the simplest ones, require a huge amount of memory to store and retrieve data during computation. This prevents a practical mapping onto platforms with very limited resources, like those deployed in the end-nodes of the Internet-of-Things (IoT). The aim of this paper is to describe an efficient compression-driven training framework for embedded DNNs. The learning algorithm, which consists of a modified version of the Stochastic Gradient Descent, limits the original training space in a (-σ, 0, σ) ternarized subspace, with σ a hyperparameter learned layer-wise. Tested on medium- and large-scale DNNs, both Convolutional Neural Networks (CNNs) and Recurrent Neural Networks (RNNs), the obtained σ-ary DNNs enable an efficient use of sparse-matrix representations, hence high compression rate (up to 77× for CNN and 95× for RNNs), at the cost of a limited accuracy loss.