Speeding up neural network execution: an application to speech recognition

Many papers have addressed the problem of speeding up neural network execution, most of them trying to reduce network size by weight and neuron pruning, and others making use of special hardware. In this paper we propose a new, different method able to reduce the computational effort needed to calculate the output activity of a neural network. The suggested technique can be applied to a wide class of connectionist models for processing of slow varying signals (for example: vocal, radar, sonar and video signals). In addition, neither specialized hardware nor big quantities of additional memory are required. For each neuron of the network, the method suggests comparing its activation value at a certain moment with the corresponding activation value computed at the previous net forward computation: if no change occurred the neuron does not perform any computation, otherwise it propagates to the connected neurons the difference of its two activations multiplied by its outcoming weights. The proposal is verified in a speech recognition framework on two main tasks with two different neural network architectures. The results show a drastic reduction of the execution time on both the neural architectures and no significant changes in recognition quality.