NEAT: Non-linearity Aware Training for Accurate, Energy-Efficient and Robust Implementation of Neural Networks on 1T-1R Crossbars

In this era of IoT, energy-efficient and adversarially secure implementation of Deep Neural Networks (DNNs) on hardware has become imperative. Memristive crossbars have emerged as an energy-efficient component of deep learning hardware accelerators due to their compact and efficient Matrix Vector Multiplication (MVM) implementation. However, they suffer from non-idealities (such as, interconnect parasitics, device variations, sneak paths) introduced by their circuit topology that degrades computational accuracy. A 1T-1R synapse, adding a transistor (1T) in series with the memristive synapse (1R), has been proposed to mitigate sneak paths in a crossbar. However, we observe that the non-linear characteristics of the transistor affect the overall conductance of the 1T-1R cell which in turn affects the MVM operation. This 1T-1R non-linearity arising from the input voltage-dependent non-linearity is not only difficult to model or formulate, but also causes a drastic performance degradation of DNNs when mapped to such crossbars. In this paper, we first analyse the non-linearity in ideal 1T-1R crossbars (excluding non-idealities such as device variations and interconnect parasitics) and propose a novel Non-linearity Aware Training (NEAT) method to address the non-linearities. Specifically, we first identify the range of network weights, which can be mapped into the 1T-1R cell within the linear operating region of the transistor. After that, we regularize the weights of neural networks to exist within the linear operating range by using an iterative training algorithm. Our iterative training significantly recovers the classification accuracy drop caused by the non-linearity. Moreover, we find that each layer has a different weight distribution and in turn requires different gate-voltage of transistor to guarantee linear operation. Based on this observation, we achieve energy-efficiency while preserving classification accuracy by applying heterogeneous gate-voltage control to the 1T-1R cells across different layers. Finally, we conduct various experiments on CIFAR10 and CIFAR100 benchmark datasets to demonstrate the effectiveness of our non-linearity aware training. Overall, NEAT yields ~20% energy gain with less than 1% accuracy loss (with homogeneous gate control) when mapping ResNet18 networks on 1T-1R crossbars. Thereafter, we integrate the 1T-1R crossbars with various non-idealities. We show that NEAT leads to more adversarially robust mappings of DNNs onto non-ideal 1T-1R crossbars than standard DNNs mapped directly onto 1R crossbars. In case of a VGG11 network on CIFAR100 dataset, we obtain 17% improvement in clean accuracy and 2-8% & 5-6% improvements in adversarial accuracies, respectively, for Fast Gradient Sign Method (FGSM) and Projected Gradient Descent (PGD) based adversarial attacks via NEAT on non-ideal 64x64 crossbars, in comparison to standard DNNs.