An Efficient Reconfigurable Framework for General Purpose CNN-RNN Models on FPGAs

Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) have made great progress in machine learning community. Combining CNN and RNN can accomplish more general and complex tasks. Many specially designed hardware accelerators on FPGA or ASIC have been proposed for CNN or RNN, yet few of them focus on CNN-RNN-based models for general purpose applications. In this paper, we propose a complete design framework for deploying general-purpose CNNRNN-based models on FPGAs. We use Deephi Aristotle and Descartes IPs to build an efficient and reconfigurable hardware system with the support of Deephi's toolchains and Xilinx SDSoC environment. We also design a CNN-RNN-based co-optimization method which can find the IP configuration to achieve the maximum throughput under the given FPGA resources and neural network models. Our implementation on the Xilinx ZU5EG FPGA achieves the throughput of 690.76GOPS and the energy efficiency of 86.34GOPS/W on LRCN network.