Restricted Boltzmann Machines and Deep Belief Networks on Sunway Cluster

Deep learning models have showed great potential in classification and recognition over the last decade. Deep Belief Networks (DBNs) have been applied in visual, voice fields due to their great feature presentation capability. However, there are a vast number of time consuming calculations in the training of DBNs. Many researches have accelerated the training of DBNs with good speedups on CPU, GPU, FPGA, etc. At the same time, the latest published Sunway(SW) many-core processor has high computing performance and dedicated heterogeneous architecture. This paper provides a DBNs training system on SW cluster and verifies SW cluster's applicability of training DBNs. We firstly optimize the Restricted Boltzmann Machines and Deep Belief Networks on Sunway processor, then build a parallelism model with linear topology to train DBNs on multiple processors. The system is implemented on the TaihuLight supercomputer and evaluated by training a DBN with 2.8 million parameters with MNIST dataset. Experimental results show that our system achieves considerable speedups on Sunway processors as compared with CPUs.