FC-RCCN: Fully convolutional residual continuous CRF network for semantic segmentation

Abstract Enlarging the spatial resolution of features generated by fully convolutional networks (FCNs) can improve the performance of semantic segmentation. To achieve this goal, deeper network with deconvolutional structure can be applied. However, when the network architecture becomes more complex, the training efficiency may degrade. To address the joint optimization problem of improving spatial resolution through deeper networks and training deeper networks more effectively, we propose a Fully Convolutional Residual Continuous CRF Network (FC-RCCN) for semantic segmentation. FC-RCCN is composed of three subnetworks: a unary network, a pairwise network, and a superpixel based continuous conditional random filed (C-CRF) network. In order to generate full spatial resolution predictions with high-quality, a residual block based unary network with multi-scale features fusion is proposed. Even though the unary network is a deeper network, the whole framework can be trained effectively in an end-to-end way using the joint pixel-level and superpixel-level supervised learning strategy which is optimized by a pixel-level softmax cross entropy loss and a superpixel-level log-likelihood loss. Besides, C-CRF inference is fused with pixel-level prediction during the test procedure, which guarantees the method’s robustness to the superpxiel errors. In the experiments, we evaluatee the power of the three subnetworks and the learning strategy comprehensively. Experiments on three benchmark datasets demonstrate that the proposed FC-RCCN outperforms previous segmentation methods and obtains the state-of-the-art performance.