Recurrent Multi-column 3D Convolution Network for Video Super-Resolution

Super-resolution (SR) aims to recover high resolution (HR) content from low resolution (LR) content, which is a hotspot in image and video processing. Since there is strong correlation between video sequence, how to effectively exploit spatio-temporal information among serial frames is significant for video super-resolution (VSR). In this paper, we propose a recurrent multi-column 3D convolution network to fuse inter-frame and intra-frame information for VSR. Specifically, we first introduce motion compensation by optical flow to align the reference frame and supporting frames. Different from the previous methods, our supporting frames is originated from the previous reconstructed SR frame, instead of front and back low resolution frames of reference frame. This is more conducive to generating visual consistent video sequence. Then, 3D multi-column block (3DMB), which is composed with different separable 3D convolutions by reducing the parameters without performance penalties, is performed to fuse spatio-temporal features and recover missing details of reference frame. Finally, consecutive HR video sequence is obtained by reconstruction module. Comparative experiments are carried out to demonstrate our proposed method outperforms other advanced methods.