Video super-resolution: from QVGA to HD in real-time

In surveillance, reconnaissance and numerous other video applications, enhancing the resolution and quality enhances the usability of captured video. In many such applications, video is often acquired from low cost legacy sensors that offer low resolution due to modest optics and low-resolution arrays, providing imagery that may be grainy and missing important details - and still face transmission bottlenecks. Many post-processing techniques have been proposed to enhance the quality of the video and superresolution is one such technique. In this paper, we extend previous work on a real-time superresolution application implemented in ASIC/FPGA hardware. A gradient based technique is used to register the frames at the sub-pixel level. Once we get the high resolution grid, we use an improved regularization technique in which the image is iteratively modified by applying back-projection to get a sharp and undistorted image. The matlab/simulink proven algorithm was migrated to hardware, to achieve 320x240 -> 1280x960, at more than 38 fps, a stunning superresolution by 16X in total pixels. This significant advance beyond real-time is the main contribution of this paper. Additionally the algorithm is implemented in C to achieve real-time performance in software with optimization for Intel I7 processor. Fixed 32 bit processing structure is used to achieve easy migration across platforms. This gives us a fine balance between the quality and performance. The proposed system is robust and highly efficient. Superresolution greatly decreases camera jitter to deliver a smooth, stabilized, high quality video.