Transfer Learning for Spectral Image Reconstruction from RGB Images

Spectral image reconstruction from RGB images has emerged as a hot topic in the computer vision community due to easy-access and low-cost acquisition compared with traditional spectral imaging acquisition methods. With the growth of the available spectral data-sets, this reconstruction problem has been effectively addressed using deep convolutional neural networks (CNN). The goal is to learn a non-linear mapping from 3-RGB bands to L spectral bands. However, these methods demand many spectral images to train the CNN to obtain a good recovery. In contrast, the proposed process consists of a pre-training step where the weights of a convolutional neural network fit with a large number of RGB image data sets available without its corresponding ground-truth spectral images, taking into account the RGB spectral response of the camera which is modeled as a non-trainable layer. Then, some layers of this pre-trained network are frozen to retrain it with the available spectral data-set to generate a spectral image with L bands. The proposed training scheme can be used with any pre-existing deep network that maps RGB to spectral images, and it is here evaluated with a “U-net” architecture. The RGB sensing is based on the Bayer filter pattern from a Nikon D90 DSLR camera. The simulated and experimental data demonstrate the effectiveness of the proposed approach compared to training without transfer learning, showing a gain of up to 4 dB, with less spectral data.