Effective and Efficient ROI-wise Visual Encoding Using an End-to-End CNN Regression Model and Selective Optimization

In neuroscience, visual encoding based on functional magnetic resonance imaging (fMRI) has been attracting much attention, especially with the recent development of deep learning. Visual encoding model is aimed at predicting subjects’ brain activity in response to presented image stimuli . Current visual encoding models firstly extract image features through a pre-trained convolutional neural network (CNN) model, and secondly learn to linearly map the extracted CNN features to each voxel. However, it is hard for the two-step manner of visual encoding model to guarantee the extracted features are linearly well-matched with fMRI voxels, which reduces final encoding performance. Analogizing the development of the computer vision domain, we introduced the end-to-end manner into the visual encoding domain. In this study, we designed an end-to-end convolution regression model (ETECRM) and selective optimization based on the region of interest (ROI)-wise manner to accomplish more effective and efficient visual encoding. The model can automatically learn to extract better-matched features for encoding performance based on the end-to-end manner. The model can directly encode an entire visual ROI containing enormous voxels for encoding efficiency based on the ROI-wise manner, where the selective optimization was used to avoid the interference of some ineffective voxels in the same ROI. Experimental results demonstrated that ETECRM obtained improved encoding performance and efficiency than previous two-step models. Comparative analysis implied that the end-to-end manner and large volume of fMRI data are potential for the visual encoding domain.