SCOTI: Science Captioning of Terrain Images for data prioritization and local image search

Abstract Planetary exploration is full of challenges. Data bandwidth is very limited between planetary rovers and ground-based data system. What’s worse, even though NASA has accumulated over 34 million images from various missions, it requires significant effort and is hardly possible for any scientist to go through all of them. In order to improve the degree of automation and the efficiency of these processes, we propose a system leveraging machine learning for planetary rovers to actively look for scientifically interesting and valuable features according to text instructions from scientists and prioritize the images captured onboard with those features for downlink. Such an image prioritization mechanism can also be naturally applied to content-based image search through text description in any local planetary image data server, allowing scientists to search for images with desired features without going through them one by one. Besides theoretical and engineering details of our proposed approach, we also present both quantitative and qualitative evaluation of the system along with some concrete examples.