Controllable digital restoration of ancient paintings using convolutional neural network and nearest neighbor

Abstract Ancient paintings are valuable culture legacy which can help archaeologists and culture researchers to study history and humanity. Most ancient artworks have damage problems, such as degradation, flaking and cracking. This work presents a novel controllable image inpainting framework with capability of incorporating suggestions from experts, which can help artists envisage how the ancient painting may have looked after a restoration. The framework leverages the content prediction power of deep convolutional neural network (CNN) and the nearest neighbor based pixel matching, where a deep CNN is designed to produce a coarse estimation of complete paintings by filling in missing regions and nearest neighbor based pixel matching is designed to map a mid-frequency estimation obtained from the deep CNN to high quality outputs in a controllable manner. In addition, we design a pixel descriptor using multi-scale neural features from different layers of a pre-trained deep network to capture different amounts of spatial context. Experimental results demonstrate that the proposed approach successfully predicts information in large missing regions and generates controllable high-frequency photo-realistic inpainting results.