Deliberation on object-aware video style transfer network with long–short temporal and depth-consistent constraints

Video style transfer, as a natural extension of image style transfer, has recently gained much interests. It has much important applications on non-photorealistic rendering and computer games. However, existing image-based methods cannot be readily extended to videos because of temporal flickering and stylization inconsistency. Therefore, the main effort of this work is to propose an efficient salient object-aware and depth-consistent video style transfer algorithm. Specifically, DenseNet is carefully extended as feed-forward backbone network for better style transfer quality. Then, through utilizing salient object segmentation and depth estimation results, depth-consistent loss and object masked long–short temporal losses are deliberately proposed at training stage. The proposed losses can preserve stereoscopic sense without salient semantic distortion and consecutive stylized frame flickering. The proposed network has been compared with several state-of-the-art methods. The experimental results demonstrate that the proposed method is more superior on achieving real-time processing efficiency, nice rendering quality and coherent stylization at the same time. Related source codes have been released on GitHub.