Unsupervised Face Manipulation via Hallucination

This paper addresses the problem of manipulating a face image in terms of changing its pose. To achieve this, we propose a new method that can be trained under the very general “unpaired” setting. To this end, we firstly propose to model the general appearance, layout and background of the input image using a low-resolution version of it which is progressively passed through a hallucination network to generate features at higher resolutions. We show that such a formulation is significantly simpler than previous approaches for appearance modelling based on autoencoders. Secondly, we propose a fully learnable and spatially-aware appearance transfer module which can cope with misalignment between the input source image and the target pose and can effectively combine the features from the hallucination network with the features produced by our generator. Thirdly, we introduce an identity preserving method that is trained in an unsupervised way, by using an auxiliary feature extractor and a contrastive loss between the real and generated images. We compare our method against the state-of-the-art reporting significant improvements both quantitatively, in terms of FID and IS, and qualitatively.