Bowtie Networks: Generative Modeling for Joint Few-Shot Recognition and Novel-View Synthesis

In this paper, we propose a novel dual-task of joint few-shot recognition and novel-view synthesis: given only one or few images of a novel object from arbitrary views with only category annotation, we aim to simultaneously learn an object classifier and generate images of that type of object from new viewpoints. While there has been increasing interest in simultaneously addressing two or more tasks, existing work mainly focuses on multi-task learning of shareable feature representations. Here, we take a different perspective --- learning a shared generative model across the dual-task. To this end, we propose bowtie networks that jointly learn 3D geometric and semantic representations with a feedback loop. Experimental evaluation on challenging fine-grained recognition datasets demonstrates that our synthesized images are realistic from multiple viewpoints and significantly improve recognition performance as ways of data augmentation, especially in the low-data regime.