Learning Convolutional Neural Networks with Deep Part Embeddings

We propose a novel concept of Deep Part Embeddings (DPEs), which can be used to learn new Convolutional Neural Networks (CNNs) for different classes. We define DPE as a neuron of a trained CNN along with its network of filter activations that is interpretable as a part of a class that the neuron contributes to. Given a new class $\mathcal{C}$, we explore the idea of combining different DPEs that intuitively constitute $\mathcal{C}$, from trained CNNs (not on $\mathcal{C}$), into a network that learns the class $\mathcal{C}$ with few training samples. An important application of our proposed framework is the ability to modify a CNN trained on n classes to learn a new class with limited training data without significantly affecting its performance on the n classes. We visually illustrate the different network architectures and extensively evaluate their performance against the baselines.