Incremental learning of object detection with output merging of compact expert detectors

Object detection has been widely applied and has good performance in recent years. However, it has a practical problem that we have to retrain the network by all images if we want to add a new task. It will require much memory to store images of all tasks and time to retrain the network. If we only use the data set of the new task to fine-tune the original model, it will perform well on the new task but catastrophic forgetting of previous tasks. This paper presents a method to address this issue and achieves incremental learning of object detection when only the current task data set is available each time. The core of our proposed solution is to train an expert detector for each task and merge the output of expert detectors by the merging module. Unlike previous works, our method has no limitation on the number of tasks added and will not cause any loss of mAP. We also use some tricks such as adding pre-classifiers, pruning networks and combining the convolutional layer and batch normalization layer to optimize the model. We conduct extensive experiments on the PASCAL VOC dataset and compare the results with the previous methods, along with a detailed empirical analysis of our approach.