Powerset fusion network for target classification in unattended ground sensors

This paper presents an end-to-end multimodal fusion approach called powerset fusion network (PFN), based on a convolutional neural network (CNN), which is used for acoustic-seismic moving target classification in unattended ground sensor (UGS) systems. We first introduce the powerset fusion to fuse different modalities sufficiently. Then fusing representations at all abstraction levels is explored by using the proposed fully hybrid framework. We also examine how to use multi-task learning (MTL) to obtain better representations, overcome missing modalities and save embedded storage space. Finally, we present a novel multimodal cost function to accelerate the learning procedure. The experimental results based on the field dataset show that the performance of PFN is superior to that of the popular deep learning fusion systems.