Multispectral periocular classification with multimodal compact multi-linear pooling

Feature-level fusion approaches for multispectral biometrics are mainly grouped into two categories: (1) concatenation and (2) element-wise multiplication. While concatenation of feature vectors has benefits in allowing all elements to interact, it is difficult to learn output classification. Differently, element-wise multiplication has the benefits in enabling multiplicative interaction, it is difficult to learn input embedding. In this paper, we propose a novel approach to combine the benefits of both categories based on a compact representation of two feature vectors’ outer product, which is called the multimodal compact multi-linear pooling technique. We first propose to expand the bilinear pooling technique for two inputs to a multi-linear technique to accommodate for multiple inputs (multiple inputs from multiple spectra are frequent in the multispectral biometric context). This fusion approach not only allows all elements to interact, enables multiplicative interaction, but also uses a small number of parameters and low computation complexity. Based on this fusion proposal, we subsequently propose a complete multispectral periocular recognition system. Employing Higher-Order-Spectra features with an elliptical sampling approach proposed by Algashaam et al., our proposed system achieves state-of-the-art performance in both our own and the IIIT Multispectral Periocular datasets. The proposed approach can also be extended to other biometric modalities.