Nonlinear Temporal Correlation Based Network for Action Recognition

Action recognition, a trending topic in current research, is important for human behavior analysis, virtual reality, and human computer interaction. Recently, Some of the latest works have achieved impressive results in action recognition by decomposing 3D convolutions into temporal and spatial convolutions, respsctively. Modelling the temporal features is important for action recognition. In this paper, we reconsider the decomposing of convolution operations. In the previous temporal convolution operations, the temporal features are extracted by simple linear transformation, and the temporal relations among adjacent frames are not fully considered. Therefore, we propose a novel temporal structure, namely, Nonlinear Temporal Extractors, to replace the existing 1D temporal convolutions. On the one hand, this operation can extract temporal features by considering the relation along the time dimension. On the other hand, this enhances network’s representation ability by increasing the nonlinearity of the network. Finally, we perform experiments on the common action classification datasets, including UCF-101, HMDB-51, and mini-Kinetics-200. Experimental results show the effectiveness of our proposed structure.