Cross-Covariance Matrix: Time-Shifted Correlations for 3D Action Recognition

Abstract In this paper, we introduce cross-covariance to form Symmetric Positive Definite(SPD) matrix-based representations for 3D action recognition. The cross-covariance is generated by the correlational statistics between the time-shifted poses, which brings more informative features and time-order structure to improve the discriminative power on actions. Due to the special manifold structure, the cross-covariance is a totally different SPD matrix representation compared to covariance. In order to utilize such cross-covariance statistics while holding the covariance statistics, we devise a fashion of expression to mix them together as a larger SPD matrix benefiting from the Riemannian geometry. Further, we extend the symmetric cross-covariance into the non-symmetric versions, where the time-order information can be embodied in the related matrix structures. Thus, the derived SPD representations are able to improve the discriminative power of actions. In addition, our representations are easy to implement and to be extended by other prevalent techniques, such as kernel methods, metric learning, etc. Experimental results show that our proposed method efficiently promotes the performance of 3D action recognition tasks. In some public databases, our method outperforms the state-of-the-art methods which are based on other feature representations, kernel matrices, temporal hierarchical features, etc.