Single-Skeleton and Dual-Skeleton Hypergraph Convolution Neural Networks for Skeleton-Based Action Recognition

In the last several years, the graph convolutional networks (GCNs) have shown exceptional ability on skeleton-based action recognition. Currently used mainstream methods often include identifying the movements of a single skeleton and then fusing the features. But in this way, it will lose the interactive information of two skeletons. Moreover, since there are some interactions between people (such as handshake, high-five, hug, etc.), the loss will reduce the accuracy of skeleton-based action recognition. To address this issue, we propose a two-stream approach (SD-HGCN). On the basis of single-skeleton stream (S-HGCN), a dual-skeleton stream (D-HGCN) is added to recognizing actions with interactive information between skeletons. The model mainly includes a multi-branch inputs adaptive fusion module (MBAFM) and a skeleton perception module (SPM). MBAFM can make the input features more distinguishable through two GCNs and an attention module. SPM may identify relationships between skeletons and build topological knowledge about human skeletons, through adaptive learning of the hypergraph distribution matrix based on the semantic information in the skeleton sequence. The experimental results show that the D-HGCN consumes less time and has higher accuracy, which meets the real-time requirements. Our experiments demonstrate that our approach outperforms state-of-the-art methods on the NTU and Kinetics datasets.