Skeleton-based Chinese sign language recognition and generation for bidirectional communication between deaf and hearing people

Abstract Chinese sign language (CSL) is one of the most widely used sign language systems in the world. As such, the automatic recognition and generation of CSL is a key technology enabling bidirectional communication between deaf and hearing people. Most previous studies have focused solely on sign language recognition (SLR), which only addresses communication in a single direction. As such, there is a need for sign language generation (SLG) to enable communication in the other direction (i.e., from hearing people to deaf people). To achieve a smoother exchange of ideas between these two groups, we propose a skeleton-based CSL recognition and generation framework based on a recurrent neural network (RNN), to support bidirectional CSL communication. This process can also be extended to other sequence-to-sequence information interactions. The core of the proposed framework is a two-level probability generative model. Compared with previous techniques, this approach offers a more flexible approximate posterior distribution, which can produce skeletal sequences of varying styles that are recognizable to humans. In addition, the proposed generation method compensated for a lack of training data. A series of experiments in bidirectional communication were conducted on the large 500 CSL dataset. The proposed algorithm achieved high recognition accuracy for both real and synthetic data, with a reduced runtime. Furthermore, the generated data improved the performance of the discriminator. These results suggest the proposed bidirectional communication framework and generation algorithm to be an effective new approach to CSL recognition.