Isolated Video-Based Arabic Sign Language Recognition Using Convolutional and Recursive Neural Networks

For natural and meaningful communication between the deaf community and the hearing population, sign language is very important. Most of the Arab sign recognition studies have focused on the identification of the sign action based on the descriptor of the feature. However, the limitation of this traditional method is the need to choose which features are important in each particular sequence. To address this issue, we propose a novel approach based on a deep learning architecture to classify video sequences of Arabic sign language, especially Moroccan sign language. Two methods of classification are applied, namely 2D convolutional recurring neural network (2DCRNN) and 3D convolutional neural network (3DCNN). Concerning the first method, a 2DCRNN model is used to extract features with a recurring network pattern to detect the relationship between frames. The second method uses a 3DCNN model learning the spatiotemporal features out of small patches. After 2DCRNN and the 3DCNN models extracted feature, the video data are classified into various classes, using a fully connected network. The proposed approach is trained over a collection of 224 videos of five individuals performing 56 different signs. The results achieved through the fourfold cross-validation technique demonstrate the performance of the proposed approach in terms of recall, F1 score, and AUROC, with the level accuracy of 92% for 2DCRNN and 99% for 3DCNN.