An Integrated CNN-LSTM Model for Bangla Lexical Sign Language Recognition

The development of Human-Computer Interaction (HCI) will not only empower the disabled with access to technology but also help to build tools for assisting the disable and their caregivers. Some disabled people are deaf, they use signs to communicate. Signs are emblematic forms which convey a specific meaning. These emblems construct the sign language, which is a non-verbal language used by the deaf community. This research attempts to build a Bangla Sign language recognition system that can recognize signs of both hands. Hands vary in shapes and sizes, as well as signs vary in orientations and motions. Accurate feature extraction is necessary for such systems. For such purposes, deep learning approaches can prove to be effective for the classification and feature engineering of images. In the beginning, an integrated CNN-LSTM model is proposed for building a Sign language Recognition System, a Bangla Sign Language (BSL) dataset consisting of Bangla lexical signs is considered. This dataset consists of 13,400 images comprising thirty-six classes of Bangla lexical signs. The model is trained using a CNN-LSTM model. This model produces a training accuracy of 90% and a testing accuracy of 88.5%. The proposed model is compared to CNN and other CNN state of the art models, namely VGG16, VGG9 and MobileNet. The CNN model and other CNN state undergo the problem of overfitting as their training accuracy is greater than the testing accuracy.