CNN-Based Real-time Hand and Fingertip Recognition for the Design of a Virtual Keyboard

As AR/VR technology develops, the demand for interface technologies such as hand gesture recognition is increasing. For control of human-computer interface, accurate hand gesture classification and accurate finger position estimations are required. The accuracy of skin-color-based hand gesture detection methods can be affected by the presence of complex backgrounds, the intensity of light, and by the skin color backgrounds. In this paper, we propose a control method of real-time virtual keyboard that functions properly regardless of background changes through CNN-based hand gesture detection and fingertip detection. As a result of applying the proposed algorithm, the RMSE (Root mean square error) was reduced by approximately 82%. Additionally, the speed of the proposed algorithm is close to 38 fps, which is sufficient for real-time virtual keyboard applications.