Toward Using Machine Learning-Based Motion Gesture for 3D Text Input

In this paper, we discuss our implementation of a gesture-based 3-dimensional typing system within virtual reality. Rather than the conventional point-and-click keyboard commonly found in immersive technologies, we explore using unique gestures with the controller to enter a specific key. To map these gestures and movements to their respective key, we utilize machine learning techniques to avoid naive hard-coded implementations. The result of the trained model is a text input system that adapts to the user’s gestures, rather than forcing the user to conform to the system’s definition of a gesture. Our goal is to work toward a viable alternative to standard virtual reality keyboards and typing systems.