Range Camera for Simple behind Display Interaction

This paper presents a method to estimate hand positions behind a display, while the person’s body is in front of the display. This allows for direct and bare hand interaction with virtual objects in an AR setting. As a sensor, we use a time-of-flight range camera, which provides depth information per pixel, but is noisy. While hand tracking has previously been attempted by localizing body parts and fitting a body model into the data, we propose a simpler algorithm. We first remove outliers, and then model the hand as an oriented box whose position and orientation is determined by a PCA algorithm. The calculated hand model is then used in a physics engine based interaction with virtual content.