I Got Your Point: An Investigation of Pointing Cues in a Spherical Fish Tank Virtual Reality Display

Pointing is a fundamental building block in human communication. While it is ubiquitous in our daily interactions within the real world, it is difficult to precisely interpret a virtual agent's pointing direction to the physical world, considering its complex and subtle gesture cues, such as the movements of the human hand and head. Fish Tank Virtual Reality (FTVR) display has the potential to provide accurate pointing cues as it creates a compelling 3D spatial effect by rendering perspective-corrected vision. In this paper, we conducted a study with pointing cues of three levels (Head-only, Hand-only, and Hand+Head) to evaluate how the head and hand gesture cues affect observers' performance in interpretation of where a virtual agent is pointing in a spherical FTVR display. The results showed that the hand gesture significantly helps people interpret the pointing both accurately and quickly for fine pointing (15°), with 19.4% higher accuracy and 1.42 seconds faster than the head cue. The combination of the head and hand yielded a small improvement on the accuracy (4.4%) with even slightly longer time (0.38 seconds) compared to the hand-only cue. However, for coarse pointing (30°), head cue appears to be sufficient with the accuracy of 90.2%. The result of this study provides guidelines on cues selection for designing pointing in the virtual environment.