On perceptual grouping of reference expressions

Understanding of reference expressions is critical in human-agent interaction through natural language. This paper extends and generalizes an existing formal model of reference expressions involving perceptual grouping, to take account of the wide variety of human-produced reference expressions. The previous model can only represent a linear process of narrowing down of the referent; e.g.: "the right ball of the three balls in the left". While psy- chological experiments indicate a general tendency towards humans using expressions of this type, it is far from the only or even always the most natural one. In particular, our model can account for non-linear expressions (relations of sets of objects, other than exclusively the subsumption-relation) that are used by humans, like: "the ball in front of the desk and behind the chair". We conducted an experiment to collect data of reference expressions in English in order to evaluate the proposed model. Our proposed model yielded an increase in both coverage and accuracy of referent identification. An algorithm is then outlined for the application of this model to other languages.