Referring Phenomena in a Multimedia Context and their Computational Treatment

A growing number of research projects has started to investigate the use of referring expressions in multimedia systems. On the one hand, the use of multiple media has led to new problems, such as a proper treatment of cross-media references. On the other hand, it has turned out that many concepts already known from natural language processing, such as cohesion, take on an extended meaning in multimedia discourse. For example, a proper treatment of referring expressions in a multimedia discourse requires an explicit representation of the syntax and semantics of the graphical discourse. As theories of NL reference become more sophisticated, it is quite natural to investigate whether these theories also encompass other media, such as graphics and pointing gestures. Several research projects have already started to transfer theories to the broader context of multimedia discourse. Examples of models that have been used for multimedia applications are Grosz and Sidner's theory of discourse structure, the centering model developed by Joshi and colleagues and Appelt's and Kronfeld's model of referring. However, there are researchers who doubt that linguistic phenomena, such as anaphora, also exist in multimedia dialogue. The reason they give is that there are no graphical devices for distinguishing between a reference-specifying and a predication-specifying part since objects and their properties are hardly separable once depicted.