The Multi-site 2009 EVALITA Spoken Dialog System Evaluation

This document presents the coordination and the ev aluation procedures for the Spoken Dialogue System Task in EVALITA 2009. Three institutions participated into the competition, Uni versity of Trento, University of Naples and Loquendo. EVALITA participants were asked to develop a SDS application operating in the sales force domain, th ey were provided with a preliminary list of scenarios indicating system acc ounting modalities and a possible list of subtasks that should made possible . The three systems were hosted on a server at Trento, 19 volunteers called all of them. The calls have been recorded, transcribed and annotated. The evalu ation work, based on scripting run on the annotations, has been mainly f ocused on assessing performance at the dialogue, task, and concept leve ls. Detailed results indicating the systems performances are reported in the paper. This document presents the coordination and the evaluation proced ures for the Spoken Dialogue System Task in EVALITA 2009. Three institutions participated into the competition, University of Trento, University o f Naples and Loquendo SpA.