Test and evaluation of a spoken dialogue system

The development of spoken dialogue systems (SDSs) requires the definition of evaluation metrics which can assess the performance of these systems at different levels and compare various SDSs. The authors present a first test, made with naive users, on an integrated dialogue system for telephone speech access to a remote data base. They describe the system architecture as well as the goals of the test, its features, the methodology used during the evaluation, and the results obtained. The SDS is shown to be effective for providing the user with the required information. The presence of spontaneous speech phenomena is frequent with naive users. The dialogue helps the user to overcome the errors due to spontaneous speech. The use of isolated words for confirmation is useful, but partially limits the interaction friendliness. >