Using acoustic paralinguistic information to assess the interaction quality in speech-based systems for elderly users

Abstract At present, there is a growing need for assisting living systems for the elderly. This type of technology must meet certain accessibility and usability requirements to be adopted successfully. Speech-based interfaces represent a very attractive option for improving the usability and acceptability of assistive technologies for elderly users because they provide a very easily accessible interface and also facilitate paralinguistic analysis of the user's voice to provide valuable information about their emotional and mental states while interacting with a computer system. This information can be used to improve the quality of the interaction between the assistive system and its user. In this study, we propose a method based on the automatic recognition of acoustic paralinguistic phenomena for estimating the quality of speech-based interactions. The proposed method automatically recognizes paralinguistic phenomena (e.g., shouting, hyper-articulation, and hesitation) during the interaction between the user and the system. The interaction is then characterized based on the occurrence of these phenomena. A model is trained using this characterization, which can estimate the quality of the interaction. Using this method, we obtained good recognition performance (F-measure around 70%) when classifying paralinguistic phenomena and there was also a strong correlation between the estimated quality of the interaction metrics and the quality of the interaction metrics reported by the users of an assistive system. The proposed method is useful for automatically estimating the quality of interactions perceived by users and it could support the construction of speech-based systems to adapt and personalize content as well as the style of the interaction with the user.