Limitations of Standard Accessible Captioning of Sounds and Music for Deaf and Hard of Hearing People: An EEG Study

Captioning is the process of transcribing speech and acoustical information into text to help deaf and hard of hearing people accessing to the auditory track of audiovisual media. In addition to the verbal transcription, it includes information such as sound effects, speaker identification or music tagging. However, it just takes into account a limited spectrum of the whole acoustic information available in the soundtrack and, hence an important amount of emotional information is lost when attending just to the normative compliant captions. In this paper it is shown, by means of behavioral and EEG measurements, how emotional information related to sounds and music used by the creator in the audiovisual work is perceived differently by normal hearing group and hearing disabled group when applying standard captioning. Audio and captions activate similar processing areas respectively in each group, although not with the same intensity. Moreover, captions require higher activation of voluntary attentional circuits, as well as language related areas. Captions transcribing musical information increase attentional activity, instead of emotional processing.