Automatic Transcription for the Hard of Hearing: Comprehension Improvement by Introducing Local Confidence Measures

In this paper we present the use of confidence measures to improve the comprehension of automatic transcription by the hard of hearing. The framework consists in live shows or live streams automatically transcribed by a large vocabulary speech recognition system. We have defined local confidence measures that can be estimated as soon as possible without having to wait for the recognition process to be completed. These measures have achieved results close to a reference of post-processed measure computed on the whole signal and currently known to be the most accurate measure. We then conducted an experiment to test the contribution of our confidence measure in improving the comprehension of an automatic transcription containing errors. We introduced several modalities to highlight words of low confidence in this transcription, which showed that these modalities can improve the comprehension of automatic transcription.