Combining Speech and Handwriting Modalities for Mathematical Expression Recognition

In this paper, we open new perspectives for mathematical expression recognition by introducing an original bimodal system. Since handwritten mathematical expression recognition is a very challenging task prone to many ambiguities, we use speech as an additional modality to circumvent limitations that are inherent to the written form. A use case scenario corresponds to lectures given in classrooms where the teacher would write and read aloud any mathematical expressions to allow a better interpretation. In addition to state-of-the-art solutions for recognizing handwriting and speech, we introduce a multilayer architecture for the merger of modalities. Specifically, the Dempster–Shafer theory is used to process the information at the symbol level. This bimodal system is evaluated on real bimodal data, the HAMEX dataset. Large improvements are observed when speech and handwriting are combined when compared to the single handwriting modality.