Combining Different Modalities in Classifying Phonological Categories

This paper concerns a new dataset we are collecting combining 3 modalities (EEG, video of the face, and audio) during imagined and vocalized phonemic and single-word prompts. We pre-process the EEG data, compute features for all 3 modalities, and perform binary classification of phonological categories using a combination of these modalities. For example, a deep-belief network obtains accuracies over 90 % on identifying consonants, which is significantly more accurate than two baseline support vector machines. These data may be used generally by the research community to learn multimodal relationships, and to develop silent-speech and brain-computer interfaces.