HMM based speech-driven 3D tongue animation

We propose a speech-driven 3D tongue animation system. Firstly, the input speech is analyzed to obtain the phoneme sequence. Next, articulatory movements are predicted from the phoneme sequence using a hidden Markov model (HMM) based framework. The HMMs are trained beforehand using a corpus of human articulatory movements, which are recorded by three electromagnetic articulograph (EMA) sensors glued on the tongue tip, tongue body, and tongue dorsum of a speaker respectively. Finally, the predicted articulatory movements are used to control the deformations of a 3D tongue model. The tongue model is a triangular mesh with three key vertices. The three vertices are chosen so that their positions are in correspondence with the three EMA sensors mentioned above. Our tongue model can achieve various tongue shapes with volume preservation. Experiments show that the generated tongue animations are realistic and synchronize well with the input speech.