Development of a Lip-Reading System Based on Mesuring Intraoral Shadow Area for a Patient with Respirator

Human communication in our normal lives is formed by verbal and non-verbal communication(for example, changing facial expression, body lauguage and speaking). If useful information needed to human communication does not exist, then not only communication becomes difficult but also life saving information may be unavailable. Patients with respirator(for example, injury to the throat)can not speak . While they cannot have voice communication with their doctors, nurses or families, treatment therapy and nursing decisions become difficult. In earlier times, instead of voice, patients communicated by means of writing. But this method required hard mental and physical work of patients. In addition, the longer the therapeutic period, the more speaking ability declines. This lengthens the period of rehabilitation. Medical treatment relies on voice communication between a patient, a medical doctor and a nurse. To support this communication, we developed a lip-reading system. We measured the changes of intraoral shadow area every1/30second during speaking a sentence and transformed them into a lip pattern on this system. We used20senteces in the experiment. Lip patterns were used for pattern recognition by a neural network. The rate of correct recognition for a normal subject and a patient was79.8%and55.6%, respectively. This system using only simple process provided good performance. We think, not only intraoral area, addtional information(for example, using respiratory flow from respirator and using shape of lip movement)can make the rate of correct recognition higher.