Self Organizing Markov Map for Speech and Gesture Recognition

Gesture and Speech based human Computer interaction is attractive attention across various areas such as pattern recognition, computer vision. Thus kind of research areas find many kind of application in Multimodal HCI, Robotics control, Sign language recognition. This paper presents head and hand Gesture as well as Speech recognition system for human computer interaction (HCI).This kind of vision based system can show the capability of computer, which understand and responding to the hand and head gesture also for Speech in form of sentence. This recognition system consists of two main modules namely 1.Gesture recognition 2.Speech recognition, Gesture recognition consists of various phases.i. image capturing, ii. Feature extraction of gesture iii.Gesture modeling (Direction, Position, generalized), 2.Speech recognition consists of various phases i. taking voice signals ii. Spectral coding iii. Unit matching (BMU) iv. Lexical decoding v.syntactic, semantic analysis.  Compared with many existing algorithms for gesture and speech recognition, SOM provides flexibility, robustness against noisy environment. The detection of gestures is based on discrete predestinated symbol sets, which are manually labeled during the training phase. The gesture-speech correlation is modelled by examining the co-occurring speech and gesture patterns. This correlation can be used to fuse gesture and speech modalities for edutainment applications (i.e. video games, 3-D animations) where natural gestures of talking avatars are animated from speech. A speech driven gesture animation example has been implemented for demonstration.