A Protocol to Evaluate Virtual Keyboards for Indian Languages

We were asked to evaluate input mechanisms for touch-screen devices with the objective of standardising one of them for each of 14 major languages of India. For this purpose, we propose a protocol that consists of a 45-minute long training session, a 20-word first-time usability test, and a longitudinal test consisting of about 30 sessions, each of which required the user to type about 10 phrases 4 to 6 words long (a total of 300 phrases). The evaluation should be done with school children from standards 4th to 7th. The course of the evaluation may last 2-4 weeks for each user. To help follow the protocol over this long period and to collate the data, we offer a tool. Currently, we provide test corpora for Assamese, Bengali, Gujarati, Hindi, Kannada, Marathi, Odia, Tamil, Telugu, and Urdu. We have ensured that each corpus represents a mix of informal communication between people, popular phrases from films, songs, poetry and public discourse, and formal texts from school books and literature. We have tagged each phrase according to typing difficulty, phrase length, and memorability and age appropriateness. We evaluated the protocol through pilot tests with 206 users in Marathi, Gujarati, Hindi, Bengali, Odia, Assamese and Tamil. In this paper, we present the original protocol, the detailed findings from the Marathi pilots, and the proposed modifications to the protocol.