Efficacy in Automated Language Translators

Abstract : This paper suggests an improved measure for evaluating the usefulness of automated machine language translators. With the Global War on Terror (GWOT), the Army has increasing interest and need for accurate language translation more than ever. Today, there are approximately 20,000 linguists with language training in either the Active Duty or Reserve components of the U.S. Army. Coalition operations and U.S. presence in Iraq, Kuwait, and other areas in the Middle East require Arabic translation. Unfortunately, the Army has never been able to maintain the number of linguists it needs, particularly in the hard-to-fill, low-density languages. Previous evaluations of machine translations usually rely on word error rate. Machine translation systems should be rated not in terms of their word error rate but in terms of human comprehension and usefulness, which is some function of word translation, syntax translation, and semantic interpretation. This study introduces a new method of evaluating human comprehension in the context of machine translation using a language translation program known as the Forward Area Language Converter (FALCon). A study was conducted where participants received seven translated articles in a random order. For each of the seven articles, the participants received a set of corresponding comprehension questions. The goal of the questions was to gear the reader toward intelligence gathering and to see if he could grasp main concepts and details. The results of this study suggest that word error rate is not an effective measure of the usefulness of a machine language translator. Comprehension tests perform better at evaluating a human's understanding of a translated document. This study further indicates strengths and weaknesses in each translator.