On-Line Assessment of Comprehension Processes

In this paper we describe a new version of a former paper-and-pencil standardized comprehension test called Test of Comprehension Processes (Vidal-Abarca, Gilabert, Martinez, & Selles, 2007). The new version has been adapted to a computer-based environment based on the moving window technique. It can be used to assess comprehension strategies of students from fifth to tenth grades (11 to 16 years old). Comprehension strategies are registered on-line using reading times and visits to relevant sections of the text during the question-answering process. Data show that the computerbased version draws similar results to those provided by the paper-and-pencil version. In addition, we identify the particular strategies deployed during the question-answering process by high, medium and low comprehenders. Keywords: comprehension test, assessment of reading processes, on-line reading measurement, moving window technique in comprehension En el presente articulo presentamos una nueva version de un test de comprension estandarizado de lapiz y papel llamado Test de Procesos de Comprension (Vidal-Abarca, Gilabert, Martinez, & Selles, 2007), el cual ha sido adaptado a un entorno electronico mediante la utilizacion una tecnica de ventana movil. Esta version electronica puede ser usada para diagnosticar estrategias de comprension de escolares entre 5? de Primaria y 4? de Educacion Secundaria Obligatoria (11 a 16 anos). Las estrategias de comprension se miden registrando tiempos de lectura y visitas a segmentos relevantes del texto de forma on-line durante el proceso de responder a preguntas del texto. Los resultados muestran en primer lugar que la version electronica es similar a la version papel y lapiz. En segundo lugar, se muestran diferentes estrategias durante el proceso de respuesta a preguntas del texto que son caracteristicas de estudiantes con estrategias de comprension de nivel alto, medio y bajo. Palabras clave: test de comprension, evaluacion de procesos de lectura, medida de variables on-line en lectura, tecnicas de ventana movil en comprension Current theoretical models describe compression as a process in which readers construct a mental representation of the text information. This process involves three groups of variables: characteristics of the readers, the task, and the text, which are organized around a set of processing strategies (Entin & Klare, 1985; Goldman, 1997; Kintsch & Kintsch, 2005; McNamara & Kintsch, 1996; Trites & McGroarty, 2005). The aim of these strategies is to obtain a better or more efficient outcome from the reading task, which requires an adjustment of the processes to the task requirements (e.g., Alderson, 2000; Goldman, 1997). Despite this process-oriented theoretical framework of comprehension, assessment tools are generally based on product variables, without considering the processes or strategies followed by readers to accomplish a task. These assessment tools have been useful to classify students in regard to their abilities, in a roughly reliable, valid and economic way (Magliano, Millis, Ozurur, & McNamara, 2007). However, in the last few years significant research efforts have been made to develop electronic procedures to assess comprehension processes using on-line variables (e.g., Magliano, Millis, The NIU R-SAT Development Team, Levinstein, & Boonthum, in press; McNamara, O'Reilly, Best, & Ozuru, 2006). As a result, tremendous advances have been made on the analysis of strategies and processes that play a role during text reading. The procedures used to capture these processes on-line have been think aloud protocols and the study of reading times. Reading times are registered through two techniques: the analysis of eye-movements and the moving-window method. These procedures are becoming more accessible because of the development of computer-based systems that can easily handle the processing demands that they require at a relatively low cost. …