Differentiated Assessment of Mathematical Competence with Multidimensional Adaptive Testing

(ProQuest: ... denotes formulae omitted.)The improvement of educational processes is a primary aim of societies, governments, and research. In a variety of large-scale assessments of student achievement (LSAs), educational attainments are assessed and associated with different individual or contextual characteristics such as socio-economic background or the structural features of school systems. The results of LSAs provide valuable information for governments, and such information can be used to evaluate the extent to which educational goals are being achieved to monitor student achievement over time and to facilitate the making of decisions on reform measures. Well-known international LSAs include the Programme for International Student Assessment (PISA), the Trends in International Mathematics and Science Study (TIMSS), and the Progress in International Reading Literacy Study (PIRLS). These assessments are based on elaborate theoretical frameworks. In these frameworks, the structure of the competence constructs of interest is typically specified in terms of complex theoretical models. The development of these models was motivated by an increasing interest in measuring competencies with a strong reference to real-life tasks in specific contexts instead of measuring general cognitive abilities (e.g., McClelland, 1973; Shavelson, 2013). Typical examples of such complex competence constructs are the constructs of student literacy in mathematics, reading, and science, which are the focuses of the PISA (e.g., the Organisation for Economic Co-operation and Development [OECD], 2013a). For instance, the definition of mathematical literacy in the PISA is based on a theoretical model that differentiates between multiple cognitive processes, mathematical content areas, and situations in which mathematical literacy can be applied.To adequately operationalize such complex definitions and thus the corresponding theoretical underpinnings, large item pools are required. The examinees' response behavior to the items that are included in such a pool is usually modeled by item response theory (IRT) models (e.g., de Ayala, 2009; Embretson & Reise, 2000). Generally, multidimensional IRT (MIRT) models can be used to model the complex structures of competence models and, hence, they make it possible to link task performance with the multiple aspects of the competence construct that are considered to affect performance (e.g., Ackerman, Gierl, & Walker, 2003; Adams, Wilson, & Wang, 1997; Hartig & Hohler, 2008; Walker & Beretvas, 2003). However, the potential of MIRT models is restricted by the resources that are available for testing. Higher demands in terms of precise measures for multiple dimensions imply higher testing effort, e.g., considerably longer testing times and/or a higher number of students being assessed. To limit testing effort and the associated costs, as well as to ensure the cooperation with the institutions that are involved, highly efficient testing procedures are needed. Multidimensional adaptive testing (MAT, e.g., Frey & Seitz, 2009; Segall, 2010) is a promising procedure for the measurement of complex competence constructs on a very high level of efficiency, while consuming considerably fewer testing resources than conventional testing with fixed test forms. MAT is a method that is used to simultaneously measure the standing of an examinee on several dimensions. In MAT, item selection is based on the previous responses of the examinee. Tailoring the item selection to the individual competence level makes it possible to substantially increase measurement efficiency. This can be used to either reduce test length or increase measurement precision (e.g., Frey & Seitz, 2011). Thus, MAT provides the necessary prerequisites to measure higher-dimensional competence constructs with a reasonable amount of testing effort and opens up the possibility of reporting more differentiated results compared to conventional methods. …