A validation and differential item functioning (DIF)study of an abbreviated dynamics concept inventory

Concept inventories (CIs) have become popular assessment tools in science, technology, engineering, and mathematicseducation. Some researchers use CI scores when looking at differences in conceptual understanding or learning gainsacross demographic groups, but very few CIs have been evaluated for measurement bias or other aspects that threaten thefair assessment of learners. The most common psychometric evaluation models are shaped primarily by the majoritydemographic group, so these models can hide biases in the assessment against minority groups. The purpose of this studywas to evaluate the extent to which the validity, reliability, and fairness evidence supports the use of the total score on a 12-item Abbreviated Dynamics Concept Inventory (aDCI) as a measure of a student’s overall conceptual understanding ofdynamics. Because of the strong relationship between the aDCI and the Force Concept Inventory, which has previouslybeen shown to include item-level gender biases, we examined threats to fair measurement across gender scores of the aDCI.We employed an argument-based validation approach which tested: (1) the fit of a single-factor latent structure for theaDCI scores via a confirmatory factor analysis (CFA), (2) the difficulty and discrimination of each item using item responsetheory, (3) the correlation between the aDCI scores and similar measures of conceptual understanding, and (4) thedifferential item functioning of the aDCI items across gender groups via a multiple-group CFA. We found that one itemhad face-level construct validity concerns and two others were slightly biased against women. Possible sources of genderbias included the question’s content and context. Our results suggest that the interpretation of a student’s total aDCI scoreshould consider the differential item functioning of two items across gender andthe construct-alignmentconcerns of a thirditem. This work highlights the importance and challenge of designing inclusive assessments and validating them with fairpsychometric models.