PAPER: Examining Sequence of Contextualized Items in Science

Student performance on science tests is often substantially influenced by the context of individual test items.  This study systematically described the sequence patterns used in science test-item contexts and obtained empirical evidence on how the sequence patterns of contextual information support or hinder student performance on science tests.  Contextual information includes supplemental information, such as a vignette or selected background facts that precede or follow a test item.  This study addressed three gaps in the literature on contextualized items by providing the following: a consistent theory-based framework for selecting test-item characteristics to be studied; new knowledge about how student performance is influenced by specific elements or features of the test items; and better understanding of subgroups of students and their performance in order to advance knowledge about how to achieve consistently equitable testing.  In particular, this study produced both theory and empirical evidence regarding how the sequence of contextual information can be attributed to differential performance between English language learners (ELL) and non-ELL groups.  Findings from the ELL and non-ELL comparisons lead to important improvements in how items are developed to ensure test fairness. The design and data analyses were framed by three research questions:  (1) How is providing the sequence of information presented in item contexts associated with student performance?  (2) Considering different levels of linguistic demands in various contexts, how is the contextual sequence linked to science performance of ELLs and non-ELLs?  (3) How do different dimensions of sequence of context influence how students perceive and respond to tasks?  This study included multiple facets of inquiry, including psychometrics, test-item creation, misconceptions, and cognitive diagnostic assessment.  It generated item-development guidelines based on dimensions of sequence of contextual information.  Items resulting from this process were designed to evoke students' stored knowledge relevant to the content and/or process skills being assessed.  The newly developed science items were field tested with a diverse population of students from middle schools and high schools in the U.S.  A range of psychometric and statistical procedures were applied to student test scores.