Exploring the Relationship between Item Exposure Rate and Test Overlap Rate in Computerized Adaptive Testing.

This paper presents a derivation of an average between-test overlap index as a function of the item exposure index, for fixed-length computerized adaptive tests. This relationship is used to investigate the simultaneous control of item exposure at both the item and test levels. Implications for practice as well as future research are also discussed. Exploring the Relationship Between Item Exposure Rate and Test Overlap Rate in Computerized Adaptive Testing The popularity of computerized adaptive tests (CATs) has increased in recent years due to the significant progress of computer technology. Many conventional paper-and-pencil (PP Bergstrom, Lunz, & Gershon, 1992; Way Zara, & Leahy, 1996); and (b) conditional item selection (e.g., Sympson & Hetter, 1985; Davey & Parshall, 1995; Stocking & Lewis, 1995, 1998). Regardless of the item exposure control method used, item exposure rate and average item overlap are two indices commonly used to track item exposure in CATs (Way, 1998). Item exposure rate refers to the relative frequency with which an item is presented across all CAT administrations, that is, the proportion of all CATs in which an item is administered. Average item overlap is defined by Way (1998) as the proportion (or percentage) of items shared by pairs of exams, averaged across all possible pairwise comparisons. It is im portant to note that Mills and Stocking (1996) use the term item overlap in referring to ''the extent to which one item may cue the correct response to another item or the extent to which two items depend on the same specific knowledge" (p. 294). To avoid confusion, and to provide a more accurate and descriptive nomenclature, we introduce the following terminology and definitions: (a) For a pairwise comparison between two fixed-length CATs that have been administered, the between-test overlap is the proportion of items on one test that also appear on the other test (i.e., the proportion of shared items); and (b) the average betiveen-test overlap is the arithmetic mean of the between-test overlaps across all possible pairwise comparisons. Furthermore, we use the terms average between-test overlap and test overlap rate interchangeably. The average between-test overlap, as defined above, is equivalent to the average item overlap defined by Way (1998). By considering both the item exposure rate and the average between-test overlap, item exposure can be monitored at the individual item level as well as the test level. Despite the importance of both item exposure rate and test overlap rate in tracking item exposure control, few studies have investigated the effects of simultaneously controlling the magnitudes of these two indices. W hile most research to date has focused on item exposure control at the individual item level, Davey and Parshall (1995) proposed a conditioned item exposure control method designed to function at both the item and test levels. Although this method reduces the amount of test overlap and is more general than methods that function only at the individual item level, it fails to control the test overlap rate exactly, that is, it fails to ensure desired levels of test security. Research with more comprehensive methods of controlling the item exposure rate and the test overlap rate simultaneously may be useful. Based on the conceptual definitions of item exposure rate and average betweentest overlap, it is to be expected that these two indices are highly related. If the average between-test overlap could be expressed as a function of item exposure rates, then it would not be necessary to undertake time-consuming pairwise comparisons of CATs to determine all between-test overlaps. Rather, the average between-test overlap could be more simply computed once the item exposure rates were known. Such a simplification would be especially efficient when the number of CATs administered (hence the number of pairwise comparisons) is large. Furthermore, if the average between-test overlap could be expressed as a function of the item exposure rates, then the relationship between these two indices could be investigated directly and easily. This, in turn, may provide insights into CAT design and im plementation considerations relevant to the sim ultaneous control of item exposure rates and average between-test overlap. The purpose of this paper is to present an analytical derivation for the mathematical form of an average between-test overlap index as a function of the item exposure index, for fixed-length CATs. This algebraic relationship is used to investigate the simultaneous control of item exposure at both the item and test levels. Implications for practice as well as future research are also discussed.