Coefficient Alpha: An Engineer's Interpretation of Test Reliability

Reliability is a fundamental concept of test construction. The most common measure of reliability, coefficient alpha, is frequently used without an understanding of its behavior. This article contributes to the understanding of test reliability by demonstrating that questions which lower reliability are inconsistent with the bulk of the test, being prone to test-taking tricks and guessing. These qualitative characteristics, obtained from focus groups, provide possible causes of lower reliability such as poorly written questions (e.g., the correct answer looks different from the incorrect answers), questions where students must guess (e.g., the topic is too advanced), and questions where recalling a definition is crucial. Quantitative findings confirm that questions lower reliability when students who answer correctly have lower overall scores than students who answer incorrectly. This phenomenon is quantified by the “gap” between these students' overall scores, which is shown to be highly correlated with other item metrics. An increasing number of concept inventory tests are being developed to assess student learning in engineering. Scores and student comments from the Statistics Concept Inventory are used to make these judgments.