Generalizability theory in psychotherapy research: The impact of multiple sources of variance on the dependability of psychotherapy process ratings

Abstract A central task of psychotherapy process measurement development is the assessment of reliability and validity. The convention of reporting intraclass correlations (ICCs) for coding procedures assumes that variance in scores can be adequately explained by differences between individuals and error resulting from differences in coders. Given the complex coding procedures that are common in psychotherapy process studies (multiple sessions may be rated by multiple coders on one or more multi-item scales), an ICC may fail to account for all of the relevant sources of variability in obtained scores. If process studies are to provide useful information about the mutative agents in psychotherapy, assessment procedures must be developed that dependably assess the constructs of interest. Generalizability theory provides a framework within which multiple sources of error can be simultaneously evaluated, thus improving the accuracy of reliability estimates and providing critical information for modification a...