Cumulating the Intellectual Gold of Case Study Research

Evaluations of public administration research have permeated Public Administration Review for over a decade (Perry and Kraemer 1986; Stallings 1986; Stallings and Ferris 1988). Two themes pervade the literature criticizing research in the field: Critics contend that knowledge is not being cumulated (Adams and White 1994; White et al. 1996) and that research in general (dissertation research in particular) is of poor quality (Adams and White 1994; Cleary 1992; McCurdy and Cleary 1984; White 1986a; White et al. 1996). Criticisms that public administration research lacks quality remain largely unchallenged (see Bailey 1992 and Box 1992, for two rare exceptions). The use of case studies as a preferred research methodology in public administration lies behind both themes. Critics contend that case studies play a limited role in knowledge cumulation (Adams and White 1994) and that they fare poorly on indicators of quality (Adams and White 1994; McCurdy and Cleary 1984). Both claims can be traced to the perception that case studies are not generalizable, a perception we show to be terribly misconceived. We contend that public administration is well suited to case studies because they satisfy the recognized need for conditional findings and in-depth understanding of cause and effect relationships that other methodologies find difficult to achieve. Case studies are a key part of the solution, not part of the problem. We will show this by first defining case studies and by outlining a typology of case studies. Next, we will show how the knowledge-cumulation problem can be solved by using meta-analysis. Finally, we will explain how the criticism that case studies have poor quality is based on a problematic definition of quality and misguided criteria. The critics' test of quality generally involves counting the number of quality criteria that are satisfied in a study. The higher the count, the better the quality of the study. That is, the criteria are additive (see McCurdy and Cleary 1984). When a whole range of quality measures are applied to any single case study, it fares badly. Similarly, when a whole range of quality measures are applied to a study using a different methodology (survey research, for example), it is also likely to fare badly. The errors of this logic lie with the application of all quality criteria to each single, isolated study pertaining to a particular point in time (or several points in time), and with the assumption that all measures of quality are appropriate for all methodologies. For example, consider a quality indicator such as the subtlety and richness of findings embedded in a meaningful historical context. A study that is not a narrative case study would fare poorly on such a criterion. Measures of quality should make sense for the methodology being used, with an understanding that good-quality studies, using a variety of methodologies, will add up to produce knowledge over time and that knowledge cumulation is actually dependent on the use of multiple methodologies. Exclusion of any methodology undermines knowledge cumulation. The charge that case studies are of poor quality stems, in part, from the perception that they are not generalizable, and thus cannot be cumulated. Both of these criticisms are untrue. Case studies are often done in clusters. Even when not so designed, they can be considered retrospectively as a group for a broader and richer analysis. When case studies are considered cumulatively, a wide variety of special conditions can be recognized to ascertain whether the findings are generalizable. Meta-analysis aims to solve the related issues of cumulation and generalizability; the cumulation of case studies using meta-analysis allows specific tests of generalizability. Meta-analysis is a method of combining findings across research studies that has become increasingly popular in the social sciences (Hedges and Olkin 1985; Hunter et al. …