An Examination of the Validity of the IHS Classification System for Migraine and Tension‐Type Headache in the College Student Population
15
Citation
0
Reference
10
Related Paper
Citation Trend
Abstract:
The validity of the International Headache Society (IHS) classification system for college‐aged students with headache was examined using cluster analysis. Undergraduate college student volunteers (N=369) underwent a structured diagnostic interview for headaches, and the sample was divided into two subsamples for purposes of replication. A hierarchical cluster analysis (Ward's method) of the headache characteristics reported by the first subsample suggested a statistically distinct three‐cluster solution, and the solution was replicated using the second subsample. It appeared that one cluster was tensionlike, while the other two were migrainelike. Nonhierarchical cluster analyses (K‐means) of the cases from each subsample revealed a similar pattern of a tensionlike and two migrainelike clusters. Identical three‐cluster solutions were found for the second subsample both by using cluster centers from the first subsample and by clustering the cases independently, suggesting that the cluster solution was not a random finding. The IHS classification system appears to lack adequate specificity and sensitivity for college‐aged students with headache who report migrainelike symptoms. Thus, the generalizability of research results using college‐aged students with headache to the adult population may be questionable.Keywords:
Cluster headache
External validity
Hierarchical clustering
External validity
Internal validity
Research Design
Cite
Citations (4)
External validity
Internal validity
Evidence-Based Medicine
Conceptual framework
Cite
Citations (11)
External validity
Internal validity
Cite
Citations (0)
The NLP community typically relies on performance of a model on a held-out test set to assess generalization. Performance drops observed in datasets outside of official test sets are generally attributed to "out-of-distribution" effects. Here, we explore the foundations of generalizability and study the factors that affect it, articulating lessons from clinical studies. In clinical research, generalizability is an act of reasoning that depends on (a) internal validity of experiments to ensure controlled measurement of cause and effect, and (b) external validity or transportability of the results to the wider population. We demonstrate how learning spurious correlations, such as the distance between entities in relation extraction tasks, can affect a model's internal validity and in turn adversely impact generalization. We, therefore, present the need to ensure internal validity when building machine learning models in NLP. Our recommendations also apply to generative large language models, as they are known to be sensitive to even minor semantic preserving alterations. We also propose adapting the idea of matching in randomized controlled trials and observational studies to NLP evaluation to measure causation.
External validity
Spurious relationship
Internal validity
Cite
Citations (0)
Introduction: Broadening eligibility criteria has been a focus to increase the generalizability of trial findings. Using upper-limb motor trials conducted early post-stroke as the illustrative domain, we sought to (1) investigate whether the published aim and conclusion statements adequately reflect the generalizability of findings and (2) explore internal validity and feasibility as constraints to achieving generalizability. Methods: We systematically applied a conceptual model of a trial sampling process to published literature from systematic review and prospective cross-sectional data. The eligibility criteria reported and used to exclude patients were classified by consensus as related to safety, internal validity, feasibility, or a combination thereof. Categorical data were reported as counts/proportions, and continuous data were reported as median (interquartile range (IQR)). Results: Thirty trials ( n = 1638 participants) were included in the published literature and 1013 patients in the prospective data set. Thirty-seven percent of trials did not describe their target population in the aim and conclusion, and 80% did not report all trial screening data. Eligibility criteria related to internal validity were the most common type reported and applied to exclude patients across both data sets. In the prospective data set, 70% of patients were excluded for more than one reason. Conclusion: Key information to support the generalizability of trial findings was insufficiently reported in published upper-limb motor research conducted early post-stroke. Broadening eligibility criteria alone is unlikely to sufficiently improve trial inclusivity due to internal validity constraints. Trials could achieve inclusivity through targeting multiple sub-populations, that in combination, produce clinically relevant results that are applicable to a broader population.
External validity
Stroke
Internal validity
Interquartile range
Cite
Citations (5)
Randomized trials play an important role in estimating the effect of a policy or social work program in a given population. While most trial designs benefit from strong internal validity, they often lack external validity, or generalizability, to the target population of interest. In other words, one can obtain an unbiased estimate of the study sample average treatment effect (SATE) from a randomized trial; however, this estimate may not equal the target population average treatment effect (TATE) if the study sample is not fully representative of the target population. This paper provides an overview of existing strategies to assess and improve upon the generalizability of randomized trials, both through statistical methods and study design, as well as recommendations on how to implement these ideas in social work research.
Internal validity
External validity
Randomized experiment
Sample (material)
Research Design
Cite
Citations (64)
Purpose: To examine the concepts of external validity and generalizability, and explore strategies to strengthen generalizability of research findings, because of increasing demands for knowledge utilization in an evidence‐based practice environment. Framework: The concepts of external validity and generalizability are examined, considering theoretical aspects of external validity and conflicting demands for internal validity in research designs. Methodological approaches for controlling threats to external validity and strategies to enhance external validity and generalizability of findings are discussed. Conclusions: Generalizability of findings is not assured even if internal validity of a research study is addressed effectively through design. Strict controls to ensure internal validity can compromise generalizability. Researchers can and should use a variety of strategies to address issues of external validity and enhance generalizability of findings. Enhanced external validity and assessment of generalizability of findings can facilitate more appropriate use of research findings.
External validity
Internal validity
Compromise
Incremental validity
Validity
Cite
Citations (227)
Clinical and epidemiologic investigations are paying increasing attention to the critical constructs of "representativeness" of study samples and "generalizability" of study results. This is a laudable trend and yet, these key concepts are often misconstrued and conflated, masking the central issues of internal and external validity. The authors define these issues and demonstrate how they are related to one another and to generalizability. Providing examples, they identify threats to validity from different forms of bias and confounding. They also lay out relevant practical issues in study design, from sample selection to assessment of exposures, in both clinic-based and population-based settings.
Cite
Citations (219)
A 10-year-old boy recently received a diagnosis of diabetes. His mother feels that he would benefit from learning about coping styles and behavior management techniques from other children who have diabetes. However, you are unaware of the effectiveness of this type of group intervention.A parent is concerned about her child developing asthma because he has had shortness of breath and wheezing at times. The child's older sister recently received an asthma diagnosis, based on similar symptoms. You are curious about the prevalence of asthma in siblings.Generalizability refers to the extent to which the results of a study apply to individuals and circumstances beyond those studied. (1) Commonly referred to as external validity, generalizability is the degree to which a given study's findings can be extrapolated to another population. Results of a study are considered generalizable if they have relevant characteristics of and implications for more individuals than those in the sample studied.Generalizability is a crucial aspect of both research design and interpretation of research in practice. In research design, the researcher must balance achieving strong internal validity with generalizability. Generalizability is one important criterion in interpreting study results and deciding whether study findings would apply to those in an individual's medical practice.Internal validity (the ability to make conclusions about relationships between variables, based on study design and methods) is closely tied to generalizability, but efforts to improve internal validity can pose a threat to generalizability. Both internal validity and generalizability are critical elements in research design but can be difficult to achieve simultaneously.Randomized, controlled trials generally have strong internal validity due to the nature of the treatment versus control group design. However, in an effort to make the two groups as similar as possible at baseline, this study design often reduces generalizability. In the case of the patient who has diabetes, a randomized, controlled trial may show that an intervention to improve coping and behavior related to diabetes management is effective. Generalizability may be reduced if the intervention was effective for only a specific group whose characteristics were very similar to those of a comparison group.Sample selection is an important component of both internal validity and generalizability. To achieve balance between them, the goal in sample selection should be to increase the heterogeneity of the sample without minimizing the internal validity of the study. One method of improving generalizability is to use methods such as random sampling. A random sample reduces any bias in the selection of the study sample, which is one of the major threats to external validity. Selection bias refers to an error or problem with the way in which participants for a study are selected.Because such sampling techniques often are not possible, it is imperative to consider a study's eligibility (or inclusion) criteria. Achieving generalizability to a wider population requires broad eligibility criteria. Examples of broad eligibility criteria include children of various ages, of diverse backgrounds, and from different patient care settings. However, this mix might not be feasible to achieve and reduces the precision of the study. Again, it is important to consider the balance of generalizability and internal validity. Broad eligibility criteria tend to improve generalizability; narrow eligibility criteria may improve the precision and feasibility of a study. (1)Another important threat to generalizability is the setting in which the research is conducted. When designing a study based solely in a hospital setting, there are implied limits in the generalizability of the study's findings because of the uniqueness of the type of setting in terms of treatment and practices. Such a limitation could be addressed by conducting research in multiple or variable settings, including clinics and private practices. (2)When reading the results of studies published in the medical literature, it is important to consider the study's generalizability. In interpreting study results, the clinician should ask questions such as: How is the sample drawn? What are the eligibility criteria? How are the setting and sample of the study related to my own setting and population?In your investigation of whether coping and behavior modification group interventions are effective in patients who have diabetes, you discover one study that indicates that a 10-week group session improved patients’ diabetes management skills significantly. However, on closer examination, you determine that the study eligibility criteria limited the sample to teenagers 16 years of age and older, who had family members who had diabetes and who lived in a predominantly rural community with limited access to health-care resources. This could raise concerns about the generalizability of this study's results to your own patient, who is 10 years old, has no family member who has diabetes, and lives in a relatively wealthy suburban neighborhood.In looking at the research available to determine the prevalence of asthma in siblings, you find a study indicating that 40% of children who received the diagnosis had a sibling who had the same diagnosis. You feel that rate to be considerably higher than what you would expect based on your years of experience as a practitioner. In looking at the sampling techniques, you determine that the authors used a convenience sample based on 300 children who had at least one sibling presenting at a clinic in one single month. You might have concerns about the generalizability of this study because the sample used in the study does not reflect the larger population, including the population of your own patients.
Internal validity
External validity
Research Design
Cite
Citations (0)
This chapter discusses strategies researchers use to address threats to the external validity (i.e. generalizability) of research results. It describes threats to external validity in terms of populations, times, and situations. Within these factors, it considers specific threats to external validity, such as: (1) interactions among different treatments or conditions; (2) interactions between treatments and methods of data collection; (3) interactions of treatments with selection; (4) interactions of situation with treatment; and (5) interactions of history with treatment. Each of these factors can limit the generalizability of research results and, therefore, the validity of claims based on those results.
External validity
Internal validity
Validity
Incremental validity
Cite
Citations (0)