Evaluation of clinical systems for treating tobacco use: A comparison of key informant interview and document examination

• Objective: To determine the level of agreement between reports from key informants and examination of clinic documents regarding the presence of tobacco treatment systems in primary care clinics. • Design: Clinic tours with interview of key informants and collection of clinic documents. • Setting and participants: 130 out of 350 clinics in 30 medical groups in the Blue Cross and Blue Shield of Minnesota primary care network were selected for potential site visits. 95 of these clinics agreed to participate. 84% of participating clinics were family medicine clinics. Key informants were either clinic managers (n = 64) or clinicians (n = 31). • Measurements: Informants reported whether the clinic used patient health questionnaires, chart stamps/labels, problem list/flow sheets, prompts/ reminders, progress notes, specialized treatment forms, patient educational materials, specialist/ community resource lists, or specific referral forms for the treatment of tobacco use. Documents were examined directly for any tobacco treatment content. • Results: Agreement was highest for the health questionnaire (kappa = 0.73) and chart stamp (kappa = 0.60). For other clinic documents, agreement was low (kappa range, ‐0.04 to 0.32). Key informants were more likely to make understatements (mean, 1.74 [SD = 1.33]/clinic) than overstatements (mean, 0.68 [SD = 0.79]/clinic) about the presence of treatment tools. The level of agreement was not associated with clinic or informant characteristics. • Conclusions: Until key informant interview methods are validated, direct examination of clinic documents is recommended to evaluate tobacco systems. Proper assessment of clinical systems is important in determining how best to proceed with efforts to improve delivery of tobacco treatment services.