Quality Assessments of HMO Diagnosis Databases Used to Monitor Childhood Vaccine Safety

Summary

Objective: To assess the quality of automated diagnoses extracted from medical care databases by the Vaccine Safety Datalink (VSD) study.

Methods: Two methods are used to assess quality of VSD diagnosis data. The first method compares common automated and abstracted diagnostic categories (“outcomes”) in 1-2% simple random samples of study populations. The second method estimates positive predictive values of automated diagnosis codes used to identify potential cases of rare conditions (e.g., acute ataxia) for inclusion in nested case-control medical record abstraction studies.

Results: There was good agreement (64-68%) between automated and abstracted outcomes in the 1-2% simple random samples at 3 of the 4 VSD sites and poor agreement (44%) at 1 site. Overall at 3 sites, 56% of children with automated cerebella ataxia codes (ICD-9 = 334) and 22% with “lack of coordination” codes (ICD-9 = 781.3) met objective clinical criteria for acute ataxia.

Conclusions: The misclassification error rates for automated screening outcomes substantially reduce the power of screening analyses and limit usefulness of screening analyses to moderate to strong vaccine-outcome associations. Medical record verification of outcomes is needed for definitive assessments.

Keywords

Databases - health maintenance organizations - International Classification of Diseases (ICD-9) - quality control - vaccines

• References

• 1 Chen RT, Glasser RP, Rhodes PH. et al. The Vaccine Safety Datalink Project: a new tool for improving vaccine safety monitoring in the United Sates. Pediatrics 1997; 99: 765-73.
• 2 Chen RT, DeStefano F, Davis RL. et al. The Vaccine Safety Datalink Project: immunization research in health maintenance organizations in the USA. WHO Bulletin 2000; 78: 186-94.
• 3 Fisher ES, Whaley FS, Krushat WH. et al. The accuracy of Medicare’s hospital claims data: progress has been made, but problems remain. Am J Public Health 1992; 82: 243-8.
• 4 Office of Statewide Health Planning and Development. Report of results from the OSHPD re-abstracting project. Sacramento, CA: Office of Statewide Health Planning and Development; 1990.
• 5 Baron JA, Weiderpass E. An introduction to epidemiological research with medical databases. Ann Epidemiol 2000; 10: 200-4.
• 6 Romano PS, Roos LL, Luft HS. et al. and the Ischemic Heart Disease Patient Outcomes Research Team A comparison of administrative versus clinical data: coronary artery bypass surgery as an example. J Clin Epidemiol 1994; 47 (03) 249-60.
• 7 Smith GS, Langlois JA, Buechner JS. Methodological issues in using hospital discharge data to determine the incidence of hospitalized injuries. Amer J of Epidemiol 1991; 134 (10) 1146-58.
• 8 Feinstein AR. Unsolved scientific problems in the nosology of clinical medicine. Arch Intern Med 1988; 148: 2269-74.
• 9 McMahon LF, Smits HL. Can Medicare prospective payment survive the ICD-9-CM Disease Classification System?. Ann of Int Med 1986; 104: 562-6.
• 10 Mullooly JP, Drew L, DeStefano F. et al. Quality of HMO vaccination databases used to monitor childhood vaccine safety. Am J Epidemiol 1999; 149: 186-94.
• 11 Mullooly JP. Misclassification model for persontime analysis of automated medical care databases. Am J Epidemiol 1996; 144: 782-92.
• 12 Farrington CP. Relative incidence estimated from case series for vaccine safety evaluation. Biometrics 1995; 51: 228-35.
• 13 Farrington CP, Nash J, Miller E. Case series analysis of adverse reactions to vaccines: A comparative evaluation. Am J Epidemiol 1996; 143: 1165-73.
• 14 Chin HL, Krall M. Implementation of a comprehensive computer-based patient record system in Kaiser Permanente’s northwest region. MD Comput 1997; 14: 41-5.
• 15 Selby JB. Linking automated databases for research in managed care settings. Ann Intern Med 1997; 127 (Pt 2) 719-24.