Extracting Findings from Narrative Reports: Software Transferability and Sources of Physician Disagreement

Abstract:

While natural language processing systems are beginning to see clinical use, it remains unclear whether they can be disseminated effectively through the health care community. MedLEE, a general-purpose natural language processor developed for Columbia-Presbyterian Medical Center, was compared to physicians' ability to detect seven clinical conditions in 200 Brigham and Women's Hospital chest radiograph reports. Using the system on the new institution's reports resulted in a small but measurable drop in performance (it was distinguishable from physicians at p = 0.011). By making adjustments to the interpretation of the processor's coded output (without changing the processor itself), local behavior was better accommodated, and performance improved so that it was indistinguishable from the physicians. Pairs of physicians disagreed on at least one condition for 22% of reports; the source of disagreement appeared to be interpretation of findings, gauging likelihood and degree of disease, and coding errors.

Keywords

Natural Language Processing - Computerized Medical Record Systems - Thoracic Radiography; Evaluation

• REFERENCES

• 1 Spyns P. Natural language processing in medicine: an overview. Meth Inform Med 1996; 35: 285-301.
• 2 Friedman C, Alderson PO, Austin JHM, Cimino JJ, Johnson SB. A general natural-language text processor for clinical radiology. Amer Med Inform Ass 1994; 1: 161-74.
• 3 Sager N, Lyman M, Bucknall C, Nhan N, Tick LJ. Natural language processing and the representation of clinical data. Amer Med Inform Ass 1994; 1: 142-60.
• 4 Baud RH, Rassinoux AM, Scherrer JR. Natural language processing and semantical representation of medical texts. Meth Inform Med 1992; 31: 117-25.
• 5 Rassinoux AM, Michel PA, Juge C, Baud R, Scherrer JR. Natural language processing of medical texts within the HELIOS environraent. Comput Meth Progr Biomed 1994; 45 (Suppl) S79-S96.
• 6 Zweigenbaum P. MENELAS: an access system for medical records using natural language. Comput Meth Progr Biomed 1994; 45: 117-20.
• 7 Haug PJ, Ranum DL, Frederick PR. Computerized extraction of coded findings from free-text radiologic reports: work in progress. Radiology 1990; 174: 543-8.
• 8 Gundersen ML, Haug PJ, Pryor TA, et al. Development and evaluation of a computerized admission diagnoses encoding system. Comput Biomed Res 1996; 29: 351-72.
• 9 Zingmond D, Lenert LA. Monitoring freetext data using medical language processing. Comput Biomed Res 1993; 26: 467-81.
• 10 Lenert LA, Tovar M. Automated linkage of free-text descriptions of patients with a practice guideline. Proceedings of the Annual Symposium on Computer Applications in Medical Care 1993; 274-8.
• 11 Gabrieli ER. Computer-assisted assessment of patient care in the hospital. J Med Systems 1988; 12: 135-46.
• 12 Hripcsak G, Friedman C, Alderson PO, Du-Mouchel W, Johnson SB, Clayton PD. Unlocking clinical data from narrative reports: a study of natural language processing. Ann Intern Med 1995; 122: 681-8.
• 13 Friedman C, Hripcsak G, DuMouchel W, Johnson SB, Clayton PD. Natural language processing in an operational clinical information system. Natural Language Engin 1995; 1: 83-108.
• 14 Jain NL, Knirsch C, Friedman C, Hripcsak G. Identification of suspected tuberculosis patients based on natural language processing of chest radiograph reports. Proceedings of the Annual Symposium on Computer Applications in Medical Care 1996; 542-6.
• 15 Knirsch CA, Jain NL, Pablos-Mendez A, Friedman C, Hripcsak G. Respiratory isolation of tuberculosis patients using clinical guidelines and an automated clinical decision support system. Infect Control Hosp Epidemiol. 1998 (in press).
• 16 Teich JM, Spurr CD, Flammini SJ, et al. Response to a trial of physician-based inpatient order entry. Proceedings of the Annual Symposium on Computer Applications in Medical Care 1993; 316-20.
• 17 Hripcsak G, Ludemann P, Pryor TA, Wigertz OB, Clayton PD. Rationale for the Arden Syntax. Comput Biomed Res 1994; 27: 291-324.
• 18 Efron B, Tibshirani RJ. An Introduction to the Bootstrap. New York: Chapman & Hall; 1993
• 19 DuMouchel W, Friedman C, Hripcsak G, Johnson SB, Clayton PD. Two applications of statistical modelling to natural language processing. In: AI and Statistics V. Fisher D, Lenz H. eds. New York: Springer-Verlag,; 1996: 413-21.
• 20 Flanagan JC. The critical incident technique. Psychol Bull 1954; 51: 327-58.
• 21 Bundens WP, Bergan JJ, Halasz NA, Murray J, Drehobl M. The superficial femoral vein: a potentially lethal misnomer. JAMA 1995; 274: 1296-8.
• 22 Yerushalmy J. The statistical assessment of the variability in observer perception and description of roentgenographic pulmonary shadows. Radiologic Clinics of North America 1969; 7: 381-92.
• 23 Jollis JG, Ancukiewicz M, DeLong ER, Pryor DB, Muhlbaier LH, Mark DB. Discordance of databases designed for claims payment versus clinical information systems: implications for outcomes research. Ann Intern Med 1993; 119: 844-50.