Abstract
Background Identifying pneumonia using diagnosis codes alone may be insufficient for research
on clinical decision making. Natural language processing (NLP) may enable the inclusion
of cases missed by diagnosis codes.
Objectives This article (1) develops a NLP tool that identifies the clinical assertion of pneumonia
from physician emergency department (ED) notes, and (2) compares classification methods
using diagnosis codes versus NLP against a gold standard of manual chart review to
identify patients initially treated for pneumonia.
Methods Among a national population of ED visits occurring between 2006 and 2012 across the
Veterans Affairs health system, we extracted 811 physician documents containing search
terms for pneumonia for training, and 100 random documents for validation. Two reviewers
annotated span- and document-level classifications of the clinical assertion of pneumonia.
An NLP tool using a support vector machine was trained on the enriched documents.
We extracted diagnosis codes assigned in the ED and upon hospital discharge and calculated
performance characteristics for diagnosis codes, NLP, and NLP plus diagnosis codes
against manual review in training and validation sets.
Results Among the training documents, 51% contained clinical assertions of pneumonia; in
the validation set, 9% were classified with pneumonia, of which 100% contained pneumonia
search terms. After enriching with search terms, the NLP system alone demonstrated
a recall/sensitivity of 0.72 (training) and 0.55 (validation), and a precision/positive
predictive value (PPV) of 0.89 (training) and 0.71 (validation). ED-assigned diagnostic
codes demonstrated lower recall/sensitivity (0.48 and 0.44) but higher precision/PPV
(0.95 in training, 1.0 in validation); the NLP system identified more “possible-treated”
cases than diagnostic coding. An approach combining NLP and ED-assigned diagnostic
coding classification achieved the best performance (sensitivity 0.89 and PPV 0.80).
Conclusion System-wide application of NLP to clinical text can increase capture of initial diagnostic
hypotheses, an important inclusion when studying diagnosis and clinical decision-making
under uncertainty.
Keywords
pneumonia - natural language processing - decision-making - diagnosis - surveillance
