Extracting Pain Care Quality Indicators from U.S. Veterans Health Administration Chiropractic Care Using Natural Language Processing

Abstract

Background Musculoskeletal pain is common in the Veterans Health Administration (VHA), and there is growing national use of chiropractic services within the VHA. Rapid expansion requires scalable and autonomous solutions, such as natural language processing (NLP), to monitor care quality. Previous work has defined indicators of pain care quality that represent essential elements of guideline-concordant, comprehensive pain assessment, treatment planning, and reassessment.

Objective Our purpose was to identify pain care quality indicators and assess patterns across different clinic visit types using NLP on VHA chiropractic clinic documentation.

Methods Notes from ambulatory or in-hospital chiropractic care visits from October 1, 2018 to September 30, 2019 for patients in the Women Veterans Cohort Study were included in the corpus, with visits identified as consultation visits and/or evaluation and management (E&M) visits. Descriptive statistics of pain care quality indicator classes were calculated and compared across visit types.

Results There were 11,752 patients who received any chiropractic care during FY2019, with 63,812 notes included in the corpus. Consultation notes had more than twice the total number of annotations per note (87.9) as follow-up visit notes (34.7). The mean number of total classes documented per note across the entire corpus was 9.4 (standard deviation [SD]  =  1.5). More total indicator classes were documented during consultation visits with (mean  =  14.8, SD  =  0.9) or without E&M (mean  =  13.9, SD  =  1.2) compared to follow-up visits with (mean  =  9.1, SD  =  1.4) or without E&M (mean  =  8.6, SD  =  1.5). Co-occurrence of pain care quality indicators describing pain assessment was high.

Conclusion VHA chiropractors frequently document pain care quality indicators, identifiable using NLP, with variability across different visit types.

Keywords

musculoskeletal pain - natural language processing - Veterans health - pain care quality

Background and Significance

Musculoskeletal pain is highly burdensome, carrying significant direct health care costs and indirect costs through functional impacts, disability, and lost productivity.[1] [2] [3] It is among the most common reasons individuals in the United States seek health care,[4] especially in the U.S. military Veteran population receiving health care in the Veterans Health Administration (VHA).[5]

In 2009, VHA issued Directive 2009-053 to promote quality pain care consistent with the VHA National Pain Management Strategy.[6] The directive outlined essential components of high-quality pain care to include (1) timely and appropriate pain assessment, (2) development and enactment of a pain treatment plan, and (3) subsequent reassessment of the effectiveness of the plan. The directive also reaffirmed VHA's commitment to providing pain care consistent with the Stepped Care Model for Pain Management,[7] which promotes early self-management with stepwise progression to primary care management and increasingly advancing specialty care management when clinically appropriate. The earliest steps of the Stepped Care Model promote the use of many guideline-recommended nonpharmacologic approaches for pain, including psychological/behavioral therapies, exercise/movement therapies, and manual therapies.[8]

One VHA specialty frequently utilizing many of these nonpharmacologic approaches for pain is chiropractic services.[9] The VHA is the largest integrated health care system offering chiropractic services, first as a covered service in 2000 and later as an integrated service at VHA facilities beginning in 2004.[10] Since 2018, VHA chiropractic care has undergone transformative expansion following legislative and executive action.[11] [12] VHA Chiropractic Program Office data show a growth of over 250% in facilities offering on-site chiropractic services from Fiscal Year (FY) 2017 (87 facilities) to FY2022 (221 facilities). Similarly, the total number of visits and patients receiving chiropractic care increased substantially from FY2017 (208,400 visits by 47,486 Veterans) to FY2022 (393,532 visits by 93,360 Veterans).

Among the backdrop of rapid expansion, a scalable, dynamic solution to monitoring the quality of chiropractic care delivered across the national VHA enterprise is needed. Current state quality assessment through the VHA Ongoing Professional Practice Evaluation (OPPE) requirements is typically measured via time-intensive, peer-conducted chart review to evaluate predetermined OPPE minimum standards. While easier to access and use, structured electronic health record (EHR) data present limitations to support in-depth evaluation of clinical care delivered across a national health system.

Natural language processing (NLP) of clinic documentation offers a potential opportunity to extract care quality data from unstructured EHR data on a national scale.[13] NLP is a diverse, integrated field utilizing computational techniques for linguistic analysis to achieve human-like language processing.[14] Information extraction from unstructured text using NLP has been successful across multiple sources, including physician reports,[15] radiology reports,[16] and patient experience surveys,[17] and across different settings, including VHA.[18] NLP has been used to evaluate pain care quality (PCQ) in VHA primary care clinics, including developing an extraction tool,[19] a PCQ indicator score,[20] and artificial intelligence methods to detect instances of pain assessment.[20] [21] The defined PCQ indicators represent essential elements of guideline-concordant, comprehensive pain assessment, treatment planning, and reassessment, and their documentation in the clinical text note may offer a measurable representation of the quality of pain care being provided.

Objectives

While the previously developed NLP algorithms were built from VA primary care progress notes, the lexicon describing PCQ indicators should theoretically be consistent across primary and specialty care clinic settings. We propose in this work to explore the application of the previously developed NLP algorithm to identify PCQ indicators in chiropractic clinics, a VHA specialty care clinic frequently evaluating and managing musculoskeletal pain conditions. The primary objective of the study was to assess PCQ patterns documented by chiropractors, describe their distributions across different clinic visit types, and compare the co-occurrence of documented pain assessment classes to a previous sample from primary care visits for pain.

Methods

Study Design, Setting, Participants, and Data Sources

This observational study was a secondary analysis of the Women Veterans Cohort Study[22]—an EHR cohort of post-9/11 men and women Veterans who served during Operation Enduring Freedom, Operation Iraqi Freedom, and Operation New Dawn (OEF/OIF/OND). Study reporting was informed by the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE).[23] This study was approved by the VA Connecticut Healthcare System Institutional Review Board.

Patients with at least one visit to a chiropractic clinic at a VHA facility from October 1, 2018 to September 30, 2019 (FY2019) were included in the sample. Patient demographic characteristics were obtained from the EHR, including age (as of October 1, 2018), sex, race, and ethnicity.

All chiropractic care visits during the study period were identified using a VHA administrative clinic stop code identifier for “Chiropractic Care.” Access to chiropractic care in the VHA most often requires the placement of a referral order to initiate a consultation. Visits were identified as consult visits if linked to an ordered consultation request in the EHR and follow-up visits if not. Any visit with an “Evaluation and Management” (E&M) current procedural terminology (CPT) code (99201–99205, 99211–99215) were identified as examination visits. Four groups of clinic visits were defined for comparison of documentation across types of visits—consult visits with an E&M code, consult visits without an E&M code, follow-up visits with an E&M code, and follow-up visits without an E&M code.

All clinical text note documents (VHA text integration utilities documents) were extracted for the identified visits to build a corpus of clinic visit notes (and sub-corpora for each clinic visit type). Only notes describing face-to-face (i.e., in-person) patient encounters (“ambulatory” or “in-hospital” visits) were retained for the NLP analysis, based on a structured visit type identifier in the EHR. Telehealth visits were uncommon in this corpus describing visits from FY2019 (pre-COVID-19 pandemic) and were not included.[24] Other notes linked to the identified visits but describing historical events, clinical reminders, telecommunications, electronic consultations, or nursing notes were excluded from the analysis.

Analysis

We applied a previously developed and described rule-based NLP algorithm using Python 3.5 to extract PCQ indicators from the corpus.[20] The algorithm performed strongly in a study of 1 year of primary care notes from pain visits (F-measure  =  91.9%, Precision  =  93.0%, Recall  =  90.9%).[20]

The algorithm uses a line-by-line analysis of text documents paired with rule-based token regular expression matching to identify annotation spans—or tagged snippets of text—mapped to PCQ indicator classes defined by a vocabulary. The vocabulary was reviewed by the study team to review appropriateness for application in chiropractic clinic documentation. A random sample of 100 chiropractic clinic notes was reviewed after the NLP algorithm was applied to generate a qualitative impression of the model's face validity[25] in the unseen corpus. Quantitative evaluation of this process was beyond the objectives of this study.

PCQ indicator classes included those describing pain assessment, treatment planning, and reassessment, for a total of 25 indicators ([Supplementary Table S1], available in the online version). We also examined the subset of 11 classes describing pain assessment to compare documentation by chiropractors to a reference sample derived from primary care documentation.[21]

Table 1 Characteristics of Women Veterans Cohort Study patients receiving VHA chiropractic care during FY2019

Variable	Total
N	11,752
Age,[a] median [Q1, Q3], y (%)	39 [34, 48]
<40 y	52.6
40–49 y	25.6
50–64 y	20.7
65 years and over	1.1
Sex (%)
Female	15.8
Male	84.2
Race (%)
White	73.5
Black or African American	15.7
Asian	2.2
Native Hawaiian or Other Pacific Islander	1.2
American Indian or Alaska Native	1.0
Mixed race	1.3
Other/unknown	5.1
Ethnicity (%)
Hispanic or LatinX	12.0
Not Hispanic or LatinX	88.0

Abbreviation: VHA, Veterans Health Administration.

^a Age as of October 1, 2018.

Study Outcomes

As an exploratory analysis, descriptive statistics were calculated for the presence of PCQ indicators across the entire corpus and across notes for each of the four unique clinic visit types.

The number of PCQ indicator classes present was calculated for each note, with a greater number of present PCQ indicator classes presumed to reflect a broader array of the quality of pain care delivered. The prevalence of notes with each individual indicator class and each total number of indicator classes was calculated as the percentage of the entire corpus and each visit type.

The within note frequency of each individual PCQ indicator class was identified, with a greater number describing more mentions of the indicator class. The mean number of mentions of each class per note were calculated for all notes and the four clinic visit types.

The presence or absence of 11 PCQ indicator classes describing pain assessment was evaluated for co-occurrence of classes—or simultaneous documentation of individual PCQ indicator classes in the same note. The percentages of all notes and notes of each visit type in the chiropractic clinic note corpus were calculated for varying levels of co-occurrence, and aggregate percentages were compared to previously published data from primary care visits for musculoskeletal pain diagnoses by Fodeh et al.[17]

Results

There were 11,752 patients who received any chiropractic care during FY2019 ([Table 1]), with 11,416 having at least one ambulatory or in-hospital visit and 336 patients excluded. Characteristics of the NLP corpus of ambulatory or in-hospital visits are shown in [Table 2], across all notes and by visit type. The entire corpus included 63,812 notes from 52,117 visits across 80 VHA facilities, with over 2.6 million annotations of PCQ indicators present across the corpus. The most common text spans that were annotated across the entire corpus and their respective PCQ indicator classes are shown in [Supplementary Table S2] (available in the online version). The most common text span by far was “pain” (Pain Mention) accounting for 12.6% of all annotations, with “low back” (Pain Site) the second most common (2.0% of all annotations).

Table 2 Corpus characteristics from ambulatory or in-hospital only chiropractic clinic visits from FY2019, by visit type

		Visit type
Variable	Total	Consult, E&M	Consult, no E&M	Follow-up, E&M	Follow-up, no E&M
Patients	11,416	4,339	2,980	5,020	7,890
Facilities	80	72	72	75	80
Visits	52,117	4,421	3,022	13,690	33,017
Notes	63,812	4,425	3,024	15,884	40,479
Annotations	2,608,565	418,148	236,736	612,413	1,341,268

Abbreviation: E&M, evaluation and management.

Follow-up visit notes made up 88.3% of the entire corpus and were more likely to not have an E&M code present (63.4% of all notes). Consult notes were more likely to have an E&M code associated with the visit (6.9% of all notes) than not (4.7% of all notes). Consult notes had more than twice the total number of annotations per note (87.9) as follow-up visit notes (34.7) on average. Across all chiropractic care notes, only 3.8% did not include any documentation of PCQ indicators. These notes were most commonly for follow-up visits and frequently included telecommunications visit notes that were miscoded as ambulatory or in-hospital visits upon manual review.

The distributions of the total number of PCQ indicator classes across the entire corpus and by visit type are shown in [Fig. 1]. The mean number of total classes documented per note across the entire corpus was 9.4 (standard deviation [SD]  =  1.5). More total indicator classes were documented during consult visits with or without E&M CPTs (consult with E&M: mean  =  14.8, SD  =  0.9; consult without E&M: mean  =  13.9, SD  =  1.2) compared to follow-up visits with or without E&M (follow-up with E&M: mean  =  9.1, SD  =  1.4; follow-up without E&M: mean  =  8.6, SD  =  1.5).

Fig. 1 Prevalence of chiropractic clinic notes with total number of pain care quality indicator classes, by visit type (with mean and standard deviation).

[Fig. 2] shows the relationship between PCQ indicator class frequency within a note (as the mean number of mentions) and the class prevalence across the entire corpus. Across all notes and across notes for each visit type, classes with higher prevalence across the corpus (or sub-corpus) were also mentioned more frequently, on average, within a note. The most common classes were consistent across all notes and visit types and included Pain Mention, Chiropractic, Pain Site, Physical Diagnosis, and Etiology. The least common classes were similarly consistent across all notes and visit types, describing Diurnal Variation, Implantables, and Mental Health. The remaining classes were mentioned with similar within note frequencies across all notes and visit types. However, they had a greater prevalence across the corpora of consult visits with or without E&M than within the corpora of follow-up notes.

Fig. 2 Pain care quality indicator class mentions per note by class prevalence across corpus, by visit type.

When examining only the subset of 11 pain assessment classes ([Fig. 3]), over 50% of all chiropractic clinic notes documented at least four different classes (consult with E&M  =  94.8%; consult without E&M  =  87.2%; follow-up with E&M  =  46.9%; follow-up without E&M  =  46.5%). Greater co-occurrence of pain assessment classes was present in all chiropractic care visit types compared to the reference sample of 1,058 primary care pain visit notes by Fodeh et al.[21]

Fig. 3 Co-occurrence of subset of pain care quality indicators describing pain assessment, organized by visit type. The height of each stacked bar represents proportion of notes with at least the specified lower bound number of pain care quality indicators. *Comparison data from primary care pain visit notes from Fodeh SJ, Finch D, Bouayad L, et al. Classifying clinical notes with pain assessment using machine learning. Med Biol Eng Comput 2018;56(7):1285–1292

Discussion

Monitoring and evaluating pain care to ensure quality is a high priority for the VHA. A challenge presents in unifying and applying solutions across multiple specialties providing health care services for musculoskeletal pain in the VHA's Stepped Care Model for Pain Management. In this study, we applied a rule-based NLP algorithm developed in primary care documentation to chiropractic care documentation as one specialty delivering musculoskeletal pain care in the VHA. A scalable solution for monitoring PCQ in VHA chiropractic clinics nationally is essential given the rapid expansion of VHA chiropractic services nationally since 2017, with adoption of modernized processes that move beyond current-state time-intensive manual chart review methods for quality monitoring. Operationalizing this process, including determining the computational infrastructure needed to support real-time (or near real-time) monitoring, is important future work.

The application of the existing PCQ indicator NLP algorithm in an unseen corpus of notes from a separate clinical discipline was based on the premises that (1) indicators of quality pain care would be consistent across disciplines managing musculoskeletal pain and (2) the terminology describing how pain care is documented is reasonably similar across disciplines. While formal validation of the algorithm's performance in the corpus of chiropractic clinic notes was not performed in this study, we took steps to ensure it was appropriate for the study's primary purpose before comparing PCQ indicator patterns across different visit types. The rule-based NLP model used a lookup vocabulary that was reviewed by the study team before application to review the PCQ indicator classes and the terminology linked to each class. After applying the NLP algorithm, we also reviewed a sample of model-annotated notes to confirm face validity and appropriateness for use in a corpus of chiropractic clinic notes. While applied in only a single VHA specialty pain care discipline in this study, application of the NLP algorithm in other nonprimary care settings is likely reasonable after ensuring appropriateness with manual review.

Our findings show that VHA chiropractors frequently document multiple PCQ indicators across notes and visits. We identified a direct relationship between the most prevalent indicator classes across the corpus and the average number of mentions of a class per note. When classes were mentioned more frequently across the entire collection of notes, they were also mentioned more frequently within individual notes. The individual classes mentioned most frequently across the corpus and within individual notes were generally expected. Their high per note prevalence may be attributable to the depth of evaluation of a patient's pain in a thorough pain history with respect to location and etiology and the use of multiple physical diagnosis procedures as part of a comprehensive musculoskeletal and neuromuscular examination to evaluate a patient's pain and functional status and inform the management plan.[26] The most common text spans also support attribution to chiropractic practice as they described pain, especially spinal pain (e.g., low back or neck), range of motion (e.g., flexion, rotation), and chiropractic treatment (e.g., spinal manipulation). The most common text spans may be useful in developing a standardized vocabulary and ontology of chiropractic care to support future NLP evaluation in the clinical discipline.

Of interest was Mental Health among the classes mentioned least frequently, given comorbid pain and mental health conditions are common in the OEF/OIF/OND Veteran population[27] [28] and those receiving VHA chiropractic care.[29] Further, recognition of the contributions of psychosocial factors, and mental health conditions as an extension of those factors, to a patient's pain experience is an expected best practice of chiropractic care consistent with a biopsychosocial model of care.[26] We suspect this finding is due to chiropractors not considering or not documenting mental health conditions or psychosocial factors related to mental health, given the vocabulary was viewed as appropriate by the study team. However, a disconnect between the algorithm's vocabulary for the Mental Health class and how chiropractors are documenting mental health considerations in their notes could also be contributory and is a potential area for future formal evaluation. Our findings suggest an opportunity may exist for quality improvement related to consideration of mental health conditions and symptoms by VHA chiropractors, while also demonstrating the generalizability of a scalable, NLP quality monitoring solution to identify potential areas for targeted quality improvement initiatives.

Our findings regarding the documentation of PCQ indicators by visit type showed expected but interesting documentation patterns. Consult visits consistently had many more PCQ indicator classes documented than follow-up visits, which was expected. However, the presence of an E&M CPT code was not necessarily reflective of more PCQ indicators documented in either consult or follow-up visits. We suspect this may reflect the “minimum components” requirements of E&M coding in use during FY2019 or may reflect a known lack of fidelity in outpatient clinician coding, of which under-coding is common.[30] [31]

When evaluating the subset of pain assessment classes for co-occurrent documentation in comparison to the reference sample of primary care notes, chiropractic clinic notes showed consistent documentation of more assessment classes across all visit types. We suspect this is likely due to the focused nature of a chiropractic care visit on one or a few musculoskeletal pain conditions—while pain may be one of many conditions being managed in a typical primary care visit. Repeating our methods in corpora of notes from other VHA pain care specialties would likely yield similar patterns of pain assessment. This highlights the benefits of the collaborative structure of the VHA in providing integrated and interdisciplinary team-based pain care with facilitated access to maximize the quality of pain assessment.[32] Important future work should also evaluate how patient factors, such as demographic characteristics, and visit factors, such as visit type or method of health care delivery (e.g., face-to-face vs. telehealth), influence documentation patterns across different VHA pain care settings.

As in any NLP study, our study is ultimately limited by the contents and quality of the text documents that are processed. While NLP of clinical notes can allow analyses that move beyond structured data elements captured in the EHR, secondary analysis is limited by the contents collected and entered at the point of care for clinical purposes by the document writer and may not fully reflect what is performed in the visit. In some cases, documentation may under-represent details of the patient–clinician encounter, while in others, historical documentation may be carried forward for reference as an over-representation. Other limitations include the use of templated text within clinic notes. For example, templated statements documenting patient consent may include text spans related to PCQ indicator classes (e.g., “…I advised patient on the benefits and risks of spinal manipulation <Treatment>…”), but not confirm an indicator was present during the visit. Our approach was to broadly explore the application of the existing algorithm in chiropractic clinic notes and thus we did not aim to discern the intentions of PCQ indicator mentions using assertion or negation.

We also recognize limitations related to short communication notes, most often due to inaccurate coding of note types or multiple notes linked to an individual visit. We attempted to limit our corpus to only ambulatory and in-hospital note types to focus our analysis on notes documenting clinical encounters. However, it was evident during review that inaccurate coding of note types led to the inclusion of other types of notes in the corpus, for example telecommunications notes describing scheduling or telephone follow-up on patient status or diagnostic test findings. Further, multiple notes could be linked to an individual visit identifier that often were addenda describing trainee participation in notes or other short, administrative communications. As our analysis was performed at the note level, each of these were considered distinct notes. Future work should consider alternative strategies, such as only evaluating notes authored by the attending chiropractor or concatenating all notes related to the same visit identifier to resolve to a single collected document per visit.

Conclusion

Our study demonstrates that a PCQ indicator NLP algorithm developed from primary care pain visit notes may have utility in VHA pain specialty care for the purposes of quality care monitoring on a national scale. We identified patterns of frequently documented PCQ indicators by VHA chiropractors that can be used to inform quality improvement initiatives related to pain evaluation and management. NLP may be a useful approach in future work to study features of pain care, including chiropractic care, not evident in structured EHR data. Developing standardized, comprehensive ontologies describing pain care and chiropractic care may enhance validity and reproducibility of results. Future applications of this NLP algorithm should investigate relationships between PCQ indicators and patient characteristics as part of EHR-based phenotyping efforts, which may inform clinical decision making and ensure delivery of high-quality and equitable pain care in VHA pain specialty clinics, including chiropractic clinics.

Clinical Relevance Statement

Natural language processing is an approach that may facilitate evaluation of pain care quality in unstructured data from specialty pain care, including chiropractic clinic documentation. Veterans Health Administration chiropractors frequently document many pain care quality indicators, with different patterns noted across consultation and follow-up visit types and visits with or without evaluation and management procedural codes. These results can inform education and quality improvement initiatives related to evaluation and management of pain in clinical practice.

Multiple-Choice Questions

1. The greatest average number of pain care quality indicator classes was documented in which type of VHA chiropractic care visit?

   • Consultation visit with an evaluation and management CPT

   • Consultation visit without an evaluation and management CPT

   • Follow-up visit with an evaluation and management CPT

   • Follow-up visit without an evaluation and management CPT

   Correct Answer: The correct answer is option a. Consultation visit with an evaluation and management CPT. In consultation visits with an evaluation and management CPT, an average of 14.8 pain care quality indicator classes were documented. Consultation visits without an evaluation and management CPT had 13.9 classes documented on average. Follow-ups with (9.1) or without (8.6) an evaluation and management CPT had fewer pain care quality indicator classes documented.

2. VHA chiropractors infrequently documented which of the following pain care quality indicator classes?

   • Pain Site

   • Physical Diagnosis

   • Mental Health

   • Etiology

   Correct Answer: The correct answer is option c. Mental Health. Chiropractors infrequently documented “Mental Health” indicators. It is unclear if our finding is due to chiropractors not considering or not documenting mental health conditions or psychosocial factors related to mental health, a disconnect between the algorithm's vocabulary for the Mental Health class and how chiropractors are documenting mental health considerations in their notes, or some combination of these and other factors.

Conflict of Interest

None declared.

Acknowledgements

The contents of this manuscript represent the view of the authors and do not necessarily reflect the position or policy of the U.S. Department of Veterans Affairs, the National Institutes of Health, nor the United States Government. This material is based upon work supported by the National Center for Complementary and Integrative Health of the National Institutes of Health under Award Number K08AT011570, the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, and Health Services Research and Development (IIR-12-118 and CIN-13-407), and the NCMIC Foundation, with resources and the use of facilities at the VA Connecticut Healthcare System. The authors have no additional conflicts of interest, financial or otherwise, to disclose.

Availability of Data and Materials

To maximize protection security of Veterans' data while making these data available to researchers, the U.S. Department of Veterans Affairs (VA) developed the VA Informatics and Computing Infrastructure (VINCI). VA researchers must log onto VINCI via a secure gateway or virtual private network connection (VPN) and use a virtual workspace on VINCI to access and analyze VA data. By VA Office of Research and Development policy, VINCI does not allow the transfer of any patient-level data out of its secure environment without special permission. Researchers who are not VA employees must be vetted and receive “without compensation” (WOC) employee status to gain access to VINCI. All analyses performed for this study took place on the VINCI platform. For questions about data access, contact study lead, Dr. Brian C. Coleman ([e-mail]) or the VA Office of Research and Development ([e-mail]).

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed and approved by the VA Connecticut Healthcare System Institutional Review Board.

• References

• 1 Hoy D, March L, Brooks P. et al. The global burden of low back pain: estimates from the Global Burden of Disease 2010 study. Ann Rheum Dis 2014; 73 (06) 968-974
• 2 Hurwitz EL, Randhawa K, Yu H, Côté P, Haldeman S. The Global Spine Care Initiative: a summary of the global burden of low back and neck pain studies. Eur Spine J 2018; 27 (Suppl. 06) 796-801
• 3 Blyth FM, Briggs AM, Schneider CH, Hoy DG, March LM. The global burden of musculoskeletal pain-where to from here?. Am J Public Health 2019; 109 (01) 35-40
• 4 St Sauver JL, Warner DO, Yawn BP. et al. Why patients visit their doctors: assessing the most prevalent conditions in a defined American population. Mayo Clin Proc 2013; 88 (01) 56-67
• 5 Goulet JL, Kerns RD, Bair M. et al. The musculoskeletal diagnosis cohort: examining pain and pain care among Veterans. Pain 2016; 157 (08) 1696-1703
• 6 US Department. of Veterans Affairs. VA Directive 2009-053 - Pain Management. Washington, DC; US Department of Veterans Affairs; 2009
• 7 Kerns RD, Philip EJ, Lee AW, Rosenberger PH. Implementation of the Veterans health administration national pain management strategy. Transl Behav Med 2011; 1 (04) 635-643
• 8 Kligler B, Bair MJ, Banerjea R. et al. Clinical policy recommendations from the VHA state-of-the-art conference on non-pharmacological approaches to chronic musculoskeletal pain. J Gen Intern Med 2018; 33 (1, suppl 1): 16-23
• 9 Lisi AJ, Salsbury SA, Hawk C. et al. Chiropractic integrated care pathway for low back pain in Veterans: results of a delphi consensus process. J Manipulative Physiol Ther 2018; 41 (02) 137-148
• 10 Lisi AJ, Brandt CA. Trends in the use and characteristics of chiropractic services in the Department of Veterans Affairs. J Manipulative Physiol Ther 2016; 39 (05) 381-386
• 11 Military Conssstruction, Veterans Affairs, and Related Agencies Appropriations Act of 2018, Pub. L. 115-141, 132 Stat. 348 (March 23, 2018)
• 12 US Department. of Veterans Affairs. VA Directive 2018-1210 - Chiropractic Care. Washington, DC: US Dept. of Veterans Affairs; 2018
• 13 Ohno-Machado L. Realizing the full potential of electronic health records: the role of natural language processing. J Am Med Inform Assoc 2011; 18 (05) 539
• 14 Nadkarni PM, Ohno-Machado L, Chapman WW. Natural language processing: an introduction. J Am Med Inform Assoc 2011; 18 (05) 544-551
• 15 Rousseau JF, Ip IK, Raja AS. et al. Can automated retrieval of data from emergency department physician notes enhance the imaging order entry process?. Appl Clin Inform 2019; 10 (02) 189-198
• 16 Sevenster M, Buurman J, Liu P, Peters JF, Chang PJ. Natural language processing techniques for extracting and categorizing finding measurements in narrative radiology reports. Appl Clin Inform 2015; 6 (03) 600-110
• 17 Nawab K, Ramsey G, Schreiber R. Natural language processing to extract meaningful information from patient experience feedback. Appl Clin Inform 2020; 11 (02) 242-252
• 18 Jones BE, South BR, Shao Y. et al. Development and validation of a natural language processing tool to identify patients treated for pneumonia across VA emergency departments. Appl Clin Inform 2018; 9 (01) 122-128
• 19 Dorflinger LM, Gilliam WP, Lee AW, Kerns RD. Development and application of an electronic health record information extraction tool to assess quality of pain management in primary care. Transl Behav Med 2014; 4 (02) 184-189
• 20 Luther SL, Finch DK, Bouayad L. et al. Measuring pain care quality in the Veterans Health Administration primary care setting. Pain 2022; 163 (06) e715 –e724
• 21 Fodeh SJ, Finch D, Bouayad L. et al. Classifying clinical notes with pain assessment using machine learning. Med Biol Eng Comput 2018; 56 (07) 1285-1292
• 22 Gaffey AE, Burg MM, Rosman L. et al. Baseline characteristics from the Women Veterans Cohort Study: gender differences and similarities in health and healthcare utilization. J Womens Health (Larchmt) 2021; 30 (07) 944-955
• 23 von Elm E, Altman DG, Egger M, Pocock SJ, Gøtzsche PC, Vandenbroucke JP. STROBE Initiative. The Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) statement: guidelines for reporting observational studies. J Clin Epidemiol 2008; 61 (04) 344-349
• 24 Roytman GR, Coleman BC, Corcoran KL, Goertz CM, Long CR, Lisi AJ. Changes in the use of telehealth and face-to-face chiropractic care in the Department of Veterans Affairs before and after the COVID-19 pandemic. J Manipulative Physiol Ther 2021; 44 (07) 584-590
• 25 Fang Q, Nguyen D, Oberski DL. Evaluating the construct validity of text embeddings with application to survey questions. EPJ Data Sci 2022; 11 (01) 39
• 26 Hawk C, Whalen W, Farabaugh RJ. et al. Best practices for chiropractic management of patients with chronic musculoskeletal pain: a clinical practice guideline. J Altern Complement Med 2020; 26 (10) 884-901
• 27 Ramsey C, Dziura J, Justice AC. et al. Incidence of mental health diagnoses in Veterans of Operations Iraqi Freedom, Enduring Freedom, and New Dawn, 2001-2014. Am J Public Health 2017; 107 (02) 329-335
• 28 Seal KH, Metzler TJ, Gima KS, Bertenthal D, Maguen S, Marmar CR. Trends and risk factors for mental health diagnoses among Iraq and Afghanistan Veterans using Department of Veterans Affairs health care, 2002-2008. Am J Public Health 2009; 99 (09) 1651-1658
• 29 Coleman BC, Corcoran KL, DeRycke EC. et al. Factors associated with posttraumatic stress disorder among Veterans of recent wars receiving Veterans affairs chiropractic care. J Manipulative Physiol Ther 2020; 43 (08) 753-759
• 30 Burks K, Shields J, Evans J, Plumley J, Gerlach J, Flesher S. A systematic review of outpatient billing practices. SAGE Open Med 2022; 10: 20 503121221099021
• 31 King MS, Sharp L, Lipsky MS. Accuracy of CPT evaluation and management coding by family physicians. J Am Board Fam Pract 2001; 14 (03) 184-192
• 32 Becker WC, Dorflinger L, Edmond SN, Islam L, Heapy AA, Fraenkel L. Barriers and facilitators to use of non-pharmacological treatments in chronic pain. BMC Fam Pract 2017; 18 (01) 41

Publication History

Received: 05 January 2023

Accepted: 27 April 2023

Accepted Manuscript online:
10 May 2023

Article published online:
02 August 2023