Download PDF
Appl Clin Inform 2012; 03(02): 164-174
DOI: 10.4338/ACI-2011-11-RA-0070
Research Article
Schattauer GmbH

Assessing Electronic Note Quality Using the Physician Documentation Quality Instrument (PDQI-9)

P.D. Stetson
1   Department of Biomedical Informatics, Columbia University
2   Department of Medicine, Columbia University
,
S. Bakken
1   Department of Biomedical Informatics, Columbia University
3   School of Nursing, Columbia University
,
J.O. Wrenn
4   Department of Medical Informatics, Vanderbilt University
,
E.L. Siegler
4   Division of Geriatrics and Gerontology, Weill Cornell Medical College
› Author Affiliations
Further Information

Publication History

received: 21 November 2011

accepted: 29 March 2012

Publication Date:
16 December 2017 (online)

Also available at
   Permissions and Reprints

Summary

Objective: To refine the Physician Documentation Quality Instrument (PDQI) and test the validity and reliability of the 9-item version (PDQI-9).

Methods: Three sets each of admission notes, progress notes and discharge summaries were evaluated by two groups of physicians using the PDQI-9 and an overall general assessment: one gold standard group consisting of program or assistant program directors (n = 7), and the other of attending physicians or chief residents (n = 24). The main measures were criterion-related validity (correlation coefficients between Total PDQI-9 scores and 1-item General Impression scores for each note), discriminant validity (comparison of PDQI-9 scores on notes rated as best and worst using 1-item General Impression score), internal consistency reliability (Cronbach’s alpha), and inter-rater reliability (intraclass correlation coefficient (ICC)).

Results: The results were criterion-related validity (r = –0.678 to 0.856), discriminant validity (best versus worst note, t = 9.3, p = 0.003), internal consistency reliability (Cronbach’s alphas = 0.87–0.94), and inter-rater reliability (ICC = 0.83, CI = 0.72–0.91).

Conclusion: The results support the criterion-related and discriminant validity, internal consistency reliability, and inter-rater reliability of the PDQI-9 for rating the quality of electronic physician notes. Tools for assessing note redundancy are required to complement use of PDQI-9. Trials of the PDQI-9 at other institutions, of different size, using different EHRs, and incorporating additional physician specialties and notes of other healthcare providers are needed to confirm its generaliz-ability.

Keywords

Electronic health record - documentation - note - quality - instrument

 

  • References

  • 1 DesRoches CM, Campbell EG, Rao SR, Donelan K, Ferris TG, Jha A. et al. Electronic health records in ambulatory care –a national survey of physicians. N Engl J Med 2008; 359 (01) 50-60.
    CrossrefPubMedGoogle Scholar
  • 2 Medicare and Medicaid programs; electronic health record incentive program.. Final rule. Fed Regist 2010; 75 (144) 44313-44588.
    PubMedGoogle Scholar
  • 3 Amarasingham R, Plantinga L, Diener-West M, Gaskin DJ, Powe NR. Clinical information technologies and inpatient outcomes: a multiple hospital study. Arch Intern Med 2009; 169 (02) 108-114.
    CrossrefPubMedGoogle Scholar
  • 4 Poon EG, Wright A, Simon SR, Jenter CA, Kaushal R, Volk LA. et al. Relationship between use of electronic health record features and health care quality: results of a statewide survey. Med Care 2010; 48 (03) 203-209.
    CrossrefPubMedGoogle Scholar
  • 5 Johnson SB, Bakken S, Dine D, Hyun S, Mendonca E, Morrison F. et al. An electronic health record based on structured narrative. J Am Med Inform Assoc 2008; 15 (01) 54-64.
    CrossrefPubMedGoogle Scholar
  • 6 Kirkland LR, Bryan CS. Osler’s service: a view of the charts. J Med Biogr 2007; 15 (Suppl. 01) 50-54.
    CrossrefPubMedGoogle Scholar
  • 7 Weed LL. The problem oriented record as a basic tool in medical education, patient care and clinical research. Ann Clin Res 1971; 3 (03) 131-134.
    PubMedGoogle Scholar
  • 8 Fries JF. Alternatives in medical record formats. Med Care 1974; 12 (10) 871-881.
    CrossrefPubMedGoogle Scholar
  • 9 Reiser SJ. The clinical record in medicine. Part 2: Reforming content and purpose. Ann Intern Med 1991; 114 (11) 980-985.
    CrossrefPubMedGoogle Scholar
  • 10 Burnum JF. The misinformation era: the fall of the medical record. Ann Intern Med 1989; 110 (06) 482-484.
    CrossrefPubMedGoogle Scholar
  • 11 Embi PJ, Yackel TR, Logan JR, Bowen JL, Cooney TG, Gorman PN. Impacts of computerized physician documentation in a teaching hospital: perceptions of faculty and resident physicians. J Am Med Inform Assoc 2004; 11 (04) 300-309.
    CrossrefPubMedGoogle Scholar
  • 12 Hartzband P, Groopman J. Off the record –avoiding the pitfalls of going electronic. N Engl J Med 2008; 358 (16) 1656-1658.
    CrossrefPubMedGoogle Scholar
  • 13 Hirschtick RE. A piece of my mind. Copy-and-paste. Jama 2006; 295 (20) 2335-2336.
    CrossrefPubMedGoogle Scholar
  • 14 Siegler EL, Adelman R. Copy and paste: a remediable hazard of electronic health records. Am J Med 2009; 122 (06) 495-496.
    CrossrefPubMedGoogle Scholar
  • 15 Thielke S, Hammond K, Helbig S. Copying and pasting of examinations within the electronic medical record. Int J Med Inform 2007; 76 (Suppl. 01) S122-S128.
    CrossrefPubMedGoogle Scholar
  • 16 Weir CR, Hurdle JF, Felgar MA, Hoffman JM, Roth B, Nebeker JR. Direct text entry in electronic progress notes. An evaluation of input errors. Methods Inf Med 2003; 42 (01) 61-67.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 17 O’Donnell HC, Kaushal R, Barron Y, Callahan MA, Adelman RD, Siegler EL. Physicians’ attitudes towards copy and pasting in electronic note writing. Journal of general internal medicine: official journal of the Society for Research and Education in Primary Care Internal Medicine 2009; 24 (01) 63-68.
    PubMedGoogle Scholar
  • 18 Gelzer R, Hall T, Liette E, Reeves MG, Sundby J, Tegen A. et al. Auditing copy and paste. J AHIMA 2009; Jan 80 (01) 26-9 quiz 31-2.
    PubMedGoogle Scholar
  • 19 Payne T, Patel R, Beahan S, Zehner J. The Physical Attractiveness of Electronic Physician Notes. AMIA Annu Symp Proc 2010: 622-626.
    PubMedGoogle Scholar
  • 20 Institute of Medicine.. Committee on Improving the Patient R Dick RS, Steen EB, Detmer DE. The computer-based patient record : an essential technology for health care. Washington, D. C.: National Academy Press; 1997
    Google Scholar
  • 21 Romm FJ, Putnam SM. The validity of the medical record. Med Care 1981; 19 (03) 310-315.
    CrossrefPubMedGoogle Scholar
  • 22 Tufo HM, Speidel JJ. Problems with Medical Records. Medical care 1971; 9 (06) 509-517.
    CrossrefPubMedGoogle Scholar
  • 23 Rosenbloom ST, Crow AN, Blackford JU, Johnson KB. Cognitive factors influencing perceptions of clinical documentation tools. J Biomed Inform 2007; 40 (02) 106-113.
    CrossrefPubMedGoogle Scholar
  • 24 Stetson PD, Morrison FP, Bakken S, Johnson SB. Preliminary development of the physician documentation quality instrument. J Am Med Inform Assoc 2008; 15 (04) 534-541.
    CrossrefPubMedGoogle Scholar
  • 25 Wrenn JO, Stein DM, Bakken S, Stetson PD. Quantifying clinical narrative redundancy in an electronic health record. J Am Med Inform Assoc 2010; 17 (01) 49-53.
    CrossrefPubMedGoogle Scholar
  • 26 Aronsky D, Haug PJ. Assessing the quality of clinical data in a computer-based record for calculating the pneumonia severity index. J Am Med Inform Assoc 2000; 7 (01) 55-65.
    CrossrefPubMedGoogle Scholar
  • 27 Coakley FV, Heinze SB, Shadbolt CL, Schwartz LH, Ginsberg MS, Lefkowitz RA. et al. Routine editing of trainee-generated radiology reports: effect on style quality. Acad Radiol 2003; 10 (03) 289-294.
    CrossrefPubMedGoogle Scholar
  • 28 Efthimiadis EN, Hammond KW, Laundry R, Thielke SM. Developing an EMR simulator to assess users’ perception of document quality. Proc 43rd Hawaii Int Conf on System Sciences –2010 2010: 1-9.
    PubMedGoogle Scholar
  • 29 Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997; 4 (05) 342-355.
    CrossrefPubMedGoogle Scholar
  • 30 Logan JR, Gorman PN, Middleton B. Measuring the quality of medical records: a method for comparing completeness and correctness of clinical encounter data. Proc AMIA Symp 2001: 408-412.
    PubMedGoogle Scholar
  • 31 Myers KA, Keely EJ, Dojeiji S, Norman GR. Development of a rating scale to evaluate written communication skills of residents. Acad Med 1999; 74 (Suppl. 10) S111-S113.
    CrossrefPubMedGoogle Scholar
  • 32 Hammond KW, Efthimiadis EN, Weir CR, Embi PJ, Thielke SM, Laundry RM. et al. Initial steps toward validating and measuring the quality of computerized provider documentation. AMIA Annu Symp Proc 2010 2010: 271-275.
    PubMedGoogle Scholar
  • 33 Bates DW. Getting in step: electronic health records and their role in care coordination. J Gen Intern Med 2010; 25 (03) 174-176.
    CrossrefPubMedGoogle Scholar
  • 34 Bates DW, Bitton A. The future of health information technology in the patient-centered medical home. Health Aff (Millwood) 2010; 29 (04) 614-621.
    CrossrefPubMedGoogle Scholar
  • 35 O’Malley AS, Grossman JM, Cohen GR, Kemper NM, Pham HH. Are electronic medical records helpful for care coordination? Experiences of physician practices. J Gen Intern Med 2010; 25 (03) 177-185.
    CrossrefPubMedGoogle Scholar
  • 36 Naik AD, Singh H. Electronic health records to coordinate decision making for complex patients: what can we learn from wiki?. Med Decis Making 2010; 30 (06) 722-731.
    CrossrefPubMedGoogle Scholar
  • 37 Meaningful Use Workgroup Request for Comments Regarding Meaningful Use Stage 2.. Health Information Technology Policy Committee; 2011; Available from: http://healthit.hhs.gov/media/faca/ MU_RFC%20_2011–01–12_final.pdf.
    PubMed
  • 38 Hammond KW, Helbig ST, Benson CC, Brathwaite-Sketoe BM. Are electronic medical records trustworthy? Observations on copying, pasting and duplication. AMIA Annu Symp Proc 2003: 269-273.
    PubMedGoogle Scholar