Keywords
Electronic health records and systems - problem-oriented medical records - evaluation - satisfaction - physician
1. Background and Significance
1. Background and Significance
Implementation of electronic health records (EHRs) to replace patients’ paper charts can benefit health care delivery [[1], [2]]. Past research provides good evidence that EHRs can improve quality, safety, and efficiency [[3]]. Clinical documentation is a key component of EHRs. Clinical documentation should support patient care and improve clinical outcomes through enhanced communication [[4]]. Clinical documentation however, can consume almost as much time as does direct patient care [[5]].
In the late 1960s, Weed proposed a problem-oriented approach to improving the structure and content of clinical documentation [[6]]. In the 1970s Weed developed and implemented PROMIS (the Problem-Oriented Medical Information System) at the University of Vermont, USA [[7]]. Over the next several decades, worldwide adoption of various forms of the problem-oriented approach and problem-oriented medical records (POMRs) occurred [[8]–[9]].
Technical and organizational aspects influence user satisfaction with EHR systems [[10]]. Especially due to recent government-mandated EHR rollouts in the USA, many physicians’ user satisfaction with EHR technology has decreased over time. Clinicians now raise concerns about the negative impact of EHRs on the quality of care [[11]–[14]]. The report “Health IT and patient safety: building safer systems for better care”, from the US Institute of Medicine, describes the potential benefits and risks of health IT [[15]]. A report from AmericanEHR and the American Medical Association (AMA) in 2014 indicated that over the previous five years, more physicians reported being dissatisfied or very dissatisfied with their EHR systems [[16]]. Due to EHR-related safety concerns, implementations should include tools to monitor and learn from experience [[17]]. Additional research suggests potential EHR disadvantages including high cost, disruption to workflows, inefficiency, and privacy concerns [[18], [19]].
The ISO defined usability as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use [[20]]. Usability of EHRs contributes to efficiency and patient safety [[21]]. In the end, clinicians must be able to effectively and efficiently complete their administrative work. If not, they will find workarounds, use the system in a different way than intended, or even refuse to use it with potentially substantial consequences. Therefore, research on usability and satisfaction of users, and in particular of physicians, is of utmost importance.
This study reports an evaluation of the use, usability, and physician satisfaction of a locally developed problem-oriented clinical notes application that replaced paper-based records in a large Dutch university medical center.
1.1 Setting
In a university-based Dutch Medical Center, the transition from paper-based clinical notes to digital clinical notes began in 2007. This 960-bed hospital serves as a referral centre for a population of approximately 2.5 million people. In 2013, the medical center had nearly 10,000 (7,227 FTE) employees of which approximately 1,000 were physicians.
2. Objectives
This research on the new problem-oriented note application: (1) evaluated system use over time, (2) evaluated usability and physician satisfaction, and (3) examined the effects of sex, age, professional experience, training hours, medical specialty, and intensity of use, on physician satisfaction.
3. Methods
3.1 Design paradigm and functionality of the clinical notes application
Authors developed an HL7 v3-based model using the conceptualization of Weed’s POMR, and the Subjective, Objective, Assessment, Plan (SOAP) headings to classify elements [[22]]. From this model, we developed a clinical notes application as a key component of the hospital homegrown EHR. This application is generic for all medical specialties. The application is primarily intended for physicians, but also other health care professionals can benefit from additional specialty-specific discrete registration forms such as: pain score, checklists, drawings, intake, growth parameters, et cetera. Authors used the application’s database, an application event log file, and a questionnaire to evaluate various aspects of system use.
3.1.1 Definitions
The application used the terms “concern”, “condition”, and “assessment” to avoid the more ambiguous terms “problem” and “diagnosis”. Types of diagnoses range from early and indefinite to final and definitive. Early diagnoses may be vague, or even expressed in natural language rather than coded. In “Modeling problem-oriented clinical notes” we presented the underlying data model [[22]]. The concept “concern” entails as any piece of information about the patient that needs attention. The concept “condition” encompasses a disease, disorder, injury, trauma, or other health-related state, such as pregnancy, aging, or stress. The concept “assessment” represents a clinician’s conclusions and working assumptions about a condition that will guide treatment of the patient.
3.1.2 Progress notes and the condition list
The application’s format for progress notes is generic for all medical specialties. A list with the medical conditions acts as the central starting point for organizing progress notes. The condition list is automatically generated from the progress notes field Assessment-Diag/Hyp. An option exists for predefined template-based notes that minimize typing. Users can link a note to one or more medical conditions by clicking the checkbox to the left of a condition. See ►[Figure 1] for a screenshot of the progress note entry screen.
Fig. 1 Screenshot of the progress note tab within the clinical notes application with the problem list (medical conditions) always in view.
To guarantee consistency, one cannot directly enter data into the medical condition list. Diagnoses (as Assessment entries) can be entered as free text or ICD10-codes. Different font properties differentiate the status of each condition. For instance, “ruled out” items have strikethrough notations and “resolved” items are gray. Progress note changes are tracked. Different views are possible such as chronological and by medical condition. The application provides options for computer generated summaries and letter generation.
3.2 Implementation of the clinical notes application
The implementation was done via an incremental approach starting in 2010. When a department indicated their readiness to go-live, they were scheduled for implementation, usually within six months. Implementation consisted of three to five weeks of introduction, eleven to fourteen weeks of preparation, two weeks of education, one week of go-live, and four weeks of aftercare and evaluation. Physicians estimated to have low digital skills received special attention during the transition. By the end of 2011, all medical specialties were using the application.
3.3 Application event log file
A separate log file automatically recorded all user-system interactions (such as mouse clicks and application events) within the clinical notes application. The complete log file was available on a Microsoft SQL server, and analyzed and queried with MS SQL management studio. The event logging began with the first go-live. Each record of the log file contains items such as patient identification, user identification, user role, performed action, and a date time stamp, but no medical data. A progress note as a whole was counted as one note at the time it was saved.
Two external professional testers validated the log file facility. They evaluated all unique log entries with the corresponding user actions and application events. Reading the log file, they could replay all user actions with the corresponding application events. The testers wrote a manual describing how to read the log file, and how to understand all performed user actions and events within the application.
3.4 EHR database
With an EHR database administrator, authors could query the clinical notes database. These queries provided more detailed information for analysis of some parts of the progress notes. In several cases, authors also validated the outcome from the log file with the actual data of the database. For example, the number of progress notes can be extracted from both the log file as well as from the EHR database.
3.5 Survey
Physicians received an on-line survey questionnaires regarding system usage and opinions. The questionnaire included questions about user education, professional experience, use, and usability. Because authors wanted to link answers from the survey to data from the log file, the survey was not anonymous. Permission to do so was granted by the privacy officer as a representative of the board of directors.
The questions used to determine user satisfaction with the application usability were taken from the Computer System Usability Questionnaire (CSUQ) from IBM [[23]]. The CSUQ was developed to assess overall user satisfaction with system usability using Likert scales. Our survey used a five-point Likert scale ranging from strongly disagree (1) to strongly agree (5). We pilot tested the questionnaire using several physicians to assess intelligibility and clarity. For the actual survey, we selected physicians who had written at least 200 progress notes in 2012. After excluding physicians who were not able to respond because of extended leave or retirement, the total population was 700. Selected physicians were invited to take part in the survey by email at the end of April 2013. A first reminder was sent two weeks later followed by a second reminder at the end of May. Because the application was browser-based and survey recipients were selected as experienced users, seven of the nineteen CSUQ questions were less appropriate for our situation, and therefore not used.
3.6 Analysis
We analyzed the log file and the EHR database using MS SQL management studio. The log file was frequently used for analyzing the use of the application, for troubleshooting, and for analysis of reported usability problems. Statistical analyses were performed with IBM SPSS Statistics 22.0 (SPSS Inc, Chicago, IL). Descriptive statistics were used to show total use and usage over time for medical specialties, users, and patients. In particular, the focus was on the use of the application by physicians, which were identified in the log file using the “responsible_provider_type” code. All selected physicians for the survey were categorized by their specialty in the following eight groups; 1-anesthesiology, 2-cardio-lung, 3-internists, 4-neurology, 5-pediatrics, 6-supporting (radiology, radiotherapy, and nuclear medicine), 7-surgeons, and 8-others. We also categorize physicians by the number of written notes. We performed a non-respondent analysis, using chi-squared tests to study differences between responders and non-responders, with respect to the medical specialty, and the number of written progress notes, which were extracted from the log file. The internal consistency of items in the questionnaire was measured using Cronbach’s Alpha. Univariate analysis of variance was used to study the effects of sex, age, experience, training, and group on overall satisfaction. The significance level was set at P<0.05. Pearson correlation was used to study the correlation between intensity of use and satisfaction scores. For the intensity of use, we looked at three aspects, total actions performed, total number of patients for whom actions are performed, and duration of use in months. Point of saturation was determined with the log file at three levels, by medical specialties, by unique users, and for unique patients, at the moment where the use of the application no longer increases.
4. Results
Data from January 2010 until October 2013 included 2,887,546 progress notes written on 219,755 patients. The 1,793 physicians wrote about 70% of all notes (2,023,546) on 201,964 patients; 30% of all notes (864,371) were written by 3,272 physician assistants, nurses, and other non-physician staff. In total, 1,654 physicians made 373,124 progress note changes.
►[Figure 2] illustrates use of the application by all users graphically for each month. Saturation of use by individual medical specialties occurred in the second half of 2011. Saturation of unique users who saved data within the application occurred by mid 2012. The number of unique patients with one or more progress notes plateaued in mid 2012. From 2013, each month 76,498 notes were entered by 970 physicians. We observed a large variation in use. Of all patients, 84% had 1 to 20 notes, while 16% of the patients had >20 notes. In 2012, the mean written notes per physician was 648, with a minimum of 1, and a maximum of 4373, and a standard deviation of 700. In 2012, each physician used the application on a mean of 288 patients per year, with a minimum of 1, and a maximum of 1776, and a standard deviation of 306. The maximum number of notes for one patient was 1,418. Of the total of 260,872 patients that had a record in the clinical notes application, 41,117 patients had no progress notes. Most of these patients had been migrated from paper records to the EHR with only allergy and medical history data. The option to code diagnoses with ICD 10 was rarely used.
Fig. 2 Usage of the clinical notes application for each month from January 2010 until October 2013 by medical specialties, unique users, and unique patients.
4.1 Survey results
The 700 selected physicians, returned 263 completed survey questionnaires. Five of the respondents were excluded due to incompleteness. ►[Table 1] shows the survey respondent characteristics. Analysis of the questionnaire thus involved 258 respondents (response rate of 37%). The median time to complete the questionnaire was 5 minutes, 23 seconds.
Table 1
Survey respondent characteristics of all physicians n=258
|
Physician characteristics
|
Frequency
|
%
|
|
Sex
|
|
Male
|
119
|
46
|
|
Female
|
139
|
54
|
|
Age in years
|
|
25–35
|
111
|
43
|
|
36–45
|
69
|
27
|
|
46–55
|
53
|
21
|
|
>55
|
25
|
10
|
|
Years in practice as physician (professional experience)
|
|
<5
|
57
|
22
|
|
5–10
|
77
|
30
|
|
11–15
|
40
|
16
|
|
16–20
|
23
|
9
|
|
>20
|
61
|
24
|
|
Aware of training possibilities
|
|
Yes
|
134
|
52
|
|
No
|
124
|
48
|
|
Training hours before use of the application
|
|
<1
|
110
|
43
|
|
1–4
|
115
|
45
|
|
>4
|
33
|
13
|
|
Number of notes
|
|
200–500
|
58
|
22
|
|
501–1000
|
82
|
32
|
|
1001–1500
|
56
|
22
|
|
1501–2000
|
33
|
13
|
|
2001–2500
|
18
|
7
|
|
>2500
|
11
|
4
|
No differences existed between respondents and non-respondents with respect to specialty (p=0.635) or the number of written progress notes (p=0.329).
Almost all users described themselves as experienced with the system (►[Table 2]). The 12 CSUQ questions enabled physicians to express their satisfaction with the use of the clinical notes application; the overall satisfaction score was 3.2. More specific responses included U1 through U7 for system usefulness (3.5) and Q1 through Q4 for system quality (2.8). The Cronbach’s Alpha for usefulness. i.e., U1 through U7, was 0.880, and for quality, i.e., Q1 through Q4, 0.787. Analysis discovered no significant effects on overall satisfaction by gender, age in years, years in practice as physician, or number of hours of training before use of the application. For the specialty group variable, we found statistically significant differences: anesthesiology had the lowest (2.8) and pediatrics the highest (3.5) overall satisfaction score (p=0.013).
Table 2
Usability satisfaction questionnaire and answers from all 258 physicians who responded
|
|
Strongly disagree
|
Disagree
|
Neutral
|
Agree
|
Strongly agree
|
Mean
|
|
Characteristic
|
|
Experience with the system
|
|
|
|
|
|
|
|
E1– I am an experienced user of the clinical notes application
|
5
|
8
|
26
|
125
|
94
|
4.1
|
|
E2– I am experienced with the possibilities of recording progress notes
|
6
|
7
|
19
|
133
|
93
|
4.2
|
|
Usefulness
|
|
U1– Overall, I am satisfied with how easy it is to use this system
|
13
|
47
|
72
|
115
|
11
|
3.2
|
|
U2– It is simple to use this system
|
3
|
13
|
32
|
183
|
27
|
3.8
|
|
U3– I can effectively complete my work using this system
|
16
|
49
|
70
|
111
|
12
|
3.2
|
|
U4– I am able to complete my work quickly using this system
|
8
|
22
|
33
|
162
|
33
|
3.7
|
|
U5– I am able to efficiently complete my work using this system
|
11
|
52
|
71
|
109
|
15
|
3.3
|
|
U6– I feel comfortable using this system
|
6
|
30
|
81
|
126
|
15
|
3.4
|
|
U7– It was easy to learn to use this system
|
3
|
15
|
76
|
143
|
21
|
3.6
|
|
Mean usefulness score
|
9
|
33
|
62
|
136
|
19
|
3.5
|
|
Quality (interface and quality)
|
|
Q1– Whenever I make a mistake using the system, I can fix it easily
|
7
|
41
|
44
|
145
|
21
|
3.5
|
|
Q2– It is easy to find the information I need
|
50
|
99
|
69
|
37
|
3
|
2.4
|
|
Q3– The organization of information on the screens is clear
|
34
|
105
|
72
|
39
|
8
|
2.5
|
|
Q4– This system has all the functions and capabilities I expect it to have
|
23
|
103
|
87
|
41
|
4
|
2.6
|
|
Mean quality score
|
29
|
87
|
68
|
66
|
9
|
2.8
|
|
Overall satisfaction
|
|
O1– Overall, I am satisfied with this system
|
19
|
70
|
94
|
67
|
8
|
2.9
|
|
Overall usability satisfaction score
|
16
|
55
|
67
|
106
|
15
|
3.2
|
There was a high correlation between the overall satisfaction score and users’ usefulness and quality ratings. No correlations were noted between overall satisfaction score and self-reported experience with the system, intensity of use, total actions performed, total number of patients for whom actions were performed, and duration of use in months.
5. Discussion
The current study evaluated implementation of a problem oriented clinical documentation that replaced paper-based records. We studied system use and physician satisfaction in a retrospective analysis over a four year period.
5.1 Principal findings
Key results of interest to other institutions include:
-
A gradual rollout with an incremental approach that enabled each hospital unit to indicate their readiness led to an acceptable overall physician satisfaction score of 3.2 from 1 (highly dissatisfied) to 5 (highly satisfied).
-
It took two years for the complete transition from paper to digital clinical notes.
-
A log file is of great value, for detailed monitoring, in the discussion with physicians, and to analyze system use patterns.
-
As might be expected, significant differences in satisfaction occurred between medical specialties.
-
Specific aspects of our problem-oriented system that authors believe facilitated adoption include: the hierarchical and multidisciplinary features of the problem list whereby admitting, differential, intermediate and final diagnoses are visible at a glance; linking capabilities whereby progress notes can simply be linked to none, one or more problems; and, a function that auto-populates the problem list using the physician-entered conditions and assessments from the content of the notes to guarantee consistency.
-
While system implementers communicated extensively via email, intranet, and regular staff meetings (quick reference cards, videos, on-line help, walk-in sessions, dos and don’ts, and face-to-face support), the survey showed that 48 percent of the physicians were unaware of training opportunities. This should be a key priority for others undertaking similar implementations.
5.2 Relation to other studies
Comparison of physician satisfaction and EHR usability with other studies is difficult. First, according to Nielsen, overall user satisfaction with a system is often seen as part of usability [[24]]. Second, user expectations for an ideal system have varied over time, possibly due to greater levels of users’ use of computers in other everyday settings. The overall EHR system satisfaction in past studies gave high ratings of up to 85 percent [[25]], while more recent reports include 67 percent being dissatisfied with system functionality [[26]]. In 2013, The American College of Physicians (ACP) and America-nEHR Partners revealed that user satisfaction with electronic health records had decreased compared with 2010 [[27]]. Wide variability exists in EHR use and satisfaction with key functions of the EHRs, such as clinical documentation and problem list usage [[19]]. However, clinical notes and diagnosis functionality, which is the main subject of this study, seems to be a good predictor independently of the EHR system related to physician satisfaction [[28]]. A recent systematic review of published EHR usability evaluations reported the lack of a formal and standardized ways of reporting results [[29]].
Other research with similar scope and focus also reported an implementation transition period of two years [[30]]. Previous research suggested that the POMR and SOAP approach should be supplemented with chronological, source- and task-oriented views [[31]]. We have taken this into account and put significant effort into the organization and presentation of information. Nevertheless, there still is room for improvement. Other research that measured physician satisfaction after 2010, with a questionnaire between one and three years after EHR implementation, also using a Likert scale from 1 to 5, reported varying satisfaction levels [[32]–[34]].
A review of EHR implementations reporting benefits and issues showed that high satisfaction is related to a number of factors, such as support provided when problems occurred and usability [[35]]. Physicians seem to be more satisfied with an easy to use EHR [[36]].
Finding the right balance between discrete data and free text is difficult given the tension between structure and flexibility in documentation [[37]]. Besides this tension, there is also a wide range of possible barriers and resistance from physicians to implement an EHR [[38]].
Development and successful adoption of clinical documentation tools also depend on the integration possibilities [[39]]. Specific for anesthesia care, research shows that difficult integration with EHRs is one of the most cited problems [[40]].
Recent studies show that many physicians are frustrated by modern EHR systems in their current form. Reports indicate that EHRs lack user-friendliness, are clunky and time-consuming, and reduce face-to-face time with the patient [[6]]. Our studied, generic easy-to-use application with predominantly free-text fields, scores sufficient in general, but lacks secondary use of EHR data. Many data can be registered discretely in modern EHR systems, but the way to register and to gain an overview of data is often unnatural and time-consuming. In a longitudinal survey of physicians lasting two years, satisfaction following implementation of modern commercial EHR systems dropped and remained below baseline [[41]]. There is still an enormous room for improvement.
5.3 Weaknesses of the study
The survey response rate (37%) is relatively low; however, our non-responders analysis did not reveal any difference between respondents and non-respondents regarding specialism and the number of notes written. Authors did not further analyze subgroups of non-respondents to see if their survey responses would have varied significantly from those reported by respondents. The questionnaire was not anonymous, which may have biased respondents to answer consistent with how they think the researcher wanted them to respond [[42]].
Satisfaction was measured at a single moment in time. Also, there may have been other factors that affected satisfaction that are not measured. Comparing use, usability and satisfaction is difficult given the many variables: how the EHR product is configured and customized; organization-specific issues such as guidelines; the nature of user training and education; and support for and timing of implementation can all affect the outcomes. There are many EHR systems available. Functionalities of those systems varies, as does the use of those functionalities [[43]]. We did not study the relation between satisfaction and functionality directly. Functional weaknesses of our system include the absence of an integrated order entry system and a clinical decision support system. In our study, we did not measure job satisfaction but it is likely that satisfaction with EHRs is associated with job satisfaction [[44]]. Other factors that influence satisfaction are the pre-implementation, technological environment, and the approach to implementations [[45]].
6. Conclusions
With a generic, relatively easy to use clinical notes application based on the ideas of POMR and SOAP, and an incremental implementation, a Dutch University Medical Center managed the transition from paper-based notes to digital notes for all medical specialties in two years. The application was in general appreciated with a neutral satisfaction score. The manuscript has listed above useful lessons for other implementors.
Clinical Relevance Statement
Clinical Relevance Statement
Following the implementation of a problem-oriented clinical notes application as a component of an electronic health record, we found a neutral to sufficient satisfaction with no significant differences between sex, age, professional experience, or training hours.
We found no correlation between user satisfaction and intensity of use or experience with the system.
How elements of EHRs are used and which aspects influence user satisfaction are very important considering the high costs of EHRs and the potential benefits in quality, safety, and efficiency, wherein the use of the log file is proved to be an essential tool.
Multiple choice question
Which instrument provides great advantages in the implementation and follow-up of a clinical documentation system?
-
Detailed planning
-
Adequate training
-
Log file
-
Incremental approach
All of the things are of great importance. but objective detailed monitoring through the use of a log file (answer C) provides great advantages.
A huge number of people are involved with the implementation of such a system, so detailed planning is essential. Also, staff must be well prepared for the challenges of working with the new system where adequate training is indispensable. An incremental approach may be useful but depends on many variables within the organization.
