Keywords
Electronic health records and systems - problem-oriented medical records - evaluation
- satisfaction - physician
1. Background and Significance
1. Background and Significance
Implementation of electronic health records (EHRs) to replace patients’ paper charts
can benefit health care delivery [[1], [2]]. Past research provides good evidence that EHRs can improve quality, safety, and
efficiency [[3]]. Clinical documentation is a key component of EHRs. Clinical documentation should
support patient care and improve clinical outcomes through enhanced communication
[[4]]. Clinical documentation however, can consume almost as much time as does direct
patient care [[5]].
In the late 1960s, Weed proposed a problem-oriented approach to improving the structure
and content of clinical documentation [[6]]. In the 1970s Weed developed and implemented PROMIS (the Problem-Oriented Medical
Information System) at the University of Vermont, USA [[7]]. Over the next several decades, worldwide adoption of various forms of the problem-oriented
approach and problem-oriented medical records (POMRs) occurred [[8]–[9]].
Technical and organizational aspects influence user satisfaction with EHR systems
[[10]]. Especially due to recent government-mandated EHR rollouts in the USA, many physicians’
user satisfaction with EHR technology has decreased over time. Clinicians now raise
concerns about the negative impact of EHRs on the quality of care [[11]–[14]]. The report “Health IT and patient safety: building safer systems for better care”,
from the US Institute of Medicine, describes the potential benefits and risks of health
IT [[15]]. A report from AmericanEHR and the American Medical Association (AMA) in 2014 indicated
that over the previous five years, more physicians reported being dissatisfied or
very dissatisfied with their EHR systems [[16]]. Due to EHR-related safety concerns, implementations should include tools to monitor
and learn from experience [[17]]. Additional research suggests potential EHR disadvantages including high cost,
disruption to workflows, inefficiency, and privacy concerns [[18], [19]].
The ISO defined usability as the extent to which a product can be used by specified
users to achieve specified goals with effectiveness, efficiency and satisfaction in
a specified context of use [[20]]. Usability of EHRs contributes to efficiency and patient safety [[21]]. In the end, clinicians must be able to effectively and efficiently complete their
administrative work. If not, they will find workarounds, use the system in a different
way than intended, or even refuse to use it with potentially substantial consequences.
Therefore, research on usability and satisfaction of users, and in particular of physicians,
is of utmost importance.
This study reports an evaluation of the use, usability, and physician satisfaction
of a locally developed problem-oriented clinical notes application that replaced paper-based
records in a large Dutch university medical center.
1.1 Setting
In a university-based Dutch Medical Center, the transition from paper-based clinical
notes to digital clinical notes began in 2007. This 960-bed hospital serves as a referral
centre for a population of approximately 2.5 million people. In 2013, the medical
center had nearly 10,000 (7,227 FTE) employees of which approximately 1,000 were physicians.
2. Objectives
This research on the new problem-oriented note application: (1) evaluated system use
over time, (2) evaluated usability and physician satisfaction, and (3) examined the
effects of sex, age, professional experience, training hours, medical specialty, and
intensity of use, on physician satisfaction.
3. Methods
3.1 Design paradigm and functionality of the clinical notes application
Authors developed an HL7 v3-based model using the conceptualization of Weed’s POMR,
and the Subjective, Objective, Assessment, Plan (SOAP) headings to classify elements
[[22]]. From this model, we developed a clinical notes application as a key component
of the hospital homegrown EHR. This application is generic for all medical specialties.
The application is primarily intended for physicians, but also other health care professionals
can benefit from additional specialty-specific discrete registration forms such as:
pain score, checklists, drawings, intake, growth parameters, et cetera. Authors used
the application’s database, an application event log file, and a questionnaire to
evaluate various aspects of system use.
3.1.1 Definitions
The application used the terms “concern”, “condition”, and “assessment” to avoid the
more ambiguous terms “problem” and “diagnosis”. Types of diagnoses range from early
and indefinite to final and definitive. Early diagnoses may be vague, or even expressed
in natural language rather than coded. In “Modeling problem-oriented clinical notes”
we presented the underlying data model [[22]]. The concept “concern” entails as any piece of information about the patient that
needs attention. The concept “condition” encompasses a disease, disorder, injury,
trauma, or other health-related state, such as pregnancy, aging, or stress. The concept
“assessment” represents a clinician’s conclusions and working assumptions about a
condition that will guide treatment of the patient.
3.1.2 Progress notes and the condition list
The application’s format for progress notes is generic for all medical specialties.
A list with the medical conditions acts as the central starting point for organizing
progress notes. The condition list is automatically generated from the progress notes
field Assessment-Diag/Hyp. An option exists for predefined template-based notes that
minimize typing. Users can link a note to one or more medical conditions by clicking
the checkbox to the left of a condition. See ►[Figure 1] for a screenshot of the progress note entry screen.
Fig. 1 Screenshot of the progress note tab within the clinical notes application with the
problem list (medical conditions) always in view.
To guarantee consistency, one cannot directly enter data into the medical condition
list. Diagnoses (as Assessment entries) can be entered as free text or ICD10-codes.
Different font properties differentiate the status of each condition. For instance,
“ruled out” items have strikethrough notations and “resolved” items are gray. Progress
note changes are tracked. Different views are possible such as chronological and by
medical condition. The application provides options for computer generated summaries
and letter generation.
3.2 Implementation of the clinical notes application
The implementation was done via an incremental approach starting in 2010. When a department
indicated their readiness to go-live, they were scheduled for implementation, usually
within six months. Implementation consisted of three to five weeks of introduction,
eleven to fourteen weeks of preparation, two weeks of education, one week of go-live,
and four weeks of aftercare and evaluation. Physicians estimated to have low digital
skills received special attention during the transition. By the end of 2011, all medical
specialties were using the application.
3.3 Application event log file
A separate log file automatically recorded all user-system interactions (such as mouse
clicks and application events) within the clinical notes application. The complete
log file was available on a Microsoft SQL server, and analyzed and queried with MS
SQL management studio. The event logging began with the first go-live. Each record
of the log file contains items such as patient identification, user identification,
user role, performed action, and a date time stamp, but no medical data. A progress
note as a whole was counted as one note at the time it was saved.
Two external professional testers validated the log file facility. They evaluated
all unique log entries with the corresponding user actions and application events.
Reading the log file, they could replay all user actions with the corresponding application
events. The testers wrote a manual describing how to read the log file, and how to
understand all performed user actions and events within the application.
3.4 EHR database
With an EHR database administrator, authors could query the clinical notes database.
These queries provided more detailed information for analysis of some parts of the
progress notes. In several cases, authors also validated the outcome from the log
file with the actual data of the database. For example, the number of progress notes
can be extracted from both the log file as well as from the EHR database.
3.5 Survey
Physicians received an on-line survey questionnaires regarding system usage and opinions.
The questionnaire included questions about user education, professional experience,
use, and usability. Because authors wanted to link answers from the survey to data
from the log file, the survey was not anonymous. Permission to do so was granted by
the privacy officer as a representative of the board of directors.
The questions used to determine user satisfaction with the application usability were
taken from the Computer System Usability Questionnaire (CSUQ) from IBM [[23]]. The CSUQ was developed to assess overall user satisfaction with system usability
using Likert scales. Our survey used a five-point Likert scale ranging from strongly
disagree (1) to strongly agree (5). We pilot tested the questionnaire using several
physicians to assess intelligibility and clarity. For the actual survey, we selected
physicians who had written at least 200 progress notes in 2012. After excluding physicians
who were not able to respond because of extended leave or retirement, the total population
was 700. Selected physicians were invited to take part in the survey by email at the
end of April 2013. A first reminder was sent two weeks later followed by a second
reminder at the end of May. Because the application was browser-based and survey recipients
were selected as experienced users, seven of the nineteen CSUQ questions were less
appropriate for our situation, and therefore not used.
3.6 Analysis
We analyzed the log file and the EHR database using MS SQL management studio. The
log file was frequently used for analyzing the use of the application, for troubleshooting,
and for analysis of reported usability problems. Statistical analyses were performed
with IBM SPSS Statistics 22.0 (SPSS Inc, Chicago, IL). Descriptive statistics were
used to show total use and usage over time for medical specialties, users, and patients.
In particular, the focus was on the use of the application by physicians, which were
identified in the log file using the “responsible_provider_type” code. All selected
physicians for the survey were categorized by their specialty in the following eight
groups; 1-anesthesiology, 2-cardio-lung, 3-internists, 4-neurology, 5-pediatrics,
6-supporting (radiology, radiotherapy, and nuclear medicine), 7-surgeons, and 8-others.
We also categorize physicians by the number of written notes. We performed a non-respondent
analysis, using chi-squared tests to study differences between responders and non-responders,
with respect to the medical specialty, and the number of written progress notes, which
were extracted from the log file. The internal consistency of items in the questionnaire
was measured using Cronbach’s Alpha. Univariate analysis of variance was used to study
the effects of sex, age, experience, training, and group on overall satisfaction.
The significance level was set at P<0.05. Pearson correlation was used to study the
correlation between intensity of use and satisfaction scores. For the intensity of
use, we looked at three aspects, total actions performed, total number of patients
for whom actions are performed, and duration of use in months. Point of saturation
was determined with the log file at three levels, by medical specialties, by unique
users, and for unique patients, at the moment where the use of the application no
longer increases.
4. Results
Data from January 2010 until October 2013 included 2,887,546 progress notes written
on 219,755 patients. The 1,793 physicians wrote about 70% of all notes (2,023,546)
on 201,964 patients; 30% of all notes (864,371) were written by 3,272 physician assistants,
nurses, and other non-physician staff. In total, 1,654 physicians made 373,124 progress
note changes.
►[Figure 2] illustrates use of the application by all users graphically for each month. Saturation
of use by individual medical specialties occurred in the second half of 2011. Saturation
of unique users who saved data within the application occurred by mid 2012. The number
of unique patients with one or more progress notes plateaued in mid 2012. From 2013,
each month 76,498 notes were entered by 970 physicians. We observed a large variation
in use. Of all patients, 84% had 1 to 20 notes, while 16% of the patients had >20
notes. In 2012, the mean written notes per physician was 648, with a minimum of 1,
and a maximum of 4373, and a standard deviation of 700. In 2012, each physician used
the application on a mean of 288 patients per year, with a minimum of 1, and a maximum
of 1776, and a standard deviation of 306. The maximum number of notes for one patient
was 1,418. Of the total of 260,872 patients that had a record in the clinical notes
application, 41,117 patients had no progress notes. Most of these patients had been
migrated from paper records to the EHR with only allergy and medical history data.
The option to code diagnoses with ICD 10 was rarely used.
Fig. 2 Usage of the clinical notes application for each month from January 2010 until October
2013 by medical specialties, unique users, and unique patients.
4.1 Survey results
The 700 selected physicians, returned 263 completed survey questionnaires. Five of
the respondents were excluded due to incompleteness. ►[Table 1] shows the survey respondent characteristics. Analysis of the questionnaire thus
involved 258 respondents (response rate of 37%). The median time to complete the questionnaire
was 5 minutes, 23 seconds.
Table 1
Survey respondent characteristics of all physicians n=258
|
Physician characteristics
|
Frequency
|
%
|
|
Sex
|
|
Male
|
119
|
46
|
|
Female
|
139
|
54
|
|
Age in years
|
|
25–35
|
111
|
43
|
|
36–45
|
69
|
27
|
|
46–55
|
53
|
21
|
|
>55
|
25
|
10
|
|
Years in practice as physician (professional experience)
|
|
<5
|
57
|
22
|
|
5–10
|
77
|
30
|
|
11–15
|
40
|
16
|
|
16–20
|
23
|
9
|
|
>20
|
61
|
24
|
|
Aware of training possibilities
|
|
Yes
|
134
|
52
|
|
No
|
124
|
48
|
|
Training hours before use of the application
|
|
<1
|
110
|
43
|
|
1–4
|
115
|
45
|
|
>4
|
33
|
13
|
|
Number of notes
|
|
200–500
|
58
|
22
|
|
501–1000
|
82
|
32
|
|
1001–1500
|
56
|
22
|
|
1501–2000
|
33
|
13
|
|
2001–2500
|
18
|
7
|
|
>2500
|
11
|
4
|
No differences existed between respondents and non-respondents with respect to specialty
(p=0.635) or the number of written progress notes (p=0.329).
Almost all users described themselves as experienced with the system (►[Table 2]). The 12 CSUQ questions enabled physicians to express their satisfaction with the
use of the clinical notes application; the overall satisfaction score was 3.2. More
specific responses included U1 through U7 for system usefulness (3.5) and Q1 through
Q4 for system quality (2.8). The Cronbach’s Alpha for usefulness. i.e., U1 through
U7, was 0.880, and for quality, i.e., Q1 through Q4, 0.787. Analysis discovered no
significant effects on overall satisfaction by gender, age in years, years in practice
as physician, or number of hours of training before use of the application. For the
specialty group variable, we found statistically significant differences: anesthesiology
had the lowest (2.8) and pediatrics the highest (3.5) overall satisfaction score (p=0.013).
Table 2
Usability satisfaction questionnaire and answers from all 258 physicians who responded
|
Strongly disagree
|
Disagree
|
Neutral
|
Agree
|
Strongly agree
|
Mean
|
|
Characteristic
|
|
Experience with the system
|
|
|
|
|
|
|
|
E1– I am an experienced user of the clinical notes application
|
5
|
8
|
26
|
125
|
94
|
4.1
|
|
E2– I am experienced with the possibilities of recording progress notes
|
6
|
7
|
19
|
133
|
93
|
4.2
|
|
Usefulness
|
|
U1– Overall, I am satisfied with how easy it is to use this system
|
13
|
47
|
72
|
115
|
11
|
3.2
|
|
U2– It is simple to use this system
|
3
|
13
|
32
|
183
|
27
|
3.8
|
|
U3– I can effectively complete my work using this system
|
16
|
49
|
70
|
111
|
12
|
3.2
|
|
U4– I am able to complete my work quickly using this system
|
8
|
22
|
33
|
162
|
33
|
3.7
|
|
U5– I am able to efficiently complete my work using this system
|
11
|
52
|
71
|
109
|
15
|
3.3
|
|
U6– I feel comfortable using this system
|
6
|
30
|
81
|
126
|
15
|
3.4
|
|
U7– It was easy to learn to use this system
|
3
|
15
|
76
|
143
|
21
|
3.6
|
|
Mean usefulness score
|
9
|
33
|
62
|
136
|
19
|
3.5
|
|
Quality (interface and quality)
|
|
Q1– Whenever I make a mistake using the system, I can fix it easily
|
7
|
41
|
44
|
145
|
21
|
3.5
|
|
Q2– It is easy to find the information I need
|
50
|
99
|
69
|
37
|
3
|
2.4
|
|
Q3– The organization of information on the screens is clear
|
34
|
105
|
72
|
39
|
8
|
2.5
|
|
Q4– This system has all the functions and capabilities I expect it to have
|
23
|
103
|
87
|
41
|
4
|
2.6
|
|
Mean quality score
|
29
|
87
|
68
|
66
|
9
|
2.8
|
|
Overall satisfaction
|
|
O1– Overall, I am satisfied with this system
|
19
|
70
|
94
|
67
|
8
|
2.9
|
|
Overall usability satisfaction score
|
16
|
55
|
67
|
106
|
15
|
3.2
|
There was a high correlation between the overall satisfaction score and users’ usefulness
and quality ratings. No correlations were noted between overall satisfaction score
and self-reported experience with the system, intensity of use, total actions performed,
total number of patients for whom actions were performed, and duration of use in months.
5. Discussion
The current study evaluated implementation of a problem oriented clinical documentation
that replaced paper-based records. We studied system use and physician satisfaction
in a retrospective analysis over a four year period.
5.1 Principal findings
Key results of interest to other institutions include:
-
A gradual rollout with an incremental approach that enabled each hospital unit to
indicate their readiness led to an acceptable overall physician satisfaction score
of 3.2 from 1 (highly dissatisfied) to 5 (highly satisfied).
-
It took two years for the complete transition from paper to digital clinical notes.
-
A log file is of great value, for detailed monitoring, in the discussion with physicians,
and to analyze system use patterns.
-
As might be expected, significant differences in satisfaction occurred between medical
specialties.
-
Specific aspects of our problem-oriented system that authors believe facilitated adoption
include: the hierarchical and multidisciplinary features of the problem list whereby
admitting, differential, intermediate and final diagnoses are visible at a glance;
linking capabilities whereby progress notes can simply be linked to none, one or more
problems; and, a function that auto-populates the problem list using the physician-entered
conditions and assessments from the content of the notes to guarantee consistency.
-
While system implementers communicated extensively via email, intranet, and regular
staff meetings (quick reference cards, videos, on-line help, walk-in sessions, dos
and don’ts, and face-to-face support), the survey showed that 48 percent of the physicians
were unaware of training opportunities. This should be a key priority for others undertaking
similar implementations.
5.2 Relation to other studies
Comparison of physician satisfaction and EHR usability with other studies is difficult.
First, according to Nielsen, overall user satisfaction with a system is often seen
as part of usability [[24]]. Second, user expectations for an ideal system have varied over time, possibly
due to greater levels of users’ use of computers in other everyday settings. The overall
EHR system satisfaction in past studies gave high ratings of up to 85 percent [[25]], while more recent reports include 67 percent being dissatisfied with system functionality
[[26]]. In 2013, The American College of Physicians (ACP) and America-nEHR Partners revealed
that user satisfaction with electronic health records had decreased compared with
2010 [[27]]. Wide variability exists in EHR use and satisfaction with key functions of the
EHRs, such as clinical documentation and problem list usage [[19]]. However, clinical notes and diagnosis functionality, which is the main subject
of this study, seems to be a good predictor independently of the EHR system related
to physician satisfaction [[28]]. A recent systematic review of published EHR usability evaluations reported the
lack of a formal and standardized ways of reporting results [[29]].
Other research with similar scope and focus also reported an implementation transition
period of two years [[30]]. Previous research suggested that the POMR and SOAP approach should be supplemented
with chronological, source- and task-oriented views [[31]]. We have taken this into account and put significant effort into the organization
and presentation of information. Nevertheless, there still is room for improvement.
Other research that measured physician satisfaction after 2010, with a questionnaire
between one and three years after EHR implementation, also using a Likert scale from
1 to 5, reported varying satisfaction levels [[32]–[34]].
A review of EHR implementations reporting benefits and issues showed that high satisfaction
is related to a number of factors, such as support provided when problems occurred
and usability [[35]]. Physicians seem to be more satisfied with an easy to use EHR [[36]].
Finding the right balance between discrete data and free text is difficult given the
tension between structure and flexibility in documentation [[37]]. Besides this tension, there is also a wide range of possible barriers and resistance
from physicians to implement an EHR [[38]].
Development and successful adoption of clinical documentation tools also depend on
the integration possibilities [[39]]. Specific for anesthesia care, research shows that difficult integration with EHRs
is one of the most cited problems [[40]].
Recent studies show that many physicians are frustrated by modern EHR systems in their
current form. Reports indicate that EHRs lack user-friendliness, are clunky and time-consuming,
and reduce face-to-face time with the patient [[6]]. Our studied, generic easy-to-use application with predominantly free-text fields,
scores sufficient in general, but lacks secondary use of EHR data. Many data can be
registered discretely in modern EHR systems, but the way to register and to gain an
overview of data is often unnatural and time-consuming. In a longitudinal survey of
physicians lasting two years, satisfaction following implementation of modern commercial
EHR systems dropped and remained below baseline [[41]]. There is still an enormous room for improvement.
5.3 Weaknesses of the study
The survey response rate (37%) is relatively low; however, our non-responders analysis
did not reveal any difference between respondents and non-respondents regarding specialism
and the number of notes written. Authors did not further analyze subgroups of non-respondents
to see if their survey responses would have varied significantly from those reported
by respondents. The questionnaire was not anonymous, which may have biased respondents
to answer consistent with how they think the researcher wanted them to respond [[42]].
Satisfaction was measured at a single moment in time. Also, there may have been other
factors that affected satisfaction that are not measured. Comparing use, usability
and satisfaction is difficult given the many variables: how the EHR product is configured
and customized; organization-specific issues such as guidelines; the nature of user
training and education; and support for and timing of implementation can all affect
the outcomes. There are many EHR systems available. Functionalities of those systems
varies, as does the use of those functionalities [[43]]. We did not study the relation between satisfaction and functionality directly.
Functional weaknesses of our system include the absence of an integrated order entry
system and a clinical decision support system. In our study, we did not measure job
satisfaction but it is likely that satisfaction with EHRs is associated with job satisfaction
[[44]]. Other factors that influence satisfaction are the pre-implementation, technological
environment, and the approach to implementations [[45]].
6. Conclusions
With a generic, relatively easy to use clinical notes application based on the ideas
of POMR and SOAP, and an incremental implementation, a Dutch University Medical Center
managed the transition from paper-based notes to digital notes for all medical specialties
in two years. The application was in general appreciated with a neutral satisfaction
score. The manuscript has listed above useful lessons for other implementors.
Clinical Relevance Statement
Clinical Relevance Statement
Following the implementation of a problem-oriented clinical notes application as a
component of an electronic health record, we found a neutral to sufficient satisfaction
with no significant differences between sex, age, professional experience, or training
hours.
We found no correlation between user satisfaction and intensity of use or experience
with the system.
How elements of EHRs are used and which aspects influence user satisfaction are very
important considering the high costs of EHRs and the potential benefits in quality,
safety, and efficiency, wherein the use of the log file is proved to be an essential
tool.
Multiple choice question
Which instrument provides great advantages in the implementation and follow-up of
a clinical documentation system?
-
Detailed planning
-
Adequate training
-
Log file
-
Incremental approach
All of the things are of great importance. but objective detailed monitoring through
the use of a log file (answer C) provides great advantages.
A huge number of people are involved with the implementation of such a system, so
detailed planning is essential. Also, staff must be well prepared for the challenges
of working with the new system where adequate training is indispensable. An incremental
approach may be useful but depends on many variables within the organization.
