A hospital-wide transition from paper to digital problem-oriented clinical notes

Summary

Objectives: To evaluate the use, usability, and physician satisfaction of a locally developed problem-oriented clinical notes application that replaced paper-based records in a large Dutch university medical center.

Methods: Using a clinical notes database and an application event log file and a cross-sectional survey of usability, authors retrospectively analyzed system usage for medical specialties, users, and patients over 4 years. A standardized questionnaire measured usability. Authors analyzed the effects of sex, age, professional experience, training hours, and medical specialty on user satisfaction via univariate analysis of variance. Authors also examined the correlation between user satisfaction in relation to users’ intensity of use of the application.

Results: In total 1,793 physicians used the application to record progress notes for 219,755 patients. The overall satisfaction score was 3.2 on a scale from 1 (highly dissatisfied) to 5(highly satisfied). A statistically significant difference occurred in satisfaction by medical specialty, but no statistically significant differences in satisfaction took place by sex, age, professional experience, or training hours. Intensity of system use did not correlate with physician satisfaction.

Conclusions: By two years after the start of the implementation, all medical specialties utilized the clinical notes application. User satisfaction was neutral (3.2 on a 1–5 scale). Authors believe that the significant factors facilitating this transition mirrored success factors reported by other groups: a generic, consistent, and transparent design of the application; intensive collaboration; continuous monitoring; and an incremental rollout.

Citation: Cillessen FHJM, de Vries Robbé PF, Biermans MCJ. A hospital-wide transition from paper to digital problem-oriented clinical note. Appl Clin Inform 2017; 8: 502–514 https://doi.org/10.4338/ACI-2016-08-RA-0137

Keywords

Electronic health records and systems - problem-oriented medical records - evaluation - satisfaction - physician

1. Background and Significance

Implementation of electronic health records (EHRs) to replace patients’ paper charts can benefit health care delivery [[1], [2]]. Past research provides good evidence that EHRs can improve quality, safety, and efficiency [[3]]. Clinical documentation is a key component of EHRs. Clinical documentation should support patient care and improve clinical outcomes through enhanced communication [[4]]. Clinical documentation however, can consume almost as much time as does direct patient care [[5]].

In the late 1960s, Weed proposed a problem-oriented approach to improving the structure and content of clinical documentation [[6]]. In the 1970s Weed developed and implemented PROMIS (the Problem-Oriented Medical Information System) at the University of Vermont, USA [[7]]. Over the next several decades, worldwide adoption of various forms of the problem-oriented approach and problem-oriented medical records (POMRs) occurred [[8]–[9]].

Technical and organizational aspects influence user satisfaction with EHR systems [[10]]. Especially due to recent government-mandated EHR rollouts in the USA, many physicians’ user satisfaction with EHR technology has decreased over time. Clinicians now raise concerns about the negative impact of EHRs on the quality of care [[11]–[14]]. The report “Health IT and patient safety: building safer systems for better care”, from the US Institute of Medicine, describes the potential benefits and risks of health IT [[15]]. A report from AmericanEHR and the American Medical Association (AMA) in 2014 indicated that over the previous five years, more physicians reported being dissatisfied or very dissatisfied with their EHR systems [[16]]. Due to EHR-related safety concerns, implementations should include tools to monitor and learn from experience [[17]]. Additional research suggests potential EHR disadvantages including high cost, disruption to workflows, inefficiency, and privacy concerns [[18], [19]].

The ISO defined usability as the extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency and satisfaction in a specified context of use [[20]]. Usability of EHRs contributes to efficiency and patient safety [[21]]. In the end, clinicians must be able to effectively and efficiently complete their administrative work. If not, they will find workarounds, use the system in a different way than intended, or even refuse to use it with potentially substantial consequences. Therefore, research on usability and satisfaction of users, and in particular of physicians, is of utmost importance.

This study reports an evaluation of the use, usability, and physician satisfaction of a locally developed problem-oriented clinical notes application that replaced paper-based records in a large Dutch university medical center.

1.1 Setting

In a university-based Dutch Medical Center, the transition from paper-based clinical notes to digital clinical notes began in 2007. This 960-bed hospital serves as a referral centre for a population of approximately 2.5 million people. In 2013, the medical center had nearly 10,000 (7,227 FTE) employees of which approximately 1,000 were physicians.

2. Objectives

This research on the new problem-oriented note application: (1) evaluated system use over time, (2) evaluated usability and physician satisfaction, and (3) examined the effects of sex, age, professional experience, training hours, medical specialty, and intensity of use, on physician satisfaction.

3. Methods

3.1 Design paradigm and functionality of the clinical notes application

Authors developed an HL7 v3-based model using the conceptualization of Weed’s POMR, and the Subjective, Objective, Assessment, Plan (SOAP) headings to classify elements [[22]]. From this model, we developed a clinical notes application as a key component of the hospital homegrown EHR. This application is generic for all medical specialties. The application is primarily intended for physicians, but also other health care professionals can benefit from additional specialty-specific discrete registration forms such as: pain score, checklists, drawings, intake, growth parameters, et cetera. Authors used the application’s database, an application event log file, and a questionnaire to evaluate various aspects of system use.

3.1.1 Definitions

The application used the terms “concern”, “condition”, and “assessment” to avoid the more ambiguous terms “problem” and “diagnosis”. Types of diagnoses range from early and indefinite to final and definitive. Early diagnoses may be vague, or even expressed in natural language rather than coded. In “Modeling problem-oriented clinical notes” we presented the underlying data model [[22]]. The concept “concern” entails as any piece of information about the patient that needs attention. The concept “condition” encompasses a disease, disorder, injury, trauma, or other health-related state, such as pregnancy, aging, or stress. The concept “assessment” represents a clinician’s conclusions and working assumptions about a condition that will guide treatment of the patient.

3.1.2 Progress notes and the condition list

The application’s format for progress notes is generic for all medical specialties. A list with the medical conditions acts as the central starting point for organizing progress notes. The condition list is automatically generated from the progress notes field Assessment-Diag/Hyp. An option exists for predefined template-based notes that minimize typing. Users can link a note to one or more medical conditions by clicking the checkbox to the left of a condition. See ►[Figure 1] for a screenshot of the progress note entry screen.

Fig. 1 Screenshot of the progress note tab within the clinical notes application with the problem list (medical conditions) always in view.

To guarantee consistency, one cannot directly enter data into the medical condition list. Diagnoses (as Assessment entries) can be entered as free text or ICD10-codes. Different font properties differentiate the status of each condition. For instance, “ruled out” items have strikethrough notations and “resolved” items are gray. Progress note changes are tracked. Different views are possible such as chronological and by medical condition. The application provides options for computer generated summaries and letter generation.

3.2 Implementation of the clinical notes application

The implementation was done via an incremental approach starting in 2010. When a department indicated their readiness to go-live, they were scheduled for implementation, usually within six months. Implementation consisted of three to five weeks of introduction, eleven to fourteen weeks of preparation, two weeks of education, one week of go-live, and four weeks of aftercare and evaluation. Physicians estimated to have low digital skills received special attention during the transition. By the end of 2011, all medical specialties were using the application.

3.3 Application event log file

A separate log file automatically recorded all user-system interactions (such as mouse clicks and application events) within the clinical notes application. The complete log file was available on a Microsoft SQL server, and analyzed and queried with MS SQL management studio. The event logging began with the first go-live. Each record of the log file contains items such as patient identification, user identification, user role, performed action, and a date time stamp, but no medical data. A progress note as a whole was counted as one note at the time it was saved.

Two external professional testers validated the log file facility. They evaluated all unique log entries with the corresponding user actions and application events. Reading the log file, they could replay all user actions with the corresponding application events. The testers wrote a manual describing how to read the log file, and how to understand all performed user actions and events within the application.

3.4 EHR database

With an EHR database administrator, authors could query the clinical notes database. These queries provided more detailed information for analysis of some parts of the progress notes. In several cases, authors also validated the outcome from the log file with the actual data of the database. For example, the number of progress notes can be extracted from both the log file as well as from the EHR database.

3.5 Survey

Physicians received an on-line survey questionnaires regarding system usage and opinions. The questionnaire included questions about user education, professional experience, use, and usability. Because authors wanted to link answers from the survey to data from the log file, the survey was not anonymous. Permission to do so was granted by the privacy officer as a representative of the board of directors.

The questions used to determine user satisfaction with the application usability were taken from the Computer System Usability Questionnaire (CSUQ) from IBM [[23]]. The CSUQ was developed to assess overall user satisfaction with system usability using Likert scales. Our survey used a five-point Likert scale ranging from strongly disagree (1) to strongly agree (5). We pilot tested the questionnaire using several physicians to assess intelligibility and clarity. For the actual survey, we selected physicians who had written at least 200 progress notes in 2012. After excluding physicians who were not able to respond because of extended leave or retirement, the total population was 700. Selected physicians were invited to take part in the survey by email at the end of April 2013. A first reminder was sent two weeks later followed by a second reminder at the end of May. Because the application was browser-based and survey recipients were selected as experienced users, seven of the nineteen CSUQ questions were less appropriate for our situation, and therefore not used.

3.6 Analysis

We analyzed the log file and the EHR database using MS SQL management studio. The log file was frequently used for analyzing the use of the application, for troubleshooting, and for analysis of reported usability problems. Statistical analyses were performed with IBM SPSS Statistics 22.0 (SPSS Inc, Chicago, IL). Descriptive statistics were used to show total use and usage over time for medical specialties, users, and patients. In particular, the focus was on the use of the application by physicians, which were identified in the log file using the “responsible_provider_type” code. All selected physicians for the survey were categorized by their specialty in the following eight groups; 1-anesthesiology, 2-cardio-lung, 3-internists, 4-neurology, 5-pediatrics, 6-supporting (radiology, radiotherapy, and nuclear medicine), 7-surgeons, and 8-others. We also categorize physicians by the number of written notes. We performed a non-respondent analysis, using chi-squared tests to study differences between responders and non-responders, with respect to the medical specialty, and the number of written progress notes, which were extracted from the log file. The internal consistency of items in the questionnaire was measured using Cronbach’s Alpha. Univariate analysis of variance was used to study the effects of sex, age, experience, training, and group on overall satisfaction. The significance level was set at P<0.05. Pearson correlation was used to study the correlation between intensity of use and satisfaction scores. For the intensity of use, we looked at three aspects, total actions performed, total number of patients for whom actions are performed, and duration of use in months. Point of saturation was determined with the log file at three levels, by medical specialties, by unique users, and for unique patients, at the moment where the use of the application no longer increases.

4. Results

Data from January 2010 until October 2013 included 2,887,546 progress notes written on 219,755 patients. The 1,793 physicians wrote about 70% of all notes (2,023,546) on 201,964 patients; 30% of all notes (864,371) were written by 3,272 physician assistants, nurses, and other non-physician staff. In total, 1,654 physicians made 373,124 progress note changes.

►[Figure 2] illustrates use of the application by all users graphically for each month. Saturation of use by individual medical specialties occurred in the second half of 2011. Saturation of unique users who saved data within the application occurred by mid 2012. The number of unique patients with one or more progress notes plateaued in mid 2012. From 2013, each month 76,498 notes were entered by 970 physicians. We observed a large variation in use. Of all patients, 84% had 1 to 20 notes, while 16% of the patients had >20 notes. In 2012, the mean written notes per physician was 648, with a minimum of 1, and a maximum of 4373, and a standard deviation of 700. In 2012, each physician used the application on a mean of 288 patients per year, with a minimum of 1, and a maximum of 1776, and a standard deviation of 306. The maximum number of notes for one patient was 1,418. Of the total of 260,872 patients that had a record in the clinical notes application, 41,117 patients had no progress notes. Most of these patients had been migrated from paper records to the EHR with only allergy and medical history data. The option to code diagnoses with ICD 10 was rarely used.

Fig. 2 Usage of the clinical notes application for each month from January 2010 until October 2013 by medical specialties, unique users, and unique patients.

4.1 Survey results

The 700 selected physicians, returned 263 completed survey questionnaires. Five of the respondents were excluded due to incompleteness. ►[Table 1] shows the survey respondent characteristics. Analysis of the questionnaire thus involved 258 respondents (response rate of 37%). The median time to complete the questionnaire was 5 minutes, 23 seconds.

Table 1

Survey respondent characteristics of all physicians n=258

Physician characteristics	Frequency	%
Sex
Male	119	46
Female	139	54
Age in years
25–35	111	43
36–45	69	27
46–55	53	21
>55	25	10
Years in practice as physician (professional experience)
<5	57	22
5–10	77	30
11–15	40	16
16–20	23	9
>20	61	24
Aware of training possibilities
Yes	134	52
No	124	48
Training hours before use of the application
<1	110	43
1–4	115	45
>4	33	13
Number of notes
200–500	58	22
501–1000	82	32
1001–1500	56	22
1501–2000	33	13
2001–2500	18	7
>2500	11	4

No differences existed between respondents and non-respondents with respect to specialty (p=0.635) or the number of written progress notes (p=0.329).

Almost all users described themselves as experienced with the system (►[Table 2]). The 12 CSUQ questions enabled physicians to express their satisfaction with the use of the clinical notes application; the overall satisfaction score was 3.2. More specific responses included U1 through U7 for system usefulness (3.5) and Q1 through Q4 for system quality (2.8). The Cronbach’s Alpha for usefulness. i.e., U1 through U7, was 0.880, and for quality, i.e., Q1 through Q4, 0.787. Analysis discovered no significant effects on overall satisfaction by gender, age in years, years in practice as physician, or number of hours of training before use of the application. For the specialty group variable, we found statistically significant differences: anesthesiology had the lowest (2.8) and pediatrics the highest (3.5) overall satisfaction score (p=0.013).

Table 2

Usability satisfaction questionnaire and answers from all 258 physicians who responded

	Strongly disagree	Disagree	Neutral	Agree	Strongly agree	Mean
Characteristic
Experience with the system
E1– I am an experienced user of the clinical notes application	5	8	26	125	94	4.1
E2– I am experienced with the possibilities of recording progress notes	6	7	19	133	93	4.2
Usefulness
U1– Overall, I am satisfied with how easy it is to use this system	13	47	72	115	11	3.2
U2– It is simple to use this system	3	13	32	183	27	3.8
U3– I can effectively complete my work using this system	16	49	70	111	12	3.2
U4– I am able to complete my work quickly using this system	8	22	33	162	33	3.7
U5– I am able to efficiently complete my work using this system	11	52	71	109	15	3.3
U6– I feel comfortable using this system	6	30	81	126	15	3.4
U7– It was easy to learn to use this system	3	15	76	143	21	3.6
Mean usefulness score	9	33	62	136	19	3.5
Quality (interface and quality)
Q1– Whenever I make a mistake using the system, I can fix it easily	7	41	44	145	21	3.5
Q2– It is easy to find the information I need	50	99	69	37	3	2.4
Q3– The organization of information on the screens is clear	34	105	72	39	8	2.5
Q4– This system has all the functions and capabilities I expect it to have	23	103	87	41	4	2.6
Mean quality score	29	87	68	66	9	2.8
Overall satisfaction
O1– Overall, I am satisfied with this system	19	70	94	67	8	2.9
Overall usability satisfaction score	16	55	67	106	15	3.2

There was a high correlation between the overall satisfaction score and users’ usefulness and quality ratings. No correlations were noted between overall satisfaction score and self-reported experience with the system, intensity of use, total actions performed, total number of patients for whom actions were performed, and duration of use in months.

5. Discussion

The current study evaluated implementation of a problem oriented clinical documentation that replaced paper-based records. We studied system use and physician satisfaction in a retrospective analysis over a four year period.

5.1 Principal findings

Key results of interest to other institutions include:

• A gradual rollout with an incremental approach that enabled each hospital unit to indicate their readiness led to an acceptable overall physician satisfaction score of 3.2 from 1 (highly dissatisfied) to 5 (highly satisfied).

• It took two years for the complete transition from paper to digital clinical notes.

• A log file is of great value, for detailed monitoring, in the discussion with physicians, and to analyze system use patterns.

• As might be expected, significant differences in satisfaction occurred between medical specialties.

• Specific aspects of our problem-oriented system that authors believe facilitated adoption include: the hierarchical and multidisciplinary features of the problem list whereby admitting, differential, intermediate and final diagnoses are visible at a glance; linking capabilities whereby progress notes can simply be linked to none, one or more problems; and, a function that auto-populates the problem list using the physician-entered conditions and assessments from the content of the notes to guarantee consistency.

• While system implementers communicated extensively via email, intranet, and regular staff meetings (quick reference cards, videos, on-line help, walk-in sessions, dos and don’ts, and face-to-face support), the survey showed that 48 percent of the physicians were unaware of training opportunities. This should be a key priority for others undertaking similar implementations.

5.2 Relation to other studies

Comparison of physician satisfaction and EHR usability with other studies is difficult. First, according to Nielsen, overall user satisfaction with a system is often seen as part of usability [[24]]. Second, user expectations for an ideal system have varied over time, possibly due to greater levels of users’ use of computers in other everyday settings. The overall EHR system satisfaction in past studies gave high ratings of up to 85 percent [[25]], while more recent reports include 67 percent being dissatisfied with system functionality [[26]]. In 2013, The American College of Physicians (ACP) and America-nEHR Partners revealed that user satisfaction with electronic health records had decreased compared with 2010 [[27]]. Wide variability exists in EHR use and satisfaction with key functions of the EHRs, such as clinical documentation and problem list usage [[19]]. However, clinical notes and diagnosis functionality, which is the main subject of this study, seems to be a good predictor independently of the EHR system related to physician satisfaction [[28]]. A recent systematic review of published EHR usability evaluations reported the lack of a formal and standardized ways of reporting results [[29]].

Other research with similar scope and focus also reported an implementation transition period of two years [[30]]. Previous research suggested that the POMR and SOAP approach should be supplemented with chronological, source- and task-oriented views [[31]]. We have taken this into account and put significant effort into the organization and presentation of information. Nevertheless, there still is room for improvement. Other research that measured physician satisfaction after 2010, with a questionnaire between one and three years after EHR implementation, also using a Likert scale from 1 to 5, reported varying satisfaction levels [[32]–[34]].

A review of EHR implementations reporting benefits and issues showed that high satisfaction is related to a number of factors, such as support provided when problems occurred and usability [[35]]. Physicians seem to be more satisfied with an easy to use EHR [[36]].

Finding the right balance between discrete data and free text is difficult given the tension between structure and flexibility in documentation [[37]]. Besides this tension, there is also a wide range of possible barriers and resistance from physicians to implement an EHR [[38]].

Development and successful adoption of clinical documentation tools also depend on the integration possibilities [[39]]. Specific for anesthesia care, research shows that difficult integration with EHRs is one of the most cited problems [[40]].

Recent studies show that many physicians are frustrated by modern EHR systems in their current form. Reports indicate that EHRs lack user-friendliness, are clunky and time-consuming, and reduce face-to-face time with the patient [[6]]. Our studied, generic easy-to-use application with predominantly free-text fields, scores sufficient in general, but lacks secondary use of EHR data. Many data can be registered discretely in modern EHR systems, but the way to register and to gain an overview of data is often unnatural and time-consuming. In a longitudinal survey of physicians lasting two years, satisfaction following implementation of modern commercial EHR systems dropped and remained below baseline [[41]]. There is still an enormous room for improvement.

5.3 Weaknesses of the study

The survey response rate (37%) is relatively low; however, our non-responders analysis did not reveal any difference between respondents and non-respondents regarding specialism and the number of notes written. Authors did not further analyze subgroups of non-respondents to see if their survey responses would have varied significantly from those reported by respondents. The questionnaire was not anonymous, which may have biased respondents to answer consistent with how they think the researcher wanted them to respond [[42]].

Satisfaction was measured at a single moment in time. Also, there may have been other factors that affected satisfaction that are not measured. Comparing use, usability and satisfaction is difficult given the many variables: how the EHR product is configured and customized; organization-specific issues such as guidelines; the nature of user training and education; and support for and timing of implementation can all affect the outcomes. There are many EHR systems available. Functionalities of those systems varies, as does the use of those functionalities [[43]]. We did not study the relation between satisfaction and functionality directly. Functional weaknesses of our system include the absence of an integrated order entry system and a clinical decision support system. In our study, we did not measure job satisfaction but it is likely that satisfaction with EHRs is associated with job satisfaction [[44]]. Other factors that influence satisfaction are the pre-implementation, technological environment, and the approach to implementations [[45]].

6. Conclusions

With a generic, relatively easy to use clinical notes application based on the ideas of POMR and SOAP, and an incremental implementation, a Dutch University Medical Center managed the transition from paper-based notes to digital notes for all medical specialties in two years. The application was in general appreciated with a neutral satisfaction score. The manuscript has listed above useful lessons for other implementors.

Clinical Relevance Statement

Following the implementation of a problem-oriented clinical notes application as a component of an electronic health record, we found a neutral to sufficient satisfaction with no significant differences between sex, age, professional experience, or training hours.

We found no correlation between user satisfaction and intensity of use or experience with the system.

How elements of EHRs are used and which aspects influence user satisfaction are very important considering the high costs of EHRs and the potential benefits in quality, safety, and efficiency, wherein the use of the log file is proved to be an essential tool.

Multiple choice question

Which instrument provides great advantages in the implementation and follow-up of a clinical documentation system?

• Detailed planning

• Adequate training

• Log file

• Incremental approach

All of the things are of great importance. but objective detailed monitoring through the use of a log file (answer C) provides great advantages.

A huge number of people are involved with the implementation of such a system, so detailed planning is essential. Also, staff must be well prepared for the challenges of working with the new system where adequate training is indispensable. An incremental approach may be useful but depends on many variables within the organization.

Conflict of Interest

The authors declare that they have no conflicts of interest in the research.

Human Subject Research Approval

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed by the privacy officer as a representative of the board of directors of the Radboud University Medical Center, Nijmegen, the Netherlands.

• References

• 1 Hillestad R, Bigelow J, Bower A, Girosi F, Meili R, Scoville R, Taylor R. Can electronic medical record systems transform health care? Potential health benefits, savings, and costs. Health Aff (Millwood) 2005; 24 (05) 1103-1117. doi: 10.1377/hlthaff.24.5.1103
• 2 Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med 2006; 144 (10) 742-752. doi:10.7326/0003–4819–144–10–200605160–00125
• 3 Jones SS, Rudin RS, Perry T, Shekelle PG. Health information technology: an updated systematic review with a focus on meaningful use. Ann Intern Med 2014; 160 (01) 48-54. doi: 10.7326/M13–1531
• 4 Kuhn T, Basch P, Barr M, Yackel T. Clinical documentation in the 21st century: executive summary of a policy position paper from the American College of Physicians. Ann Intern Med 2015; 162 (04) 301-303.
• 5 Ammenwerth E, Spötl HP. The time needed for clinical documentation versus direct patient care. A work-sampling analysis of physicians’ activities. Methods Inf Med 2009; 48 (01) 84-91. doi: 10.3414/ME0569
• 6 Weed LL. Medical records that guide and teach. N Engl J Med 1968; 278 (11) 593-600. doi: 10.1056/NEJM196803142781105
• 7 Schultz JR. A history of the PROMIS Technology: An effective human interface. In: Goldberg A. editor. A History of Personal Workstations. Reading, MA: AMC Press; 1988
• 8 Walker HK, Hurst JW, Woody MF. (eds) Applying the Problem Oriented System. Medcom Press: New York; 1973
• 9 Mellner C, Selander H, Wolodarski J. The computerized problem-oriented medical record at Karolinska Hospital: Format and function, users’ acceptance and patient attitude to questionnaire. Methods Inf Med 1976; 15 (01) 11-20.
• 10 Simons SM, Cillessen FH, Hazelzet JA. Determinants of a successful problem list to support the implementation of the problem-oriented medical record according to recent literature. BMC Med Inform Decis Mak. 2016 16: 102. doi: 10.1186/s12911–016–0341–0
• 11 Friedberg MW, Chen PG, Van Busum KR, Aunon F, Pham C, Caloyeras JP, Mattke S, Pitchforth E, Quigley DD, Brook RH, Crosson FJ, Tutty M. Factors affecting physician professional satisfaction and their implications for patient care, health systems and health policy. Rand Corporation. 2013 http://www.rand.org/content/dam/rand/pubs/research_reports/RR400/RR439/RAND_RR439.pdf Last accessed March 2016.
• 12 Rosenbaum L. Transitional chaos or enduring harm? The EHR and the disruption of medicine. N Engl J Med 2015; 373 (17) 1585-1588.
• 13 Love JS, Wright A, Simon SR, Jenter CA, Soran CS, Volk LA, Bates DW, Poon EG. Are physicians’ perceptions of healthcare quality and practice satisfaction affected by errors associated with electronic health record use?. J Am Med Inform Assoc 2012; 19 (04) 610-614. doi: 10.1136/amiajnl-2011–000544
• 14 Bowman S. Impact of electronic health record systems on information integrity: quality and safety implications. Perspect Health Inf Manag. 2013 10, 1c. http://perspectives.ahima.org/impact-of-electronic-health-record-systems-on-information-integrity-quality-and-safety-implications Last accessed February 2016.
• 15 Institute of Medicine. Health IT and Patient Safety: Building Safer Systems for Better Care. Washington, DC: The National Academies Press; 2011
• 16 AmericanEHR Partners. Physicians Use of EHR Systems 2014. http://www.americanehr.com/research/reports/Physicians-Use-of-EHR-Systems-2014.aspx Last accessed March 2016.
• 17 Meeks DW, Smith MW, Taylor L, Sittig DF, Scott J M, Singh H. An analysis of electronic health record-related patient safety concerns. Journal of the American Medical Informatics Association 2014; 21 (06) 1053-1059. doi: 10.1136/amiajnl-2013–002578
• 18 Menachemi N, Collum TH. Benefits and drawbacks of electronic health record systems. Risk Manag Healthc Policy 2011; 4: 47-55. doi: 10.2147/RMHP.S12985
• 19 Makam AN, Lanham HJ, Batchelor K, Samal L, Moran B, Howell-Stampley T, Kirk L, Cherukuri M, Santini N, Leykum LK, Halm EA. Use and satisfaction with key functions of a common commercial electronic health record: a survey of primary care providers. BMC Med Inform Decis Mak 2013; 13: 86. doi: 10.1186/1472–6947–13–86
• 20 the International Organization for standardization. https://www.iso.org/obp/ui/#iso:std:iso:9241:-11:ed-1:v1:en Last accessed October 2016.
• 21 Staggers N, Xiao Y, Chapman L. Debunking health IT usability myths. Appl Clin Inform 2013; 4 (02) 241-250. doi: 10.4338/ACI-2013–03-IE-0016
• 22 Cillessen FH, de Vries Robbe PF. Modeling problem-oriented clinical notes. Methods Inf Med 2012; 51 (06) 507-515. doi: 10.3414/ME11–01–0064
• 23 Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. International Journal of Human-Computer Interaction 1995; 7 (01) 57-78. doi: 10.1080/10447319509526110
• 24 Nielsen J. What is Usability? In: Usability Engineering. San Francisco, CA: Morgan Kaufmann Publishers; 1993: 23-48
• 25 Jamoom E, Beatty P, Bercovitz A, Woodwell D, Palso K, Rechtsteiner E. Physician adoption of electronic health record systems: United States, 2011. NCHS Data Brief 2012; 98: 1-8.
• 26 Manchikanti L, Benyamin RM, Falco FJ, Hirsch JA. Metamorphosis of medicine in the United States: a carrot and stick policy of electronic medical records. Pain Physician 2014; 17 (06) E671-E680.
• 27 American College of Physicians. Survey of Clinicians: User Satisfaction with Electronic Health Records Has Decreased Since. 2010 https://www.acponline.org/newsroom/ehrs_survey.htm Last accessed March 2016.
• 28 Menachemi N, Powers T, Au DW, Brooks RG. Predictors of physician satisfaction among electronic health record system users. J Healthc Qual 2010; 32 (01) 35-41. doi: 10.1111/j.1945–1474.2009.00062.x
• 29 Ellsworth MA, Dziadzko M, O’Horo JC, Farrell AM, Zhang J, Herasevich V. An appraisal of published usability evaluations of electronic health records via systematic review. Journal of the American Medical Informatics Association. 2016 ocw046. doi: http://dx.doi.org/10.1093/jamia/ocw046
• 30 Payne TH, TenBroek AE, Fletcher GS, Labuguen MC. Transition from paper to electronic inpatient physician notes. J Am Med Inform Assoc 2010; 17: 108e11. doi: 10.1197/jamia.M3173
• 31 Bossen C. Evaluation of a computerized problem-oriented medical record in a hospital department: does it support daily clinical practice?. Int J Med Inform 2007; 76 (08) 592-600. doi: 10.1016/j.ijmedinf.2006.04.007
• 32 Jia-lin L, Siru L, Fei L. Physician satisfaction with electronic medical record in a huge hospital (China). Stud Health Technol Inform 2013; 192: 920.
• 33 Alharthi H, Youssef A, Radwan S, Al-Muallim S, Zainab A. Physician satisfaction with electronic medical records in a major Saudi Government hospital. Journal of Taibah University Medical Sciences 2014; 9 (03) 213-218. doi: 10.1016/j.jtumed.2014.01.004
• 34 Tilahun B, Fritz F. Comprehensive Evaluation of Electronic Medical Record System Use and User Satisfaction at Five Low-Resource Setting Hospitals in Ethiopia. JMIR Medical Informatics 2015; 3 (02) e22. doi: 10.2196/medinform.4106
• 35 Nguyen L, Bellucci E, Nguyen LT. Electronic health records implementation: an evaluation of information system impact and contingency factors. Int J Med Inform 2014; 83 (11) 779-796. doi: 10.1016/j.ijmedinf.2014.06.011
• 36 Wright E, Marvel J. Electronic health records: postadoption physician satisfaction and continued use. Health Care Manag (Frederick) 2012; 31 (03) 259-267. doi: 10.1097/HCM.0b013e3182619e90
• 37 Rosenbloom ST, Denny JC, Xu H, Lorenzi N, Stead WW, Johnson KB. Data from clinical notes: a perspective on the tension between structure and flexible documentation. J Am Med Inform Assoc 2011; 18 (02) 181-186. doi: 10.1136/jamia.2010.007237
• 38 Ajami S, Bagheri-Tadi T. Barriers for adopting electronic health records (EHRs) by physicians. Acta Inform Med 2013; 21 (02) 129-134. doi: 10.5455/aim.2013.21.129–134
• 39 Ho YX, Gadd CS, Kohorst KL, Rosenbloom ST. A qualitative analysis evaluating the purposes and practices of clinical documentation. Appl Clin Inform 2014; 5 (01) 153-168. doi:10.4338/ACI-2013–10-RA-0081.
• 40 Trentman TL, Mueller JT, Ruskin KJ, Noble BN, Doyle CA. Adoption of anesthesia information management systems by US anesthesiologists. J Clin Monit Comput 2011; 25: 129. doi:10.1007/s10877–011–9289-x
• 41 Hanauer DA, Branford GL, Greenberg G, Kileny S, Couper MP, Zheng K, Choi SW. Two-year longitudinal assessment of physicians’ perceptions after replacement of a longstanding homegrown electronic health record: does a J-curve of satisfaction really exist?. Journal of the American Medical Informatics Association. Jul 2016; ocw077; doi: 10.1093/jamia/ocw077
• 42 Podsakoff PM, MacKenzie SB, Lee JY, Podsakoff NP. Common method biases in behavioral research: a critical review of the literature and recommended remedies. J Appl Psychol 2003; 88 (05) 879-903. doi: 10.1037/0021–9010.88.5.879
• 43 Ancker JS, Kern LM, Edwards A, Nosal S, Stein DM, Hauser D, Kaushal R. How is the electronic health record being used? Use of EHR data to assess physician-level variability in technology use. J Am Med Inform Assoc. 2014 doi: 10.1136/amiajnl-2013–002627.
• 44 Jones CD, Holmes GM, Lewis SE, Thompson KW, Cykert S, DeWalt DA. Satisfaction with electronic health records is associated with job satisfaction among primary care physicians. Inform Prim Care 2013; 21 (01) 18-20. doi: 10.14236/jhi.v21i1.27
• 45 O’Connell RT, Cho C, Shah N, Brown K, Shiffman RN. Take note(s): differential EHR satisfaction with two implementations under one roof. J Am Med Inform Assoc 2004; 11 (01) 43-49. doi: 10.1197/jamia.M1409