Factors Influencing Problem List Use in Electronic Health Records—Application of the Unified Theory of Acceptance and Use of Technology

Abstract

Background Problem-oriented electronic health record (EHR) systems can help physicians to track a patient's status and progress, and organize clinical documentation, which could help improving quality of clinical data and enable data reuse. The problem list is central in a problem-oriented medical record. However, current problem lists remain incomplete because of the lack of end-user training and inaccurate content of underlying terminologies. This leads to modifications of diagnosis code descriptions and use of free-text notes, limiting reuse of data.

Objectives We aimed to investigate factors that influence acceptance and actual use of the problem list, and used these to propose recommendations, to increase the value of problem lists for (re)use.

Methods Semistructured interviews were conducted with physicians, heads of medical departments, and data quality experts, who were invited through snowball sampling. The interviews were transcribed and coded. Comments were fitted in constructs of the validated framework unified theory of acceptance user technology (UTAUT), and were discussed in terms of facilitators and barriers.

Results In total, 24 interviews were conducted. We found large variability in attitudes toward problem list use. Barriers included uncertainty about the responsibility for maintaining the problem list and little perceived benefits. Facilitators included the (re)design of policies, improved (peer-to-peer) training to increase motivation, and positive peer feedback and monitoring. Motivation is best increased through sharing benefits relevant in the care process, such as providing overview, timely generation of discharge or referral letters, and reuse of data. Furthermore, content of the underlying terminology should be improved and the problem list should be better presented in the EHR system.

Conclusion To let physicians accept and use the problem list, policies and guidelines should be redesigned, and prioritized by supervising staff. Additionally, peer-to-peer training on the benefits of using the problem list is needed.

Keywords

problem-oriented medical records - facilitators and barriers - electronic health records and systems - secondary use - adoption - qualitative

Background and Significance

One of the core elements in electronic health record (EHR) systems is the problem list, which is a list of active and inactive problems relevant to the current care of the patient.[1] Most often, these problems are coded based on an underlying terminology. A problem list is central in the problem-oriented medical record (POMR), which is defined as “a structured organization of patient information per presented medical problem.”[1] POMR has proven to be successful in providing a structure that helps physicians record their notes about patients, which helps them to get a good understanding of the medical history of the patient.[2] [3] [4] Accurate problem lists that provide a concise summary of patient problems can therefore help physicians to track a patient's status and progress, and organize clinical reasoning and documentation.[3] [5] [6]

Although physicians and patients acknowledge the value of an accurate problem list,[7] [8] problem lists often remain incomplete, due to lack of responsibility, incomplete and inaccurate content of the underlying terminology, and because of little technical and administrative support.[5] [9] [10] Subsequently, physicians often modify these coded problems to their preferences, by adding details for these problems in clinical notes.[11] For instance, if the problem code is glaucoma but the modified free text is suspected glaucoma, the problem list indicates by its code that the patient has glaucoma, despite the patient not being confirmed to have that diagnosis. These alterations might inhibit patient care, as they can change the meaning of the code. Consequently, structured data from problem lists cannot be reused reliably.[12] [13] [14] [15]

To better understand acceptance and use of technologies in health care, such as EHRs, researchers have frequently studied facilitators and barriers for health care professionals.[16] [17] [18] [19] [20] To our knowledge, little attention has been given to factors associated with acceptance and use of problem lists.[5] [21] Therefore, it is still unclear to what extent health care professionals are adopting and accepting the problem list for clinical tasks, and potentially for reuse purposes. According to the unified theory of acceptance and use of technology (UTAUT), a validated framework that is often applied in health care context,[16] [22] [23] [24] the use of a new technology is influenced by performance expectancy, effort expectancy, social influence, and facilitating conditions.[16] [25] [26] Investigating factors influencing acceptance and actual problem list use using constructs of UTAUT could help determine what barriers need to be conquered and what facilitators need to be implemented to increase problem list use.

Objectives

The purpose of this study is to explore factors influencing the acceptance and actual use of the problem list. Removing existing barriers and implementing facilitators can help optimize problem list use. That, in turn, will eventually increase the quality of clinical data, enabling (re)use.

Methods

Study Design

In the Netherlands, health care providers make use of the diagnosis thesaurus (DT) to select the best-fitting problem code for their patients. The DT is an interface terminology SNOMED CT used in EHRs of a growing number of Dutch hospitals.[27] We recruited two groups of participants from Amsterdam UMC, that both use the same EHR system (Epic) since 2016:

• (1) Physicians and physicians in training who use the EHR system daily.

• (2) Leaders who play a role in sharing organization's culture and strategic vision, including heads of medical departments and quality managers.

The participants were contacted via email and snowball sampling was applied. The participants were asked to read a participation sheet and sign an informed consent form.

Data Collection

Face-to-face or telephone semistructured interviews were conducted with participants between October and December 2019. The interview guide started with collecting demographics, including gender, age, and work experience in the field and with the EHR system. Interview questions were based on UTAUT constructs and are shown in Appendix A. We also included questions related to discrete or structured registration in general, to better understand what factors cause participants to refrain from coding or modify diagnosis code descriptions. Answers of the participants were summarized during the interviews to ensure validation of the answers. Similarity of answers on the same questions by different participants also strengthens the validity.[28] Interviews were held until no new information was provided, at which point we determined theoretical saturation had been achieved.

Analysis

E.K. transcribed and coded interviews according to the grounded theory open coding approach. This is a qualitative research method used to generate a theory from gathered data.[29] This process started with open coding to create oversight of collected data, then categorizing the codes into clusters of codes to relate codes to UTAUT constructs (i.e., axial coding). For example, answers related to EHR training are first coded as “training” and then reorganized to the UTAUT construct “facilitating conditions.” The codes were assigned using the Atlas.ti 8.4.4 software. If multiple comments described similar constructs, one example comment was selected to represent the opinions of the participants within the UTAUT construct. This approach is comparable to that of a study of BenMessaoud et al.[30] Coded comments were discussed with other authors (R.C. and N.F.K.). Based on consensus, each coded comment finally related to at least one UTAUT construct. Next, we aimed to explain these comments in terms of facilitators and barriers.

In our application of UTAUT, performance expectancy is the degree to which an individual believes that using the problem list will help him or her to attain gains in their clinical tasks, also compared with the previous methods (e.g., free text). Effort expectancy is the degree of effort, such as time and usability, associated with problem list use. Social influence is the degree to which physicians perceive that peers and key figures, such as colleagues and heads of medical departments, believe he or she should use the problem list.[25] The presence of facilitating conditions can be defined as the organizational and technical infrastructure that exists to support problem list use.[31]

Results

In total, 75 physicians were invited. After 24 (32%) physicians participated in the interviews, we reached saturation and held no further interviews. Ten interviews were conducted via telephone, and 14 interviews were conducted face-to-face. The median duration of an interview was 30.5 (15–70) minutes. The characteristics of the participants are shown in [Table 1].

Table 1 Demographics of the interview participants (n = 24)

Demographic		N (%)
Occupation	Physician	14 (58)
	Physician in training	2 (8)
	Head of medical department	4 (17)
	Physician and data quality manager	4 (17)
Gender	Male	18 (75)
	Female	6 (25)
Working years (physician)[a]	< 5 y	2 (8)
	5 < 10 y	5 (21)
	10 < 15 y	11 (46)
	> 15 y	6 (25)
Working years with Epic	Since 2015	15 (63)
	Since 2016	9 (37)
Age group	20–30	1 (4)
	30–40	5 (21)
	40–50	11 (46)
	50–60	4 (17)
	60–70	3 (12)
Specialty	Nonsurgical (including number of participants): anesthesiology (2), ear, nose, throat (ENT) (3), internal medicine (hematology (1), infection diseases (1) oncology (1)), neurology (1), obstetrics and gynecology (2), orthopedics (1), pediatrics (6), urology (1)	19 (79)
	Surgical (including number of participants): dental surgery (1), neurosurgery (1), plastic surgery (1), intensive care (2)	5 (21)

^a The physicians in training were placed in the subgroup < 5 years.

Interview Outcomes

The interview topics were organized using the UTAUT model ([Fig. 1]). Example comments for each construct are included in [Table 2]. The construct Image was omitted, because this was not addressed in any comments. Perceived ease of use and ease of use were combined, because the definitions of these constructs are similar.[25]

Fig. 1 Unified theory of acceptance and use of technology (UTAUT) model adapted from the UTAUT model of Venkatesh et al.[25]

Table 2 UTAUT constructs with corresponding quotes of the physicians

Subthemes	Pros	Cons
Performance expectancy
Perceived usefulness	P1: “It is just a short time investment, but once it works, your preparation for clinic is faster, you always have a complete overview, you miss nothing, your letters are ready in no time (...) you no longer have to search through your old notes. It is just complete.”	C1: “I cannot imagine the profit from the problem list. So I don't need it because I'm not aware of the potential benefits.” C2: “Especially because the problem list is not kept up to date, it is not something I fully trust.”
Outcome expectancy	P2: “For me it is a handy tool to gather the information in 1 place, so it helps me if a patient of mine is admitted to just call the physician and say, look, the information is right there.”	C3: “In principle the intentions are there, but they (physicians) do not see the added value of it yet.”
Relative advantage	P3: “If you are going to record things in a discrete manner, that is just not as fast as writing on a piece of paper with a half-illegible handwriting (...) there must eventually be some sort of realization that it may take more time at the front gate, but that it ultimately saves time in the end.”	C4: “I think that if you just think and write good notes, then that is actually clear (...) those automatic click lists cause all kinds of noise.” C5: “I don't use it if I want to know how my patient is doing. Then I read the notes. I also think those letters are super ugly. Those automatic letters with hardly any proper language and also a lot of things that are not useful for you as a physician.”
Extrinsic motivation	P4: “I think everyone is too busy (...) it just has to do with motivation and a kind of obligation that you feel that you should have your patients records adequately documented.” P5: “We cannot get the complications from the central registration, and that is why we think that this should be done via the problem list.” P6: “That is of course the only reason why you want to fill it in a uniform manner, so that you can reproduce it for whatever purpose.”	C6: “Everyone knows that you have to use that problem list, and in fact you use it, but as limited as possible, because it is an obligation. Because you cannot create a financial code otherwise.” C7:“You only feel this necessity if you want to do something with the data yourself.” C8:“It has no added value for me. The diseases that I find interesting, I put them in a patient list, and I will find those back again very quickly.” C9:“The core task of the EHR system should be that most professionals can easily see the status. It should not be the goal to be able to send bills or have some scientific registration system set up.”
Job-fit	P7: “It appears in my status and it is also quickly generated from the problem list in my letter.”	C10: “The needs of the subspecialties are not the same (…) and as soon as patients have multiple problems, it is very complicated to get an immediate overview.”
Effort expectancy
(Perceived) Ease of use	P8: “I register it because you can easily create financial codes from the problem list.”	C11:“A lot of terms cannot be found. (…) And suppose it is in there, then you just have to type just exact the right thing to find that. So you will almost have to know what is exactly in there, if you want to find it.” C12: “The terminology of the problems is not something we recognize. That makes it very difficult to find.” C13: “These terms do not cover the problem. (...) so I change the description.” C14: “What we do is always adjust the free text. Therefore it may be that within the free text it is nicely registered and that it works well for our status, while it is a completely different problem for the underlying ICD code.” C15: “It is a phase you are in (...) I write ‘analysis’ in the display field and I do that consciously because patients get to see this information in their patient record, in (the application) MyChart.”
Complexity		C16: “Someone comes in with a set of symptoms which could fit to four different terms in the terminology.” C17: “It is quite complex. It is also very comprehensive and it is very easy to add something (to the problem list). And for many people it is difficult to throw something out.” C18: “I think it is not covering (the diseases), it is not complete, incorrectly described, and very difficult in use that it takes me too much time. So I just don't use it.”
Social influence
Subjective norm	P9: “Head of medical departments must be asked to ensure that their department propagates the right policy. I think that is step 1 (...) and then the following step is that we talk to each other that we are not doing it correctly.” P10: “They (physicians) know I (head of medical department) am constantly insisting (to use it). And then it gradually gets better.”	C19: “There is currently not much supervision or control.” C20: “I (head of medical department) can play a role in telling them (physicians) it is important, but the product must be good then.”
Social factors	P11: “If you can show (to your colleagues) something like ‘this is the problem list and it is useful, because’ then you can convince people and they will use it.” P12: “It is a matter of repeating, repeating, repeating, repeating and motivating and lecturing each other. That is no fun. But it eventually helps.” P13: “Not everybody has to be happy and interested, you just need one interested (physician) per specialty and from different layers in hierarchy.” P14: “Of course you can do it with negative feedback, but in my experience that won't work. I think you would convince people if you take them with you in your way of thinking.” P15: “It would not be a solution for us to show a graph every week showing who had the least problems filled in on the problem list. It is mostly positive learning methods. You do well if you register it, and you remind each other now and then.”	C21: “Physicians can get away with it by just putting a problem on the problem list that they thought it was at the beginning, while that might not be the case anymore.”
Facilitating conditions
Compatibility		C22: “There are two competing systems, the medical history and problem list.”
Perceived behavioral control	P16: “You need to agree upon what you want, measure that, and improve based on that.” P17: “I think there should be a point of contact if you want to add something, because it doesn't exist (…).”	C23: “The big objection is that the problem list is not updated. And that is also because no individual physician is responsible for the problem list. (...) I am responsible for my own notes, so I ensure that they are up-to-date and correct. And I think that is a central problem with the problem list. There is not one owner and everyone can mess with it.” C24: “I do not know if that (a point of contact) is possible or to who I should go to if I want to add something.”
Facilitating conditions	P18: “I think it would be good if you had some sort of body in the hospital that says: ‘this is the way we want to work with the problem list, and these are the rules and that is what we stick to.” P19: “We have a manual for all physician assistants.” P20: “We have work agreements, and with that we try to limit the number of problems to two or three, because otherwise you lose the overview.” P21: “Rules do not lead to better behavior, it is more about internal motivation.” P22: “We see the recorded problems with patient transfer. So, if something is not listed, then we will adjust it.” P23: “If you want to improve problem list use, you have to sit down with the individuals and do it. But something like the e-mail that was recently sent out, which is completely free of obligation (...). I think that just doesn't work.” P24: “There must be a certain profit to be seen. Someone must explain next to us how we use the problem list and also ensure that we can use the problem list but also that it helps us. There must be something in it for us.” P25: “I think training is a form of imposing sanctions (…) I would want to prevent that everyone at the department has to be (re)trained” P26: “My suggestion would be to put someone there who can do that (fill in the problem list), and who will do that for the entire hospital. Then the physicians are also satisfied.”	C25: “With the introduction of Epic, I feel there has been no supervision and training on how to use the problem list.” C26: “None of my colleagues, including myself, know what the rules are. It has not been established.” C27: “Epic is a system that says that everything is possible, meaning that everyone will start doing it their own way.” C28: “When the EHR was introduced, nobody knew what the rules were. So, we actually had to think for ourselves a little about how we were going to work with the EHRs. And that particularly applies to the problem list.” C29: “There are other priorities. That service center is overloaded. They are just busy with other things.”

Abbreviations: EHR, electronic health record; UTAUT, unified theory of acceptance and use of technology.

Note: “It” refers to the problem list.

Barriers and Facilitators

This section discusses barriers and facilitators that were derived from quotes displayed in [Table 2]. Examples of barriers have been described under “cons” and examples of facilitators have been described under “pros.” The quotes are indicated with Cn (cons) and Pn (pros).

Barriers

Performance Expectancy

Physicians reported barriers toward the value of the problem list for doing their job. First, most physicians were unaware of the added value of problem list compared with their current registration methods, for example, free-text notes (C1, C3, C4, C5). Second, physicians reported they do not trust problem list completeness as terms do not get updated once the diagnosis has been confirmed, for example, by diagnostic tests (C2). Some physicians mentioned that they only use the problem list to select the correct reimbursement code (C6) (Dutch: Diagnose Behandeling Combinatie). Furthermore, not all physicians agreed that reuse of data was a potential motivation to problem list use. Some physicians mentioned having other, similar, methods to collect data for research purposes, for example, Excel files. Others reported that the primary goal of health care is to treat the patient, and that reuse of data is therefore no incentive to problem list use (C7–C9). Moreover, specialties have different demands on what should be documented on the problem list (C10). For example, dermatologists would be interested in all dermatology-related problems, while that would be irrelevant for cardiologists.

Effort Expectancy

Professionals agreed that the content and presentation of the problem list in the EHR system itself showed room for improvement to reduce documentation efforts. First, the functionality to query the underlying terminologies to choose best-fitting terms was found ineffective (C11). Searching only succeeded if physicians typed in exact terms. Second, the DT does not include terms that are used in practice (C12). Consequently, physicians who use the problem list always choose a similar concept, but change the description (C13). This modified description is displayed to the physician in the problem list of a patient record. However, few physicians considered this dangerous for health care while, for example, decision support is triggered by coded problems and not by modified descriptions (C14). In addition, physicians reported the need to modify the terms because the diagnosis is not always immediately known from the beginning and patients have insights in their data too,[32] which thus requires to describe uncertainty about the diagnosis (C15). Physicians considered the DT too extensive to unequivocally determine the most applicable term (C16). Furthermore, physicians find it difficult to delete irrelevant or inactivate passing problems and they find it too time-consuming to code problems on the problem list (C17, C18).

Social Influence

Several physicians reported the lack of organizational policy including enforcement mechanisms and the lack of supervision as a barrier for proper implementation of the problem list (C19). According to a head of department, this is the result of the unawareness of the relative advantage. Promoting the problem list among colleagues therefore remains a low priority (C20). As a result, physicians avoid updating the problem list after they have fulfilled the minimal requirement to put one diagnosis on the problem list (C21).

Facilitating Conditions

Physicians reported that the EHR system contains a variety of similar registration options, such as the medical history (C22). Nonetheless, most professionals agreed that even if the content of the problem list is complete, they would still also describe clinical information in notes. They reported that clinical information cannot fully be captured in a predefined set of terms (C23).

Physicians reported uncertainty about responsibility of keeping the problem list up-to-date, and only a few physicians were aware of the possibility to request new terms to further complete the DT (C24).

None of the physicians had received specific guidance or training on effective problem list utilization (C25). Physicians reported that they do not have the knowledge to use the problem list or know what information is appropriate to document on the problem list (C26). Consequently, physicians created workaround processes, which resulted in proliferation of various methods to document findings (C27, C28). In addition, some physicians reported that the support of the EHR team is limited (C29).

Facilitators

Performance and Effort Expectancy

Some physicians reported that using the problem list supports preparing clinical tasks by having a complete patient overview (P1–P4). It also has a potential in reusing data (P6). For instance, for the generation of discharge and referral letters (P7). Furthermore, it facilitates the capture of complications and of financial codes (P5, P8).

Social Influence

Physicians reported that heads of medical departments or supervising physicians should propagate the policy (P9, P10). However, motivation is increased if the benefits relevant to care are shown, and physicians are reminded repeatedly (P11, P12). To ensure benefits are shared, physicians reported the need for “champions” within specialties (P13). Champions are motivated colleagues that have an intrinsic interest and commitment to implement change.[33]

Physicians also reported they needed to receive positive feedback instead of negative feedback from their supervisors or colleagues, and reminders on how and why to use the problem list (P14, P15).

Facilitating Conditions

Some physicians reported that problem list use should not only be motivated through positive feedback, but also monitored to indicate which specialties perform better than others (P16). Furthermore, physicians need a point of contact to allow them to request for new terms in the terminology (P17).

Other reported facilitating conditions were written and enforced policies and guidelines, which might help professionals in keeping the problem list up-to-date (P18–P20). However, some physicians disagreed with such central policies, and reported that problem list use should be mainly stimulated through intrinsic motivation (P21). Some physicians reported that, within their specialty, supervising staff regularly checks whether the problem list is up-to-date. These check moments were: during multidisciplinary consultations, after daily clinic consultations, and whenever a patient needs to be transferred to another specialty (P22).

Additionally, physicians reported that the best way to promote problem list use is to show-and-tell in small training sessions, that is, peer-to-peer training by “champions” (P23). These sessions should focus on benefits relevant to care. For instance, one physician reported that trainers should exemplify how problem list data can be reused within specialties (P24). Furthermore, physicians could be retrained during their career. To determine who should be retrained, problem list use could be monitored. One head of department specifically mentioned that heads would be motivated to stimulate physicians to properly use the problem list, if that means the physicians do not need to be retrained (P25).

Last, several physicians mentioned the need for administrative support, such as scribes, who maintain the problem list on behalf of the physician (P26).

Discussion

From our study, it became clear that there was a wide variation in attitudes toward problem list use, as most constructs of UTAUT contained mixed positive and negative comments. The physicians to whom the relative advantage was known, felt that up-to-date and accurate problem lists can increase efficiency of consultations, because it supports preparing clinical tasks by having a complete patient overview. Furthermore, the problem list has a potential value in reusing clinical data. Contrary, to some physicians the relative advantage was too low compared with free text because they do not recognize the benefits or, if they recognize the benefits, they have not experienced the advantage in their daily practice. Consequently, the use and maintenance of problem lists did not get priority.

Physicians that recognized the relative advantage also acknowledged some barriers reported by physicians that did not recognize the relative advantage, namely the lack of accurate content of the DT and inefficient search functionality of the terms within the EHR system, leading to modifying diagnosis code descriptions. Lacking content and poor presentation has also been reported by previous studies that examined factors influencing problem list use.[5] [7] [8] [10] [21] [34] [35] These studies also support our findings with regard to the lack of policies and guidelines[5] [35] and lack of training.[5] [21] [34] In our study, we further identified how training can be improved: by involving “champions” in small peer groups. However, in addition to improved training it is important that usability issues are solved,[35] [36] [37] because it will be difficult to train physicians to use the problem list appropriately if the problem list is insufficient.[36] [38] Furthermore, the time available to keep the problem list up-to-date and to promote problem list use is often a restriction,[5] although this might depend on prioritization of maintaining the problem list, which we believe will increase once benefits are shared.

Finally, it is remarkable that only a few physicians recognized the risk of modifying diagnosis code descriptions. One conceivable explanation for this unawareness could be that the physicians use workarounds and other collection methods, for example, Excel, for reuse purposes and therefore do not experience the need to keep underlying diagnosis codes up-to-date.

Recommendations

In [Table 3], we present recommendations to help establish wider agreement on problem list use, thus reducing inaccuracies and making problem list information accessible for reuse, including the need to solve usability issues and the need to increase priority through benefit sharing. During interviews, we noted disagreement on whether problem list use should be monitored or whether it should only be motivated through peer feedback. Without monitoring, physicians could fall back to old habits and use the problem list incorrectly.[1] Additionally, monitoring could help identifying which specialties might need extra support and training. We also consider the implementation of regular checks a possibility to ensure that the problem list is kept up-to-date. Some participants suggested medical scribes, who help maintaining the problem list. However, literature states that to balance the increased expenses, additional revenue from increased patient visits is needed for scribes to be financially prudent.[39]

Table 3 Recommendations to increase problem list use

	Recommendations
Organizational level, i.e., policy creators, EHR-trainers and hospital board members
Alignment with workflow	Create guidelines and policies explaining who is responsible for the maintenance of the problem list and how to use it; share the guidelines with (supervising) staff; introduce the guidelines in small training sessions per specialty
Training and support	Initial EHR system training should include problem list aspects and should focus on the advantages of problem list use in the care process; implement retraining programs; offer ongoing onsite support; implement new (super-user) training programs by training tech-savvy and enthusiastic physicians, the “champions,” so they can share their knowledge with their colleagues
Monitoring and feedback	Monitor problem list use on a specialty level to discover misuse of problem list and act on it by inviting head of medical departments and/or physicians for retraining programs
Increase motivation	Motivate physicians and increase prioritization for heads of medical departments through knowledge and benefit sharing in small training sessions; host meetings per specialty and give examples of how data can be reused, relevant for those specialties
Individual level, i.e., physicians and head of medical departments
Peer feedback	Monitor problem list use within the specialty to discover misuse of the problem list. Heads of medical departments and supervisors should notify their colleagues in case of lack of problem list use.
Knowledge sharing and demonstration	Motivate colleagues through knowledge sharing and demonstration of benefits that save time
Stimulate problem list use	Implement daily moments to update the problem list, for instance during patient meetings, on transfer of patients to another specialty and at the end of daily clinic
System level, i.e., EHR system vendors and terminology holders
Terminology	Improve the content of the terminology together with physicians; facilitate a change proposal process for improving the content of the terminology and keep user up-to-date whenever new terms are added to the terminology
EHR functionality	Include search functionality of the terminology in EHR systems
Availability similar methods	Reduce the availability of similar registration methods in EHR systems (e.g., medical history)

Abbreviation: EHR, electronic health record.

In parallel with this study, Amsterdam UMC established a committee that aims to improve data quality in the EHRs to reduce administrative burden and to improve problem list use. We discussed our recommendations with committee member and all were considered relevant for future implementation. In fact, some recommendations were already considered by this committee, such as setting up a key performance indicator to measure if problem list use increases by excluding the medical history functionality in the EHR system and the implementation of new training programs for physicians in training who can share the knowledge with their colleagues. Also, each specialty is required to assign a “point of contact” in 2020, who is responsible for teaching others how to work with the EHR system and problem list. This is in agreement with our recommendation of finding “champions” who should communicate knowledge and benefits of problem list use. Future research will focus on whether implementation of current methods from the committee combined with our recommendations indeed improves problem list use.

Strengths and Limitations

Although this research provided some meaningful findings, it has its limitations. First, although a qualitative approach was considered the best method to achieve our objectives, such approach has drawbacks. For example, answers of the participants may have been socially desirable or prone to recollection bias. Furthermore, we did not present the transcripts to the participants for comments (i.e., member-checking).[40] However, we are confident that we collected accurate results as we used the validated UTAUT framework, which is commonly used to determine factors influencing acceptance of (new) technologies in medical context[16] [22] [25] [26] we received similar answers among the participants about problem list use and we reached saturation. Additionally, literature[5] [7] [8] [10] [21] [34] states factors influencing problem list use that correspond to our outcomes. Therefore, although we examined one organization that uses one software system, other hospitals that use the same EHR system or the same approach to populate the problem list might still benefit from the presented results and recommendations.

Conclusion

This study was unique in the application of the UTAUT model to examine factors that affect acceptance and problem list usage by physicians. The results of this study will help health care institutions and EHR system and terminology providers to understand barriers and facilitators to acceptance and use of problem lists. With this understanding, strategic plans and training programs can be developed to persuade physicians to employ problem list use in their routine practice. The results show that some physicians were attracted to the benefits, but to convince all physicians to use the problem list, usability issues should be solved and facilitators are required. Facilitators include assigning “champions” who can convince colleagues to use the problem list through positive experience sharing in small peer groups and top-down control from supervisors or other directing health care providers, who need to prioritize and follow the redesigned policies and guidelines. Future research should focus on evaluating implementation of the proposed recommendations.

Clinical Relevance Statement

Only when data on problem lists are accurate, one can profit from potential EHR benefits such as automatic generation of discharge and referral letters and reuse of clinical data for decision support. To establish an accurate problem list, health care management should redesign training programs, policy, and guidelines, aiming at optimization of problem list use. Furthermore, EHR and terminology developers should ensure that presentation of the problem list is efficient and the content of the underlying terminology is complete.

Multiple Choice Questions

1. Which of the statements about problem lists is true, based on this study?

   • I. Problem lists often remain incomplete, due to lack of benefits.

   • II. Problem lists often remain incomplete, due to lacking presentation in EHR systems and incomplete content.

     • I is true, II is false.

     • I is true, II is true.

     • I is false, II is false.

     • I is false, II is true.

   Correct Answer: The correct answer is option d. Benefits of problem list usage are clear, but not (yet) recognized by physicians.

2. Physicians acknowledge problem list benefits, but what are the best methods to support these benefits?

   • Through peer-to-peer training and explicit and formal policies describing problem list use.

   • Through peer-to-peer training and by trusting the physician to decide which problems belong on the problem list.

   • Through plenary training sessions and by trusting the physician to decide which problems belong on the problem list.

   • Through plenary training sessions and explicit and formal policies describing problem list use.

   Correct Answer: The correct answer is option a. Peer-to-peer training will stimulate benefit and knowledge sharing, thus potentially increase prioritization of problem list use. Policies and guidelines are needed to guide physicians in problem list use and to establish who is responsible for maintaining the problem list. It is important to trust physicians to decide which problems belong on the problem list, but, as shown in this study, without formal policies problem list use is poorly supported.

Conflict of Interest

None declared.

Acknowledgment

We thank the study participants, without whom our research would not have been possible.

Protection of Human and Animal Subjects

The study was performed in compliance with the World Medical Association Declaration of Helsinki on Ethical Principles for Medical Research Involving Human Subjects, and was reviewed and approved by the Medical Ethical Testing Committee (METC) of Amsterdam UMC.

• References

• 1 Simons SM, Cillessen FH, Hazelzet JA. Determinants of a successful problem list to support the implementation of the problem-oriented medical record according to recent literature. BMC Med Inform Decis Mak 2016; 16 (102) 102
• 2 Salmon P, Rappaport A, Bainbridge M, Hayes G, Williams J. Taking the problem oriented medical record forward. Proc AMIA Annu Fall Symp 1996; 463-467
• 3 Hartung DM, Hunt J, Siemienczuk J, Miller H, Touchette DR. Clinical implications of an accurate problem list on heart failure treatment. J Gen Intern Med 2005; 20 (02) 143-147
• 4 Buchanan J. Accelerating the benefits of the problem oriented medical record. Appl Clin Inform 2017; 8 (01) 180-190
• 5 Wright A, Maloney FL, Feblowitz JC. Clinician attitudes toward and use of electronic problem lists: a thematic analysis. BMC Med Inform Decis Mak 2011; 11 (36) 36
• 6 Simborg DW, Starfield BH, Horn SD, Yourtee SA. Information factors affecting problem follow-up in ambulatory care. Med Care 1976; 14 (10) 848-856
• 7 Hose B-Z, Hoonakker PLT, Wooldridge AR. , et al. Physician perceptions of the electronic problem list in pediatric trauma care. Appl Clin Inform 2019; 10 (01) 113-122
• 8 Wright A, Feblowitz J, Maloney FL. , et al. Increasing patient engagement: patients' responses to viewing problem lists online. Appl Clin Inform 2014; 5 (04) 930-942
• 9 Holmes C, Brown M, Hilaire DS, Wright A. Healthcare provider attitudes towards the problem list in an electronic health record: a mixed-methods qualitative study. BMC Med Inform Decis Mak 2012; 12 (127) 127
• 10 Wright A, McCoy AB, Hickman T-TT. , et al. Problem list completeness in electronic health records: a multi-site study and assessment of success factors. Int J Med Inform 2015; 84 (10) 784-790
• 11 Walji MF, Kalenderian E, Tran D. , et al. Detection and characterization of usability problems in structured data entry interfaces in dentistry. Int J Med Inform 2013; 82 (02) 128-138
• 12 Andrews JC, Bogliatto F, Lawson HW, Bornstein J. Speaking the same language: using standardized terminology. J Low Genit Tract Dis 2016; 20 (01) 8-10
• 13 Rosenbloom ST, Miller RA, Johnson KB, Elkin PL, Brown SH. Interface terminologies: facilitating direct entry of clinical data into electronic health record systems. J Am Med Inform Assoc 2006; 13 (03) 277-288
• 14 Rosenbloom ST, Miller RA, Johnson KB, Elkin PL, Brown SH. A model for evaluating interface terminologies. J Am Med Inform Assoc 2008; 15 (01) 65-76
• 15 Horsky J, Drucker EA, Ramelson HZ. Accuracy and completeness of clinical coding using ICD-10 for ambulatory visits. AMIA Annu Symp Proc 2018; 2017: 912-920
• 16 Kim S, Lee K-H, Hwang H, Yoo S. Analysis of the factors influencing healthcare professionals' adoption of mobile electronic medical record (EMR) using the unified theory of acceptance and use of technology (UTAUT) in a tertiary hospital. BMC Med Inform Decis Mak 2016; 16 (01) 12
• 17 Ajami S, Bagheri-Tadi T. Barriers for adopting electronic health records (EHRs) by physicians. Acta Inform Med 2013; 21 (02) 129-134
• 18 Abbott PA, Foster J, Marin HdeF, Dykes PC. Complexity and the science of implementation in health IT--knowledge gaps and future visions. Int J Med Inform 2014; 83 (07) e12-e22
• 19 McGinn CA, Grenier S, Duplantie J. , et al. Comparison of user groups' perspectives of barriers and facilitators to implementing electronic health records: a systematic review. BMC Med 2011; 9 (46) 46
• 20 Vishwanath A, Scamurra SD. Barriers to the adoption of electronic health records: using concept mapping to develop a comprehensive empirical model. Health Informatics J 2007; 13 (02) 119-134
• 21 Martin PM, Sbaffi L. Electronic health record and problem lists in Leeds, United Kingdom: variability of general practitioners' views. Health Informatics J 2019; 0 (00) 1460458219895184
• 22 Hennington A, Janz BD. Information systems and healthcare XVI: physician adoption of electronic medical records: applying the UTAUT model in a healthcare context. Comm Assoc Inform Syst 2007; 19: 60-80
• 23 Ifinedo P. Technology acceptance by health professionals in Canada: an analysis with a modified UTAUT model. 2012 45th HICSS; 2012
• 24 Wills MJ, El-Gayar OF, Bennett D. Examining healthcare professionals' acceptance of electronic medical records using UTAUT. Issues Inf Syst 2008; 9 (02) 396-401
• 25 Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. Manage Inf Syst Q 2003; 425-478
• 26 Palau-Saumell R, Forgas-Coll S, Sánchez-García J, Robres E. User acceptance of mobile apps for restaurants: an expanded and extended UTAUT-2. Sustainability 2019; 11 (1210): 1-24
• 27 Dutch Hospital Data. Diagnosethesaurus. 2020 . Available at: https://www.dhd.nl/producten-diensten/diagnosethesaurus/Paginas/Diagnosethesaurus.aspx . Accessed March 23, 2020
• 28 Griffee DT. Research tips: interview data collection. J Dev Educ 2005; 28 (03) 36-37
• 29 Walker D, Myrick F. Grounded theory: an exploration of process and procedure. Qual Health Res 2006; 16 (04) 547-559
• 30 Benmessaoud C, Kharrazi H, MacDorman KF. Facilitators and barriers to adopting robotic-assisted surgery: contextualizing the unified theory of acceptance and use of technology. PLoS One 2011; 6 (01) e16395
• 31 Malik Bader Alazzam A, Samad Hasan Basari ASS, Ibrahim YM, Ramli MR, Naim MH. Trust in stored data in EHRs acceptance of medical staff: using UTAUT2. Int J Appl Eng Res. 2016; 11 (04) 2737-2748
• 32 Anderson MO, Jackson SL, Oster NV. , et al. Patients typing their own visit agendas into an electronic medical record: pilot in a safety-net clinic. Ann Fam Med 2017; 15 (02) 158-161
• 33 Miech EJ, Rattray NA, Flanagan ME, Damschroder L, Schmid AA, Damush TM. Inside help: an integrative review of champions in healthcare-related implementation. SAGE Open Med 2018; 6: 2050312118773261
• 34 Chen E, Garcia-Webb M. An analysis of free-text alcohol use documentation in the electronic health record: early findings and implications. Appl Clin Inform 2014; 5 (02) 402-415
• 35 Holmes C. The problem list beyond meaningful use. Part I: the problems with problem lists. J AHIMA 2011; 82 (02) 30-33 , quiz 34
• 36 Kim MS, Clarke MA, Belden JL, Hinton E. Usability challenges and barriers in EHR training of primary care resident physicians. International Conference on Digital Human Modeling and Applications in Health, Safety, Ergonomics and Risk Management; Springer; 2014
• 37 Karsh B-T, Weinger MB, Abbott PA, Wears RL. Health information technology: fallacies and sober realities. J Am Med Inform Assoc 2010; 17 (06) 617-623
• 38 Kruse CS, Kristof C, Jones B, Mitchell E, Martinez A. Barriers to electronic health record adoption: a systematic literature review. J Med Syst 2016; 40 (12) 252
• 39 Carnes KM, de Riese CS, de Riese WT. A cost-benefit analysis of medical scribes and electronic medical record system in an academic urology clinic. Urol Pract 2015; 2 (03) 101-105
• 40 Varpio L, Ajjawi R, Monrouxe LV, O'Brien BC, Rees CE. Shedding the cobra effect: problematising thematic emergence, triangulation, saturation and member checking. Med Educ 2017; 51 (01) 40-50