Usability Assessment of Secure Messaging for Clinical Document Sharing between Health Care Providers and Patients

Abstract

Background Web-based patient portals feature secure messaging systems that enable health care providers and patients to communicate information. However, little is known about the usability of these systems for clinical document sharing.

Objective This article evaluates the usability of a secure messaging system for providers and patients in terms of its ability to support sharing of electronic clinical documents.

Methods We conducted usability testing with providers and patients in a human–computer interaction laboratory at a Midwestern U.S. hospital. Providers sent a medication list document to a fictitious patient via secure messaging. Separately, patients retrieved the clinical document from a secure message and returned it to a fictitious provider. We collected use errors, task completion, task time, and satisfaction.

Results Twenty-nine individuals participated: 19 providers (6 physicians, 6 registered nurses, and 7 pharmacists) and 10 patients. Among providers, 11 (58%) attached and sent the clinical document via secure messaging without requiring assistance, in a median (range) of 4.5 (1.8–12.7) minutes. No patients completed tasks without moderator assistance. Patients accessed the secure messaging system within 3.6 (1.2–15.0) minutes; retrieved the clinical document within 0.8 (0.5–5.7) minutes; and sent the attached clinical document in 6.3 (1.5–18.1) minutes. Although median satisfaction ratings were high, with 5.8 for providers and 6.0 for patients (scale, 0–7), we identified 36 different use errors. Physicians and pharmacists requested additional features to support care coordination via health information technology, while nurses requested features to support efficiency for their tasks.

Conclusion This study examined the usability of clinical document sharing, a key feature of many secure messaging systems. Our results highlight similarities and differences between provider and patient end-user groups, which can inform secure messaging design to improve learnability and efficiency. The observations suggest recommendations for improving the technical aspects of secure messaging for clinical document sharing.

Keywords

personal health records - secure messaging - communication - interfaces and usability - electronic health records and systems

Background and Significance

Secure electronic messaging tools within Web-based patient portals are being implemented rapidly in health care,[1] and there is growing enthusiasm among providers and patients to use this form of communication.[2] [3] [4] [5] The percentage of nonfederal acute care hospitals in the United States that offer secure messaging services (e.g., viewing, downloading, and transmitting data, such as clinical documents, between providers and patients or providers only) has been increasing steadily in recent years.[6]

Secure messaging systems, which are often offered in conjunction with personal health record (PHR) systems, can help providers reach patients during vulnerable periods (e.g., after hospital discharge) when they may be at a higher risk for an adverse drug event.[3] [7] [8] Secure messaging systems can also help patients to take a proactive role in their care, thus mitigating some medication safety and quality issues.[7] [9] For example, secure messaging can increase patients' medication adherence,[10] prompt patients to refill prescriptions,[11] and improve clinical outcomes for glycemic control[12] and blood pressure.[13] Moreover, secure messaging can also improve communication efficiency between providers and patients. One study[14] found that the time for providers to update a medication record was reduced when patients sent information via secure messaging, compared with fax or phone. Patients also perceive benefits of secure messaging: “24/7” access, to providers, reduced wait times, and a permanent record of their health care inquiries.[15]

To maximize the value of secure messaging, there are usability deficits that need to be addressed.[1] [15] [16] For example, prior studies indicate that secure messaging could be improved with better designs for navigation, user preference settings, and preformulated subject text.[15] Usability assessments of secure messaging tools are limited; usability testing of these systems and need to evaluate a wider range of end-user types (e.g., providers and patients) and include more rigorous assessments of usability barriers.[16] Despite using the same (or similar) secure messaging systems, providers and patients have different goals, needs, and processes, which should be reflected in the design of the interface. There is also a continued need for usability evaluations as secure messaging features expand. To date, little is known about the usability of secure messaging systems for clinical document sharing. Sharing clinical documents is critical for information exchange between providers and patients when electronic health records (EHRs) are not fully interoperable with each other.[17] Clinical document sharing allows providers and patients to communicate supplementary information, providing a more comprehensive patient health narrative.

While some health information technology (IT) research presents usability evaluations of documents that capture clinical information,[18] we did not find any literature that evaluates the usability of sharing clinical documents via secure messaging. Such evaluation is important, because secure messaging could be useful for a wide range of clinical documents, including a patient's treatment plan, medications, imaging studies, and laboratory reports. By identifying use errors and subsequently improving secure messaging designs, health care systems will likely realize increased efficiency and satisfaction related to clinical document sharing.[7] This improvement may consequently increase adoption of these technologies and enhance their clinical effectiveness.

Objective

Our objective was to evaluate the usability of a secure messaging system for providers and patients, and specifically, the system's ability to support attaching, sending, and receiving clinical documents via secure messaging. The clinical document in this study was an interactive electronic portable document format (PDF) file with the patient's medication list, intended to help support medication reconciliation activities.[19]

Methods

This study was part of a larger body of research concerning asynchronous communication between providers and patients for medication reconciliation.[19] The larger study focused on the design and usability of an electronic tool for medication reconciliation.

Study Setting and Design

Usability testing sessions were completed at the Human–Computer Interaction and Simulation Laboratory, located within the Health Services Research and Development (HSR&D) department in a major Department of Veterans Affairs (VA) Medical Center in the Midwest United States.[20] The Institutional Review Board at Indiana University and the VA Research and Development Committee both approved the research methods (Protocol #1412128959).

We used qualitative and quantitative methods to evaluate the usability of a secure messaging system that is implemented in VA facilities across the United States. The secure messaging system connects providers interacting with the VA's EHR system to patients interacting with their PHR portal, My HealtheVet. Providers access secure messaging via a link through the EHR while patients must log in to their PHR portal to access the secure messaging feature. A standardized script and task list were used to maintain consistency across usability sessions. Interface designs for the secure messaging system were nearly identical for provider and patient users. Participants were instructed to use a think-aloud protocol via an instructional video[21] and asked to verbalize their reactions, whether positive, neutral, or negative.[22] Their verbalizations, face, and computer screen activity were video recorded via the Morae usability testing software (Version 3.3.2, TechSmith Corporation, Okemos, Michigan, United States). Participants sat at a computer workstation, while the moderator sat at a separate workstation behind a partition that enabled remote viewing of the participant's computer screen.

Usability tasks were constructed to align with provider and patient roles ([Table 1]). Providers were instructed to navigate to the secure messaging system from their EHR, create a new message, attach a clinical document, and send the clinical document to a fictitious patient. Separately, patients were expected to navigate to the secure messaging system from their PHR, retrieve the attached clinical document, and then save and attach the document in a reply. Providers and patients were expected to make changes to the clinical document (i.e., medication list), which is reported in detail elsewhere.[19] Task goals were given to participants at the start of the session, and it was expected that they would discover the subtasks of each task goal on their own. Moderators assisted participants only if participants indicated that they were unable to continue with the task. The total time allotted was 30 minutes for providers and 90 minutes for patients.

Table 1 Usability testing tasks

Provider tasks
0. Prior to testing, have providers log in to the computer
1. Task #1 goal: Navigate to secure messaging   • Log into electronic health record (EHR)
• Search for patient record
• Open secure messaging application
2. Task #2 goal: Create message with a new attachment, then send to fictitious patient
• Create new message
• Add recipient
• Add subject category
• Upload clinical document
• Attach clinical document
• Write message
• Click send
Patient tasks
1. Task #1 goal: Navigate to secure messaging system within PHR portal
• Log in to PHR portal
• Navigate to secure messaging inbox
• Recognize new message (from study moderator acting as a provider)
• Open new message
2. Task #2 goal: Download clinical document from provider
• Download attachment
• Save attachment
• Open attachment
3. Task #3 goal: Respond to message with a new attachment, then send to provider
• Navigate to inbox
• Reply to original message OR create new message
o If replying to the original message, go to next step
o If creating a new message:     ▪ Add recipient
▪ Add subject category
• Upload and attach clinical document
• Write message
• Click send

Abbreviation: PHR, personal health record.

Note: These tasks were developed to assess the usability of the secure messaging system for providers and patients. The tasks goals (shown in boldface font) listed below represent the overarching tasks that participants were asked to complete for usability testing. The listed steps beneath each task goal are the anticipated steps one would take in a scenario for providers sending a clinical document to a fictitious patient, and a patient response when receiving a clinical document from a fictitious provider. Participants needed to figure out the subtask steps on their own to complete the goal. Participants may take alternative steps or task work flows to achieve the same end goals.

Participants and Recruitment

Providers were recruited at clinical staff meetings and via email within the following strata: registered nurses, physicians, and inpatient and outpatient pharmacists. Prior experience with secure messaging was not required. Providers were offered a clinically related book in recognition of their time.

Patients were recruited in person at discharge from inpatient care and via posted flyers and mailed letters from the hospital within 30 days of discharge. It was required that patients were at least 18 years old, with self-reported access to a computer and the Internet. Prior secure messaging experience was not necessary, but patients had to be already enrolled in the PHR portal program. Enrollment included a brief, in-person tutorial and a brochure about secure messaging. Patients were excluded if they did not pass the Callahan Screener for Cognitive Impairment.[23] Patients received a USD 25 gift card for completing at least one task and a second USD 25 gift card for completing the entire session. The study had a recruitment goal of 10 patients.[24]

Data Collection and Analysis

Demographic data were collected, including prior experience with the VA secure messaging system and frequency of use. Videos were analyzed for task completion rates, efficiency, qualitative feedback, and use errors. A task was completed if the goal(s) in [Table 1] were achieved. Efficiency was measured by the time to complete each task, which included any time with the think-aloud protocol and interacting with the moderator. One patient had extreme difficulty with the tasks; the moderator deviated from protocol to assist the patient with completing the tasks. The deviation added significant unscripted moderator–participant interaction, which invalidated accurate analysis of time data, thus we omitted this patient's data from the time analysis as an outlier. Data from this participant were retained in the analyses of qualitative feedback, use errors, and satisfaction ratings.

Qualitative usability feedback was grouped by positive and negative sentiments spoken aloud. These sentiments were organized by frequency and severity to determine key aspects of the system that facilitated or hindered usability. Observed use errors were captured using the following definition from the Food and Drug Administration medical usability guidelines:

“User action or lack of action that was different from that expected by the manufacturer and caused a result that (1) was different from the result expected by the user and (2) was not caused solely by device failure and (3) did or could result in harm.”[25]

Observed use errors were organized based on the clinical implications of addressing (or not addressing) the use error. Relevant system failures unrelated to use were also captured. Usability analyses were led by an industrial engineer (M.A.J.) and human–computer interaction researcher (H.P.).

At the end of the session, participants completed the 19-item Post-System Study System Usability Questionnaire (PSSUQ)[26] with 7-point Likert-scaled responses from left (“1 = Strongly Disagree”) to right (“7 = Strongly Agree”). We reverse coded the original PSSUQ scale to conform to common convention that a higher rating is traditionally associated with higher satisfaction. Subscales for the PSSUQ analysis were based on factor analyses from Lewis,[26] which gives equal weight to each item in the scale. “N/A” responses were excluded from summary calculations. Descriptive statistics were calculated using SPSS Statistics 24 (IBM Corporation, Armonk, New York, United States).

Results

Participant Demographics

Participants included 29 individuals ([Table 2]). No patients were excluded due to cognitive impairment.

Table 2 Demographic data of study participants

	Health care providers (n = 19)	Patients (n = 10)
Age	38 (30–65)	59 (48–75)
Gender (frequency, % male)	7 (36.8)	10 (100)
Years of clinical experience	10 (1–41)	N/A
Role (frequency, %)	6 (31.6) Physicians	N/A
	6 (31.6) Registered nurses
	3 (15.8) Inpatient pharmacists
	4 (21.1) Outpatient pharmacists
Clinical environment (frequency, %)	7 (36.8) Inpatient	N/A
	12 (63.2) Outpatient
Previous experience with secure messaging (frequency, % yes) and current usage (frequency, %)	11 (58)	2 (20)
	• 5 Physicians	• 1 (10) Daily
	o 2 (18) Daily	• 1 (10) Weekly
	o 3 (27) Weekly
	• 6 Registered nurses
	o 5 (45) Daily
	o 1 (9) Weekly
	• 0 Pharmacists

Note: Data are shown as median (range) unless otherwise indicated.

Task Completion and Efficiency

For task #1, 11 (58%) providers successfully accessed the secure messaging system without prompting or assistance from the moderator; 5 (26%) needed direction to correctly open the secure messaging system; and 3 (16%) were unable to open the secure messaging system due to system failures (i.e., account was not authorized since they were a new user). Of the five providers who needed directions to open the system, three (16%) could not remember how to access the secure messaging system through the EHR and all required a moderator to log in to the system for them. Providers completed task #1 within 1.2 (0.3–3.2) minutes.

For task #2, 11 (58%) providers successfully attached the clinical document and sent it to a fictitious patient without moderator assistance. There were no consistent trends between experience and provider task completion rates. Of the 11 provider participants who completed tasks #1 and #2 without moderator assistance, 7 had prior experience with secure messaging and 4 did not. The 8 providers who needed moderator assistance were evenly split with experience and no experience. Providers completed task #2 within 2.7 (1.2–10.7) minutes.

The task completion time for each provider role is shown in [Table 3]. Although the sample sizes were not intended for statistical comparisons, descriptive statistics suggest that physicians completed tasks faster than other providers. The largest time gap between participant roles seemed to be between physicians and nurses, even though all nurses (6/6) and nearly all physicians (5/6) indicated that they had experience with secure messaging systems. No pharmacists had prior experience with secure messaging, except via the tutorial at enrollment, yet their task time was still faster compared with the nurses.

Table 3 Task time data and moderator assistance percentages for the different categories of providers per task

	Physicians (n = 6)	Nurses (n = 6)	Pharmacists (n = 7)	All providers (n = 19)
Task 1 time to completion (min)	0.7 (0.3–2.5)	1.3 (0.6–3.2)	1.1 (0.5–2.6)	1.2 (0.3–3.2)
Task 2 time to completion (min)	1.6 (1.2–5.5)	7.8 (2.4–10.7)	3.4 (1.3–4.9)	2.7 (1.2–10.7)
Total # of participants completing all tasks without moderator assistance	4 (66%)	3 (50%)	4 (57%)	11 (58%)

Note: Values are presented as median (range) in minutes for tasks and frequency (percentage) of participants that completed the task without moderator assistance.

No patients completed any task without moderator assistance. Over half of patients (60%) expressed difficulty finding secure messaging within the PHR interface. For task #1, patients (n = 9) spent 3.6 (1.2–15.0) minutes to access the secure messaging inbox and for task #2, 0.8 (0.5–5.7) minutes to retrieve the clinical document from the secure message. For task #3, patients required 6.3 (1.5–18.2) minutes to return the attachment via secure messaging.

Qualitative Findings

Providers expressed some concern regarding the limitations of secure messaging for provider–provider communication and information coordination. Two (11%) providers mentioned that secure messaging limits coordination because it prevents forwarding of secure messages to other members of the patient care team. Two (11%) other providers mentioned concerns that patients may send multiple messages over time with attachments of medication information. In the VA secure messaging system, the messages contain quoted text underneath, which contains the previous message text, but do not nest (or group in a thread) related messages. This format omits attachments from previous messages, which could result in a loss of information over time. Two (11%) providers mentioned that it would be nice if secure messaging were fully integrated with the EHR to streamline functions, better track medication activity, and patient care. Two (11%) providers expressed a desire for tutorials or training on how to effectively use secure messaging beyond the initial training they received when signing up for the service.

Seven (37%) providers voluntarily expressed that it was easy to learn how to use the secure messaging system in the EHR, and six (32%) described it as “simple.” Two (11%) providers expressed they would enjoy communicating with their patients about their medications with secure messaging. One provider remarked that the interface “seems a lot like [Microsoft] Outlook, which I am comfortable with.” A pharmacist liked that the system used common design conventions, saying, “It had all the things you would expect from an email messaging function.”

Patients were not as vocal as providers during the think-aloud sections of usability testing. However, some patients expressed negative feedback regarding the attachment process, such as confusion with opening an attachment (60%) and accessing a downloaded clinical document (40%). Some (20%) patients wanted training or tutorials. Only two (11%) patients offered positive remarks on the secure messaging system, noting that it was easy to learn and simple to use.

Use Errors and System Failures with Secure Messaging

We found 3 distinct use errors as providers navigated to secure messaging (provider task #1) and 20 distinct errors while providers used the system to send a document (provider task #2). These use errors were grouped by category. There were a total of 9 observed use errors for physicians (3 and 6 errors for tasks #1 and 2, respectively); 9 use errors for registered nurses[1] [8]; and 8 use errors for pharmacists.[1] [7] Main findings from physician use errors related to attachment issues with the clinical document, accessing secure messaging via the EHR, and difficulties coordinating with other providers (e.g., could not forward a message). The latter need was also mentioned by pharmacists. Other use errors for pharmacists related to documentation (e.g., to track medication changes, document attachments). Use errors for nurses focused predominantly on a need for shortcuts for more efficient work (e.g., streamlined methods for attaching documents) and help with error recovery (e.g., the ability to recall sent messages).

With patients we found the following: 12 distinct types of use errors when locating the secure message (patient task #1); 5 distinct use errors while opening the document (patient task #2); 23 distinct use errors while replying with an updated document (patient task #3, e.g., [Fig. 1]); and 5 distinct use errors related to unclear text within the secure messaging system (e.g., misunderstanding error messages, accidental navigation away from PHR portal). Moderator probes revealed that errors related to unclear instructions were caused by usability instead of with task instructions. [Table 4] describes the most severe and/or frequent use errors.

Fig. 1 The error message (in red type) that appears when a user tries to send a blank secure message.

Table 4 Use errors for secure messaging

	Use error	Frequency and user type	Description and potential implications
1	Difficulty navigating to secure messaging	3 (16%) Providers  1 Nurse  1 Pharmacist  1 Physician 7 (70%) Patients	Three providers had difficulty accessing secure messaging from the EHR. For patients, after logging on to the PHR, 7 patients had difficulty finding the secure message feature and navigating to the associated screen. Patients looked in a range of incorrect locations on the PHR homepage, including the bottom of the page, “patient information” link, and the search bar
2	Difficulty locating clinical documents after downloading them from secure messaging	4 (40%) Patients	Patients were required to download the clinical document and reupload it during the usability session. When downloading, the clinical documents were automatically saved to a temporary folder from the PHR portal. Four patients did not recognize this situation and did not change the location of saving the document. This resulted in confusion and frustration when patients tried to locate it later on, and the default requires users to recognize a need to modify the save as path before saving. This aspect of the system was not dependent on the internet browser and was an aspect of the PHR portal itself
3	Challenges with steps to attach the clinical document	1 (5%) Providers  1 Nurse 7 (70%) Patients	One provider and 7 patients did not know what steps to take to attach the clinical document. Some tried to drag and drop or copy and paste clinical documents into a secure message and it did not work. The provider tried to copy and paste the file and also tried to drag and drop the file (similar to shortcuts for attachments when sending emails). The patients clicked the instructions link (see [Fig. 1]) to upload the files. To correctly upload the clinical document, participants had to click an attachment button and then browse for the correct file to upload. The attachment button does not have any visual cues that indicate it is a button. To correctly upload the clinical document, participants had to click an attachment button and then browse for the correct file to upload. The lack of shortcuts or additional modes of attachment were not immediately apparent
4	Unsure if clinical document is attached	2 (11%) Providers  2 Nurses 2 (20%) Patients	Two providers and 2 patients accidentally sent messages without attachments. Of these participants, 1 provider and 1 patient noted uncertainty regarding correctly attaching the clinical document
5	Risk of a blank secure message	4 (21%) Providers  2 Nurses  2 Pharmacists 6 (60%) Patients	Four providers and 6 patients attempted to send messages with a clinical document attached without text in the message field, which prompts automated error text (“The message cannot be blank”) and prevents message sending ([Fig. 1]). The error text is small and embedded on the interface, which lead to 2 of the 4 providers and 3 of the 6 patients to be unable to resolve the error and send the message to the intended recipient

Abbreviations: EHR, electronic health record; PHR, personal health record.

Note: The use errors listed below were the most frequently detected during testing and/or have the highest potential risk of a negative clinical outcome. The use errors are organized by the flow of the tasks and the order that the error could appear when sending a clinical document attachment via secure messaging.

In addition to use errors, three distinct system failures were observed: file size limitations; unexpected, accidental logout mid-task; and system error with sending messages. For example, three providers and one patient could not attach the clinical documents because they were too large. The secure message system file size for a single attachment cannot exceed 3 MB, and the total size of the (maximum of 4) attachments cannot exceed 6 MB. Other system failures included the system logging out two patients during use and preventing one patient from sending a secure message for an undetermined reason.

Satisfaction Responses

Satisfaction results are shown in [Table 5]. Providers who were first-time users (n = 8) and experienced users (n = 11) both gave positive satisfaction ratings for secure messaging (median = 5.9, interquartile range [IQR] = 1.1; median= 5.5, IQR = 1.0, respectively). In contrast to first-time provider users, there appeared to be wider variation in satisfaction ratings among first-time patient users (median = 5.4, IQR = 3.0, n = 8). The physician role had the lowest satisfaction ratings but smallest IQR. For the subfactors of the PSSUQ, registered nurses had the lowest IQR of the providers for the interface quality subfactor, but all other subfactors seemed relatively consistent across provider roles.

Table 5 Satisfaction scores for secure messaging, as measured with the validated, 19-item Post-Study System Usability Questionnaire (PSSUQ)

Score	All providers (n = 19)	Physicians (n = 6)	Registered nurses (n = 6)	Pharmacists (n = 7)	Patients (n = 10)
Overall satisfaction (Items 1–19)	5.9 (5.5–6.3)	5.5 (5.2–6.0)	6.1 (5.1–6.5)	5.9 (5.6–7.0)	6.0 (3.8–6.6)
System usefulness (Items 1–8)	6.0 (5.5–6.6)	5.9 (5.0–6.3)	6.3 (5.3–7.0)	6.0 (5.9–7.0)	5.8 (4.1–6.8)
Information quality (Items 9–15)	5.8 (5.3–6.0)	5.3 (4.9–5.6)	5.9 (4.8–6.3)	6.0 (5.8–7.0)	6.0 (3.4–6.7)
Interface quality (Items 16–18)	6.0 (5.0–6.0)	5.5 (4.8–6.0)	6.0 (3.9–6.5)	6.0 (5.3–7.0)	6.0 (4.0–6.7)

Note: Results are shown as median (interquartile range).

Participants rated each item on a 7-point Likert-type scale where 1 = strongly disagree and 7 = strongly agree.

Three PSSUQ items were marked not applicable by participants for items regarding error recovery and instructions, when they did not perceive an error with the system. All three of these items (numbered 9–11) were in the information quality subscale. These items were:

• The system gave error messages that clearly told me how to fix problems (N/A for 15 providers and 1 patient).

• Whenever I made a mistake using the system, I could recover easily and quickly (N/A for 9 providers and 1 patient).

• The information (such as online help, onscreen messages, and other documentation) provided with this system was clear (N/A for 10 providers and 1 patient).

The N/A values were not included in the statistics.

Discussion

We evaluated the usability of a nationally implemented secure messaging system to assess its ability to help providers and patients send and receive clinical documents. Many usability studies in health care informatics research only focus on one homogeneous end-user group (e.g., nurses or patients only).[16] A strength of this research is that it explores multiple clinical viewpoints and diverse end-users (providers and patients) when assessing usability. In addition, this appears to be the first study to date that examines the usability of an interface for clinical document sharing, a key feature of many secure messaging systems. In this work, we identified three distinct areas for improving the usability and technical functions of secure messaging for clinical document sharing: (1) interface efficiency; (2) training and documentation to enhance learnability and memorability for novice and intermittent users, respectively; and (3) interface personalization to support differences in end-user roles.

First, our research presents specific usability and interface considerations that impact the efficiency of clinical document sharing. Our results (the “Task Completion and Efficiency” section) demonstrate that secure messaging tasks are inefficient as related to clinical document sharing: it took almost 5 minutes for providers to attach and send a clinical document, not including the time to fill out the clinical document. Providers are already time-constrained for patient appointments, and it is critical for clinical document sharing to function seamlessly with their work processes. The need for a more efficient secure messaging system was also apparent when providers repeatedly switched between EHR and secure messaging interfaces to view the information from both the clinical document and patient record. If providers could send a secure message while viewing the patient record simultaneously, it may improve efficiency and reduce cognitive effort. The process for storing clinical documents could also be more efficient: the current system requires users to store and retrieve files locally on a computer, which was time consuming for participants. Future designs of secure messaging could provide more flexibility for attaching clinical documents within the secure message or a link to a subportal that contains patient-reported information. These embedded features could instead store health information on a secure server that is connected to patient databases so that messaging can more easily synchronize with EHR data. These recommendations can be generalized to other areas of clinical document sharing (e.g., sharing laboratory results, medical images, vaccination lists, medication records) with remote patients and providers.

There are also some technical limitations of the secure messaging and larger EHR system that should be addressed when implementing clinical document sharing to enhance efficiency. For instance, acceptable file sizes should be expanded to allow sharing of larger clinical documents. Based on our study, 3 MB is too small; some providers needed approximately 20 MB to share an interactive PDF medication list. Settings for system timeouts, approximately 15 minutes, should also be increased to 30 minutes since users may take longer to fill out forms and scan in additional clinical documentation.[19] System timeouts mid-task could result in frustration and decreased satisfaction with secure messaging.

A second major finding of this research was the need for better learnability and memorability for novice and intermittent users to support their use of the secure messaging system. This finding was suggested by the frequent need for moderator assistance to complete tasks (the “Task Completion and Efficiency” section); a lack of notable differences with task completion rates for novice versus experienced providers who use the system infrequently (the “Participant Demographics” and “Task Completion and Efficiency” sections); and greater variation in satisfaction for new versus experienced patient users of secure messaging (the “Satisfaction Responses” section). One key area for improving learnability of the interface is the attachment feature: both novice providers and patients attempted to attach clinical documents in the same way they would attach a file when sending an email (see [Table 4]). By continuing to incorporate recommended interaction designs from similar functioning messaging platforms,[27] users may be able to learn features faster. Furthermore, ensuring quick learnability is beneficial for many health care systems, where teaching novice medical professionals is a top priority and turnover rate may be higher.[28] The memorability of secure messaging system features could be improved for intermittent users by giving better feedback to users about the state of the system and their actions, such as through refreshing the user interface and other visual cues to draw users' attention to errors that occur (see [Fig. 1]). In addition, including help and documentation resources closer to the point of action could help intermittent users and/or patients who may have cognitive impairments. Based on the key areas of difficulty, help and documentation resources should cover how to add message recipients and attach clinical documents through an overlay tutorial, and add more salient text regarding the clinical document requirements for the file types and attachment sizes. Although these help and documentation resources may help novice and intermittent users, it is important to note that they should not be implemented as a substitute for improving usability.

A third key area for improving secure messaging and associated clinical document sharing is to provide role-based interface personalization. This study was unique compared with similar health IT studies,[29] [30] in that we compared quantitative (task completion rates, time to complete tasks, satisfaction ratings) and qualitative data (use errors) for physician, nurse, and pharmacist clinical roles. Findings from these roles can inform personalization options of the secure messaging interface for clinical document sharing. In the following text, we review the findings by role and suggest some ideas for personalization.

We found that physicians completed tasks the fastest, yet still required assistance despite nearly all (5/6) having secure messaging experience. Results for physicians (see the “Task Completion and Efficiency” section) suggest that they may be more familiar with the secure messaging system overall (hence the faster completion times overall), but not as familiar with specific steps necessary for attaching clinical documents resulting in the need for assistance and lower satisfaction ratings (see the “Use Errors and System Failures with Secure Messaging” section). Findings suggest that physicians may not be as interested in using clinical document sharing unless it is necessary; the lack of a live-connection and the inefficiencies with the system were both cited as barriers from physicians for incorporating clinical document sharing into patient care. Since clinical document sharing tasks are often offloaded to support staff,[31] an interface personalization feature specifically for physicians could allow them to delegate clinical document sharing tasks to other members of the care team and streamline their care processes.

Nurses required the most time to complete tasks, despite all nurse participants having secure messaging experience. This finding, combined with their requests for better features to facilitate efficient work processes (see the “Use Errors and System Failures with Secure Messaging” section), suggests that nurses encounter significant time pressure, further underscoring the need for health IT systems to be designed to reduce the time to complete basic tasks. Our results indicate that clinical document sharing was not adequately designed for nurses in particular, even though they are the most frequent users of the secure messaging system. The reasons for this finding are unclear, but could be that nurses have a different workflow or mental model compared with physicians and pharmacists for how to use secure messaging in patient care.

Lastly, pharmacists' results were similar to physicians' in that they were concerned with coordination activities, specifically how to communicate via secure messaging yet still track changes and document a patient's care process within this modality. Although all providers can save entire secure message threads as patient notes, which are attached to a patient's record in the EHR, this process is inefficient and may result in providers saving superfluous or redundant information. A potential solution for personalizing the interface for pharmacist activity would be specific folders or flags within the secure messaging system to save different types of clinical documents for a patient, and have these synced to the EHR system. For example, if a patient frequently shared their medication reconciliation lists and blood glucose meter logs, each of these clinical documents could be organized by category to facilitate better tracking and retrieval of information for pharmacists.

We recognize that this research is limited in that firm conclusions about provider role differences cannot be drawn from our smaller samples of providers, and we suggest future investigations to address the unique needs and similarities of each provider type and test some of the proposed ideas. Our work is also limited in that we examined specific aspects of secure messaging (sending, receiving, and attaching electronic clinical documents) and we could not modify the interface to iteratively test our suggested design improvements for clinical document sharing. Future research could evaluate a broader range of secure messaging features and examine navigation patterns and mental models of end-users. Our research methods were limited in that the time to complete tasks included moderator interactions, which is an inherent limitation of the think-aloud method,[21] and participant recruitment could have been expanded for both sample size and diversity. Although the sample size for provider roles and patients reached minimum recommendations for finding at least 90% of usability issues,[24] larger samples could provide power for statistical comparisons between roles. Our patient sample is closely aligned with the VA patient population, with less than 10% of the overall VA patient population being female veterans[32] and many VA patients being older in age; nevertheless, a more diverse patient sample could help to extrapolate findings beyond the VA system.

Future work could also involve remote usability testing with patients in their home environment. We conducted the study in a hospital-based laboratory, which facilitated a controlled evaluation. Although this hospital-based setting simulated the working environment of providers, it did not simulate patient home environments. At home, patients may use different devices, Web browsers, and have secure messaging features set up differently. They may also have distractions or other environmental conditions that influence their use of the system. Patients might also print, complete, and then scan the clinical document at home, but this was not captured in our study. Additionally, long-term usability studies could be helpful to track the time to create messages to evaluate how satisfaction and perceived communication quality changes as users experience with the system increases.

Conclusion

Secure messaging systems are often incorporated without much testing, with usability issues hindering health IT adoption. This study evaluated clinical document sharing, an important feature of secure messaging systems, for the largest health care network in the United States. Key findings for improving secure messaging systems include: usability improvements to increase provider efficiency with secure messaging; help and documentation resources for secure messaging for novice and intermittent users; and tailored interface features to support different provider roles. An important contribution of this research is its concurrent examination of physicians', nurses', and pharmacists' performance of secure messaging tasks, revealing insights for each of these roles. In addition to highlighting key usability issues, this article provides recommendations to improve the technical and functional aspects of secure messaging. Software developers can use these recommendations to design secure messaging systems to fit end-user needs to facilitate clinical document sharing and provider–patient communication.

Clinical Relevance Statement

This research may aid the standardization of secure messaging features for clinical document sharing, thus facilitating provider–patient communication. Our results highlight key similarities and differences between usability evaluation results (i.e., efficiency, task completion rates, satisfaction ratings, qualitative feedback, and use errors) of clinical document sharing for both provider and patient end-users.

Multiple Choice Question

Which of the following is a usability heuristic that could be used to improve the design of secure messaging systems?

• Make all secure messaging systems function exactly like email.

• Improve the visibility of alerts related to system failures.

• Use 12-point font across the interface.

• Add instructional pop-ups for novice users.

Correct Answer: The correct answer is option b. Option a is incorrect because there is not one universal email platform with set standards, and email platforms and standards evolve over time. Option c is incorrect because 12-point font is quite small for interfaces and the font size should be adaptable with the size of the interface (e.g., mobile vs. monitor) and the type of information (e.g., title vs. subheading). Option d is incorrect because research has shown that modal dialog boxes (i.e., pop-ups) tend to be a nuisance to users and they are often unread and exited. Option b is the best answer because it recognizes a common issue in health IT, yet describes it in general terms. Option b is demonstrated in the results of the manuscript (see [Table 4]) as well as cited in the literature.³³

Conflict of Interest

None.

Acknowledgments

We thank the study participants. We would like to thank Amy Linksy, MD, for her assistance with writing the grant proposal that ultimately funded the larger research study encompassing this work. We also wish to thank the following VA operational partners: Kim Nazi, PhD; Maureen Layden, MD; and Eric Spahn, PharmD. We would also like to thank Rachel Dismore for her assistance with recruiting participants and managing IRB documentation, and Khoa Nguyen, PharmD, for helping moderate some usability sessions and providing pharmacy expertise. Views in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or U.S. government.

Protection of Human and Animal Subjects

This study was performed in compliance with standards set by the Institutional Review Boards at Indiana University and the VA Research and Development Committee. These organizations reviewed and approved the research methods.

Authors' Contributions

A.R. led the grant writing and study design, with contributions from A.Z., A.L., and S.S. M.J. developed the tasks scenarios with input from coauthors; moderated most usability sessions; conducted the analyses for use errors; and drafted the initial manuscript. B.P. recruited patient participants; assisted with demographic, task completion, and efficiency data analyses; and drafted most figures and tables. H.P. moderated some usability sessions and analyzed the satisfaction data. All authors contributed to and approved the final version of the manuscript.

• References

• 1 Cronin RM, Davis SE, Shenson JA, Chen Q, Rosenbloom ST, Jackson GP. Growth of secure messaging through a patient portal as a form of outpatient interaction across clinical specialties. Appl Clin Inform 2015; 6 (02) 288-304
• 2 Heyworth L, Clark J, Marcello TB. , et al. Aligning medication reconciliation and secure messaging: qualitative study of primary care providers' perspectives. J Med Internet Res 2013; 15 (12) e264
• 3 Shenson JA, Cronin RM, Davis SE, Chen Q, Jackson GP. Rapid growth in surgeons' use of secure messaging in a patient portal. Surg Endosc 2016; 30 (04) 1432-1440
• 4 Kruse CS, Argueta DA, Lopez L, Nair A. Patient and provider attitudes toward the use of patient portals for the management of chronic disease: a systematic review. J Med Internet Res 2015; 17 (02) e40
• 5 Haun JN, Patel NR, Lind JD, Antinori N. Large-scale survey findings inform patients' experiences in using secure messaging to engage in patient-provider communication and self-care management: a quantitative assessment. J Med Internet Res 2015; 17 (12) e282
• 6 Henry J, Pylypchuk Y, Patel V. Electronic capabilities for patient engagement among US non-federal acute care hospitals: 2012–2015. The Office of the National Coordinator for Health Information Technology; 2016
• 7 Schnipper JL, Gandhi TK, Wald JS. , et al. Design and implementation of a web-based patient portal linked to an electronic health record designed to improve medication safety: the Patient Gateway medications module. Inform Prim Care 2008; 16 (02) 147-155
• 8 Heyworth L, Paquin AM, Clark J. , et al. Engaging patients in medication reconciliation via a patient portal following hospital discharge. J Am Med Inform Assoc 2014; 21 (e1): e157-e162
• 9 Tang PC, Ash JS, Bates DW, Overhage JM, Sands DZ. Personal health records: definitions, benefits, and strategies for overcoming barriers to adoption. J Am Med Inform Assoc 2006; 13 (02) 121-126
• 10 Chow CK, Redfern J, Hillis GS. , et al. Effect of lifestyle-focused text messaging on risk factor modification in patients with coronary heart disease: a randomized clinical trial. JAMA 2015; 314 (12) 1255-1263
• 11 Osborn CY, Mayberry LS, Wallston KA, Johnson KB, Elasy TA. Understanding patient portal use: implications for medication management. J Med Internet Res 2013; 15 (07) e133
• 12 Kuo A, Dang S. Secure messaging in electronic health records and its impact on diabetes clinical outcomes: a systematic review. Telemed J E Health 2016; 22 (09) 769-777
• 13 Green BB, Cook AJ, Ralston JD. , et al. Effectiveness of home blood pressure monitoring, Web communication, and pharmacist care on hypertension control: a randomized controlled trial. JAMA 2008; 299 (24) 2857-2867
• 14 Raghu TS, Frey K, Chang Y-H, Cheng M-R, Freimund S, Patel A. Using secure messaging to update medications list in ambulatory care setting. Int J Med Inform 2015; 84 (10) 754-762
• 15 Haun JN, Lind JD, Shimada SL. , et al. Evaluating user experiences of the secure messaging tool on the Veterans Affairs' patient portal system. J Med Internet Res 2014; 16 (03) e75
• 16 Irizarry T, DeVito Dabbs A, Curran CR. Patient portals and patient engagement: a state of the science review. J Med Internet Res 2015; 17 (06) e148
• 17 Ferranti JM, Musser RC, Kawamoto K, Hammond WE. The clinical document architecture and the continuity of care record: a critical analysis. J Am Med Inform Assoc 2006; 13 (03) 245-252
• 18 Choi M, Lee HS, Park JH. Usability of academic electronic medical record application for nursing students' clinical practicum. Healthc Inform Res 2015; 21 (03) 191-195
• 19 Russ AL, Jahn MA, Patel H. , et al. Usability evaluation of a medication reconciliation tool: embedding safety probes to assess users' detection of medication discrepancies. J Biomed Inform 2018; DOI: 10.1016/j.jbi.2018.05.002.
• 20 Russ AL, Weiner M, Russell SA, Baker DA, Fahner WJ, Saleem JJ. Design and implementation of a hospital-based usability laboratory: insights from a Department of Veterans Affairs laboratory for health information technology. Jt Comm J Qual Patient Saf 2012; 38 (12) 531-540
• 21 Nielsen J. Demonstrate Thinking Aloud by Showing Users a Video. Available at: https://www.nngroup.com/articles/thinking-aloud-demo-video2014 . Accessed February 23, 2017
• 22 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37 (01) 56-76
• 23 Callahan CM, Unverzagt FW, Hui SL, Perkins AJ, Hendrie HC. Six-item screener to identify cognitive impairment among potential subjects for clinical research. Med Care 2002; 40 (09) 771-781
• 24 Nielsen J. Why you only need to test with 5 users. Useit Com Alertbox; 2000
• 25 FDA. Applying Human Factors and Usability Engineering to Medical Devices. Silver Spring, MA: Food and Drug Administration; 2016
• 26 Lewis JR. IBM computer usability satisfaction questionnaires: psychometric evaluation and instructions for use. Int J Hum Comput Interact 1995; 7 (01) 57-78
• 27 Cooper A, Reimann R, Cronin D. About face 3: the essentials of interaction design. John Wiley & Sons; 2007
• 28 The VA and Academic Medicine: Partners in Health Care, Training, and Research. (2014). Available at: https://www.aamc.org/download/385612/data/07182014.pdf
• 29 Thompson MJ, Reilly JD, Valdez RS. Work system barriers to patient, provider, and caregiver use of personal health records: A systematic review. Appl Ergon 2016; 54: 218-242
• 30 Urowitz S, Wiljer D, Dupak K. , et al. Improving diabetes management with a patient portal: a qualitative study of diabetes self-management portal. J Med Internet Res 2012; 14 (06) e158
• 31 Jahn M, Caldwell BS. Developing Distributed Expertise Coordination Models of Diabetes Care Teams. In: Proceedings of 2018 Human Factors and Ergonomics Society Annual Meeting, Philadelphia, PA (In Press)
• 32 U.S. Department of Veterans Affairs. VA Utilization Profile. Web 2017; 2017: 20