Applying User-Centered Design Methods to the Development of an mHealth Application for Use in the Hospital Setting by Patients and Care Partners

Abstract

Introduction Developing an optimized and user-friendly mHealth application for patients and family members in the hospital environment presents unique challenges given the diverse patient population and patients' various states of well-being.

Objective This article describes user-centered design methods and results for developing the patient and family facing user interface and functionality of MySafeCare, a safety reporting tool for hospitalized patients and their family members.

Methods Individual and group usability sessions were conducted with specific testing scenarios for participants to follow to test the usability and functionality of the tool. Participants included patients, family members, and Patient and Family Advisory Council (PFAC) members. Engagement rounds were also conducted on study units and lessons learned provided additional information to the usability work. Usability results were aligned with Nielsen's Usability Heuristics.

Results Eleven patients and family members and 25 PFAC members participated in usability testing and over 250 patients and family members were engaged during research team rounding. Specific themes resulting from the usability testing sessions influenced the changes made to the user interface design, workflow functionality, and terminology.

Conclusion User-centered design should focus on workflow functionality, terminology, and user interface issues for mHealth applications. These themes illustrated issues aligned with four of Nielsen's Usability Heuristics: match between system and the real world, consistency and standards, flexibility and efficiency of use, and aesthetic and minimalist design. We identified workflow and terminology issues that may be specific to the use of an mHealth application focused on safety and used by hospitalized patients and their families.

Keywords

user-centered design - patient safety - patient- and family-centered care - mHealth

Introduction

Efforts to make e-Health tools more usable for patients and care partners (i.e., family and friends) have included a focus on the design of patient portals and mHealth applications.[1] [2] [3] mHealth “apps” initially gained traction as consumer products that were part of the quantified self-movement and there is increasing recognition that mHealth apps may aid in self-management in the ambulatory setting.[1] mHealth applications are increasingly being used in the hospital setting, yet there is a need for more research around patient/care partner usability needs in the hospital setting.[2] The Apple Research Kit is one example of a tool that is innovating ways in which patient-facing research is conducted at major hospital institutions and offers useful solutions that can be applied to many patient-facing apps (https://goo.gl/KYxVfG). Several academic medical centers have designed patient portals for the acute care setting.[4]

The learning opportunities that e-Health tools for inpatient use open up for researchers and health care systems are numerous and can help provide a new source of information on patients and their care partners. Recent studies show that there is patient willingness to participate in systems that allow them to communicate safety events.[5] [6] Patients showed high levels of involvement in both participating in questionnaires about safety and utilizing electronic tools to report safety concerns. However, both studies showed that it is difficult to standardize data collection of patient-reported safety concerns, and developed electronic tools are not always immediately generalizable across a variable participant population. In developing e-Health tools, it is important to remember that hospitalized patients have different characteristics than the general population, and within a given population of hospitalized patients there can be wide variation of socioeconomic status, baseline use of technology, and literacy and e-Health literacy levels.[3] Care partners of hospitalized patients may experience emotional stress and increased learning needs related to their loved one's hospitalization.[4] [7] [8] [9] These emotional and situational factors add to the importance of conducting participatory design when developing an mHealth app for hospitalized patients and their families.[10] e-Health literacy is related but different than general literacy levels, and furthermore a patient's lowered health status when hospitalized could impact his or her ability to interact with new technology.[11] In addition, appropriate use of consumer health terms are especially important for any eHealth tools.[12] [13] Participatory design with end-users may lead to better engagement, and more engaged patients have been linked to better outcomes, shorter length of stay, and decreased costs compared with less engaged patients.[2] Given these reasons, we identified the need to apply user-centered design methods for the development of the patient- and care partner-facing user interface and functionality of the MySafeCare Safety Reporting Application (MySafeCare [MSC] App).

We performed this study at a large academic medical center in the Northeast United States. The MSC App is a Web-based and mobile-enabled safety reporting application that facilitates real-time communication of concerns and worrisome events from hospitalized patients and care partners to clinical staff. While there are several ways to capture and address safety events identified by clinicians, there are minimal systemic efforts and systems in place to learn more about safety issues from the patients' and care partners' perspective.[5] [6] [14] [15] Patient safety largely remains a clinician-centric topic despite evidence of a unique patient perspective in detecting safety events.[14] A handful of systems exist, yet, overall, direct and real-time reporting from patients/care partners in the hospital setting remains a critical gap in patient safety data sets and was a motivation for the development and user-centered design of MSC.[5] [6]

The MSC App has two components: a Patient and Carepartner Facing Component and a Clinical Dashboard Component. The Patient and Family Facing Component is designed to run on a mobile device, tablet, or laptop computer. It provides the user (patient, friend, or family) with the option to identify themselves or remain anonymous, indicate their type of concern through the 9 concern category icons, enter the level of severity of the concern, and enter any free text to explain in their own words. See [Fig. 1] for screenshots of the final version of the Patient and Carepartner Facing Component. The Patient and Carepartner Facing Component also includes a Compliments category for compliments about staff or care received. The Clinical Dashboard displays the submissions to appropriate clinical staff members (nurse and medical directors, patient relations, and/or clinical staff). We have published prior work on the user-centered design of the Clinical Dashboard.[16] This article describes the usability testing conducted with hospitalized patients and care partners for the iterative design and refinement of the Patient and Carepartner Facing Component of the MSC App. This study was approved by the hospital's Institutional Review Board (IRB).

Fig. 1 Version 2 patient-facing app overall screenshots of three separate pages for easier user navigation.

Methods

The user-centered design method allows for direct end-user engagement throughout the design process, facilitating the development of a tool that can provide the most beneficial outcome possible.[10] Part of this method includes conducting extensive usability testing on an initial version to identify the enhancements needed around usability and functionality. An initial MSC paper-based prototype was designed with a patient representative, patient and family relations director, and clinical staff from our institution during a Hackathon and refined into a wireframe prototype. This wireframe prototype was further refined based on preliminary stakeholder interviews with Patient and Family Advisory Council (PFAC) members (a hospital-sponsored council comprised of former patients and family members aimed at improving patient experience) and clinical staff and used to develop MSC Version 1. The following data sources informed the initial identification and design of the 9 concern categories used in the App (e.g., medication, pain, infection control, and medical device issues): (1) publications of patient-reported service quality incidents,[17] (2) a hazard and near-miss reporting system,[18] (3) patient reports on service quality and safety, with a focus on waits and delays, communication between staff and patients, and environmental issues and amenities,[19] and (4) preliminary stakeholder interviews with PFAC members and clinical staff. All of this work produced Version 1 of the MSC App. The methods below describe the user-centered design process performed on Version 1.

MySafeCare App Version 1 Usability Testing

We conducted individual scenario-based usability sessions, interactive group usability sessions, and patient/care partner engagement rounds between February 2015 and September 2015 on Version 1 of the MSC App. The participants included hospitalized patients, care partners, and members of PFAC. Targeted clinical units included the medical intensive care unit (MICU), oncology unit, and intermediate vascular unit.

Individual Usability Sessions

Scenario-based usability sessions were conducted as 15- to 20-minute sessions in which participants were asked to complete tasks in the MSC App for a set of predefined scenarios. Patients who were alert and clinically stable (per their nurse) were asked to participate and care partners, if present, were invited to participate as well. Given the level of patient acuity in the MICU, only care partners completed the usability sessions on the MICU. Scenarios were developed by research team members, including health systems engineering and clinical experts, to reflect typical situations for patients while allowing for systematic evaluation of specific tasks. Each scenario included two user tasks. In total, 18 scenarios with 36 tasks were developed to enable sufficient testing of each of the concern categories as well as other content within the application (see [Table 1] for the main targeted functionality for testing and an example task). Each task mapped to a specific feature that was being tested in the scenario. Each participant was given one of the 18 scenarios. Two researchers were always present for the sessions: one to conduct the session and the other to take detailed notes and record time of completion for tasks.

Table 1 Types of features targeted for testing and example tasks

Main targeted features for testing

• Anonymous versus identified submission

• User type (patient, family member, or friend)

• Concern level

• Concern timeline

• Entering multiple concerns in one submission

Example tasks from usability testing

1. You want to remain anonymous

2. You are the patient

3. You are very concerned with your safety

4. Your safety concern happened more than 2 d ago

5. You have not shared your safety concern with the Care Team

6. You do not plan to share your concern with the Care Team

7. You are concerned that: “When people come into your room they always wear yellow gowns. Then someone comes in without wearing a yellow gown”

8. If you choose to do so, please complete the background information about yourself

Participants were asked to “think-aloud” as they completed the scenarios on an iPad. Using a predefined paper-based template to maintain consistency of the procedures and types of data captured between participants, a researcher captured: participants think-aloud comments, task completion successes and errors, instances when assistance was required to complete the task, and amount of time required to complete the given task.

Group Usability Sessions

Hour-long interactive group usability sessions were also conducted with the hospital's PFAC. PFAC participants were shown the MSC App and were given two scenarios to walk through to simulate submitting a concern. All participants were given the same scenarios to facilitate group discussion after completing the submission. Detailed notes of the dynamic interactive group discussions were recorded by the research team.

Engagement Rounds

Finally, we conducted biweekly engagement rounds to patient rooms and family waiting room areas from April 2015 to October 2015. MSC App Version 1 was available for use on clinical units during this period, while iterative design and development continued to inform refinements for Version 2. Engagement rounds were conducted to inform patients and care partners of the availability of the MSc App, how to access it, and answer any additional questions. Patients and care partners that were interested and engaged would often ask to use the app right then. Any questions, concerns, issues, or difficulties that came up during these engagement rounds were noted by the team members and triangulated with data from our individual and group usability testing to further inform iterative development.

Analysis of Usability Work and Theme Development

Three steps, each with several substeps, were used to identify specific requirements for application revisions: (1) Theme Development: (a) compilation of all notes from usability sessions and engagement rounds, (b) coding of discrete issues, (c) grouping of discrete issues into categories, (d) identification of theme for each category; (2) Association to Nielson Heuristics: (a) user-driven requirements and identified issues, categories, and themes were discussed for group consensus and mapped to Nielson Heuristics during team meetings; (3) Versioning of the MSC App: (a) translation of MSC App issues for each theme into specific requirements to inform the development of Version 2 of the MSC App, (b) implementation of specific requirements for Version 2, and (c) continuation of the iterative user-centered design process for new version of the MSC App.

Detailed documentation from the usability testing sessions and engagement rounds—particularly difficulties or issues users experienced when completing the scenarios and “think-aloud” comments—were analyzed in depth. Two research team members performed the thematic qualitative coding of the data with one team member as the primary coder performing the initial coding and the second team member refining and validating the coding. The coded themes were discussed and confirmed with the larger research team for consensus. QSR International's NVivo 10 software for qualitative data analysis was used to capture and organize the results. Data collection continued until the iterative analysis of issues indicated data saturation had been reached. Nielsen's 10 heuristics identify general guidelines for developing a user-friendly tool; the themes identified in our usability analysis of the MSC App were evaluated in the context of how they related to Nielsen's 10 Usability Heuristics.[20]

Results

MySafeCare App Version 1 Usability Testing

A total of 11 individual usability sessions, 3 small group usability sessions with 25 members of PFAC, and over 250 interactions with patients and families during engagement rounds were conducted to inform usability of the Patient and Carepartner Facing Component of the MSC App. Each PFAC session had on average 6 to 10 participants, and consisted of a combination of former patients and care partners. Over the course of 7 months, up to 286 participants interacted with the MSC App.

Analysis of Usability Work and Theme Development

• Step 1: Theme Development

Twelve discrete issues were categorized during analysis of the user-centered design sessions on Version 1 as necessary revisions for Version 2 of the MSC App. These were further categorized into three distinct themes which fell under the headings of terminology, workflow functionality, and user interface design ([Table 2]).

Table 2 Identified Version 1 application issues, associated themes, and example revisions for Version 2

Themes	Issues	Example revisions
Workflow functionality	1. Informed consent text in introduction is too long to read/understand 2. Need to capture if user “Accepts” or “Does not accept” the informed consent information 3. When entering more than one concern in a single submission it is not intuitive to click the “plus” button. Additional functionality is needed to associate each concern with a severity level 4. A category to submit a compliment should be added	For issue #3: Logic revised so that user is prompted to choose either a Concern or Compliment which automatically navigates to appropriate submission page Submission page includes buttons labeled “Add Another” and “Remove” to allow for additional submissions or modifications by the user
Terminology	The term “safety concern” is confusing. A phrase that is better understood by users is “worrisome or concerning event” Concern category labels are confusing. The addition of “My” is clearer to users that the categories relate to their care and experience The use of the following terms in the app generated confusion: “caregiver,” “medical device,” “infection” category, “self.” More clear and directed terminology includes “family, “hygiene,” and “I am the patient” 4. Specification of how to answer the optional background questions is needed based on user type (patient, family, friend)	For issue #1: All references to “safety concern” were revised to “worrisome or concerning event,” including consent forms, posted flyers, and engagement rounds discussion. Additionally, the question “Would you say your stay has gone perfectly? If not, we would like to know” was added to the first page to convey that we are seeking a broad spectrum of safety and quality issues
User interface design	1. “Minus” and “Plus” buttons for adding/deleting a concern are not labeled and therefore hard to find and understand their function 2. The icons and labels for the concern categories are too small and hard to read/see, especially on smaller devices 3. The hospital logo is not in an obvious position (off to the side), and there is too much extra white space on the introduction page 4. All contents of the application are on one long page, which requires cumbersome scrolling and often results in users missing sections of the application	For issue #4: The application was modified to stratify content across 7 short pages based on the natural grouping of questions to limit the amount of scrolling required of users

After review of data and discussions to achieve group consensus with research team members, the final high-level themes identified as major design implications when creating a mobile application for patient/family use in an acute care hospital setting were:

1. Terminology to match hospitalized patient and family health literacy and shared understanding.

2. Simplified intuitive workflow functionality for patients and families who may experience barriers to navigating apps while in the hospital setting.

3. User interface design for mobile devices.

• Step 2: Association to Nielson Heuristics

[Fig. 2] shows the four heuristics that we found were specifically identified by hospitalized patients and care partners as needing refinement during our in-depth participatory design: match between system and the real world, consistency and standards, flexibility and efficiency of use, and aesthetic and minimalist design.

Fig. 2 MySafeCare as an mHealth tool compared with Nielsen's Usability Heuristics.

• Step 3: Versioning of the MSC App.

Workflow Functionality

Three issues associated with the workflow of the MSC App were highlighted during both the individual and group usability testing sessions. The scenario walk-through was challenging for users of Version 1 when trying to enter more than one concern at a time. Specific issues included that end-users had difficulty in navigating to the “plus” and “minus” buttons that denote adding or removing a concern, and end-users were confused how to choose a severity level when more than one concern was involved. These issues required revision to the sequence of items presented to the end-user and branching logic for a simpler and more intuitive workflow. [Fig. 3] illustrates these issues on Version 1 and the updated Version 2.

Fig. 3 MySafeCare Patient and Family Facing Component–Version 1 with annotations for refinement based on user feedback, and Version 2.

Terminology

During engagement rounds, our research team members used the term “safety concern” to describe to the purpose of the MSC App to patients and care partners. However, this was consistently met with confusion; “safety concern” was interpreted as an event in which actual harm had occurred, while the intent of the MSC App is to capture experiences in which the patient or family felt threatened and could potentially lead to a harmful event. The phrase “worrisome or concerning event or situation” was identified as the best phrase to reduce this semantic mismatch. Through usability testing, we found that the content of Version 1 of the application did not sufficiently account for health literacy levels of the patient and family population. One concern, category labeled Medical Device, raised confusion since not all patients or care partners are aware of what is considered a medical device. This was reconfigured into the “My Room” category to more accurately capture safety issues relating to a patient's immediate surroundings.

[Table 3] depicts the initial Version 1 concern categories and the final Version 2 categories and subcategories. Notably, “Medical Device” was changed to “My Room,” and “Infection” was changed to “My Hygiene.” The use of “My” in each concern category is one example of a finding and design change from a PFAC group session aimed at personalizing the application for users so that they understand this is asking about their experiences. Both PFAC and patient/care partner participants also confirmed that a free text box should be available for users to explain the concern in their own words.

Table 3 Final categories, subcategories, and icons for the MySafeCare App Version 2

Version 1	Version 2 (Final)
Concern categories	Concern categories	Icon images	Associated subcategories
			I/my family caregivers don't know my plan of care
			I/my family caregivers feel like my care team isn't following my plan of care
Plan of Care	My Plan		I/my family caregivers feel like there is a problem with my plan of care
			I/my family caregivers have other treatment concerns (please explain in box below)
			I was given the wrong medication or dose
			I was almost given the wrong medication or dose
Medication	My Medication		I was not given my medication on time
			I missed a medication
			I/my family/caregivers have other medication concerns (please explain in box below)
			My medical device is not working
			My medical device will not stop beeping
Medical Device	My Room		My medical device seems excessive My room is not clean
			I/my family caregivers have other medical device concerns (please explain in box below)
			I/my family caregivers don't feel respected
			My and/or my family caregivers' needs are being ignored
Communication	My Communication		I/my family caregivers am/are concerned about the communication between my care team about my plan of care
			I/my family caregivers am/are concerned about how my care team communicates with me about my plan of care
			I/my family caregivers have other communication concerns (please explain in box below)
			A member of my care team did not wash his/her hands
			A member of my care team did not wear gloves
Infection	My Hygiene		A member of my care team did not follow the precautions on the door
			I/my family caregivers have other infection concerns (please explain in box below
			My and/or my family caregivers' privacy is/are being ignored
Privacy	My Privacy		I/my family caregivers have other privacy concerns (please explain in box below)
			I/my family caregivers feel that my care team is not managing my pain to my expectations
Pain	My Pain		My pain is well controlled but I/my family caregivers am/are concerned about the medication
			Nobody has asked me or my family caregivers if I am in pain
			I/my family caregivers have other pain management concerns (please explain in box below)
			I am waiting too long for help going to the bathroom
			I am waiting too long for help turning and moving in bed
			I am waiting too long for my procedure
Waiting Time	My Waiting Time		I am waiting too long to be transferred
			I am waiting too long to be discharged
			I/my family caregivers have other waiting time concerns (please explain in box below)
Other	Other

We also observed variation in how patients and care partners categorized safety concerns from the provided scenarios. Specifically, 6 out of the 11 participants that took part in the individual usability testing sessions selected a different concern category for a given scenario than what was intended by the research team when drafting the scenario. For example, a given scenario was: “Your mother who visits frequently doesn't understand your care plan.” This was intended to fall under the category “Plan of Care (My Plan in Version 2)”; however, the patient interpreted it as a Communication issue. Another scenario was: “When people come into your room they always wear yellow gowns. Someone came in today without wearing a yellow gown.” This was intended to be an Infection concern (My Hygiene in Version 2), but was entered under the Plan of Care (My Plan in Version 2) concern category. This mismatch in interpretations highlights the need to learn more about patient and care partner perceptions on safety and what they deem threatening. It also highlights that a given safety concern can be categorized into more than one categories, and these categorizations may be perceived differently by different individuals. This finding emphasized the need for the free text box to allow the user to express the concern in his or her own words to ensure better communication of the issue at hand.

User Interface Design

Some of the user interface problems that were identified included the visibility of concern category icons, appropriate labeling of navigation features, and the amount of scrolling required to complete the submission. Version 1 was formatted to have all content on one page, which included lengthy informed consent information, questions to identify the user, concern submission content, and optional background questions. While this approach was intended to decrease the need for users to navigate multiple pages, the usability testing feedback identified that it was cumbersome. This intensive scrolling was an issue for two reasons: (1) some users were unaware they had to scroll for more content and may have stopped without knowing how to proceed if a research team member was not present to point it out, and (2) patients that reported they were particularly ill, tired, or had just undergone testing or treatment of some sort often stated that they did not have the energy for the level of reading and scrolling that was required to get through the MSC App. Version 2 was redesigned with page breaks based on user feedback, implemented at natural partitions in application content so that each page required minimum or no scrolling with the “Next” button clearly visible.

Overall screenshots of versions 1 and 2 of the MSC App are shown in [Figs. 1] and [4], respectively. Version 1 is all one long page, whereas in Version 2 each screenshot represents a different page.

Fig. 4 Version 1 patient-facing app overall screenshots of single page requiring scrolling.

Discussion

The hospital can be a very stressful environment for both patients and their family. Introducing new elements like the MSC App requires analysis of, and integration with, the users and their specific needs and restrictions. The three major high-level themes identified through usability testing, restated below, reflect on the importance of end-user engagement and may be applied to other types of mHealth applications for hospitalized patients:

1. Simplified intuitive workflow functionality for patients and families who may experience barriers to navigating apps while in the hospital setting.

2. Terminology to match hospitalized patient and family health literacy and shared understanding.

3. User interface design for mobile devices.

In our usability testing, we identified key issues and achieved data saturation quickly in identifying the types of challenges and struggles users experienced with Version 1. Patients and care partners identified issues and suggested better approaches to improve the design, confirming the value and need for participatory design.

The workflow issues we identified aligned with Nielsen's flexibility and efficiency of use heuristic. For example, users identified challenges to entering more than one Compliment or Concern and general navigation of the application. In response, we revised the logic configured within the app to provide more intuitive and efficient navigation consistent with end-user preferences, while still allowing the user the flexibility to enter multiple concerns or compliments and revise prior data entries.

Our user interface design changes aimed to enhance the aesthetic and minimalist design of the app to decrease user burden for patients and care partners with varied levels of comfort with technology and levels of illness. The user interface issues we identified likely apply to mobile app development overall, as confirmed by design guidance published by Apple Research Kit.[21] We identified major usability issues in Version 1 due to excessive scrolling required of the user to navigate the application. One major barrier to use was the length of the informed consent text required by the IRB to inform users that MSC is a research study and provide guidance to the user in case of a serious safety concern or incident. Participants noted that patients who were sicker would not have the energy to read the lengthy text. In response, we collaborated with the IRB to revise the text and provided a link to a video we created communicating the informed consent information. We also mimicked the style of common mobile apps through pagination of different sections[21] based on the types of questions, resulting in a few questions per page to minimize scrolling, and applied branching logic to the questions to decrease user burden.

Notable, there are a few differences between the MSC Safety Reporting Tool and other research apps targeted at patient users that we are aware of, specifically relating to the informed consent text and account creation upon initial use of the app. Account creation is typical on many other research apps (e.g., Apple Research Kit), and users of Apple Research Kit apps can bypass the informed consent information after the initial login. With MSC, patients and care partners may submit anonymously, therefore we do not capture user signatures or create accounts (in the app or for IRB consent) to preserve anonymity of participants. The requirement that patients/families trusted that their anonymity was preserved informed our approach to use a Web-based platform instead of requiring downloading to a device, as is typical of apps. We found that provision of all required informed consent information while designing anonymous reporting applications that are intuitive, easy to navigate, and quick is challenging, but possible. We expect that other patient-facing apps may have similar requirements and recommend future work to identify best practices for mHealth apps that garner patient trust and are enabled to preserve anonymity while meeting other research or operational needs.

Making sure that information and terminology is consistent and understandable across varying users is essential to the success of an application and our findings reflect Nielsen's Heuristics of “match between system and the real world” and “consistency and standards.” MSC targets “near-miss” and “unsafe condition” cases as defined by the Agency for Healthcare Research and Quality (AHRQ) Common Format[21]; in other words, situations/events that are worrying and concerning to patients and care partners have the potential to turn into actual safety events, and could be mitigated in real-time and used to improve our safety systems overall.[14] Tailoring the phrasing and word choice of the application content to effectively describe the intent of the application to the end-user is a major priority. The semantic mismatch we observed in patient/care partner understanding of the phrase “safety concern” is a great example of how there can be impactful differences in interpretation between clinical/research staff and patients and care partner, in this case around patient safety concepts. We learned that when a patient hears “safety” they think “physical harm” that has already occurred and tended to focus primarily on medication-related events and not consider other types of near misses, such as communication concerns. We found our participatory design approach, particularly our patient and care partner engagement rounds, essential to understand how best to convey that MSC intends to capture incidents not only after they occur, but at any point in time in which an experience as a patient does not seem appropriate. These engagement rounds complemented the scenario-based usability sessions to identify themes from a large sample of hospitalized patients, and allowed for observation of how well the application was being used—and understood—“in the wild.” We note that struggles observed during usability sessions could be magnified for patients that are more acutely ill or that have just undergone a treatment, thus highlighting the importance of this method for designing an electronic tool for use in the hospital setting.

Based on our findings, we recommend that participatory user-centered design of mHealth apps for patients and care partners in the hospital setting particularly emphasize four heuristics: match between system and the real world, consistency and standards, flexibility and efficiency of use, and aesthetic and minimalist design. We found the process of addressing these four heuristics with end-users improved design aspects aligned with all of Nielsen's heuristics, and this should be confirmed with future work. For example, simplifying app design workflow to provide flexibility and efficiency of use, such as including easier navigation between pages, may also enable users to recover if they become lost or enter data in error and may increase visibility of system status. A key principal in our design was to make MSc efficient and flexible to balance the collection of reliable, structured, coded data while not preventing partial data collection that included enough information for a real-time response or continuous learning. We emphasized efficiency by utilizing short, structured questions with predefined coded values, but also made these questions optional and allowed for unstructured, flexible data entry in free text form, and in doing so provided control and freedom to users. Finally, Nielsen's “recognition rather than recall” heuristic is interesting to note from our study's perspective because while it is important in all systems, it could be considered a defining design characteristic and assumed feature of an app targeted at infrequent or one-time end-users with no expectation of training or baseline knowledge, such as hospitalized patients.

A final noteworthy challenge is how to make hospitalized patients aware of new initiatives related to patient engagement. Based on this study and prior studies, our team has learned that providing viewable information in the patient room to notify patients/families is useful when combined with the conduct of daily engagement rounds.[22] [23] Patient engagement in the hospital setting represents a challenging and evolving area within implementation science.

Limitations

A limitation of this study was that usability sessions were not audio or video recorded, which may have allowed for deeper analysis of think-aloud comments. Video recording was out of the scope of this study (our devices were Apple iPads which do not allow for screen recording), though may have added to our data collection on errors and time spent navigating the application. However, all sessions were conducted with two researchers, with one exclusively responsible for detailed note taking and time keeping. Additionally, systems for patient-reported safety concerns will have an inherent limitation in that interpretation of safety issues is complex. Safety issues are very often multifactorial, so the design of a system to capture concerns is a learning process. Finally, while usability testing did not test users' ability to access the app on their own device, we did observe patients/care partners accessing the app on their own during engagement rounds.

Conclusion

Three major themes resulted from in-depth participatory design work regarding major areas to address when developing a mobile patient safety app for use in a hospital setting: workflow functionality, terminology, and user interface design. Workflow issues highlighted functionality in the app that were not intuitive or easy to navigate for hospitalized patients. We identified terminology issues that are likely specific to the domain of patient safety from the patient and care partners' perspective. The user interface design issues identified were consistent with existing guidelines for mobile applications, confirming the dependability of our findings.[23] Future work should investigate if similar themes surface during user-centered design of other patient-facing applications.

We illustrated that the issues identified for these themes specifically aligned with four of Nielsen's Usability Heuristics: match between system and the real world, consistency and standards, flexibility and efficiency of use, and aesthetic and minimalist design. While several user issues identified may not be limited to the inpatient population, the experience of being acutely ill in the hospital may enhance barriers to use. Our findings support “in-the-wild” testing in designing an mHealth app intended for use by patient and care partners in the hospital setting. We recommend more research to determine best practices and guidelines for apps targeting these populations. Our future work includes further optimization of MSC through implementation and evaluation across different clinical units to collect usability feedback from varying patient populations.

Multiple Choice Questions

1. Which of the following related to the patient and care partner perspective of care concerns and safety reporting is true:

   • A patient or family perspective of safety threats may differ from a clinician's perspective of safety threats

   • Fear of retaliation never occurs for patients or families that have safety incidents or threats they want to report

   • Only patients and families with low health literacy need to be educated about how to report safety threats

   • Care concerns from the patient and family perspective are narrow in scope and are not safety threats

   Correct Answer: The correct answer is option a. Patients and families have a unique and valuable perspective of care concerns and safety threats and these perspectives may differ from clinicians' traditional perspectives of patient safety.

2. Which of the following are key criteria for design of mHealth applications used by patients and care partners in the hospital setting?

   • Simple, intuitive design and language that fits literacy level requirements

   • Requires user training conducted by a full-time devoted IT resource

   • Includes medical abbreviations and terminology used by clinicians

   • Identifies patients and care partners through a unique identifier

   Correct Answer: The correct answer is option a. Hospitalized patients and family members are under emotional stress, which can lead to increased learning needs, therefore necessitating simple mHealth tools with low learning curves and simple, direct language. Additionally, patients and family members may have a wide range of literacy and health literacy levels.

Conflict of Interest

None.

Protection of Human and Animal Subjects

This study was reviewed by and performed in compliance with the Institutional Review Board.

• References

• 1 Majmudar MD, Colucci LA, Landman AB. The quantified patient of the future: Opportunities and challenges. Healthc (Amst) 2015; 3 (03) 153-156
• 2 Prey JE, Woollen J, Wilcox L. , et al. Patient engagement in the inpatient setting: a systematic review. J Am Med Inform Assoc 2014; 21 (04) 742-750
• 3 Goggins KM, Wallston KA, Nwosu S, Schildcrout JS, Castel L, Kripalani S. ; Vanderbilt Inpatient Cohort Study (VICS). Health literacy, numeracy, and other characteristics associated with hospitalized patients' preferences for involvement in decision making. J Health Commun 2014; 19 (Suppl. 02) 29-43
• 4 Shankar KN, Hirschman KB, Hanlon AL, Naylor MD. Burden in caregivers of cognitively impaired elderly adults at time of hospitalization: a cross-sectional analysis. J Am Geriatr Soc 2014; 62 (02) 276-284
• 5 Schneider EC, Ridgely MS, Quigley DD. , et al. Developing and Testing the Health Care Safety Hotline: A Prototype Consumer Reporting System for Patient Safety Events. Final Report (Prepared by RAND Corporation under Contract HHSA2902010000171). Rockville, MD; 2016
• 6 Lawton R, O'Hara JK, Sheard L. , et al. Can patient involvement improve patient safety? A cluster randomised control trial of the Patient Reporting and Action for a Safe Environment (PRASE) intervention. BMJ Qual Saf 2017; 26 (08) 622-631
• 7 Andresen M, Guic E, Orellana A, Diaz MJ, Castro R. Posttraumatic stress disorder symptoms in close relatives of intensive care unit patients: Prevalence data resemble that of earthquake survivors in Chile. J Crit Care 2015; 30 (05) 1152.e7-1152.e11
• 8 Lloyd RB, Rosenthal LJ. Acute traumatic and depressive symptoms in family members of hospitalized individuals with delirium. Int J Psychiatry Med 2015; 50 (02) 191-202
• 9 Bužgová R, Kozáková R, Sikorová L. Assessment of quality of life of family members of inpatients with end-stage disease. J Palliat Care 2015; 31 (04) 250-257
• 10 Dykes P, Stade D, Chang F. , et al. Participatory Design and Development of a Patient-centered Toolkit to Engage Hospitalized Patients and Care Partners in their Plan of Care. AMIA Annual Symposium Proceedings; 2014
• 11 Fidopiastis CM, Rizzo AA, Rolland JP. User-centered virtual environment design for virtual rehabilitation. J Neuroeng Rehabil 2010; 7: 11
• 12 Taha J, Sharit J, Czaja SJ. The impact of numeracy ability and technology skills on older adults' performance of health management tasks using a patient portal. J Appl Gerontol 2014; 33 (04) 416-436
• 13 Sarkar U, Karter AJ, Liu JY. , et al. The literacy divide: health literacy and the use of an internet-based patient portal in an integrated health system-results from the diabetes study of northern California (DISTANCE). J Health Commun 2010; 15 (Suppl. 02) 183-196
• 14 Weingart SN, Pagovich O, Sands DZ. , et al. What can hospitalized patients tell us about adverse events? Learning from patient-reported incidents. J Gen Intern Med 2005; 20 (09) 830-836
• 15 Hincapie AL, Slack M, Malone DC, MacKinnon NJ, Warholak TL. Relationship Between patients' perceptions of care quality and health care errors in 11 countries: a secondary data analysis. Qual Manag Health Care 2016; 25 (01) 13-21
• 16 Couture B, Cleveland J, Ergia A. , et al. User-Centered Design of the Clinical Dashboard for the MySafeCare Patient Safety Reporting System. AMIA Annual Symposium Proceedings; 2015:1. Available at: https://knowledge.amia.org/59310-amia-1.2741865/t005-1.2744350/f005-1.2744351/2247185-1.2745295/2247185-1.2745296?timeStamp=1457468408316 . Accessed March 8, 2016
• 17 Weingart SN, Pagovich O, Sands DZ. , et al. Patient-reported service quality on a medicine unit. Int J Qual Health Care 2006; 18 (02) 95-101
• 18 Currie LM, Desjardins KS, Levine ES. , et al. Web-based hazard and near-miss reporting as part of a patient safety curriculum. J Nurs Educ 2009; 48 (12) 669-677
• 19 Weissman JS, Schneider EC, Weingart SN. , et al. Comparing patient-reported hospital adverse events with medical record review: do patients know something that hospitals do not?. Ann Intern Med 2008; 149 (02) 100-108
• 20 Nielsen J. Heuristic evaluation. In Nielsen J, Mack RL. (Eds.), Usability Inspection Methods, New York, NY: John Wiley & Sons; 1994. b
• 21 Apple Inc. Apple Research Kit. Available at: http://www.apple.com/researchkit/ . Accessed March 8, 2016
• 22 Dalal AK, Dykes PC, Collins S. , et al. A web-based, patient-centered toolkit to engage patients and caregivers in the acute care setting: a preliminary evaluation. J Am Med Inform Assoc 2016; 23 (01) 80-87
• 23 Dykes P, Stade D, Chang F. , et al. Participatory Design and Development of a Patient-centered Toolkit to Engage Hospitalized Patients and Care Partners in their Plan of Care. Proceedings of the American Medical Informatics Association Annual Fall Symposium; November 16–19, 2014 . Washington, DC