Background and Significance
Patient-reported outcome (PRO) measures are an essential component of quality measurement, quality improvement, and capturing the voice of the patient in clinical care and research.[1 ]
[2 ] In 2004, the National Institutes of Health (NIH) endorsed the importance of PROs by initiating the Patient-Reported Outcomes Measurement Information System (PROMIS). The primary goal of PROMIS is to standardize the measurement of common symptoms, functions, and other aspects of self-reported health to enable efficient and interpretable clinical trial and clinical practice PRO applications.[3 ]
[4 ] Each PROMIS measure addresses a specific symptom such as fatigue, physical function, dyspnea, social function, and so on. Currently, PROMIS includes over 300 measures of physical, mental, and social function in both adult and pediatric populations.[5 ]
A critical component of PROMIS has been the development of computer-adaptive testing measures (CATs).[6 ] CATs were developed using item response theory so that they administer maximally informative items (questions) selected from a large bank of items regarding a specific symptom. In short, the items in each CAT are tailored to the respondent based on their responses to prior items. [Fig. 1 ] shows a schematic representation of CAT administration. Each successive item, or question, is selected based on a probabilistic model that takes into account the statistical properties of the items themselves as well as the participant's responses. This statistical approach is effective for maximizing measurement precision while minimizing measure length.[6 ] This reduced length enables precise scores without undue participant response burden.[7 ] Response burden is associated with less accurate PRO completion, especially for patients with low literacy or clinics with high patient volume, as patients may run out of time before completing a measure or quickly fill in answers to reach the end of longer measures.[8 ] Fixed-length PRO measures are frequently longer in length than CATs and take longer to administer.[9 ]
Fig. 1 Computer-adaptive testing event loop. The survey begins with an assumption of an average T-score of 50, the general population norm. Based on the patient's response to the first question, the next item is selected to give maximal additional information. The cycle is repeated until the confidence in the result is sufficiently high (in other words, the standard error is sufficiently low) or the maximum number of questions is reached.
PROMIS CATs addressing pain, fatigue, physical function, social function, and affect have each demonstrated clinical validity across a wide range of health conditions.[10 ]
PROMIS CATs were initially accessed and administered via a web-based application called Assessment Center (https://www.assessmentcenter.net/ ).[4 ]
[11 ] However, Assessment Center was not originally designed for clinical use, thus did not have features to integrate with any electronic health record (EHR). Consequently, early methods of integrating PROMIS CATs with EHRs relied on a multistep, manual process. For example, patients could first be sent a weblink in a secure patient-messaging portal. Then, the patient would log into Assessment Center and complete the assigned CATs. Next, their provider would log onto their Assessment Center account to access the patient's scores. Finally, the provider would enter the scores into the patient's notes within the EHR. This multistep, manual (“loose”) coupling of CATs and the EHR has several disadvantages: (1) it requires patients to deal with multiple systems, each with a different look-and-feel; (2) it increases both complexity and privacy risk by maintaining patient identifiers and survey schedules in multiple systems; (3) it often results in solutions that are not easily leveraged across different contexts (e.g., a medical assistant manually transferring scores from an Assessment Center report into the EHR); (4) it results in a delay between PRO administration and entry into the EHR which limits PRO use in clinical encounters. For EHR software that supports PRO collection such as Epic (Epic Systems, Verona, WI), this loose coupling also means that PROMIS CAT scores are not displayed alongside other PRO results and cannot utilize Epic's data visualization features.
To address the problems of PROMIS CAT integration into EHRs across multiple health systems, the EHR Access to Seamless Integration of Patient-Reported Outcomes (EASIPRO) consortium funded by the NIH National Center for Advancing Translational Sciences (NCATS) pooled the efforts of nine institutions. This case report describes the technical approach of one specific institution, Northwestern Medicine (NM). NM was the first hospital system to fully integrate PROMIS CATs into Epic. The project implementing PROMIS CATs into NM and Epic was titled Northwestern Medicine Patient-Reported Outcomes (NMPRO).
Objectives
In order for clinicians to utilize PROMIS CAT scores in clinical encounters, the optimal solution is to integrate CATs seamlessly and automatically into the EHR workflow.[12 ] However, CATs require real-time score computation which is generally housed outside of the EHR. Thus, the main objective of NMPRO was to develop custom software that tightly integrated Assessment Center functionality with Epic EHR for seamless, automatic administration and display of PROMIS CAT scores in clinical encounters.
Methods
For the design and software development phase that kicked off NMPRO, an interdisciplinary project team of 24 members was formed. The project team consisted of representatives from the institution's health care clinical information technology (IT) department who specialized in research-focused programming, research informatics, academic specialists from PROMIS, hospital quality assurance, clinic management, and clinician champions from two departments: Orthopedic Surgery and the Cancer Center. These team members were structured into two groups: (1) the working group, or members of IT and research informatics who would build NMPRO software which met on a biweekly basis, and (2) the steering committee, consisting of the remaining academic and clinical partners whose expertise would inform development which met with the working group on a quarterly basis. The joint project team collaborated on all aspects of the project including determination of design criteria, workflow modeling, software architecture, middleware design and development, testing, pilot implementation, and monitoring of hospital-wide implementation. By bringing all stakeholders into one team, development time was significantly reduced because clinical and academic concerns could be addressed in earlier design phases rather than after software development.
Design Criteria
To begin, the project team identified design criteria (see [Table 1 ]). These design criteria were determined first by the clinical members of the project team, including providers and clinical staff, specifying their initial needs which were further refined by iterative discussions among all team members. Although patients were not directly represented in the design phase, academic partners from the PROMIS team referenced their previous patient-centered implementation experiences as well as published literature on the patient experience to ensure that patient needs were being addressed (e.g., in Section 6 of [Table 1 ]).
Table 1
Northwestern Medicine patient-reported outcomes' design criteria
Process step
The system should support
1. Ordering
a. Computer-adaptive tests (CATs) orderable by clinician using normal Epic electronic health record (EHR) ordering process
b. CATs can be triggered based on several kinds of events, such as a clinic visit or a surgical procedure
2. Scheduling
a. Creation of preset timed series of CATs that are ordered or triggered once, but delivered to patient at specified intervals
3. Bundling
a. Multiple patient-reported outcome (PRO) instruments bundled into a single orderable unit
b. Ability to combine CATs and conventional, fixed-length PROs into a single bundle
4. Monitoring
a. Clinician able to monitor completion status of CATs
5. Notification
a. Patient notified that a CAT is due
b. Clinician notified when CAT is complete and results ready for review
c. Selected results identified and pushed to designated staff for immediate attention (i.e., severe Depression score referred to social work)
d. Results routed to different staff for different CAT domains or score ranges
6. Completion
a. Patient able to complete CAT through EHR portal with familiar look and feel
b. Each CAT item has vertical layout of response options as recommended by psychometricians
c. Patient able to complete CAT in the waiting area at arrival at an appointment if it was not completed in the EHR portal
d. Patient able to stop and restart without loss of prior answers
e. Patient able to complete CAT in clinic even if they have not activated EHR portal account
7. Result delivery
a. Result delivered to the clinician faster than patient can walk from the waiting area to the examination room.
8. Result storage and display
a. Result stored with other survey-type data in the EHR, not in a generic, catch-all result type
b. Results displayed using EHR-internal display routines
c. Graphing and trending of results over time
d. Results transferred to the hospital's Enterprise Data Warehouse with other EHR results
9. Alerting
a. Triggering of clinical decision support rules based on results
10. Scaling
a. System is scalable to entire enterprise
Software Architecture and Middleware Development
Next, the clinical IT team members defined the software architecture necessary to achieve the stated design criteria. Based on the hospital's existing IT infrastructure, they identified three components as needing development: a way to access Assessment Center's CAT administration functions separately from Assessment Center's study management functions, custom middleware within the hospital's existing middleware framework to manage the connection between Epic and Assessment Center, and custom code within Epic (see [Fig. 2 ]). Academic team members from PROMIS were crucial in this phase, as their knowledge of the theoretical background and statistical structure of CATs ensured that CATs could be accurately administered within the software.
Fig. 2 Project design architecture. The main software developed for this project was the custom survey management middleware (top left) housed within an existing Northwestern Memorial Healthcare enterprise service bus (“NMH Framework”). The NMPRO project also developed custom code for multiple aspects of Epic (blue box). NMPRO made use of the newly developed Assessment Center API (green). Patient CAT scores were stored within the NMH Research Database within the NMH Framework for access by the middleware and Epic (bottom). API, application programming interface; NMPRO, Northwestern Medicine Patient-Reported Outcomes; PROMIS, Patient-Reported Outcomes Measurement Information System.
Assessment Center Application Programming Interface
To address the first hurdle, the Assessment Center Application Programming Interface (AC-API) was created to support the administration of individual CATs without requiring use of Assessment Center study management functions.[4 ]
[11 ]
[13 ]
Northwestern Medicine Patient-Reported Outcomes Middleware
Next was the creation of custom middleware. The NMPRO middleware was housed within the hospital's existing general-purpose middleware framework for integrating external software systems into Epic. The primary role of the NMPRO middleware was to manage and map states between Epic and AC-API. CATs are highly stateful: when a patient responds to a given item, their cumulative score is adjusted, and the adjusted score is then used to determine the next item to display, or whether to stop item administration altogether (see [Fig. 3 ]).
Fig. 3 Swimlane diagram of the initiation of a PROMIS CAT survey. CAT, computer-adaptive test; EHR, electronic health record; PRO, patient-reported outcome measures.
The second major role of the middleware was creating a survey object within Epic that can be mapped to the appropriate PROMIS CAT items and the CAT final score. We constructed each CAT as a very large questionnaire with each question corresponding to an item in the CAT item bank (e.g., 173 items for the Physical Function item bank). The AC-API directed the middleware to display the appropriate item according to the statistical model (e.g., 4–12 specific items as described in [Fig. 1 ]). In the end, this results in a survey that Epic would recognize as having many unanswered items, but the AC-API would be able to score. The Epic model of a CAT is shown in [Fig. 4 ]. Besides the questions representing the item bank, an additional scoring question was created so that the AC-API could report the final score of the CAT.
Fig. 4 Data model for integration of CATs into conventional EHR format. A dummy questionnaire is created in the EHR that duplicates all items in the CAT item bank. The CAT is modeled as a fixed-length PRO with many unanswered questions. CAT, computer-adaptive test; EHR, electronic health record; PRO, patient-reported outcome.
Finally, the middleware supported combining multiple CATs, or even CATs and other PROs, into a single assessment to be presented to the patient and clinician. For example, in [Fig. 5 ], four PROMIS CATs are combined into one assessment and scores are displayed simultaneously to a clinician in Epic's user interface.
Fig. 5 Screenshot of PROMIS data displayed by native EHR survey module, as seen by a provider. This display went through several iterations based on feedback from clinicians during software development. EHR, electronic health record; PROMIS, Patient-Reported Outcomes Measurement Information System.
Notably, as seen in [Fig. 3 ], the data transferred between the framework and the AC-API included only session, current question, and final score. Therefore, it is possible to host the AC-API in the cloud or on a publicly accessible server with minimal privacy risk. Even so, to eliminate the need for any computing outside of our clinical network, we implemented the AC-API on a virtual machine that ran within our clinical server farm to maximize data security.
Adjustments to Epic
Within Epic, the overall strategy was to “trick” the EHR into treating CATs like other, nonadaptive, fixed-length PROs. This allowed use of Epic's existing ordering, monitoring, and data display functions. Specifically, the survey data structure in Epic was modified to include a flag indicating whether the PRO was conventional (i.e., a fixed-length PRO using Epic's existing survey functions) or CAT. Epic's outpatient survey administration module was modified to check for this flag. If a CAT was detected, control of survey administration was passed to a new CAT module that referenced the AC-API. Once the CAT was completed, the data were stored in the EHR for immediate viewing by clinicians and control returned to the EHR survey module.
The display of CATs within Epic was carefully considered because PROMIS CATs were validated with choices arranged vertically and only one item displayed at a time; we ensured that they were displayed the same way within Epic's existing user interface. This also supports accurate completion on patients' mobile devices and clinic tablets.[14 ]
[Fig. 6 ] shows the final layout for an individual CAT item after iterative feedback from the PROMIS psychometricians within the steering committee.
Fig. 6 A screenshot of a PROMIS question as seen by a patient. This display went through several iterations based on feedback from psychometricians during software development. PROMIS, Patient-Reported Outcomes Measurement Information System.
During development, we became concerned about potential data analyst input errors when configuring PROMIS CATs within the middleware due to the large number of questions we built within Epic. To reduce the risk of analyst input errors, we built a data validation tool that compared the Assessment Center data to the Epic data to identify discrepancies. This quality assurance task required deep integration into the Epic EHR and involved modifying standard Epic software code (see [Fig. 2 ]). We did our best to ensure that modifications were compatible with newer versions of Epic; still, one Epic update conflicted with our customizations which prevented patients from completing CATs for a few days while the problem was identified and resolved. Our testing and validation procedures for Epic updates were modified to include an extra step of checking NMPRO functionality before installation.
Implementation and Monitoring
Existing work describes piloting and implementation of NMPRO in two departments: orthopaedics and oncology.[15 ]
[16 ] These publications include step-by-step instructions and guidance for implementation in other hospital systems. Notably, each of the two piloting clinics developed different workflows and needs regarding the same software, depending on their day-to-day operations.[17 ] For example, orthopaedics ordered PROs to track success of specific surgical procedures such as total joint replacement. Thus, relevant PRO domains such as physical function, pain, and more were ordered for each patient both presurgery and at specified intervals postsurgery. On the other hand, oncology used PROs to regularly monitor symptoms before every scheduled visit. Patients that screened positive for tobacco use were automatically referred to a smoking cessation treatment program.[18 ] Severe PRO scores were set to trigger alerts to members of a patient's care team; for example, social workers would receive a notice when a patient reported moderate-to-severe depression or anxiety.[19 ]
[20 ] Similar cancer symptom monitoring programs have since become available through Epic using PROs other than PROMIS CATs.[21 ]
In terms of project monitoring, an executive steering committee was launched after pilot testing to oversee NMPRO use across the hospital system. The steering committee included 20 members, 4 of whom were involved in the original piloting phase of NMPRO. For the first year of implementation, this group met monthly to actively monitor implementation progress and make iterative changes to software, policy, and educational materials.
The NMPRO software was deployed across 9 specialties and 27 physical locations, supporting over 40 unique questionnaire builds to date. The participating specialties included orthopaedics, oncology, urology, cardiology, dermatology, behavioral health, chronic pain, general surgery, and endocrine surgery. These specialties varied in their patient uptake from 22% (urology) to 57% (endocrine surgery) with an average uptake of 25%. When a clinician requested integration of a new PROMIS measure, IT was able to make it available within three business days. Over 2 years of hospital-wide use, 793 providers collected 70,446 PROs from patients.
Finally, drawing from the work of this project, Epic has developed an application for administering, scoring, and viewing PROMIS CATs.[22 ]
[23 ]
[24 ] This app makes PROMIS CATs accessible to any hospital using Epic without the need for an existing middleware framework as was required by NMPRO. Other hospital systems have implemented PROMIS CATs in different EHRs through independent software that relies on the AC-API.[13 ]
[25 ]
Lessons Learned
Two specific themes emerged as lessons learned over the course of NMPRO: first, communication within the interdisciplinary team was essential to the project's success, and second, the implementation of the project into real-world clinical use required immense effort. Finally, we touch on a few future directions for this work.
NMPRO involved collaboration among two different IT teams (research and informatics), academics of the PROMIS team, and clinicians and clinical staff involved in pilot testing the implementation. We discovered quickly that interdisciplinary teamwork and communication were necessary for the project's success. For example, these groups brought differing experiences and expectations surrounding collaboration in terms of timelines, definitions of success, and amount of resources necessary to accomplish software and clinical goals. These differences were resolved through frequent face-to-face meetings with all team members, a committee charter that explicitly stated the project's goals and their completion timeline, and support from leadership of each group in managing the expectations of their group members. Specifically, the IT team regularly and clearly communicated what deliverables to expect and when, which improved relations with clinical and academic partners. In turn, academic and clinical partners prepared educational materials and specific, actionable feedback to guide software development at appropriate times to address each project goal.
When development ended and the project moved into piloting, our next lesson became clear: real-world use of our software required adjustments to clinical workflow and new thinking on the part of clinicians and patients alike. First, introducing PROs to a clinical practice resulted in significant changes in workflow that required active tailoring for optimal clinic function.[17 ] For example, incomplete PROs were initially administered by medical assistants in the exam room; after feedback from staff that often did not have sufficient time for PROs in the exam room, PROs were instead administered via tablets in the waiting room. Through intensive qualitative work such as interviews with clinicians and staff in the pilot clinics, we developed planning and change-management tools to help clinics identify potential issues and to develop proactive strategies for addressing them, which are freely and publicly available.[16 ] Beyond clinical workflow changes, we had to obtain buy-in from clinicians and patients alike. Use of the software was optional and clinicians did not universally participate—for example, 386 oncologists opted into the tobacco use screener PRO compared with 274 who opted into regular symptom monitoring alerts. Existing literature discusses strategies for obtaining clinician buy-in.[16 ]
[26 ]
[27 ] Other research on electronic PRO implementation demonstrates a strong need to address patient engagement[28 ]
[29 ]—a challenge we faced as evidenced by our 25% patient PRO uptake rate which, though low, is typical.[30 ] Within our clinics, medical assistants struggled to communicate the importance of PROs to patients.[17 ] While strategies for obtaining patient buy-in is covered in previously cited materials, we note that including patient representatives in our design process may have improved the patient experience and thus patient buy-in. It is also worth noting that patients are more open to completing PROs when their clinician references them in their appointment.[8 ]
[31 ] The challenges of bringing software into real-world use should not be underestimated, and the authors recommend thorough review of existing literature on the clinical implementation of electronic PRO systems before attempting them.
The project also left many areas for future research and development. In the course of the project, several features were identified, but were determined to be out of scope for the initial phase. These included (1) a user-friendly graphic for results over time posted to the patient portal (which has since been implemented[32 ]); (2) improved workflows to support clinician interpretation of CAT scores[33 ]; (3) the ability for a clinician to leave a note on a CAT score; (4) integration with the inpatient EHR module. The need for specific features will likely vary within each hospital setting. We were able to address many specific needs early in the project because multidisciplinary stakeholders were included in the design phase; however, desire for new features will inevitably emerge later, only after piloting and implementation. In this sense, designing and implementing electronic PRO software for real-world use will always be an iterative process.
Conclusion
NMPRO succeeded in achieving seamless integration of PROMIS CATs into Epic. NMPRO also informed other projects by members of the EASIPRO consortium in integrating PROMIS into other EHRs.[15 ]
[34 ]
[35 ] NMPRO's software architecture as described here directly informed the development of the Epic PROMIS CAT application, which currently provides all functionality described in this article to any hospital system that uses Epic.
This case report underscores informatics fundamentals: to be successful, clinical projects need to simultaneously address technical details such as data structures as well as sociotechnical issues such as clinical workflow and patient and provider user experience. The inclusion of clinical and academic stakeholders in early design and development improved our process, but developing software for live hospital systems will always involve iteration to some extent.
