Decision-Centered Design of Patient Information Visualizations to Support Chronic Pain Care

Keywords

electronic health records and systems - clinical decision support - data visualization - ambulatory care/primary care - cognition

Background and Significance

Primary care clinicians often make care decisions under significant time constraints,[1] [2] and in environments characterized by missing, scattered, erroneous, and conflicting information.[3] Increasingly, clinical decision support systems help clinicians make sense of this information chaos by executing rules that focus attention on recommended tests or treatments.[4] [5] [6] Yet, for many common clinical decisions, clearly applicable clinical guidelines are not available to guide decision making. For example, patients may have multiple chronic conditions, conditions with unclear diagnoses, or psychosocial complications for which existing guidelines do not apply.[7] [8] Moreover, clinicians may have tried guideline-based treatments in the past without improvement in patient outcomes, and thus are left searching for other treatment options.[9]

Patients with chronic pain are especially challenging cases for consistently delivering high-quality care. These patients often have multiple mental or physical health comorbidities. Moreover, today, the United States remains in the midst of an opioid crisis involving misuse, substance use disorder, and opioid overdose. In apparent response to this crisis, opioid prescribing for chronic noncancer pain has begun to decrease in recent years.[10] [11] Yet, millions of patients still suffer from chronic pain. Thus, their health care providers require high-quality information and usable tools to help them efficiently choose the best treatments for improving their patients' pain and overall well-being.[9] For complex conditions like chronic pain, electronic health records (EHRs) often contain large amounts of relevant historical patient data that a clinician may find useful in treating patients.[12] In such complex decision-making contexts, clinicians may benefit from patient information displays that help them organize and make sense of large amounts of patient information on past and current treatments, outcomes, and new treatment options.[3] [13]

Researchers have applied information visualization techniques in various ways to support understanding individual patients, populations of patients, and data over time.[14] [15] [16] [17] [18] [19] [20] [21] [22] [23] [24] When attempting to understand individual patients, clinicians often need to quickly identify events before, after, or during a particular point in time and then focus on the details of important events.[16] [25] For example, in trying to identify a treatment for a patient suffering from an acute exacerbation of chronic low back pain, clinicians may benefit from visualizations that help them quickly identify what treatments have been tried in the past and whether painful symptoms subsided alongside those treatments.[26] [27] Having identified a time in the patient's history where pain was well managed, the clinician may then be interested in drilling down to more detailed information on the events, treatments, and outcomes surrounding that point in time. Prior visualization approaches often focus on making sense of patterns in patient data, with limited actionable guidance to clinicians. Yet, providing actionable guidance is a fundamental criterion for successful adoption of decision support.[4] [28] [29] Thus, in today's world, where EHRs are widely used, but clinicians, vendors, and researchers have concerns about usability and utility,[16] [30] [31] new visual information displays are needed that allow clinicians to explore longitudinal data about their patients, and provide actionable clinical decision-making guidance. These displays should be designed with an understanding of clinicians' day-to-day information needs.

Objective

The objective of this article is to describe a decision-centered design process, and resultant interactive patient information displays, to support key decision requirements in chronic pain care.[32] We identified, and designed the information displays to support, four key decision requirements: (1) the need to understand current and past treatment plans (particularly medications), (2) the need to identify treatment options, (3) the need to identify trends and changes in patient condition, and (4) the need to assess risk of opioid misuse. Primary care clinicians typically care for many patients with chronic pain, often have limited training in chronic pain treatment, and report low satisfaction in delivering chronic pain care.[33] [34] [35] [36] [37] Moreover, because patients may have longstanding pain conditions with unclear diagnoses and/or comorbid conditions that complicate treatment choices, clinicians may benefit from visual displays that allow them to review and understand historical treatments and outcomes in the context of possible new treatment options they may order.[38] [39] Based on our understanding of clinical information availability and use, perceptions, judgments, and decisions during primary care visits for chronic pain, we iteratively designed a novel patient data visualization, the Chronic Pain Treatment Tracker, to support clinicians in reviewing current and past treatments and choosing appropriate new treatments for pain. We hope that in sharing this novel design concept, we will inspire others to adapt and extend this approach in other contexts.

Methods

Overview

We designed the Chronic Pain Treatment Tracker in three stages (see [Fig. 1]). First, we conducted critical decision method interviews[40] with adult primary care clinicians who care for patients with chronic noncancer pain. These interviews and subsequent analysis produced key clinical decision requirements related to chronic pain care. Second, we conducted a half-day multidisciplinary design workshop based on these decision requirements. Through group ideation, discussion, and sketching, the workshop produced a list of information needs that supported the decision requirements and design seeds[41] for patient information visualizations to support chronic pain care. We conceptualized design seeds as approaches to organizing information, visually displaying information, or navigating between information elements.[32] Third, we designed an interactive prototype based on the design seeds and information needs. We will briefly describe the first two stages of the design process. However, they are described in more detail in previously published papers.[32] [42] This article will detail, and present the results of, the third stage of the design process, the visual design of the prototype Chronic Pain Treatment Tracker.

Fig. 1 Stages of design used to create the Chronic Pain Treatment Tracker.

Critical Decision Method Interviews

We recruited 10 adult primary care clinicians who currently treat the chronic noncancer pain conditions of at least 5 patients. Participating clinicians worked in four clinics that span rural, suburban, and urban areas in the United States. Nine of the 10 clinicians were physicians, while the 10th was a nurse practitioner. The clinicians were 50% male, with an average age of 48 years, and an average of 15 years in practice. We recruited clinicians by e-mail, phone, and in-person presentations to clinic-wide physician meetings. Each clinician in the sample completed three interviews, each occurring within 3 days of a visit by a patient with chronic noncancer pain. Clinicians were compensated up to $500 for their time. Clinicians provided written informed consent before participating. Each interview lasted approximately 60 minutes and was audio recorded and transcribed. Each clinician's first interview included general questions about their patient population and approach to chronic pain treatment, including tools or aids they use when delivering care. The remainder of the interview (and subsequent interviews) used an adapted critical decision method interview technique,[40] in which clinicians recalled a recent patient visit and cocreated, with the interviewer, a timeline of key events in the patient's care history. The interviewer then asked probing questions to understand the clinicians' information needs, actions, goals, and decision-making strategies around these key events.

We qualitatively analyzed the interview transcripts to identify key decision requirements for clinical decision support. For the purposes of this study, we conceptualized decision requirements as challenging decision-making tasks and/or cognitive demands that clinicians encountered when managing chronic noncancer pain. First, eight researchers on the team independently reviewed the same two transcripts and identified topics of interest. This set of topics was compiled into a draft codebook, and four researchers refined the codebook through an iterative process of coding using the draft codebook, discussion, and consensus. Once the codebook was finalized, each transcript was coded individually by two researchers who then met and reached consensus on all codes. After judging that thematic saturation had been reached, we analyzed the coded data to extract higher-level concepts (i.e., decision requirements) from the coded data.

Multidisciplinary Design Workshop

We conducted a half-day design workshop to expand on the decision requirements by identifying associated information needs and design seeds. We conceptualized information needs as information that supports clinical decision requirements (e.g., information that aids in assessment, diagnosis, or treatment of pain). There were 14 total participants in the workshop, including 9 researchers from the project team, with expertise in informatics, human factors, behavioral science, engineering, and medicine. Of three physicians on the research team, two were primary care specialists and one was a pain specialist. Additional workshop participants included five primary care physicians, four of whom had also participated in the critical decision method interviews. Sixty percent of these additional workshop participants were female, with an average age of 43 years. All five of the additional workshop participants had Doctor of Medicine degrees, and had years of experience ranging from 10 to 26 years with an average of 15 years in practice. After being introduced to the decision requirements, participants worked in teams of three to five multidisciplinary members and rapidly sketched low-fidelity prototypes of patient information visualizations to support the requirements. Each small group had at least two members from the project team and at least one nonresearcher participant. Each small group presented their prototypes to the larger group for feedback and discussion before reconvening for additional refinement. Discussion focused on design concepts as well as the underlying intent and rationale. Following the workshop, two researchers thematically analyzed notes, video recordings, and the prototype designs generated during the workshop. Through this process they coded the content, analyzed the codes for commonalities and differences, and ultimately arrived at a list of information needs and design seeds through consensus.

The abovementioned interview, thematic analysis, full set of key decision requirements, and design workshop are described in more depth elsewhere.[32] [42]

Prototype Design

The Chronic Pain Treatment Tracker was iteratively designed specifically to support four identified decision requirements, and the associated information needs and design seeds ([Table 1]). The prototypes were developed by a research team member who is a user experience designer with a human factors background and 15 years' experience designing prototypes and conducting user testing on a range of interfaces, including EHRs and clinical decision support. The designer participated in the qualitative analysis of the clinician interviews and the design workshop, and therefore had a deep understanding of the decision requirements and information needs. In addition, the designer's knowledge of clinical workflow and EHR use was informed by previous research and design projects including studies evaluating clinical decision support for medication therapy management,[43] designing a prototype for consult management for the Veteran's Health Administration,[44] and evaluating a modular decision support application for colorectal cancer screening.[45] Exploratory concepts started as sketches with pencil and paper. These concepts were brought to the larger research team for feedback. Designs were refined based on several rounds of feedback. Often, new design ideas were generated during the feedback sessions and incorporated into the designs. As the design concepts matured, they were built into prototypes using Axure, a prototyping tool.[46]

Table 1 Focal decision requirements and associated information needs and design seeds[32]

Decision requirement (DR) supported by this design	Information needs associated with the DR	Design seeds associated with the DR
1. Understand current and past treatment plans (particularly medications)	• Current medications • Past medications	• Present key information aggregated and organized in a single view • Organize information in tables
2. Identify treatment options	• Effectiveness of treatments for this patient • Treatments that have been tried in the past and reasons for discontinuing	• Use visual cues to focus attention on treatment options not yet tried
3. Identify trends and changes in patient condition	• Pain medication use over time • Patient outcomes (e.g., pain and function) over time	• Use time-based displays to depict trends and highlight anomalies • Provide interactive drill-down capability for relevant details
4. Assess risk of opioid misuse or abuse	• Urine drug screen results • Opioid-related risk assessments • Opioid treatment agreement • Prescription drug monitoring program report information	• Create tools to help summarize opioid-related risks

In the design process, the Gestalt principles of similarity, proximity, and common region were leveraged to group information that decision makers process together. For example, in our designs, current treatments are grouped together and separate from past treatments and possible future treatments. Consistent with best design practices, the design process incorporated color as a redundant cue to tie display elements together,[47] aimed to include no more than five or six colors in a display,[48] and used appropriate size fonts (i.e., 12 point) for viewing screens from standard distances.[49] General usability principles (e.g., items are grouped into logical zones and headings are used to distinguish between zones) and information visualization principles (e.g., information follows a logical organization) were considered during the design process.[50]

Results

The resulting Chronic Pain Treatment Tracker ([Fig. 2]) is an exploratory prototype that supports users in understanding current and past treatment plans (decision requirement 1), and in identifying treatment options (decision requirement 2). At-a-glance, it presents key information in a single view, thereby aligning with one of the design seeds. The Chronic Pain Treatment Tracker goes beyond traditional medication lists to display the whole treatment plan, including nonmedication treatments. Treatments are organized by six types: (1) oral medications; (2) topical medications; (3) referrals; (4) interventions such as surgery, injections, medical equipment, or Rest, Ice, Compression, and Elevation therapy; (5) integrative medicine such as meditation; and (6) lifestyle changes such as exercise and nutrition. We organized the treatments by type to highlight the different modalities of treatment that are available. By doing this, clinicians can easily identify where holes in the treatment plan may exist. The data displayed in the prototype is based on a patient use case from an interview with a participant clinician. Additional details were added from the clinicians on our research team, based on typical patients they encounter who have chronic pain.

Fig. 2 Chronic Pain Treatment Tracker.

Current treatments are listed at the top of the display with the following basic information:

• the name of the treatment,

• the condition for which it is being used,

• the dose, quantity, and frequency for medications,

• appointment progress for referrals, and

• an area for clinicians to leave notes.

This last area is important, because it provides clinicians with a flexible space for leaving important reminders, notes to others, or comments on treatment progress. Too often, EHR documentation interfaces are rigid, constraining the form and type of information that can be recorded. Dropdown menus and radio buttons may force clinicians to choose options that do not fit, resulting in documentation that is incomplete or inaccurate. Providing clinicians with space for additional notes is one way to address this problem.

Past treatments are displayed in their own list below the current treatments. Past treatments are often difficult to find in the medical record if they have been removed from (or were never included in) the standard medication list. The past treatment list provides an easy way for clinicians to understand what treatments have been tried in the past without foraging through notes or relying on patient recall. For each past treatment, basic information is presented, including:

• the name of the treatment,

• the condition for which it is being used,

• the date it was last ordered, and

• the reason it was discontinued.

The reason for discontinuation is important to clinicians as they consider what treatments to explore next. Treatments may be discontinued for various reasons such as ineffectiveness, allergies, contraindications, insurance constraints, or patient preference.

Possible future treatments are highlighted on the right side of the display to aid clinicians in identifying potential treatment options. In this case, recommendations were generated by the clinicians on our research team based on the clinical case that was used to populate our design with content. In the discussion section, we describe other ways recommendations could be generated.

Treatments to be cautious about are listed in the bottom right side of the display. The caution list highlights potentially risky treatments at the point in time when clinicians need it most—as they are making decisions about what treatments to try next. The caution list has the potential to avoid disruptions in the ordering process stemming from issues such as contraindications, allergies, and drug–drug interactions.

Additional details about each treatment can be accessed by clicking on the treatment, which triggers a modal view. See [Fig. 3] for an overview of the four types of modal views.

Fig. 3 Overview of modal views.

The current treatment modal view allows clinicians to access additional details about each current treatment. To help identify trends in treatment history, pain, and function (decision requirement 3), it features a notional time-based display. To facilitate risk assessment of opioid treatments (decision requirement 4), it presents summary information for prescription drug monitoring data, urine drug screen results, and treatment agreement contracts. To support an understanding of the current treatment plan (decision requirement 1), it presents excerpts from progress notes pertaining to the current treatment. Identifying relevant notes is a process that is cumbersome today. Clinicians must guess which notes contain the information for which they are looking, resulting in tedious searching, clicking, opening, scanning, and closing as they try to find the right note. This feature aims to reduce that burden.

The past treatment modal view allows clinicians to access additional information about each past treatment. Similar to the current treatment modal view, it includes a notional time-based display for treatment history and links to relevant notes. It also includes clinician comments about treatment effectiveness and links to relevant tests and imaging. This view primarily supports clinicians' requirement to understand past treatments (decision requirement 1) and identify treatment options (decision requirement 2).

The possible future treatment modal view allows clinicians to access additional decision support for each potential treatment option (decision requirement 2). Examples of information that could be included in this view include:

• Ease for patient (e.g., proximity, cost).

• Highlights from drug resources, including a link for more information.

• Contact information for referrals.

• Previous patient experience (if any) with each treatment.

• Links to patient education materials.

• A description explaining why the treatment was recommended.

• A link to order.

The caution modal view allows clinicians to access additional details about treatments that should be avoided or approached with caution. Specifically, it provides an explanation about why a treatment has been identified as risky. The explanation promotes transparency in the system and encourages providers to make their own judgments regarding which treatments should be avoided. The information provided in this view also supports providers in the task of identifying treatment options by eliminating potentially problematic options (decision requirement 2).

Discussion

In this study, we applied a decision-centered design approach with iterative user-centered design methods and rich data from clinicians to produce an interactive visual patient interface to support care for individual patients with chronic pain.[10] [11] [38] [39] [51] As the volume and variety of data contained in EHRs continues to grow, clinicians increasingly need well-designed, visual-based tools to help them navigate a patient's history, sift through relevant contextual information, and identify the most promising treatments. This is particularly true in the case of complex and prevalent chronic conditions, like chronic pain, diabetes, and substance use disorder. The prototype tool we developed and present here offers a promising approach for providing such clinical decision support.

Our multistage design process led to novel patient information displays with potential for future implementation and evaluation. Increasingly, EHRs support integrated third-party applications that use standardized communication interfaces[52] [53] and thus allow for novel designs to be presented within a clinician's normal EHR system environment. This helps reduce time burden, loss of context, and data sharing associated with switching between an EHR and a nonintegrated clinical decision support application. Therefore, the designs presented here, as well as other designs that reflect the decision requirements we identified, could be developed independently and integrated in EHRs.

This study has several strengths and some limitations. We applied a rigorous and iterative process of engaging clinicians, researchers, and designers in codesigning interfaces that could be more usable and useful to clinicians than most current EHR information displays. However, the anticipated users of our designs are primary care clinicians that work in community and academic health systems in the United States, and our data came from clinicians in only four clinics across two health systems, so we are aware that our designs may not reflect the unique requirements and needs of some clinicians. Still, we focused our design on a particularly prevalent and costly health care challenge in chronic pain.[54] Therefore, if implemented, our designs have the potential to positively impact many patients and providers. With that said, the focus on chronic pain may limit the generalizability of our findings and designs to other conditions.

Finally, our study is limited by the fact that we did not implement the Chronic Pain Treatment Tracker with clinicians or patients. Given known challenges in EHR data completeness, accuracy, and standardization,[55] [56] [57] implementing the Chronic Pain Treatment Tracker with real-world patient data will require overcoming several such challenges. First, clinicians consistently noted the need for context in patient evaluations and clinical decision making, including understanding the rationale behind various treatments or their discontinuation. In EHRs, this information is best found in text-based clinical notes, which are difficult to incorporate into tools like the Chronic Pain Treatment Tracker due to their unstructured nature. One way to address this may be via natural language processing to identify and extract useful clinical context from notes. Additionally, populating tools like the Chronic Pain Treatment Tracker requires data from numerous areas in the EHR. For example, medication order data are stored separately from data capturing orders for referrals like physical therapy. As a result, the technical pipeline for populating each data element must be constructed individually, an arduous implementation task. To overcome this challenge, systems could build discrete data elements dedicated to Chronic Pain Treatment Tracker functionality. This would allow for technical segmentation between the process by which these discrete data elements are populated and the process by which they are retrieved for viewing in the Chronic Pain Treatment Tracker. This approach has the potential to make retrieval more uniform across data elements. The source of the recommended “Possible Future” treatments would also need to be addressed in implementation, as the mechanism for deriving this list has not been created. In implementation, such a mechanism could be as simple as a list of common treatments not previously tried, a more complex set of decision roles, or even a predictive model based on outcomes. As the designs transition to software development, we will continue to iterate with a focus on a broader range of patient data, practical constraints of the EHR, and additional user feedback.

At the time of this writing, we are in the process of evaluating the Chronic Pain Treatment Tracker. While initial feedback looks promising, we cannot yet report if these designs would be perceived as usable or adoptable, or positively impact care quality. Still, this article describes a rigorously developed design and vision for a visual patient information interface that could go far beyond typical EHR displays to provide clinicians with information they need to make high-quality patient care decisions.

Based on both the strengths and limitations of this study, there are several potential future studies. As mentioned above, because EHR systems increasingly support standardized application programming interfaces to allow for third-party app integration, future work may involve development and integration of the Chronic Pain Treatment Tracker with real-world EHR data and systems. This would then allow for pragmatic studies of the tool's impact on decision making and patient outcomes embedded in the environment of use.

Conclusion

The goal of this study was to apply a rigorous design strategy to develop a tool to facilitate clinical sensemaking for a challenging chronic condition. Driven by a decision-centered design approach, we focused on supporting the most difficult decisions and demands clinicians face when treating patients with chronic pain, including understanding the current treatment plan, identifying suitable treatment options, identifying trends, and assessing opioid-related risks. Given the complexities of pain and a digital environment characterized by missing, scattered, erroneous, and conflicting information, these challenges can be formidable. We aimed to design a tool that could help clinicians organize and make sense of the information available to them. The resulting prototype, the Chronic Pain Treatment Tracker, is a novel approach to decision support because it focuses less on algorithmic-based guidelines that are pushed to providers, and more on sensemaking. That is, the tool presents clinicians with the information they need in a structure that promotes quick uptake, understanding, and action.

Clinical Relevance Statement

These findings help health care administrators, electronic medical record developers, and health care providers identify critical information required for decision making in chronic pain care delivery. These findings also provide a launching point for future development of decision support tools beyond chronic pain care and the design elements to include for succinct communication and sensemaking.

Multiple Choice Questions

1. Primary care providers require which of the following information to making treatment decisions for patients with chronic pain?

   • Quickly understand current and past treatment plans (particularly medications).

   • Identify treatment options and assess risks of opioid misuse and abuse.

   • Identify trends and changes in patient condition.

   • All the above.

   Correct Answer: The correct answer is option d because each of the components from a, b, and c are considered critical information primary care providers use when making a treatment decision. Providers need to have an understanding of the prior and current treatments used as to identify what remaining treatment options are available in the current visit. Providers need to know what future treatment options are available for this particular patient. The provider also needs to know whether this patient has any risks related to opioids. Lastly, providers need to know how pain management has progressed overtime and assess the patient's pain and function.

2. The primary objective of this study was to:

   • Identify critical treatment information for primary care providers and their patients with chronic pain.

   • Design a succinct display for primary care providers to review pertinent information and support treatment of multiple chronic conditions, including pain, diabetes, and hypertension.

   • Describe the iterative design process used and visual displays developed to support decision making for clinicians caring for patients with chronic pain.

   • Develop visualization tools for population health professionals interested in assessing population risk among a cohort of patients with chronic pain.

   Correct Answer: The correct answer is option c. The focus of this article is to present the process and resultant visual designs for pain decision support in primary care. The purpose was not to design a more general tool that addressed other conditions, to identify treatment information generally, nor to address population health needs.

• References

• 1 Østbye T, Yarnall KS, Krause KM, Pollak KI, Gradison M, Michener JL. Is there time for management of patients with chronic diseases in primary care?. Ann Fam Med 2005; 3 (03) 209-214
• 2 Yarnall KS, Pollak KI, Østbye T, Krause KM, Michener JL. Primary care: is there enough time for prevention?. Am J Public Health 2003; 93 (04) 635-641
• 3 Beasley JW, Wetterneck TB, Temte J. , et al. Information chaos in primary care: implications for physician performance and patient safety. J Am Board Fam Med 2011; 24 (06) 745-751
• 4 Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ 2005; 330 (7494): 765
• 5 Garg AX, Adhikari NK, McDonald H. , et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA 2005; 293 (10) 1223-1238
• 6 Berner ES. Clinical Decision Support Systems. Vol. 233. New York: Springer; 2007
• 7 Lugtenberg M, Burgers JS, Clancy C, Westert GP, Schneider EC. Current guidelines have limited applicability to patients with comorbid conditions: a systematic analysis of evidence-based guidelines. PLoS One 2011; 6 (10) e25987
• 8 Tinetti ME, Green AR, Ouellet J, Rich MW, Boyd C. Caring for patients with multiple chronic conditions. Ann Intern Med 2019; 170 (03) 199-200
• 9 Wyatt KD, Stuart LM, Brito JP. , et al. Out of context: clinical practice guidelines and patients with multiple chronic conditions: a systematic review. Med Care 2014; 52 (Suppl. 03) S92-S100
• 10 U.S. Prescribing Rate Maps | Drug Overdose | CDC Injury Center [Internet]. 2017 . Available at: https://www.cdc.gov/drugoverdose/maps/rxrate-maps.html . Accessed August 8, 2018
• 11 Bohnert ASB, Guy Jr GP, Losby JL. Opioid prescribing in the United States before and after the Centers for Disease Control and Prevention's 2016 Opioid Guideline. Ann Intern Med 2018; 169 (06) 367-375
• 12 Weiskopf NG, Rusanov A, Weng C. Sick patients have more data: the non-random completeness of electronic health records. AMIA Annu Symp Proc 2013; 2013: 1472-1477
• 13 Tawfik AA, Kochendorfer KM, Saparova D, Al Ghenaimi S, Moore JL. “I Don't Have Time to Dig Back Through This”: the role of semantic search in supporting physician information seeking in an electronic health record. Perform Improv Q 2014; 26 (04) 75-91
• 14 West VL, Borland D, Hammond WE. Innovative information visualization of electronic health record data: a systematic review. J Am Med Inform Assoc 2015; 22 (02) 330-339
• 15 Plaisant C, Mushlin R, Snyder A, Li J, Heller D, Shneiderman B. LifeLines: using visualization to enhance navigation and analysis of patient records. Proc AMIA Symp 1998; 76-80
• 16 Wang TD, Plaisant C, Quinn AJ, Stanchak R, Murphy S, Shneiderman B. Aligning temporal data by sentinel events: discovering patterns in electronic health records. Proc SIGCHI Human Factors Comp Sys 2008: 457-466
• 17 Martins SB, Shahar Y, Goren-Bar D. , et al. Evaluation of an architecture for intelligent query and exploration of time-oriented clinical data. Artif Intell Med 2008; 43 (01) 17-34
• 18 Klimov D, Shahar Y, Taieb-Maimon M. Intelligent interactive visual exploration of temporal associations among multiple time-oriented patient records. Methods Inf Med 2009; 48 (03) 254-262
• 19 Radhakrishnan K, Monsen KA, Bae SH, Zhang W. ; Clinical Relevance for Quality Improvement. Visual analytics for pattern discovery in home care. Appl Clin Inform 2016; 7 (03) 711-730
• 20 Sorbello A, Ripple A, Tonning J. , et al. Harnessing scientific literature reports for pharmacovigilance. Prototype software analytical tool development and usability testing. Appl Clin Inform 2017; 8 (01) 291-305
• 21 Wanderer JP, Gruss CL, Ehrenfeld JM. Using visual analytics to determine the utilization of preoperative anesthesia assessments. Appl Clin Inform 2015; 6 (04) 629-637
• 22 Aigner W, Miksch S. CareVis: integrated visualization of computerized protocols and temporal patient data. Artif Intell Med 2006; 37 (03) 203-218
• 23 Rind A, Federico P, Gschwandtner T, Aigner W, Doppler J, Wagner M. Visual analytics of electronic health records with a focus on time. New Perspectives in Medical Records. Switzerland: Springer; 2017: 65-77
• 24 Wagner M, Slijepcevic D, Horsak B, Rind A, Zeppelzauer M, Aigner W. KAVAGait: Knowledge-Assisted Visual Analytics for Clinical Gait Analysis. IEEE Trans Vis Comput Graph 2019; 25 (03) 1528-1542
• 25 Shneiderman B. The eyes have it: a task by data type taxonomy for information visualizations. Proc IEEE Symp Vis Lang 1996: 336-343
• 26 Keim D, Kohlhammer J, Ellis G, Mansmann F. , eds. Mastering the Information Age: Solving Problems with Visual Analytics. Goslar, Germany: Eurographics Association; 2010
• 27 Aigner W, Miksch S, Schumann H, Tominski C. Visualization of Time-Oriented Data. London: Springer Science & Business Media; 2011
• 28 Kannry J, McCullagh L, Kushniruk A, Mann D, Edonyabo D, McGinn T. A framework for usable and effective clinical decision support: experience from the iCPR randomized clinical trial. EGEMS (Wash DC) 2015; 3 (02) 1150
• 29 Kantor M, Wright A, Burton M. , et al. Comparison of computer-based clinical decision support systems and content for diabetes mellitus. Appl Clin Inform 2011; 2 (03) 284-303
• 30 Middleton B, Bloomrosen M, Dente MA. , et al; American Medical Informatics Association. Enhancing patient safety and quality of care by improving the usability of electronic health record systems: recommendations from AMIA. J Am Med Inform Assoc 2013; 20 (e1): e2-e8
• 31 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
• 32 Harle CA, Apathy NC, Cook RL. , et al. Information needs and requirements for decision support in primary care: an analysis of chronic pain care. AMIA Annu Symp Proc 2018; 2018: 527-534
• 33 Leverence RR, Williams RL, Potter M. , et al; PRIME Net Clinicians. Chronic non-cancer pain: a siren for primary care--a report from the PRImary Care MultiEthnic Network (PRIME Net). J Am Board Fam Med 2011; 24 (05) 551-561
• 34 Matthias MS, Parpart AL, Nyland KA. , et al. The patient-provider relationship in chronic pain care: providers' perspectives. Pain Med 2010; 11 (11) 1688-1697
• 35 Bergman AA, Matthias MS, Coffing JM, Krebs EE. Contrasting tensions between patients and PCPs in chronic pain management: a qualitative study. Pain Med 2013; 14 (11) 1689-1697
• 36 Bendtsen P, Hensing G, Ebeling C, Schedin A. What are the qualities of dilemmas experienced when prescribing opioids in general practice?. Pain 1999; 82 (01) 89-96
• 37 Institute of Medicine Committee on Advancing Pain Research, Care, and Education. Relieving Pain in America: A Blueprint for Transforming Prevention, Care, Education, and Research. Washington, DC: National Academies Press; 2011
• 38 Militello LG, Hutton RJ. Applied cognitive task analysis (ACTA): a practitioner's toolkit for understanding cognitive task demands. Ergonomics 1998; 41 (11) 1618-1641
• 39 Militello LG, Gary K. Decision-centered design. In: The Oxford Handbook of Cognitive Engineering. New York: Oxford University Press; 2013: 261-285
• 40 Crandall B, Klein G, Klein GA, Hoffman RR. Working Minds: A Practitioner's Guide to Cognitive Task Analysis. Cambridge, MA: MIT Press; 2006
• 41 Militello LG, Patterson ES, Saleem JJ, Anders S, Asch S. Supporting macrocognition in health care: improving clinical reminders. In: Naturalistic Decision Making and Macrocognition. London: Ashgate Publishing; 2012: 203-220
• 42 Militello LG, Anders S, Downs SM. , et al. Understanding how primary care clinicians make sense of chronic pain. Cogn Technol Work 2018; 20 (04) 575-584
• 43 Snyder E, Jaynes H, Gernant S. , et al. Clinical decision support for medication therapy management: recommendations from a heuristic evaluation. AMIA Annu Symp Proc 2018; 2018: 1846-1847
• 44 Savoy A, Militello LG, Patel H. , et al. A cognitive systems engineering design approach to improve the usability of electronic order forms for medical consultation. J Biomed Inform 2018; 85: 138-148
• 45 Militello LG, Diiulio JB, Borders MR. , et al. Evaluating a modular decision support application for colorectal cancer screening. Appl Clin Inform 2017; 8 (01) 162-179
• 46 Axure; 2019 . Available at: https://www.axure.com/ . Accessed August 15, 2019
• 47 Wickens CD, Hollands JG, Banbury S, Parasuraman R. Engineering Psychology & Human Performance. New York: Psychology Press; 2015
• 48 Carter RC, Cahill M-C. Regression models of search time for color-coded information displays. Hum Factors 1979; 21 (03) 293-302
• 49 Health and Human Services. Standards and usability guidelines. Available at: https://webstandards.hhs.gov/guidelines/ . Accessed April 14, 2019
• 50 Dowding D, Merrill JA. The development of heuristics for evaluation of dashboard visualizations. Appl Clin Inform 2018; 9 (03) 511-518
• 51 Sedlmair M, Meyer M, Munzner T. Design study methodology: reflections from the trenches and the stacks. IEEE Trans Vis Comput Graph 2012; 18 (12) 2431-2440
• 52 Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J Am Med Inform Assoc 2016; 23 (05) 899-908
• 53 Bloomfield Jr RA, Polo-Wood F, Mandel JC, Mandl KD. Opening the Duke electronic health record to apps: implementing SMART on FHIR. Int J Med Inform 2017; 99: 1-10
• 54 Gaskin DJ, Richard P. The economic costs of pain in the United States. J Pain 2012; 13 (08) 715-724
• 55 Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc 1997; 4 (05) 342-355
• 56 Thiru K, Hassey A, Sullivan F. Systematic review of scope and quality of electronic patient record data in primary care. BMJ 2003; 326 (7398): 1070
• 57 Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc 2013; 20 (01) 144-151