How Do Experienced Physicians Access and Evaluate Laboratory Test Results for the Chronic Patient? A Qualitative Analysis

Abstract

Background Electronic health records may present laboratory test results in a variety of ways. Little is known about how the usefulness of different visualizations of laboratory test results is influenced by the complex and varied process of clinical decision making.

Objective The purpose of this study was to investigate how clinicians access and utilize laboratory test results when caring for patients with chronic illness.

Methods We interviewed 10 attending physicians about how they access and assess laboratory tests when following up patients with chronic illness. The interviews were audio-recorded, transcribed verbatim, and analyzed qualitatively.

Results Informants preferred different visualizations of laboratory test results, depending on what aspects of the data they were interested in. As chronic patients may have laboratory test results that are permanently outside standardized reference ranges, informants would often look for significant change, rather than exact values. What constituted significant change depended on contextual information (e.g., the results of other investigations, intercurrent diseases, and medical interventions) spread across multiple locations in the electronic health record. For chronic patients, the temporal relations between data could often be of special interest. Informants struggled with finding and synthesizing fragmented information into meaningful overviews.

Conclusion The presentation of laboratory test results should account for the large variety of associated contextual information needed for clinical comprehension. Future research is needed to improve the integration of the different parts of the electronic health record.

Keywords

laboratory information systems - electronic health records and systems - qualitative - interfaces and usability - requirements analysis and design

Background and Significance

In medicine, laboratory tests hold a central position in patient care and are used in diagnosing, treating, monitoring, and preventing disease.[1] Failure to properly comprehend laboratory test results may lead to medical errors and cause harm to the patient. Regrettably, such mistakes are not uncommon.[2] [3] [4] As patients receive more treatment,[5] health records can swell up to the point where it becomes challenging for clinicians to gain and maintain an overview of the patient's medical history, thereby increasing the risk of oversights.[6] [7] Chronic patients pose a special challenge as they often require extensive treatment over long periods of time, thereby accumulating large volumes of clinical data. The electronic health record (EHR) can potentially help clinicians manage the ever-increasing flow of information by tailoring presentation format to their information needs.[8] However, this effect seems all but guaranteed. The introduction of the EHR has in some cases even been shown to make certain clinical tasks more difficult.[9] [10] [11] [12] Every patient's medical history might be both lengthy and complex, containing multiple data types related to each other in complicated ways. It has proven difficult to produce systems capable of presenting it all in a meaningful way.[13] [14]

The EHR is commonly organized into separate modules, compartmentalizing different types of information from each other (e.g., radiological findings, doctors notes, laboratory results).[15] When it comes to accessing laboratory test results, most EHRs offer a graph interface[16] in addition to the traditional table. Several studies have investigated how line graphs and tables stack up to each other.[17] [18] [19] [20] Some have found that graphs outperform tables on parameters such as speed of assessment and user satisfaction. Others have not found any difference, or gotten the opposite result.

In real clinical conditions, laboratory test results are rarely reviewed in isolation, but rather used in conjunction with other clinical data. From previous research, we know that the relationship between presentation and comprehension of laboratory results can be unstable, depending on factors such as user preferences, characteristics of the data being displayed, and subtle implementation details.[19] [20] However, most of these studies have only evaluated simple use cases, asking test subjects who were often not clinicians to perform elementary data extraction tasks. Little is known about how the usefulness of different visualizations of laboratory test results is influenced by the complex and varied process of clinical decision making. Proper design of the EHR relies on a thorough understanding of the complex tasks it is meant to help perform.[21] [22] [23]

Objective

In this study, we investigate how test results are accessed and utilized by physicians when caring for patients. Furthermore, we seek to identify whether the different laboratory modules in the EHR facilitate such use. We focus on patients with chronic illness because they often have complex medical histories spanning long time periods and can, therefore, be difficult to assess. We believe such knowledge is needed to inform the design of future systems for presenting laboratory test results.

Methods

Setting

The study was conducted at St. Olavs Hospital (SOH), Trondheim, Norway. SOH is one of four university teaching hospitals in Norway, providing specialized care for approximately 715,000 inhabitants. In Norway, a general practitioner (GP) is the patient's primary caretaker, referring the patient to hospital specialists when needed. The GP office and the hospital have separate EHR systems, thus coordination of care relies on electronic referral letters and epicrises. The current EHR system used at SOH is DocuLive (Siemens). Physicians document their patient encounters as free-text clinical notes and access previous notes from a chronologically oriented document list. Laboratory test results are accessed through three separate software modules; the graph, table, and list interfaces ([Fig. 1]). The graph interface visualizes time series of laboratory results from one or more tests as line graphs. The y-axis is linear and represents the numerical value of each test result, while the x-axis represents the time. Details of each test result are shown when clicked on the graph interface. In the table interface, samples are presented in separate columns organized in chronological order. Each row represents a particular test and rows presenting medically related tests are collocated. The table interface can present nine columns at a time. The list interface presents each blood sample as a chronological list of links. The list also includes the time of the sample and who ordered it. By clicking on one of the links, the results from that sample are shown. In addition to laboratory results, the list and table interfaces may also contain microbiology and pathology reports.

Fig. 1 The available interfaces in the EHR. Table (A) and graph interface (B) in the top row. Bottom row depicts both the sample view (C) and result view (D) of the list interface.

Only test results both ordered and analyzed at SOH are shown in the table and graph interface. If a test is analyzed, but not ordered at SOH (eg. if a GP has requested an analysis), the result is only available in the list interface. Tests analyzed elsewhere may be presented as free text in the GPs' electronic referral letters or as scanned external documents in a separate folder of the patient's EHR.

Data Collection

We conducted 10 focused interviews[24] with physicians representing five different medical specialties (two from each specialty). We chose interviews for data collection because we were interested in the physicians' reasons and reflections about how they used laboratory results in caring for patients with chronic illness. Focused interviews were considered appropriate because interview topics were nonsensitive and related to situations the informants were highly familiar with.

Informants were recruited through direct approach. We used a stratified purposeful sampling strategy. The major sampling criteria were that the informants should be specialists and have extensive experience with caring for patients with chronic illness. To broaden the sample, we chose specialists from five different specialties in which we expected that the laboratory test results would play an important role in patient follow-up (see [Table 1] for details). Within this selection we sought variety in age and experience.

Table 1 Informants

Gender	8 men, 2 women
Specialty	2 gastroenterologists, 2 hematologists, 2 nephrologists, 2 endocrinologists, and 2 specialists in infection medicine
Age	Mean 50 y, Min 39, Max 67
Experience	Mean 24 y, Min 11, Max 40

After familiarizing with the informant and informing about the project, the interviews started with open-ended questions regarding patients with chronic illnesses pertaining to the informant's specialty (e.g., “What chronic illnesses are typical among your patients?”). It was up to each informant to define what they considered to be a chronic patient. The types of patients discussed are noted in [Table 2]. Then, the interview revolved around the follow-up of such patients, what role laboratory tests played in the follow-up, and how informants would access and use such test results (e.g., “How do you prepare yourself before a consultation?, What role does laboratory test results play for that patient?”).

Table 2 Typical patients by specialty

Specialty	Typical chronic patient
Gastroenterology	Morbus Crohn, ulcerative colitis, nonviral hepatitis
Hematology	Myelomatosis, myelodysplasia, acute and chronic leukemia, lymphoma, hemolytic anemia
Nephrology	Chronic kidney disease
Endocrinology	Diabetes, hyperthyroidism, hypothyroidism, pituitary disease
Infection medicine	HIV, hepatitis B and C

Abbreviation: HIV, human immunodeficiency virus.

To stimulate and focus reflection, informants were subsequently showed anonymized visualizations of laboratory test results pulled from the hospital's current EHR ([Fig. 1]). The informants were first asked what they thought about the visualizations in general and then asked to reflect on how the visualization conveyed important aspects of laboratory test results discussed earlier (“How do you like this?,” “How does this convey xxx?,” “Why?,” “Why not?”).

The interviews were conducted by two of the authors (T.T. and B.L.) between February and May 2016. All interviews were conducted at the informant's workplace during working hours. Interviews lasted on average 44 minutes (min, 35, max, 60). Some of the informants were on call, and were briefly interrupted during the interview (4 informants, time interrupted was not added to interview length). The interviews were audio-recorded and transcribed verbatim.

Data Analysis

Systematic text condensation (STC) was applied to analyze the data. STC is a strategy derived from psychological phenomenological analysis.[25] [26] The STC consisted of four steps. Step 1: Overviewing: The complete data set was read through by two of the authors (T.T. and B.L.). A set of themes were agreed upon through an iterative approach of trial and error. Step 2: Coding: All meaning units were identified and coded (decontextualization) using inclusion rules based on the themes. Coding was done independently by two of the authors (T.T. and B.L.). There was much disagreement after the first round of coding, leading to an adjustment of coding rules. In the subsequent round of coding, there was little disagreement. Step 3: Condensing: Using the codes as a starting point, codes were synthesized into constructed statements, representing the code group. Step 4: Synthesizing: These statements were further developed into descriptions accompanied by illustrative quotations and validated against the original material (recontextualization). All three authors participated in steps 3 and 4.

Results

Data analysis revealed three main topics. When reviewing laboratory test results, informants would usually prefer using the table, rarely using the list interface and reserving the graph for cases where they felt a thorough investigation was warranted (initial orientation). When reviewing the chronic patient, special emphasis was placed on perceiving change in test results rather than on absolute values because chronic patients could often be expected to have many abnormal values (sensing change). To assess such results accurately, informants needed contextualizing information located outside the laboratory modules (making sense of change). Informants would struggle retrieving such information, especially when reviewing historic test results.

Initial Orientation

The information needs of physicians treating chronic patients varied widely. The follow-up of a patient with chronic illness was not necessarily that complicated. However, many described the preparations for follow-ups as time-consuming, having to read several EHR notes, and look up results from supplementary investigations.

I6: “Because I've done this for many years, gaining an overview is pretty straightforward. Of course, some patients are very complicated. In such cases you have to prepare more thoroughly.”

When accessing laboratory test results, informants would usually use either the table or the graph interface. Both were appreciated but for different reasons and in different situations. When informants needed to orient themselves in the patient's laboratory test results, they would look for a quick impression of the most recent test samples. If not finding any cause for concern here, many would leave it at that and not investigate any further. Informants felt the table was well suited for this purpose. They also appreciated that results from all tests could be seen together, because different tests sometimes had to be viewed in conjunction when determining their clinical significance. However, some felt the table was strenuous to use if they needed an overview of more test results than those provided at the first page. In addition, it could be difficult to evaluate the rate of change between results because test samples were usually taken at irregular intervals.

I9: “I use this [the table interface] a lot because it gives a good overview when assessing, leukocytes versus CRP, you get a good impression of several variables simultaneously.”

I8: “I prefer seeing the numbers. The only drawback is that when you browse backwards in the patient history, you have to flip through many page [in the table interface]. In such cases you easily lose overview.”

I3: “[When talking about the table] What's missing is the timespans. Your really have to look closely at the dates to catch how long it's been between samples.”

In the graph interface, the entire evolution of test results could easily be seen. However, before being shown any content, users had to import test series manually by clicking on labels in a potentially long list of available tests. The menu did not give any indication as to what tests would be interesting to investigate, forcing the user to potentially click through many tests before seeing anything useful. A typical strategy was to use the table interface to gain an initial overview and reserve the graph interface for special investigations, if needed.

I1: “I mostly use the table interface. Then I get a lot of different test results, and then I do this [retrieves a single test result in the graph interface] to get the full history”

Fragmented and hardly accessible information made it difficult and time-consuming for the informants to get an overview of laboratory test results. Only tests ordered and analyzed at the hospital would be accessible through the table and graph interface. Tests ordered by an external party such as a GP could only be found in the list interface or as free text in referral letters—if available at all. The list interface was not considered very useful and was, in general, used only when informants suspected it contained results not available elsewhere.

I4: “To look at the tests that the GP has ordered, we have to use a specific EHR module [the list interface] and open each and every sample one by one. Perhaps look at as many as fifty samples to find a particular one. The list doesn't convey which tests are included in each sample, so you just have to go through the list. There might be piles of samples you have to go through before you find the one result you are interested in. It's terribly annoying.”

Sensing Change

The ability to see how laboratory test results evolved over time was highly valued. Although the latest test results were described as the most important, all results could be of interest—even results from before the time of diagnosis.

I1: “Often, it's fine to see a limited time period, but sometimes it's interesting to see long time spans. Especially when following the M-component. Seeing how it has been in the past. And with anemia, chronic anemia, see how it was long ago. This also applies when we get a new referral. How did it use to be? Are there any old results we can use for comparison? Often we already have some test results. Every sample is useful when considering a diagnosis. How has this evolved? What's the illness trajectory like?”

Laboratory test results were presented in relation to standardized reference ranges that are based on the healthy population. Informants' assessments of results, however, were more sophisticated than merely recognizing values within or beyond reference ranges. For certain illnesses, a test result beyond reference ranges would be considered normal, while a change in such a “normal abnormality” would be unexpected and thus considered abnormal. Even a change within the reference ranges could be relevant.

I7: “At any time, about half of our patients' test results are flagged as abnormal. Results are usually somewhat out of line. For example, all of our patients have hyperkalemia and anemia, but that doesn't really trouble us. They are supposed to have that.”

I2: “Hypothyreosis can cause TSH to rise to a hundred. However, healthy people mostly have levels between a half and four. In such cases, variation between one and three could still hint at real change.”

In general, the informants emphasized the importance of perceiving significant changes in a patient's test results. What constituted significant change depended on the context. For example, if a blood test was used to monitor the disease activity of a patient in remission, small changes could be important, because they could suggest a beginning relapse. If the disease was highly active, values had to vary more before they attracted attention.

I4: “Knowing the exact values is always of interest. At least if the patient is undergoing treatment. If they aren't treated it may not be as important, as we know it will then fluctuate a lot.”

Making Sense of Change

The informants stressed the need for comprehensive contextual information to assess laboratory test results correctly. Examples of such information were past and current medication, compliance to treatment, intercurrent diseases, surgical procedures, results from other medical investigations (radiology, pathology, microbiology, endoscopy, other laboratory test results), hospitalizations, and psychosocial status. Although many details in the patient's medical history could be relevant, a few would typically be enough to provide an appropriate context to a specific result. As with laboratory test results, relevant contextual information was spread across different locations in the EHR, making it difficult for informants to find the key pieces of information most relevant to them. Informants reported that they relied heavily on summary notes when orienting themselves in the patient's medical history.

I1: “Usually, the patient has been admitted at some point. In that case, there is a summary note or epicrisis. Some patients visit the outpatient clinic only. In such cases you merely have problem notes. I believe a summary note should be made once a year. It makes it easier for others to discern the patient history.”

Gaining an overview could be especially challenging if a patient suffered from multiple afflictions, spanning different specialties. Interest in the patient's health was typically limited to a certain chronic illness pertaining to the particular specialty of the informant. Hence, information generated by other health professionals could provide more clutter than clarification.

I5: “There are patients who sometimes need to be managed by other specialists as well. This includes analyzing other laboratory tests that aren't that relevant for me. The possibility to adjust the presentation format of the laboratory system so that I would only see what's relevant for myelomatosis today, so that I could get the overview. (...) What's most relevant. Let's say, today I would like to put on my ‘myeloma glasses’, or today I want to use my ‘acute leukemia glasses’.”

Many informants would use temporal relations between data points as guidance when filtering out information, exploiting that information originating from the same time period would often be thematically related. If, for example, noticing some interesting result in the graph interface, informants would look for the appropriate context in notes made around the time of sampling. Some informants would use these connections when navigating in the EHR as a whole. They would examine tests relevant to the disease of interest in the graph interface and look for flares. Knowing when a flare occurred would then help them locate important notes.

I6: “Take autoimmune hepatitis as an example. Suddenly test results start rising. I know there have been similar incidents in the past. When I look at the line graph I immediately see when it happened (...) Then I go back and read what caused the flare and what action I took.”

When informants opened up a new module, they would initially be presented with the most recent information available, forcing them to spend a lot of time navigating when looking for historic data spread across different modules. The graph interface could effectively convey how test results evolved over long periods of time. However, some informants felt it was difficult to make use of this because there was no easy way to connect what they saw in the graph to information located elsewhere in the EHR.

I8: “[When reviewing abnormal test results in the graph interface] They could have had a pneumonia because of flu, or some other sickness that explains the abnormal results. For example, I have to know about hospital admissions in order to say something about its relevance. I don't know who has ordered the samples or in what context.”

I5: “It would be great to see the time of treatment in the graph. [pointing at the graph] Let's say we gave some treatment here, and here we gave the next, or something like that. It would make the graph much more useful. Today there's no connection between the graph and therapy. This connection is crucial in clinical assessment.”

Discussion

Our results indicate that to make sense of test results, physicians may need comprehensive contextual information, as well as information about the temporal evolution of test results, either individually or compared with other parameters. The results also indicate that physicians struggled retrieving such information from the EHR.

Making sense of laboratory data involves forming a mental overview of them. The close link between sense-making and overview formation is apparent in Bossen and Jensen's[27] definition of overview formation as the process through which “health care professionals arrive at a sufficiently informed, accountable and coherent understanding of a situation, so that they are capable of acting consciously and with confidence.” That is, to care for a patient a clinician must form and maintain an overview on which clinical decisions can be made.

In the EHR, laboratory test results would be presented in relation to standardized reference ranges, which are based on expected values in healthy individuals. Informants could not simply rely on these, because patients with a chronic condition could have test results that were permanently outside these ranges. This corresponds to the general guidelines for the monitoring of chronic disease as outlined by Glasziou et al,[28] who emphasizes how “normality” is a relative concept for the chronic patient. Especially when using laboratory tests for monitoring, the informants would attend to changes rather than absolute values.

The EHR contained a wide variety of contextual data which were potentially needed when reviewing laboratory test results. However, an adequate context could usually be formed using a limited number of key information items, closely resembling the process of “bundling” as described by Gorman et al and Ash at al[29] [30] who define “bundles” as “organized, highly selective collections of information—to help solve problems and maintain situational awareness.” The challenge would often be to find these necessary pieces of information while filtering away piles of irrelevant data. When gathering context, informants described an information-seeking strategy closely resembling Bates' berrypicking model.[31] Typically, they would first gain an overview over the most recent events and let findings direct further exploration. No module in the EHR could adequately display all aspects of the patient's medical history. Physicians would therefore have to switch back and forth between modules in the EHR while gaining context. If, for example, a gastroenterologist noticed signs of liver damage while looking in the table, he would usually have to leave the table interface to find an explanation. Perhaps some drug has caused an adverse reaction? Has there been some recent change in medication? If so, what was the reason for this change?

Providing efficient means of filtering in the ever-growing medical record is a well-recognized challenge.[32] [33] [34] Systems providing more problem-oriented views of the medical history have been constructed and favorably received by clinicians,[35] [36] [37] but are taxing to create, and often depend on an underlying data structure not always present in the EHR.[15] The informants in our study would use time as a means of filtering, exploiting that temporal relations between parameters may be indicative of causal relations. However, the EHR did not provide any efficient way of exploring data in this manner if the data resided in different modules of the EHR (which was often the case). Varpio et al[38] suggest that the fragmented organization of data in many EHRs obscures the connections between data points and hampers clinical sense-making. Our findings support this notion. In general, the EHR provided many views of the patient's medical history, but lacked efficient ways of tying them together. This problem became especially apparent when dealing with the chronic patient. For acute problems, relevant information was likely located close to the current time. In these situations, a switch between modules was straightforward because each module would typically emphasize recent information, thereby automatically giving the user a relevant view. It was only when data became buried deep in lists and paginators that switching became a hassle. Even though none of the available laboratory interfaces alleviated this problem, informants would only point this out when discussing the graph interface. This was probably because the graph was primarily used for exploring historic test results. What interface the informants preferred would often be determined by how well the interface enabled the informants to connect test results to the context they considered most appropriate. Because users were not given efficient ways of contextualizing what they saw in the graph, its usefulness was severely diminished.

Shneiderman[39] recommends that systems should be designed to provide an overview first; details should be shown later and only on demand. An overview is made from details, but the process of overview formation is not well understood, and it is unclear how details should be presented to make overview formation as effective and efficient as possible.[40] Systems highlighting temporal relations, such as Lifelines,[41] have proven beneficial. Still, the lack of temporal views continues to be a recognized weakness of many EHRs.[42] Because of the long list of potentially important parameters, timelines can easily become cluttered,[14] making it difficult to create timelines suited for heterogeneous patient groups. Some have attempted to solve this problem by creating timelines that enable users to filter data by medical problem, utilizing advanced data-mining techniques.[36] Perhaps a timeline could provide an adequate overview simply by conveying a temporal representation of available data, facilitating orientation and navigation by time, thereby supporting the clinicians' natural information-seeking behavior.

Conclusion

To assess laboratory test results, clinicians need to construct a context in which results can be understood. Because chronic patients often have permanently abnormal test results, clinicians cannot rely solely on standardized reference ranges. The EHR should be able to present the evolution of laboratory test results, because change is a strong indicator of true abnormality. In addition, values must be interpreted in relation to background information, such as the results of other investigations, intercurrent diseases, and medical interventions. The usefulness of different visualizations of laboratory test results hinges on how it succeeds in presenting raw test results in relation to relevant contextual information.

It is highly situation dependent for which details are needed to understand the evolution of test results. Therefore, the EHR must provide potent means for identifying key pieces of information while filtering away redundant and irrelevant information. In the chronic patient, information from the same time period will often be semantically related. The EHR should take advantage of this by emphasizing the temporal relation between laboratory test results and other data points. Other studies should explore how such functionality affects the clinical sense-making of laboratory test results.

Clinical Relevance Statement

• Clinicians do not have fixed preferences for the presentation of laboratory test results. Optimal presentation depends on what aspects of the data they are interested in.

• Clinicians need to establish a context for interpreting a patient's laboratory test results.

• Simple static reference ranges are often not sufficient when assessing laboratory test results for the chronic patient. Values must be interpreted in relation to the evolution of previous results, the results of other investigations, intercurrent diseases, and medical interventions.

• The EHR should be able to easily connect laboratory test results to other types of information in the EHR. An overview emphasizing the temporal relations between data points might be beneficial.

Multiple Choice Question

When assessing laboratory test results in the electronic health record, clinicians require knowledge of:

• The evolution of test results over time.

• Previous radiological findings.

• Previous medical interventions.

• All or none of the above, depending on the situation.

Correct Answer: Option d is the correct answer. The necessary context for laboratory test results is entirely situation dependent. All of the options can be very relevant, but none has to be.

Conflict of Interest

None.

Acknowledgments

The authors thank all the study participants.

Authors' Contributions

Conception and design of the study: T.T., B.L., and M.H. Data collection: T.T. and B.L. Data analysis and interpretation: T.T., B.L., and M.H. Drafting and revising the article for important intellectual content: T.T., B.L., and M.H. Approving the final version of the submission: T.T., B.L., and M.H.

Protection of Human and Animal Subjects

The study protocol was approved by the Norwegian Center for Research Data (NSD, 45615/3/HIT). All informants gave their informed consent prior to interviews.

• References

• 1 Burtis CA, Edward RA, Bruns DE. Tietz Textbook of Clinical Chemistry and Molecular Diagnostics. St. Louis, MO: Elsevier Health Sciences; 2012
• 2 Roy CL, Poon EG, Karson AS. , et al. Patient safety concerns arising from test results that return after hospital discharge. Ann Intern Med 2005; 143 (02) 121-128
• 3 Edelman D. Outpatient diagnostic errors: unrecognized hyperglycemia. Eff Clin Pract 2002; 5 (01) 11-16
• 4 Bonini P, Plebani M, Ceriotti F, Rubboli F. Errors in laboratory medicine. Clin Chem 2002; 48 (05) 691-698
• 5 Murray CJ, Lopez AD. Measuring the global burden of disease. N Engl J Med 2013; 369 (05) 448-457
• 6 Beasley JW, Wetterneck TB, Temte J. , et al. Information chaos in primary care: implications for physician performance and patient safety. J Am Board Fam Med 2011; 24 (06) 745-751
• 7 Singh H, Spitzmueller C, Petersen NJ, Sawhney MK, Sittig DF. Information overload and missed test results in electronic health record-based settings. JAMA Intern Med 2013; 173 (08) 702-704
• 8 Jha AK. The promise of electronic records: around the corner or down the road?. JAMA 2011; 306 (08) 880-881
• 9 Poissant L, Pereira J, Tamblyn R, Kawasumi Y. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review. J Am Med Inform Assoc 2005; 12 (05) 505-516
• 10 Viitanen J, Hyppönen H, Lääveri T, Vänskä J, Reponen J, Winblad I. National questionnaire study on clinical ICT systems proofs: physicians suffer from poor usability. Int J Med Inform 2011; 80 (10) 708-725
• 11 Lium JT, Faxvaag A. Removal of paper-based health records from Norwegian hospitals: effects on clinical workflow. Stud Health Technol Inform 2006; 124: 1031-1036
• 12 Himmelstein DU, Wright A, Woolhandler S. Hospital computing and the costs and quality of care: a national study. Am J Med 2010; 123 (01) 40-46
• 13 Rind A, Wang TD, Aigner W. , et al. Interactive information visualization to explore and query electronic health records. Comp Interaction 2013; 5 (03) 207-298
• 14 West VL, Borland D, Hammond WE. Innovative information visualization of electronic health record data: a systematic review. J Am Med Inform Assoc 2015; 22 (02) 330-339
• 15 Clynch N, Kellett J. Medical documentation: part of the solution, or part of the problem? A narrative review of the literature on the time spent on and value of medical documentation. Int J Med Inform 2015; 84 (04) 221-228
• 16 Sittig DF, Murphy DR, Smith MW, Russo E, Wright A, Singh H. Graphical display of diagnostic test results in electronic health records: a comparison of 8 systems. J Am Med Inform Assoc 2015; 22 (04) 900-904
• 17 Meyer J. Performance with tables and graphs: effects of training and a Visual Search Model. Ergonomics 2000; 43 (11) 1840-1865
• 18 Schaubroeck J, Muralidhar K. A meta-analysis of the relative effects of tabular and graphic display formats on decision-making performance. Hum Perform 1991; 4 (02) 127-145
• 19 Bauer DT, Guerlain S, Brown PJ. The design and evaluation of a graphical display for laboratory data. J Am Med Inform Assoc 2010; 17 (04) 416-424
• 20 Torsvik T, Lillebo B, Mikkelsen G. Presentation of clinical laboratory results: an experimental comparison of four visualization techniques. J Am Med Inform Assoc 2013; 20 (02) 325-331
• 21 Zhang J, Walji MF. TURF: toward a unified framework of EHR usability. J Biomed Inform 2011; 44 (06) 1056-1067
• 22 Kushniruk AW, Patel VL. Cognitive and usability engineering methods for the evaluation of clinical information systems. J Biomed Inform 2004; 37 (01) 56-76
• 23 Ratwani RM, Fairbanks RJ, Hettinger AZ, Benda NC. Electronic health record usability: analysis of the user-centered design processes of eleven electronic health record vendors. J Am Med Inform Assoc 2015; 22 (06) 1179-1182
• 24 Merton RK, Kendall PL. The focused interview. Am J Sociol 1946; 51 (06) 541-557
• 25 Malterud K. Systematic text condensation: a strategy for qualitative analysis. Scand J Public Health 2012; 40 (08) 795-805
• 26 Giorgi A. The descriptive phenomenological psychological method. J Phenomenol Psychol 2012; 43 (01) 3-12
• 27 Bossen C, Jensen LG. How physicians' achieve overview': a case-based study in a hospital ward. In Proceedings of the 17th ACM conference on Computer Supported Cooperative Work & Social Computing. ACM; 2014 February 15, pp. 257–268
• 28 Glasziou P, Irwig L, Mant D. Monitoring in chronic disease: a rational approach. BMJ 2005; 330 (7492): 644-648
• 29 Ash JS, Gorman PN, Lavelle M. , et al. Bundles: meeting clinical information needs. Bull Med Libr Assoc 2001; 89 (03) 294-296
• 30 Gorman P, Ash J, Lavelle M. , et al. Bundles in the wild: managing information to solve problems and maintain situation awareness. Library Trends 2000; 49 (02) 266-289
• 31 Bates MJ. The design of browsing and berrypicking techniques for the online search interface. Online Rev 1989; 13 (05) 407-424
• 32 Berner ES, Moss J. Informatics challenges for the impending patient information explosion. J Am Med Inform Assoc 2005; 12 (06) 614-617
• 33 Bath PA. Health informatics: current issues and challenges. J Inf Sci 2008; 34 (04) 501-518
• 34 Braun LM, Wiesman F, van den Herik HJ, Hasman A, Korsten E. Towards patient-related information needs. Int J Med Inform 2007; 76 (2-3): 246-251
• 35 Chang KC, Overhage JM, Hui SL, Were MC. Enhancing laboratory report contents to improve outpatient management of test results. J Am Med Inform Assoc 2010; 17 (01) 99-103
• 36 Hirsch JS, Tanenbaum JS, Lipsky Gorman S. , et al. HARVEST, a longitudinal patient record summarizer. J Am Med Inform Assoc 2015; 22 (02) 263-274
• 37 Hsu W, Taira RK, El-Saden S, Kangarloo H, Bui AA. Context-based electronic health record: toward patient specific healthcare. IEEE Trans Inf Technol Biomed 2012; 16 (02) 228-234
• 38 Varpio L, Rashotte J, Day K, King J, Kuziemsky C, Parush A. The EHR and building the patient's story: a qualitative investigation of how EHR use obstructs a vital clinical activity. Int J Med Inform 2015; 84 (12) 1019-1028
• 39 Shneiderman B. The eyes have it: A task by data type taxonomy for information visualizations. In The Craft of Information Visualization; 2003, pp. 364–371
• 40 Hornbæk K, Hertzum M. The notion of overview in information visualization. Int J Hum Comput Stud 2011; 69 (7–8): 509-525
• 41 Plaisant C, Mushlin R, Snyder A, Li J, Heller D, Shneiderman B. LifeLines: using visualization to enhance navigation and analysis of patient records. In The Craft of Information Visualization; 2003, pp. 308–312
• 42 Samal L, Wright A, Wong BT, Linder JA, Bates DW. Leveraging electronic health records to support chronic disease management: the need for temporal data views. Clin Prob Lists Electronic Health Record 2014; 24: 279