Keywords
clinical decision support - heuristics - data processing - expert system - algorithm
Introduction
We focus on essential information cues and make decisions based on our bounded rationality in our daily decision-making.[1]
[2] Cognitive limitations of working memory result in the use of heuristics. Heuristics provide a take-the-best cue approach by minimizing information processing and selecting the best optimization. They have been commonly thought to create bias in decision-making.[1] Research has demonstrated in many fields that the take-the-best approach or less-is-more and computational model perform faster and more accurately than the traditional approach of information overload.[3]
[4]
[5]
[6]
[7] A common heuristic is “less is more.”[3] Unnecessary information creates information overload and makes the clinical decision flawed. The “less is more” heuristic has the potential for more accurate inferences and better predictions in an information space that creates unnecessary information overload.[3]
[8] Unnecessary information increases the chances of error.[1] This is especially relevant in clinical medicine.
Expert clinicians use many variations of a “rule of thumb”—or shortcut mental models—for complex cases to balance timeliness and accuracy during decision-making.[9]
[10] Much of the resulting work has incorporated the idea of optimization by considering the role of organizing information in sound reasoning. However, the medical community was quick to guard against heuristics owing to bias and faulty decision-making.[11]
[12] This bias is based on clinicians ignoring a vital piece of information. Despite this, the last three decades of heuristics research has revealed that clinical experts make better decisions by ignoring some information and focusing on the most relevant indicators for optimal clinical outcome.[13] An example of this approach involves fast-and-frugal tree (FFT) heuristics. The FFT is a type of shortcut mental model that involves an algorithm approach. The “fast-and-frugal” approach to decision-making has achieved widespread popularity in various areas, including decision analysis inpatient care.[4]
[13]
[14] The overall goal of the heuristic is to develop robust and straightforward decisions using minimal but crucial information. This results in more effective information processing and better decision-making in complex domains such as infectious diseases (ID) medicine. In clinical medicine, ID represents a challenging and complex domain.[15]
[16]
[17] The complexity of emerging diseases such as coronavirus disease 2019 (COVID-19), environmentally persistent organisms, and increasing antibiotic resistance contribute to the ID domain's complex information environment.[18] For example, understanding and dealing with pathogens with limited information can make clinicians apprehensive. However, the less-is-more approach supports limited information processing for better decisions. Therefore, clinical decision support systems (CDSSs) are crucial to support ID clinicians. CDSS interventions in the ID domain include microbiology understanding, using information visualizations, and optimizing treatment and therapy outcomes.[16]
[19]
[20]
Previous research in the design of health information systems focused on cognitively supporting clinicians by creating user-friendly CDSS embedded within the electronic health records (EHRs).[21]
[22] Most current EHRs are not designed to help clinicians' cognition.[20]
[22] Clinician's cognition refers to the higher cognitive skills to ponder complex clinical problems. Previous research has shown that the complex approach of showing all information leading to cognitive overload may not be suitable for heuristics-based decision-making by experts.[23]
[24]
[25] Such systems with unnecessary information cues on the screen will create information overload and may increase errors.[26] Complaints from clinicians center around the inability of information technology (IT) to support the high level of reasoning required to execute complex clinical tasks.[27] As we are designing and implementing our health IT system and CDSSs, exploring and understanding the value and power of heuristics in the system design are essential. Thus, there is a gap in understanding how expert clinicians process complex cases using a shortcut mental model.
In the present study, we explored complex decision-making by ID clinicians and focused on fast and frugal heuristics to simplify complex decisions into simple shortcut mental models. We hypothesized that ID clinicians use their vast experience to apply simple heuristics to reduce the practical complexity of complex cases.
Methods
Study Design
In this study, we used cognitive task analysis (CTA) and heuristics-based decision modeling. We conducted CTA, a systematic method for understanding and describing experts' complex reasoning in performing complex tasks.[27] We used “combinatorics” of CTA, combining the critical decision method (CDM) with essential incident interviews.[24] Combinatorics is a method to compare, converge, merge, and adapt methods from CTA. CTA data can be analyzed in a myriad of ways. CTA can better understand the overall data when merged. We have used the RATS (relevance of study question, appropriateness of qualitative methodology, transparency of procedure, and soundness of interpretive approach) protocol to analyze data from transcriptions of the interviews qualitatively.[28] The RATS protocol ensures CTA data curation and analysis are performed in an organized format for data reproducibility. The RATS protocol provides standardized guidelines for qualitative research methods, including CTA, cognitive work analysis, and other human factors methodology. We used the FFT algorithm to reduce complex decisions into more straightforward decision steps for decision modeling. Our main goal was to understand the value of the FFT algorithm in reducing complex cases. We used the SQUIRE (Standards for Quality Improvement Reporting Excellence) reporting guidelines for the results.[29]
Settings
We conducted the study at the University of Utah and the Salt Lake City Veterans Administrations Hospital. The study was approved by the respective institutional review boards (IRBs). All participants provided oral consent, and the University of Utah IRB approved verbal consent.
Participants
We conducted the cognitive interviews and decision-modeling tasks with 10 ID expert clinicians, defined as board-certified ID physicians with more than 5 years of full-time work experience in the ID field.
Procedure
Interviews
The first author conducted semistructured interviews based on the CDM, a type of CTA.[24] Each ID expert was asked to describe a recent complex case challenging diagnostics or treatment. The first author explained to the ID experts the purpose and objective of the study. Then, ID experts were told to think about a very complex case that took considerable effort and time.
A semistructured interview script was piloted with two ID clinicians and was refined. At the end of the interview, participants were asked to provide basic demographic information. Each interview lasted around 50 minutes. All interviews were recorded and transcribed. Audio transcripts were transcribed using a third-party company called Datagain. The company uses software and manual checks to ensure the integrity of transcription. Datagain is a Health Insurance Portability and Accountability Act (HIPAA)-compliant transcription company that uses human transcription for health care data. All patient identifiers were removed.
Decision Model
To understand fast-and-frugal decisions, we provided an FFT ([Fig. 1]). There is at least one exit leaf in the FFT algorithm at every decision node (rectangular box in [Fig. 1]). For every checked cue, at least one outcome can lead to a decision. Thus, an FFT algorithm leads to the well-established simple heuristics strategy for comparison called “take the best.”[30] Each rectangular box represents a decision node and, based on binary (yes/no) decisions, another node starts. Therefore, FFT heuristics depend on a search-stop-decision rule. After our interviews, we asked each participant to construct the complex case verbally. The research team member explained to each clinician completing the FFT in detail. Each clinician then filled out an FFT to practice based on a sample complex case. After the sample case FFT was filled, the research team member explained the correct procedure and meanings of different nodes in detail. Once the ID experts verbally clarified their understanding of filling out the FFT, the research team member provided them with an empty FFT to complete for the specific complex case. The participants also verbalized in detail the rationale of the FFT algorithm. We noted the time to complete the algorithm and calculated the number of information cues for each decision made at each step/node of the algorithm. We hypothesize that ID experts will take more time than usual to deliberate on a complex case. Therefore, collecting time data on completing the FFTs can give us a better understanding of the overall decision-making process. The time data will confirm if experts are using heuristics to solve a complex case. Generally, complex cases should take longer than simple ones that follow evidence-based guidelines.
Fig. 1 Fast-and frugal-tree decision algorithm used in the present study. Each node branches out based on binary decisions.
Data Analysis
Interviews
The research team conducted a qualitative analysis of the interview transcripts. The research team developed a codebook for conducting the iterative thematic analysis. First, all research team members met, identified, and parsed data sections for creating the codebook. After renaming each concept, the team members discussed, merged, and finally developed a group of terms to be included in the codebook. Once the codebook was created, the team members conducted an iterative analysis. The iterative inductive analysis independently identified the following three parts of transcripts: (1) sections related to FFT mental quirks focusing on sensemaking, (2) sections related to cognitive strategies, and (3) sections about coping with the complexity of the case. Three researchers independently conducted data analyses and then met as a team to complete the full transcripts. Each researcher had more than 10 years of experience in qualitative coding. The research team first created a sentence structure extraction from the CTA. The analysis revealed the texts and sentences related to heuristics. Once the specific paragraphs and sentences were identified, the research team conducted the content analysis. The content analyses were conducted in four phases: (1) initial review, (2) data coding, (3) synthesis, and (4) grouping codes into meaningful sentences.[31]
[32] The researchers coded the heuristics independently and later met to discuss the codes. The researchers then examined the codes, merged similar codes, and finally reached a consensus on a meaningful sentence representing if–then logic. One of the simplest ways our brain understands heuristics is by creating a cause–effect or if–then relationship from the FFT trees. All FFT algorithms have if–then decision trees. Thus, the if–then clause can support the most effective way to store information for future purposes. For example, if a transcript focused on “looking at the big picture instead of wallowing in given data,” then an if–then logic was formulated such as “if big picture conflicts with minutia, go with big-picture.” The multidisciplinary research team sought a group consensus at the end of each iteration, and the resulting codes were used for subsequent iterations. Finally, the team consensually merged all codes into if–then logic–based meaningful sentences. To reduce any biases in coding, the interrater reliability of Cohen's kappa was also calculated. We used the qualitative software program Atlas ti 7.0 to conduct data analyses. Atlas ti version 19.1 was used for data analysis.
Decision Model Results
Two researchers (doctors) analyzed the FFT algorithms from ID physician's transcripts. Each FFT decision node was analyzed for a specific number of information cues. Decision nodes were examined with the cues that were vital for decision-making. For example, if “patient with other signs of systemic infections, increased WBC, positive culture from blood,” three information cues were recorded. The research team calculated the number of information cues for each complex case and the time to complete each fast-and-frugal decision algorithm.
Results
Interviews
The ID experts (n = 10) had an average of 19 years of experience; two were female, and eight were male. The results matched the study goal of understanding if expert ID clinicians can reduce complex information into FFT algorithms. The results demonstrate that ID clinicians were very efficient in developing the FFT for all the 10 complex cases. The content analysis produced 27 meaningful if–then logic sentences, which the ID experts used to deal with complex cases. For example, antibiotic resistance is a global issue. Our results showed several if–then sentences related to topics such as “if repeated antibiotics then are more conservative about antibiotic use” and “if no reason to change, do the usual.” The complete list is provided in [Table 1]. The final interrater reliability of Cohen's kappa was 0.86.
Table 1
If–then heuristics from a qualitative data analysis of interviews conducted in the present study
|
If big-picture conflicts with minutia, go with big-picture
|
|
If applicable clinical pathway exists, follow the pathway
|
|
If no reason to change, do the usual
|
|
If abnormal context, do not expect normal results
|
|
If cause makes sense, consider it. Otherwise, look for causes that make sense
|
|
If repeated antibiotics, then be more conservative about antibiotics use
|
|
If no cause, then continue to look for cause and watch
|
|
If context + history is not compelling, then be conservative and watch
|
|
If no reason to deviate from the conservative path, continue on that path
|
|
If stuck, hope for luck
|
|
If risk is high, go with safest bet
|
|
If patient likes it, then try that first
|
|
If context (family or personal History) takes us into dangerous space, then consider other options
|
|
If in over your head, punt
|
|
If don't have definitive answer, even if doesn't affect care, then have low threshold for requesting more input
|
|
If sunk cost, leave it on ocean floor
|
|
if no diagnosis, GET ONE
|
|
If actions do not meet standard of care AND harm comes, THEN you are at fault
|
|
If can eliminate diagnosis, then can more quickly reach decision
|
|
If can categorize, then make assumptions
|
|
If find a trail, then follow it closely
|
|
If matches pattern, then follow typical course for pattern
|
|
If have data but not answer, then take best guess based on data
|
|
If safe to get more information, then wait and get more information
|
|
If history available, then get it.
|
|
If unexpected, then react accordingly and in proportion to risk
|
|
If context conflicts with guidelines, then go with context
|
Example Cases
Two of the problematic cases are described in the following:
“Patient is a 62-year-old, white male Veteran who presented to the VA with a history of back pain. It turns out that for the past, maybe even a couple of months, he's been falling and hurting his back and has been going to the clinic. They first treated him for the usual injuries, brain, et cetera, but the pain continued after about another month. They went ahead and got an MRI. T10, T11 spine, and the disc space between them were destroyed, and osteomyelitis was diagnosed. The care team saw the MRI with the radiologist, which turned out to be an infection. The risk factor may have been that the patient had done dental work about four to six weeks ago, so he may have been bacteremia at that time, which transmitted into the disk space. We're going to ask interventional radiology to put a needle in that space and see if we can get some tissue for microbiology and pathology. The care team started Vancomycin and third-generation cephalosporin, but the infection is unclear.”
“ The patient, a 57-year-old male, came to the VA 45 days ago for community-acquired pneumonia. However, after being treated for ten days, he developed a Vancomycin-resistant enterococcus (VRE) infection. The care team started the patient on Daptomycin based on clinical guidelines. However, after being treated for 15 days, the VRE persisted. There are no guidelines for patients who did not respond to Daptomycin. The options were to give patients another off-label antibiotic or keep the patient on Daptomycin. It was unclear when to stop the medication and if starting other medications at this point can significantly improve or worsen the overall situation for the patient.”
Decision Model
The 10 ID experts completed the FFT algorithm for each complex case in 1.65 ± 0.37 minutes. All ID experts average reduced their complex cases to simple heuristics in less than 2 minutes.
For the decision model, the ID experts analyzed 5.4 ± 3.1 information cues. One ID expert used five decision nodes, while another used four nodes. The remaining ID experts used only three decision nodes to complete the FFT algorithm. Two examples of FFTs are provided in [Fig. 2]. Each of the 10 FFT algorithms generated is an independent case and is unrelated to each other.
Fig. 2 Two examples of fast-and-frugal algorithms (A, B). The infectious diseases experts interviewed in the present study were able to reduce the complex cases within each decision node.
Our results show that even very complex cases described by ID experts were solved using a very short time. However, this also may be due to diverse complex cases unique to each ID expert.
Discussion
Our findings are similar to those of previous studies on the efficacy and effectiveness of heuristics.[30]
[33]
[34]
[35] Previous studies also found that the fast and frugal algorithm supports efficient and effective strategies for experts to deal with complex problems. Studies found that improving judgment clarifies the importance of understanding the degree to which different heuristics can be employed in the clinical domain and their actual usage. We agree that other domains of medicine may end with different types of heuristics.
One performance benefit of using heuristics could be redundant processing information, as repetitive tasks become quicker and easier to execute and understand and faster to communicate.[36] The results from this study may also be explained within fuzzy trace theory (FTT).[37] FTT is focused on the fact that judgment and decision-making rely preferentially on the gist representations of information instead of verbatim presentations. The preference to operate on the crudest gist increases with experience or expertise, as seen in our ID clinicians' study. For example, in one of the most complex cases described in the present study ([Fig. 2A, B]), the clinician focused on vital information cues from patient history such as urinary tract infections, sepsis/unstable, and blood pressure/mental status that correlated those with previous experience in treating similar patients. This gist-based thinking is not merely the retrieval of instances experienced in the past with other identical ID patients. Still, it is instead the distillation of the meaning of past experiences into an intuitive, bottom-line interpretation.
The clinical training is exhaustive and ensures that physicians adhere to clinical guidelines. Therefore, it can be mentioned that FFTs can be a simple projection of the clinical guidelines or rules. However, in this study using CDM, we ensured that clinicians focus only on a complex case. A case is complex when clinical guidelines are exhausted and no evidence-based information is available. The study focused on understanding ID experts' mechanisms to manage complex cases. The study findings still point toward the simple FFTs that experts use, similar to the if–then logic of clinical guidelines. These findings have significant implications for the health IT system design.
Implications for Health IT System Design
The current design of the Health IT system does not reflect the high-level reasoning of clinical experts.[38] As a result, the disjointed EHRs and their lack of usability in available CDSS have introduced new medical errors into the health care system.[39]
[40]
[41]
[42]
[43]
[44]
[45] EHR usability and CDSS design remain critical challenge in health care, with implications for medical errors, patient safety, and clinician burnout.[46]
[47] Current system designers assume that more information supports clinical reasoning and will thus lead to better decisions.[46]
[48] As a result, the current system design includes cluttered interfaces with extra information that does not support novice or expert clinicians. Even simple pharmacogenomics decision support systems are cluttered with laboratory results without pertinence.[43]
[49]
[50]
[51] However, results from this study suggest that heuristics may require fewer data while simultaneously producing better results using the concept of less is more. A concern, particularly from a liability perspective, is whether ignored information could cause biases. Our current CDSSs do not have the full capabilities to understand the higher cognition of expert clinicians. We provide future designers with some opportunities to create innovative and intuitive design features that may support these if–then statements.
The results from this study can potentially improve future designs of health IT. For example, “If matches pattern, then goes with the pattern.” This can be translated to provide clinicians with pattern-matching analytics. If patients' laboratory values are consistent and within normal limits with the start of the antibiotic, then the interface can show the clinician a time-series graph of the usual pattern. Such visualization can help the clinician understand the patient's overall situation. Another example is on the heuristics “If repeated antibiotics then are more conservative about antibiotics use.” The current systems do not differentiate between different types of medications in terms of providing alerts. Advanced system designers may provide specific overprescribing of antibiotics-related alerts to ID clinicians. Also, the study informs designers to be aware of different cognitive biases that arise from shortcut thinking, which may be detrimental to patient safety. With the advent of machine learning and artificial intelligence, it is possible to have automated systems check for such biases related to heuristics.[52]
[53] For example, system designers may assume that using FFT heuristics could cause ID experts to ignore information such as critical laboratory values. The system can flag any unusual data for the clinician using advanced machine learning that warrants immediate attention. Heuristics may offer an opportunity to mimic the higher cognitive thinking of clinicians. However, we must acknowledge the bias and errors that may come with heuristics. Therefore, future machine learning algorithms may check for biases and fix them automatically.[54]
[55] For example, if clinicians use heuristics and overlook some test results, the machine learning algorithm may prompt an intelligent alert to clinicians to double-check parameters. Our results' if–then statements may prove that system designers may design interfaces with pivotal information. At the same time, background machine learning checks should focus on reducing errors and biases resulting from simple heuristics thinking. Previous research has shown that not considering specific cues and not learning from reading extensively different aspects of patients' history may result in misdiagnosis. Our study also emphasizes the rationale for considering more information in deliberation. However, the current cluttered EHR design may hinder clinicians from looking into all aspects of the patient. Future intuitive interfaces taking consideration of heuristics and artificial intelligence methods may fill in the gap.[56]
[57]
[58]
Future studies may evaluate the coverage of the 27 if–then logic heuristics for their usefulness in system design. Developing a metric to quantify the outcome of these heuristics in system design may be necessary. Displaying critical information that matches the high-level reasoning techniques used by clinical experts could ultimately improve the design of health IT systems and save lives by reducing errors. Future studies should validate and possibly extend the heuristics highlighted in this study. For example, knowing what specific information ID experts ignore when creating advanced CDSS tools will be essential. Future studies may also look into other specialized fields such as cardiology or rheumatology to identify similar FFT heuristics. The results may validate this study's results in terms of matching the higher cognition of expert clinicians.
Limitations
Even though experts may focus on limited crucial information, we acknowledge that similar outcomes may not be the same for novice clinicians who might require more information cues to feel confident. More research into the area of novice clinicians is needed as well. Also, the smaller sample size of 10 participants may curb the findings' generalizability. There could be variabilities in coding heuristics statements. However, the researchers piloted and created a coding list to reduce biases. Also, in this study, we did not evaluate whether the FFTs generated were the efficient rule. We assumed that expert ID clinicians would make the optimum decision to improve their patients. Future studies may test these FFT algorithms for evaluations. We acknowledge that the culture of antibiotic stewardship affects how ID experts make treatment decisions. Antibiotic resistance is a global issue leading to resistant bacteria due to overprescribing antibiotics. Many of the overprescribing allude to the culture of prescribing to ensure clinicians feel safe. Such training on overprescribing may influence the study results. However, we only focused this study on ID clinicians with expertise and awareness of antibiotic resistance issues. Also, many if–then logic from the results may not be feasible to be translated into improving health IT system design. Any CTA study comes with recall and recency bias. We acknowledge that recall bias can be a problem. However, we asked each clinician to tell us a case that is freshly remembered with enough details in their minds. Some of the decision heuristics were nonspecific and may not be applied to the actual design of an intuitive interface. Not all of these heuristics can be used to improve health system design. Also, these results may not apply to novice clinicians. The if–then statements are based on the mental shortcut that expert clinicians use. The sentences may not represent complete and total explanations of the specific situation. The data analysis team included ID clinicians. Complexity is a phenomenon that is unique to a particular individual. One complex case to an ID clinician may not be complex for another ID clinician. We acknowledge the personal bias in the cases presented. Our research cannot claim that the resulting if–then statements can be used as building blocks for health IT system design. However, this research provides opportunities for future studies to incorporate heuristics into the health interface systems design for intuitive design.
Conclusion
In this study, we have demonstrated that ID experts can reduce very complex cases using FFT heuristics using if–then logic. Our results identified 27 if–then logic–based heuristics that ID clinicians use while dealing with highly complex cases. All 10 ID expert clinicians completed the FFT heuristics in less than 2 minutes. Relying on the concept of less is more as a tool for medical decision-making may help clinicians make accurate, transparent, and quick decisions. Expert clinicians may deal with complex cases better with the information presented in a more methodological and at-the-point of care style. Future health care system designers should consider adopting the idea of “less is more” in the health IT system design to reduce errors by matching system design to the higher-level cognitive abilities of the expert clinicians who will use the systems.
Clinical Relevance Statement
Clinical Relevance Statement
Our results focus on the expert clinicians' decision-making within the context of less-is-more heuristics. Future health IT system designers should focus on intuitive design by creating interfaces that match the cognition of expert clinicians found in our results. The current frustrations with information overload in the graphical interface can be solved by understanding heuristics and using less vital information cues for safer clinical practice.
