Download PDF
Appl Clin Inform 2016; 07(02): 604-623
DOI: 10.4338/ACI-2015-12-RA-0182
Research Article
Schattauer GmbH

Feasibility of population health analytics and data visualization for decision support in the infectious diseases domain

A pilot study
Don Roosan
1   Department of Biomedical Informatics, University of Utah, 421 Wakara Way, Salt Lake City, UT 84108, USA
,
Guilherme Del Fiol
1   Department of Biomedical Informatics, University of Utah, 421 Wakara Way, Salt Lake City, UT 84108, USA
2   IDEAS Center for Innovation, VA Salt Lake City Health System, 500 Foothill Drive, Salt Lake City, UT 84108, USA
,
Jorie Butler
2   IDEAS Center for Innovation, VA Salt Lake City Health System, 500 Foothill Drive, Salt Lake City, UT 84108, USA
,
Yarden Livnat
3   Scientific Computing and Imaging Institute, Department of Computer Sciences, University of Utah, 72 S Central Campus Dr, Salt Lake City, UT 84112, USA
,
Jeanmarie Mayer
2   IDEAS Center for Innovation, VA Salt Lake City Health System, 500 Foothill Drive, Salt Lake City, UT 84108, USA
,
Matthew Samore
1   Department of Biomedical Informatics, University of Utah, 421 Wakara Way, Salt Lake City, UT 84108, USA
2   IDEAS Center for Innovation, VA Salt Lake City Health System, 500 Foothill Drive, Salt Lake City, UT 84108, USA
,
Makoto Jones
2   IDEAS Center for Innovation, VA Salt Lake City Health System, 500 Foothill Drive, Salt Lake City, UT 84108, USA
,
Charlene Weir
1   Department of Biomedical Informatics, University of Utah, 421 Wakara Way, Salt Lake City, UT 84108, USA
2   IDEAS Center for Innovation, VA Salt Lake City Health System, 500 Foothill Drive, Salt Lake City, UT 84108, USA
› Author AffiliationsThis project was supported by the Agency for Healthcare Research and Quality (grant R36HS023349) and Department of Veterans Affairs Research and Development (grant CRE 12–230). Dr. Islam was supported by National Library of Medicine training grant (T15-LM07124) and partially supported by Houston Veterans Affairs Health Services Research & Development Center for Innovations in Quality and Effectiveness and Safety (IQuESt).
Further Information

Publication History

received: 31 December 2015

accepted: 01 May 2016

Publication Date:
16 December 2017 (online)

Also available at
   Permissions and Reprints

Summary

Objective

Big data or population-based information has the potential to reduce uncertainty in medicine by informing clinicians about individual patient care. The objectives of this study were: 1) to explore the feasibility of extracting and displaying population-based information from an actual clinical population’s database records, 2) to explore specific design features for improving population display, 3) to explore perceptions of population information displays, and 4) to explore the impact of population information display on cognitive outcomes.

Methods

We used the Veteran’s Affairs (VA) database to identify similar complex patients based on a similar complex patient case. Study outcomes measures were 1) preferences for population information display 2) time looking at the population display, 3) time to read the chart, and 4) appropriateness of plans with pre-and post-presentation of population data. Finally, we redesigned the population information display based on our findings from this study.

Results

The qualitative data analysis for preferences of population information display resulted in four themes: 1) trusting the big/population data can be an issue, 2) embedded analytics is necessary to explore patient similarities, 3) need for tools to control the view (overview, zoom and filter), and 4) different presentations of the population display can be beneficial to improve the display. We found that appropriateness of plans was at 60% for both groups (t9=-1.9; p=0.08), and overall time looking at the population information display was 2.3 minutes versus 3.6 minutes with experts processing information faster than non-experts (t8= -2.3, p=0.04).

Conclusion

A population database has great potential for reducing complexity and uncertainty in medicine to improve clinical care. The preferences identified for the population information display will guide future health information technology system designers for better and more intuitive display.

Keywords

Health information systems - uncertainty - big data - population decision support systems - information display
 

  • References

  • 1 Mandl KD, Kohane IS. Escaping the ehr trap — the future of health it. New England Journal of Medicine 2012; 366 (24) 2240-2242 doi:doi:10.1056/NEJMp1203102.
    CrossrefPubMedGoogle Scholar
  • 2 Mitka M. Physicians cite problems with ehrs. JAMA 2014; 311 (18) 1847-1847 doi:10.1001/jama.2014.5030.
    PubMedGoogle Scholar
  • 3 Wu HW, Davis PK, Bell DS. Advancing clinical decision support using lessons from outside of healthcare: An interdisciplinary systematic review. BMC Med Inform Decis Mak 2012; 12: 90 doi:10.1186/1472–6947–12–90.
    CrossrefPubMedGoogle Scholar
  • 4 Welch BM, Kawamoto K. Clinical decision support for genetically guided personalized medicine: A systematic review. Journal of the American Medical Informatics Association 2013; 20 (02) 388-400.
    CrossrefPubMedGoogle Scholar
  • 5 Hemmerich JA, Elstein AS, Schwarze ML, Moliski EG, Dale W. Risk as feelings in the effect of patient outcomes on physicians’ future treatment decisions: A randomized trial and manipulation validation. Soc Sci Med 2012; 75 (02) 367-376 doi:10.1016/j.socscimed.2012.03.020.
    CrossrefPubMedGoogle Scholar
  • 6 Woolley A, Kostopoulou O. Clinical intuition in family medicine: More than first impressions. Ann Fam Med 2013; 11 (01) 60-66 doi:10.1370/afm.1433.
    CrossrefPubMedGoogle Scholar
  • 7 Islam R, Weir C, Fiol GD. editors. Heuristics in managing complex clinical decision tasks in experts’ decision making. Healthcare Informatics (ICHI), 2014 IEEE International Conference on. 2014: 15-17 Sept. 2014.
    Google Scholar
  • 8 Redmond ROster. Questioning protocol. JAMA Intern Med 2014; 174 (05) 667-667 doi:10.1001/jamainternmed.2014.107.
    CrossrefPubMedGoogle Scholar
  • 9 Sistrom CL, Dreyer K, Weilburg JB, Perloff JN, Tompkins CP, Ferris TG. Images of imaging: How to process and display imaging utilization for large populations. American Journal of Roentgenology. 2015: W405-W420.
    PubMedGoogle Scholar
  • 10 Fauci AS, Morens DM. The perpetual challenge of infectious diseases. N Engl J Med 2012; 366 (05) 454-461 doi:10.1056/NEJMra1108296.
    CrossrefPubMedGoogle Scholar
  • 11 Fong IW. Challenges in infectious diseases. Springer; 2013
    Google Scholar
  • 12 Sullivan T. Antibiotic overuse and clostridium difficile: A teachable moment. JAMA Intern Med 2014; 174 (08) 1219-1220 doi:10.1001/jamainternmed.2014.2299.
    CrossrefPubMedGoogle Scholar
  • 13 Luo J, Wu M, Gopukumar D, Zhao Y. Big data application in biomedical research and health care: A literature review. Biomedical informatics insights 2016; 08: 1.
    PubMedGoogle Scholar
  • 14 Weber GM, Mandl KD, Kohane IS. Finding the missing link for big biomedical data. JAMA 2014; 311 (24) 2479-2480 doi:10.1001/jama.2014.4228.
    PubMedGoogle Scholar
  • 15 Tan SSL, Gao G, Koch S. Big data and analytics in healthcare. Methods of Information in Medicine 2015; 54 (06) 546-547 doi:10.3414/ME15–06–1001.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 16 Donderi DC. Visual complexity: A review. Psychol Bull 2006; 132 (01) 73-97 doi:10.1037/0033–2909.132.1.73.
    CrossrefPubMedGoogle Scholar
  • 17 Miller A, Scheinkestel C, Steele C. The effects of clinical information presentation on physicians’ and nurses’ decision-making in icus. Appl Ergon 2009; 40 (04) 753-761 doi:10.1016/j.apergo.2008.07.004.
    CrossrefPubMedGoogle Scholar
  • 18 Klimov D, Shahar Y, Taieb-Maimon M. Intelligent visualization and exploration of time-oriented data of multiple patients. Artif Intell Med 2010; 49 (01) 11-31 doi:10.1016/j.artmed.2010.02.001.
    CrossrefPubMedGoogle Scholar
  • 19 Koch SH, Weir C, Westenskow D, Gondan M, Agutter J, Haar M. et al. Evaluation of the effect of information integration in displays for icu nurses on situation awareness and task completion time: A prospective randomized controlled study. Int J Med Inform 2013; 82 (08) 665-675 doi:10.1016/j.ijmedinf.2012.10.002.
    CrossrefPubMedGoogle Scholar
  • 20 Djulbegovic B, Beckstead JW, Elqayam S, Reljic T, Hozo I, Kumar A. et al. Evaluation of physicians’ cognitive styles. Med Decis Making 2014; 34 (05) 627-637 doi:10.1177/0272989X14525855.
    CrossrefPubMedGoogle Scholar
  • 21 Visser W. Designing as construction of representations: A dynamic viewpoint in cognitive design research. Human–Computer Interaction 2006; 21 (01) 103-152.
    CrossrefPubMedGoogle Scholar
  • 22 Mirel B, Eichinger F, Keller BJ, Kretzler M. A cognitive task analysis of a visual analytic workflow: Exploring molecular interaction networks in systems biology. J Biomed Discov Collab 2011; 06: 1-33 doi:10.5210/disco.v6i0.3410.
    CrossrefPubMedGoogle Scholar
  • 23 Shneiderman B, Plaisant C, Hesse B. Improving health and healthcare with interactive visualization methods. IEEE Computer Special Issue on Challenges in Information Visualization 2013; 01: 1-13.
    PubMedGoogle Scholar
  • 24 West VL, Borland D, Hammond WE. Innovative information visualization of electronic health record data: A systematic review. Journal of the American Medical Informatics Association 2015; 22 (02) 330-339.
    PubMedGoogle Scholar
  • 25 Syroid ND, Agutter J, Drews FA, Westenskow DR, Albert RW, Bermudez JC. et al. Development and evaluation of a graphical anesthesia drug display. ANESTHESIOLOGY-PHILADELPHIA THEN HAGERSTOWN 2002; 96 (03) 565-575.
    PubMedGoogle Scholar
  • 26 Miller A, Sanderson P. editors. Designing an information display for clinical decision making in the intensive care unit. Proceedings of the Human Factors and Ergonomics Society Annual Meeting. 2003. SAGE Publications.
    Google Scholar
  • 27 Longhurst CA, Harrington RA, Shah NH. A ‘green button’for using aggregate patient data at the point of care. Health Affairs 2014; 33 (07) 1229-1235.
    CrossrefPubMedGoogle Scholar
  • 28 Jing X, Cimino JJ. A complementary graphical method for reducing and analyzing large data sets. Case studies demonstrating thresholds setting and selection. Methods of Information in Medicine 2014; 53 (03) 173-185 doi:10.3414/ME13–01–0075.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 29 Kopanitsa G, Hildebrand C, Stausberg J, Englmeier KH. Visualization of medical data based on ehr standards. Methods of Information in Medicine 2013; 52 (01) 43-50 doi:10.3414/ME12–01–0016.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 30 Murdoch TB, Detsky AS. The inevitable application of big data to health care. JAMA 2013; 309 (13) 1351-1352.
    CrossrefPubMedGoogle Scholar
  • 31 Livnat Y, Gesteland P, Benuzillo J, Pettey W, Bolton D, Drews F. et al. Epinome – a novel workbench for epidemic investigation and analysis of search strategies in public health practice. AMIA Annu Symp Proc 2010; 2010: 647-651.
    PubMedGoogle Scholar
  • 32 Garvin JH, Duvall SL, South BR, Bray BE, Bolton D, Heavirland J. et al. Automated extraction of ejection fraction for quality measurement using regular expressions in unstructured information management architecture (uima) for heart failure. J Am Med Inform Assoc 2012; 19 (05) 859-866 doi:amiajnl-2011–000535 [pii]10.1136/amiajnl-2011–000535.
    CrossrefPubMedGoogle Scholar
  • 33 Krall MAGAV, Samore MH. Big data and population-based decision support. In: Greenes RA. editor. Clinical decison support: The road to braod adoption. 2nd. ed. Oxford, UK: Elsevier Inc; 2014: 363-378.
    Google Scholar
  • 34 Wolf JR. Do it students prefer doctors who use it?. Computers in Human Behavior 2014; 35 (00) 287-294 doi:http://dx.doi.org/10.1016/j.chb.2014.03.020.
    CrossrefPubMedGoogle Scholar
  • 35 National Guideline C. Clinical practice guideline for the use of antimicrobial agents in neutropenic patients with cancer: 2010 update by the infectious diseases society of america. Agency for Healthcare Research and Quality (AHRQ): Rockville MD; http://www.guideline.gov/content.aspx?id=25651 Accessed 7/11/2015.
    Google Scholar
  • 36 Wickens CD, Carswell CM. The proximity compatibility principle: Its psychological foundation and relevance to display design. Human Factors: The Journal of the Human Factors and Ergonomics Society 1995; 37 (03) 473-494.
    CrossrefPubMedGoogle Scholar
  • 37 South BR, Shen S, Leng J, Forbush TB, DuVall SL, Chapman WW. editors. A prototype tool set to support machine-assisted annotation. Proceedings of the 2012 Workshop on Biomedical Natural Language Processing. 2012. Association for Computational Linguistics.
    Google Scholar
  • 38 Clark J. How to peer review a qualitative manuscript. Peer review in health sciences 2003; 02: 219-235.
    PubMedGoogle Scholar
  • 39 Altman DG. Practical statistics for medical research. CRC press; 1990
    Google Scholar
  • 40 Le T, Reeder B, Thompson H, Demiris G. Health providers’ perceptions of novel approaches to visualizing integrated health information. Methods Inf Med 2013; 52 (03) 250-258.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 41 Park CL, O’Neill PA, Martin DF. A pilot exploratory study of oral electrical stimulation on swallow function following stroke: An innovative technique. Dysphagia 1997; 12 (03) 161-6 doi:10.1007/PL00009531.
    CrossrefPubMedGoogle Scholar
  • 42 Ellis G, Dix A. An explorative analysis of user evaluation studies in information visualisation. Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization; Venice. Italy: 1168152: ACM; 2006: 1-7.
    Google Scholar
  • 43 Shneiderman B. Enabling visual discovery. Science 2014; 343 6171 614-614 doi:10.1126/science.1249670.
    PubMedGoogle Scholar
  • 44 Wang TD, Wongsuphasawat K, Plaisant C, Shneiderman B. Extracting insights from electronic health records: Case studies, a visual analytics process model, and design recommendations. Journal of Medical Systems 2011; 35 (05) 1135-1152.
    CrossrefPubMedGoogle Scholar
  • 45 Wongsuphasawat K. editor. Finding comparable patient histories: A temporal categorical similarity measure with an interactive visualization. IEEE Symposium on Visual Analytics Science and Technology (VAST);. 2009
    Google Scholar
  • 46 Onuoha OC, Arkoosh VA, Fleisher LA. Choosing wisely in anesthesiology: The gap between evidence and practice. JAMA Intern Med 2014; 174 (08) 1391-1395 doi:10.1001/jamainternmed.2014.2309.
    CrossrefPubMedGoogle Scholar
  • 47 Lobach D SG, Bright TJ. et al. Enabling health care decisionmaking through clinical decision support and knowledge management. Evidence report/technology assessments. Rockville (MD): Agency for Healthcare Research and Quality US. 2012
    PubMedGoogle Scholar
  • 48 Elstein AS. Thinking about diagnostic thinking: A 30-year perspective. Adv Health Sci Educ Theory Pract 2009; 14 (Suppl. 01) 7-18 doi:10.1007/s10459–009–9184–0.
    CrossrefPubMedGoogle Scholar
  • 49 Tinetti ME, Bogardus Jr ST, Agostini JV. Potential pitfalls of disease-specific guidelines for patients with multiple conditions. N Engl J Med 2004; 351 (27) 2870-2874 doi:351/27/2870 [pii]10.1056/NEJMsb042458.
    CrossrefPubMedGoogle Scholar
  • 50 Elstein AS. On the origins and development of evidence-based medicine and medical decision making. Inflamm Res 2004; 53 (Suppl. 02) S184-S189 doi:10.1007/s00011–004–0357–2.
    PubMedGoogle Scholar
  • 51 Gigerenzer G, Hertwig R, Pachur T. Heuristics: The foundations of adaptive behavior. Oxford University Press, Inc; 2011
    Google Scholar
  • 52 Wegwarth O, Gaissmaier W, Gigerenzer G. Smart strategies for doctors and doctors-in-training: Heuristics in medicine. Med Educ 2009; 43 (08) 721-728 doi:10.1111/j.1365–2923.2009.03359.x.
    CrossrefPubMedGoogle Scholar
  • 53 Gorini A, Pravettoni G. An overview on cognitive aspects implicated in medical decisions. Eur J Intern Med 2011; 22 (06) 547-553 doi:10.1016/j.ejim.2011.06.008.
    CrossrefPubMedGoogle Scholar
  • 54 Crebbin W, Beasley SW, Watters DA. Clinical decision making: How surgeons do it. ANZ J Surg 2013; 83 (06) 422-4208 doi:10.1111/ans.12180.
    CrossrefPubMedGoogle Scholar
  • 55 Markman KD, Klein WM, Suhr JA. Handbook of imagination and mental simulation. Psychology Press; 2012
    Google Scholar
  • 56 Phansalkar S, Weir CR, Morris AH, Warner HR. Clinicians’ perceptions about use of computerized protocols: A multicenter study. Int J Med Inform 2008; 77 (03) 184-193 doi:10.1016/j.ijmedinf.2007.02.002.
    CrossrefPubMedGoogle Scholar
  • 57 Luker KR, Sullivan ME, Peyre SE, Sherman R, Grunwald T. The use of a cognitive task analysis-based multimedia program to teach surgical decision making in flexor tendon repair. Am J Surg 2008; 195 (01) 11-15 doi:10.1016/j.amjsurg.2007.08.052.
    CrossrefPubMedGoogle Scholar
  • 58 Baxter GD, Monk AF, Tan K, Dear PR, Newell SJ. Using cognitive task analysis to facilitate the integration of decision support systems into the neonatal intensive care unit. Artif Intell Med 2005; 35 (03) 243-257 doi:10.1016/j.artmed.2005.01.004.
    CrossrefPubMedGoogle Scholar
  • 59 Pieczkiewicz D, Finkelstein S, Hertz M. editors. The influence of display format on decision-making in a lung transplant home monitoring program-preliminary results. Engineering in Medicine and Biology Society, 2003. Proceedings of the 25th Annual International Conference of the IEEE. 2003. IEEE.
    Google Scholar
  • 60 O’Hare D, Wiggins M, Williams A, Wong W. Cognitive task analyses for decision centred design and training. Ergonomics 1998; 41 (11) 1698-1718 doi:10.1080/001401398186144.
    CrossrefPubMedGoogle Scholar
  • 61 Winter A, Strbing A. Model-based assessment of data availability in health information systems. Methods of Information in Medicine 2008; 47 (05) 417-424 doi:10.3414/ME9123.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 62 Arnott D. Cognitive biases and decision support systems development: A design science approach. Information Systems Journal 2006; 16 (01) 55-78.
    CrossrefPubMedGoogle Scholar
  • 63 Islam R, Weir C, Del Fiol G. Clinical complexity in medicine: A measurement model of task and patient complexity. Methods of Information in Medicine 2016; 55 (01) 14-22 doi:10.3414/ME15-01-0031.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 64 Wongsuphasawat K, Plaisant C, Taieb-Maimon M, Shneiderman B. Querying event sequences by exact match or similarity search: Design and empirical evaluation. Interacting with Computers 2012; 24 (02) 55-68.
    CrossrefPubMedGoogle Scholar
  • 65 Carroll LN, Au AP, Detwiler LT, Fu TC, Painter IS, Abernethy NF. Visualization and analytics tools for infectious disease epidemiology: A systematic review. J Biomed Inform 2014; 51: 287-298 doi:10.1016/j.jbi.2014.04.006.
    CrossrefPubMedGoogle Scholar
  • 66 Rind A. Interactive information visualization to explore and query electronic health records. Foundations and Trends® in Human–Computer Interaction 2013; 05 (03) 207-298 doi:10.1561/1100000039.
    CrossrefPubMedGoogle Scholar
  • 67 Smith A, Malik S, Shneiderman B. Visual analysis of topical evolution in unstructured text: Design and evaluation of topicflow. Applications of social media and social network analysis. Springer International Publishing; 2015: 159-175.
    Google Scholar
  • 68 Monroe M, Lan R, Lee H, Plaisant C, Shneiderman B. Temporal event sequence simplification. IEEE Transactions on Visualization and Computer Graphics 2013; 19 (12) 2227-2236.
    CrossrefPubMedGoogle Scholar
  • 69 Shneiderman B, Plaisant C, Hesse BW. Improving healthcare with interactive visualization. Computer 2013; 46 (05) 58-66.
    CrossrefPubMedGoogle Scholar
  • 70 Carroll LN, Au AP, Detwiler LT, Fu T-c, Painter IS, Abernethy NF. Visualization and analytics tools for infectious disease epidemiology: A systematic review. Journal of Biomedical Informatics 2014; 51 (00) 287-298 doi:http://dx.doi.org/10.1016/j.jbi.2014.04.006.
    CrossrefPubMedGoogle Scholar
  • 71 Puget A, Mejino Jr JLV, Detwiler LT, Franklin JD, Brinkley JF. Spatial-symbolic query engine in anatomy. Methods of Information in Medicine 2012; 51 (06) 463-478 doi:10.3414/ME11–01–0047.
    Article in Thieme ConnectPubMedGoogle Scholar
  • 72 Wongsuphasawat K, Guerra JAGómez, Plaisant C, Wang TD, Taieb-Maimon M, Shneiderman B. editors. Lifeflow: Visualizing an overview of event sequences. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems; 2011. ACM.
    Google Scholar
  • 73 Gallego B, Walter SR, Day RO, Dunn AG, Sivaraman V, Shah N. et al. Bringing cohort studies to the bedside: Framework for a ‘green button’ to support clinical decision-making. 2015; 33 (07) 1229-1235.
    PubMedGoogle Scholar
  • 74 Spinellis D, Androutsellis-Theotokis S. Software development tooling: Information, opinion, guidelines, and tools. IEEE Software 2014; (06) 21-23.
    PubMedGoogle Scholar
  • 75 Lankow J, Ritchie J, Crooks R. Infographics: The power of visual storytelling. John Wiley & Sons. 2012
    PubMedGoogle Scholar
  • 76 Wang MQBaldonado, Woodruff A, Kuchinsky A. editors. Guidelines for using multiple views in information visualization. Proceedings of the working conference on Advanced visual interfaces. 2000. ACM.
    Google Scholar
  • 77 Zuk T, Schlesier L, Neumann P, Hancock MS, Carpendale S. editors. Heuristics for information visualization evaluation. Proceedings of the 2006 AVI workshop on BEyond time and errors: novel evaluation methods for information visualization. 2006. ACM.
    Google Scholar
  • 78 Gotz D, Wang F, Perer A. A methodology for interactive mining and visual analysis of clinical event patterns using electronic health record data. Journal of Biomedical Informatics 2014; 48: 148-159.
    CrossrefPubMedGoogle Scholar
  • 79 Crowley RS, Naus GJ, Stewart Iii J, Friedman CP. Development of visual diagnostic expertise in pathology: An information-processing study. Journal of the American Medical Informatics Association 2003; 10 (01) 39-51.
    CrossrefPubMedGoogle Scholar
  • 80 Schubert CC, Denmark TK, Crandall B, Grome A, Pappas J. Characterizing novice-expert differences in macrocognition: An exploratory study of cognitive work in the emergency department. Ann Emerg Med 2013; 61 (01) 96-109 doi:10.1016/j.annemergmed.2012.08.034.
    CrossrefPubMedGoogle Scholar
  • 81 Craig C, Klein MI, Griswold J, Gaitonde K, McGill T, Halldorsson A. Using cognitive task analysis to identify critical decisions in the laparoscopic environment. Hum Factors 2012; 54 (06) 1025-1039.
    CrossrefPubMedGoogle Scholar
  • 82 Christensen RE, Fetters MD, Green LA. Opening the black box: Cognitive strategies in family practice. Ann Fam Med 2005; 03 (02) 144-150 doi:10.1370/afm.264.
    CrossrefPubMedGoogle Scholar
  • 83 Arocha JF, Patel VL, Patel YC. Hypothesis generation and the coordination of theory and evidence in novice diagnostic reasoning. Medical Decision Making 1993; 13 (03) 198-211.
    CrossrefPubMedGoogle Scholar