A History of the INTERNIST-1 and Quick Medical Reference (QMR) Computer-Assisted Diagnosis Projects, with Lessons Learned

Summary

The INTERNIST-1/Quick Medical Reference (QMR) diagnostic decision support project spans four decades, from 1971-onward. This paper describes the history of the project and details insights gained of relevance to the general clinical and informatics communities.

Keywords

Medical imaging - imaging informatics - PACS - image analysis workstations

 

• References

• 1 Collen MF. A History of Medical Informatics in the United States: 1950 to 1990. Bethesda, MD: Amer Med Informatics Assoc; 1995: 489 pages. ISBN 0-9647743-0-5..
• 2 Miller RA. Medical diagnostic decision support systems: past, present, and future. J Am Med Inform Assoc 1994; 01: 827.
• 3 Miller RA. Computer-assisted diagnostic decision support: history, challenges, and possible paths forward. Adv Health Sci Educ Theory Pract 2009; Sep; 14 Suppl (Suppl. 01) 89-106 Epub 2009 Aug 12. PMID: 19672686..
• 4 Pople HE, Myers JD, Miller RA. DIALOG:A model of diagnostic logic for internal medicine. In: Proc Fourth Internat Joint Conf Artif Intel. Cambridge, Massachusetts: MIT Artificial Intelligence Laboratory Publications. 1975: 848-55.
• 5 Miller RA, Schaffner KF. The logic of problemsolving in clinical diagnosis: a course for secondyear medical students. J Med Ed 1982; 57: 63-5.
• 6 Miller RA, Pople Jr HE, Myers JD. INTERNIST-1, An Experimental Computer-based Diagnostic Consultant for General Internal Medicine. N Engl J Med 1982; 307: 468-76.
• 7 Pople Jr HE. Heuristic methods for imposing structure on ill-structured problems: the structuring of medical diagnostics. In Szolovits P. (ed): Artificial Press; (AAAS Symposium Series, no. 51). 1982: 119-190.
• 8 Miller RA. Diagnostic and Prognostic Decision making Systems:A Survey of Recent Developments in the United States. In: Koehler CO, Tautu P, Wagner G. editors. Der Beitrag der Informationsver- arbeitung zum Fortschritt der Medizin. Medizinische Informatik und Statistic series, #50. Proceedings of the 28th Jahrestagung, Deutsche Gesellschaft fur Medizinische Dokumentation, Informatik und Statistik. Heidelberg, Germany: Springer Verlag; 1983
• 9 Miller RA. INTERNIST-1/CADUCEUS: Problems Facing Expert Consultant Programs. Methods Inf Med 1984; 23: 9-14.
• 10 Miller RA. A Computer-based Patient Case Simulator. Clin Res 1984; 32: 651A.
• 11 First MB, Soffer LJ, Miller RA. QUICK (QUick Index to Caduceus Knowledge): Using the INTERNIST-1/CADUCEUS Knowledge Base As An ElectronicTextbook of Medicine. Comp Biomed Res 1985; Apr; 18 (02) 137-65.
• 12 Miller RA, Schaffner KF, Meisel A. Ethical and Legal Issues Related to the Use of Computer Programs in Clinical Medicine. Ann Intern Med 1985; 102: 529-36.
• 13 Miller RA. INTERNIST-1, An Experimental Program for Consultation in General Internal Medicine: A Critical Review of its Approach to the Use of Laboratory Information for Diagnosis. In: Proc 32nd Meeting Japan Soc Clin Path. Matsumoto, Japan; September, 1985
• 14 Masarie Jr FE, Miller RA, Myers JD. INTERNIST-1 PROPERTIES:RepresentingCommonSenseandGood MedicalPracticeinaComputerizedMedicalKnowledge Base. Comp Biomed Res 1985; 18: 458-79.
• 15 Miller RA, Masarie Jr FE, Myers JD. “Quick Medical Reference” for diagnostic assistance. MD Computing 1986; 03: 34-48.
• 16 Heckerman D, Miller RA. Towards a better understanding of the INTERNIST-1 knowledge base. In: Salamon R, Blum B, Jorgenson M. editors. Proc MEDINFO 86. Washington, D.C: 1986: 22-6.
• 17 Miller RA, McNeil MA, Challinor S, Masarie Jr FE, Myers JD. Status Report: The INTERNIST-1 / Quick Medical Reference Project. West J Med 1986; 145: 816-22.
• 18 Miller RA. Computer based Diagnostic Decision making. Medical Care 1987; 25 (12) S148-S152.
• 19 Bankowitz RA, Blumenfeld BH, Miller RA. User variability in Abstracting and entering printed case histories with Quick Medical Reference (QMR). Proc 11th Annu Symp Comput Appl Med Care. Washington, DC: IEEE Press; 1987: 68-73.
• 20 Parker RC, Miller RA. Using causal knowledge to create simulated patient cases: The CPCS project as an extension of INTERNIST 1. Proc 11th Annu Symp Comput Appl Med Care. Washington, DC: IEEE Press; 1987: 473-80.
• 21 Lincoln M, Turner C, Hesse B, Miller RA. A comparison of Clustered Knowledge Structures in Iliad and in Quick Medical Reference. Proc 12th Annu Symp Comput Appl Med Care. Washington, DC: IEEE Press; 1988: 131-5.
• 22 Giuse NB, Giuse DA, Miller RA. Computer assisted multi-center creation of medical knowledge Care. Washington, DC: IEEE Press; 1988: 583-90.
• 23 Giuse NB, Bankowitz RA, Giuse DA, Parker RC, Miller RA. Medical Knowledge Base Acquisition: The Role of the Expert Review Process in Disease Profile Construction. Proc 13th Annu Symp Comput Appl Med Care 1989; 105-9.
• 24 Wagner MM, Bankowitz RA, McNeil MA, Challinor SM, Janosky JE, Miller RA. The diagnostic importance of the history and physical examination as determined by the use of a medical decision support system. Proc 13th Annu Symp Comput Appl Med Care 1989; 139-44.
• 25 Bankowitz RA, McNeil MA, Challinor SM, Parker RC, Kapoor WN, Miller RA. A computerassisted medical diagnostic consultation service: implementation and prospective evaluation of a prototype. Ann Intern Med 1989; 110: 824-32.
• 26 Miller RA, Masarie Jr FE. Use of the Quick Medical Reference (QMR) program as a tool for medical education. Methods Inf Med 1989; 28: 340-5.
• 27 Parker RC, Miller RA. Creation of a knowledge base adequateforsimulatingpatientcases:addingdeepknowledge to the INTERNIST-1/QMR Knowledge Base. Methods Inf Med 1989; 28: 346-51.
• 28 Bankowitz RB, McNeil MA, Challinor SM, Miller RA. Effect of a computer-assisted general medicine diagnostic consultation service on house staffdiagnosticstrategy. Methods Inf Med 1989; 28: 352-6.
• 29 Giuse NB, Giuse DA, Miller RA. Medical knowledge base construction as a means of introducing students to medical informatics. Proc Internat Symp Med Informat Educ. IMIA:Victoria, BC May 1989; 228-32.
• 30 Miller RA, Masarie Jr FE. The demise of the “Greek Oracle” model for medical diagnostic systems. Methods Inf Med 1990; 29: 1-2.
• 31 Miller RA. Why the standard view is standard: people, not machines, understand patients’ problems. J Med Philos 1990; 15: 581-91.
• 32 Giuse DA, Giuse NB, Miller RA. Towards computer assisted maintenance of medical knowledge bases. Artif Intel Med 1990; 02: 21-33.
• 33 Mabry ME, Miller RA. Distinguishing drug toxicity syndromes from medical diseases: a QMR computer-based approach. Proc 14th Annu Symp Comput Appl Med Care. Washington, DC:IEEE Computer Society Press 1990; 65-71.
• 34 Miller RA, Giuse NB. The medicine in medical informatics: medical knowledge bases. Acad Med 1991; 66 (01) 15-7.
• 35 Giuse DA, Giuse NB, Bankowitz RA, Miller RA. Heuristic determination of quantitative data for knowledge acquisition in medicine. Comput Biomed Res 1991; 24 (03) 261-72.
• 36 Shiffman S, Wu AW, Poon AD, Lane CD, Middleton B, Miller RA. et al. Building as peechinter facetoamedical diagnostic system. IEEE Expert 1991; 06: 41-50.
• 37 Bankowitz RA, Miller JK, Janosky J. A prospective analysis of inter-rater agreement between a physician and a physician’s assistant in selecting QMR vocabulary terms. Proc Annu Symp Comput Appl Med Care 1991; 609-11.
• 38 Berman L, Miller RA. Problem area formation as an element of computer-aided diagnosis: a comparison of two strategies within Quick Medical Reference (QMR). Methods Inf Med 1991; 30 (02) 90-5.
• 39 Masarie Jr FE, Miller RA, Bouhaddou O, Giuse NB, Warner HR. An interlingua for electronic interchange of medical information: using frames to map between clinical vocabularies. Comp Biomed Res 1991; 24 (04) 379-400.
• 40 Miller RA, Jamnback L, Giuse NB, Masarie Jr FE. Extending the capabilities of diagnostic decision support programs through links to bibliographic searching: addition of “Canned MeSH Logic” to the Quick Medical Reference (QMR) program for use with Grateful Med. Proc Annu Symp Comput Appl Med Care 1991; 150-5.
• 41 Miller RA, Masarie FE. The Quick Medical Reference (QMR) Relationships Function: description and evaluation of a simple, efficient “multiple diagnoses” algorithm. Proc MEDINFO 92. Geneva, Switzerland:IMIA 1992; 512-8.
• 42 DeCillis A, Giuse DA, Thayer EH, Giuse NB, Miller RA. Using speech recognition in medical knowledge acquisition. Proc MEDINFO 92. Geneva, Switzerland:IMIA 1992; 1323-9.
• 43 Giuse NB, Giuse DA, Miller RA, Bankowitz RA, Janosky JE, Davidoff F. et al. Evaluating Consensus Among Physicians in Medical Knowledge Base Construction. Methods Inf Med 1993; 32: 137-45.
• 44 Giuse DA, Giuse NB, Miller RA. Consistency enforcement in medical knowledge base construction. Artif Intel Med 1993; 05: 245-52.
• 45 Giuse NB, Giuse DA, Bankowitz RA, Miller RA. Comparing contents of a knowledge base to traditional information sources. Proc Annu Symp Comput Appl Med Care 1993; 626-30.
• 46 Giuse DA, Giuse NB, Miller RA. Knowledge base editing with QMR-KAT. Proc Annu Symp Comput Appl Med Care 1993; 935.
• 47 Giuse DA, Giuse NB, Miller RA. Evaluation of Long-Term Maintenance of a Large Medical Knowledge Base. JAm Med InformAssoc 1995; 02: 297-306.
• 48 Aliferis CF, Cooper GF, Miller RA, Buchanan BG, Bankowitz RA, Giuse NB. A temporal analysis of QMR. J Am Med Inform Assoc 1996; 03: 79-91.
• 49 Rassinoux A-M, Miller RA, Baud RH, Scherrer J-R. Modeling principles for QMR medical find- ings. Proc Annu Symp Comput Appl Med Care 1996; 264-8.
• 50 Miller RA. Evaluating evaluations of medical di- agnostic systems. J Am Med Inform Assoc 1996; Nov-Dec; 03: 429-31.
• 51 Miller RA. A heuristic approach to the multiple diagnoses problem. In: Keravnou E, Garbay C, Baud R, Wyatt J, editors. Artificial Intelligence in Medicine, Proceedings of AIME 97. Lecture Notes in Artif icial Intelligence Series, #1211. Springer:Berlin 1997; 187-98.
• 52 Rassinoux A-M, Miller RA, Baud RH, Scherrer J-R. Compositional and enumerative designs for medical language representation. Proc Annu Symp Comput Appl Med Care 1997; 620-4.
• 53 Rassinoux A-M, Miller RA, Baud RH, Scherrer J-R. Modeling concepts in medicine for medical language understanding. Methods Inf Med 1998; Nov; 37 (4-5): 361-72.
• 54 Shwe MA, Middleton B, Heckerman DE, Henrion M, Horvitz EJ, Lehmann HP. et al. Probabilistic diagnosis using a reformulation of the INTERNIST1/QMR knowledge base. I. The probabilistic model and inference algorithms. Methods Inf Med 1991; Oct; 30 (04) 241-55.
• 55 Middleton B, Shwe MA, Heckerman DE, Henrion M, Horvitz EJ, Lehmann HP. et al. Probabilistic diagnosis using a reformulation of the INTERNIST1/QMR knowledge base. II. Evaluation of diagnostic performance. Methods Inf Med 1991; Oct; 30 (04) 256-67.
• 56 Shwe M, Cooper G. An empirical analysis of likelihood-weighting simulation on a large, multiply connected medical belief network. Comput Biomed Res 1991; Oct; 24 (05) 453-75.
• 57 Shiffman S, Lane CD, Johnson KB, Fagan LM. The integration of a continuous-speech-recognition system with the QMR diagnostic program. Proc Annu Symp Comput Appl Med Care 1992; 767-71.
• 58 Berner ES, Webster GD, Shugerman AA, Jackson JR, Algina J, Baker AL. et al. Performance of Four Computer-Based Diagnostic Systems. N Engl J Med 1994; Jun 23; 330 (25) 1792-6.
• 59 van Ginneken AM, Liem EB, Moorman PW. Integrating QMR with a computer-based patient record. ProcAnnu Symp Comput Appl Med Care 1993; 98-102.
• 60 Gozum ME. Emulating cognitive diagnostic skills without clinical experience: a report of medical students using Quick Medical Reference and Iliad in the diagnosis of difficult clinical cases. Proc Annu Symp Comput Appl Med Care 1994; 991.
• 61 Welford CR. A comprehensive computerized patient record with automated linkage to QMR. Proc Annu Symp Comput Appl Med Care 1994; 814-8.
• 62 Wolfram DA. An appraisal of INTERNIST-I. Artif Intell Med 1995; Apr; 07 (02) 93-116.
• 63 Berner ES, Jackson JR, Algina J. Relationships among performance scores of four diagnostic decision support systems. J Am Med Inform Assoc 1996; May-Jun; 03 (03) 208-15.
• 64 Arene I, Ahmed W, Fox M, Barr CE, Fisher K. Evaluation ofquickmedicalreference(QMR)asateaching tool. MD Comput 1998; Sep-Oct; 15 (05) 323-6.
• 65 Lemaire JB, Schaefer JP, Martin LA, Faris P, Ainslie MD, Hull RD. Effectiveness of the Quick Medical Reference as a diagnostic tool. CMAJ 1999; Sep 21; 161 (06) 725-8.
• 66 Berner ES, Maisiak RS, Cobbs CG, Taunton OD. Effects of a decision support system on physicians’ diagnostic performance. J Am Med Inform Assoc 1999; Sep; 06 (05) 420-7.
• 67 Berner ES, Maisiak RS. Influence of case and physician characteristics on perceptions of decision support systems. J Am Med Inform Assoc 1999; Sep-Oct; 06 (05) 428-34.
• 68 Friedman CP, Elstein AS, Wolf FM, Murphy GC, Franz TM, Heckerling PS. et al. Enhancement of clinicians’ diagnostic reasoning by computer-based consultation: a multisite study of 2 systems. JAMA 1999; Nov 17; 282 (19) 1851-6.
• 69 Schaffner KF. Medical informatics and the concept of disease. Theor Med Bioeth 2000; Jan; 21 (01) 85-101.
• 70 Shortliffe EH. Computer-based medical consultation: MYCIN. New York: Elsevier; 1976
• 71 Newell A, Simon HA. Human Problem Solving. Prentice Hall: Englewood Cliffs, NJ; 1972
• 72 Barnett GO. The application of computer-based medical record systems in ambulatory practice. N Engl J Med 1984; 310 (25) 1643-50.
• 73 Berner ES, Graber ML. editors. Diagnostic Error in Medicine. Adv Health Sci Educ Theory Pract. 2009. Sep; 14 Suppl 1 1-112 Epub 2009 Aug 12. ISSN 1382-4996 (Print) 1573-1677 (Online).
• 74 Newman-Toker DE, Pronovost PJ. Diagnostic Errors— The Next Frontier for Patient Safety. JAMA 2009; 301 (10) 1060-2.
• 75 Bernstein LM, Siegel ER, Goldstein CM. The hepatitis knowledge base. A prototype information transfer system. Ann Intern Med 1980; Jul; 93 (1 Pt 2): 169-81.
• 76 Clancey WJ, Leitsinger R. NEOMYCIN: reconfiguring a rule-based expert system for application to teaching. Proc 7th Internat Joint Conf Artific Intel. Vancouver, BC, Canada. San Francisco: Morgan Kaufmann Inc; 1981: 829-36.
• 77 Sackett DL, Rosenberg WM, Gray JA, Haynes RB, Richardson WS. Evidence based medicine: what it is and what it isn’t. BMJ 1996; Jan 13; 312 (7023): 71-2.
• 78 Friedman C. A “Fundamental Theorem” of Biomedical Informatics. J Am Med Inform Assoc 2009; 16: 169-70.
• 79 Stead WW, Haynes RB, Fuller S, Friedman CP, Travis LE, Beck JR. et al. Designing medical informatics research and library—resource projects to increase what is learned. JAm Med InformAssoc 1994; 01 (01) 28-33.