Validation, Clinical Trial, and Evaluation of a Radiology Expert System

Abstract:

The PHOENIX Radiology Consultant is a rule-based expert system which assists physicians in planning radiological work-up strategies. This article describes the methods used to create and validate the system’s knowledge base. The feasibility and acceptability of PHOENIX were tested for two years in a clinical trial. During this period, the system was used 1,421 times, an average of 13.7 times per week, primarily by medical students and nonradiologist physicians. Much of the system’s use occurred at night and on weekends, when the radiology department was not fully staffed. Several physicians were enlisted to further evaluate the utility of the system. The results of their evaluation indicate that an expert system that helps physicians select diagnostic-imaging studies can serve as a useful and informative component of a radiology information system, and is particularly useful for medical students and physicians in training.

Key-Words

Expert Systems - Rule-based Systems - Artificial Intelligence - Validation - Evaluation

• REFERENCES

• 1 Wyatt J, Spiegelhalter D. Evaluating medical expert systems: what to test and how?. Med Inform 1990; 15: 205-17.
• 2 Kahn Jr CE, Messersmith RN, Jokich MD. PHOENIX: An expert system for selecting diagnostic imaging procedures. Invest Radiol 1987; 22: 978-80.
• 3 Miller PL, Sittig DF. The evaluation of clinical decision support systems: what is necessary versus what is interesting. Med Inform 1990; 15: 185-90.
• 4 Scott Jr WW, Scott PP. Consultation in Diagnostic Imaging. Philadelphia: B. C. Decker; 1985
• 5 MacNeil BJ, Abrams HL. eds. Brigham and Women’s Hospital Handbook of Diagnostic Imaging. Boston: Little Brown; 1986
• 6 Straub WH. ed. Manual of Diagnostic Imaging: A Clinician’s Guide to Clinical Problem Solving. Boston: Little Brown; 1984
• 7 Rubin P. ed. Clinical Oncology for Medical Students and Physicians: A Multidisciplinary Approach. 6th ed.. New York: American Cancer Society; 1983
• 8 Greenes RA. Computer-aided diagnostic strategy selection. Radiol Clin N Am 1986; 24: 105-20.
• 9 Miller PL, Shaw C, Rose JR, Swett HA. Critiquing the process of radiologic differential diagnosis. Comp Meth Prog Biomed 1986; 22: 21-5.
• 10 Swett HA, Miller PL. ICON: a computer-based approach to differential diagnosis in radiology. Radiology 1987; 163: 555-8.
• 11 Mars NJI, Miller PL. Knowledge acquisition and verification tools for medical expert systems. Med Decis Making 1987; 07: 6-11.
• 12 Shwe MA, Tu SW, Fagan LM. Validating the knowledge base of a therapy planning system. Meth Inform Med 1989; 28: 36-50.
• 13 Nguyen TA, Perkins WA, Laffey TJ, Pecora D. Knowledge base verification. AI Magazine 1987; 02: 69-75.
• 14 Suwa M, Scott AC, Shortliffe EH. An approach to verifying completeness and consistency in a rule-based expert system. AI Magazine 1982; 02: 16-21.
• 15 Miller PL. The evaluation of artificial intelligence systems in medicine. Comp Meth Prog Biomed 1986; 22: 5-11.
• 16 Kahn Jr CE, Kovatsis PG, Messersmith RN, Lehr JL. Automated entry of radiology requisition information using artificial-intelligence techniques. Am J Roentgenol 1989; 153: 1085-8.
• 17 Sanders GD, Lyons EA. The potential use of expert systems to enable physicians to order more cost-effective diagnostic imaging examinations. J Digital Imaging 1991; 04: 112-22.
• 18 Rossi-Mori A, Pisanelli DM, Ricci FL. Evaluation stages and design steps for knowledge-based system in medicine. Med Inform 1990; 15: 191-204.