Download PDF
Methods Inf Med 2000; 39(04/05): 311-318
DOI: 10.1055/s-0038-1634398
Original Article
Schattauer GmbH

Validation of ICTERUS, a Knowledge-based Expert System for Jaundice Diagnosis

G. Molino
1   Laboratory of Clinical Informatics, San Giovanni Battista Hospital of Turin, Turin, Italy
,
M. Marzuoli
1   Laboratory of Clinical Informatics, San Giovanni Battista Hospital of Turin, Turin, Italy
,
F. Molino
1   Laboratory of Clinical Informatics, San Giovanni Battista Hospital of Turin, Turin, Italy
,
S. Battista
1   Laboratory of Clinical Informatics, San Giovanni Battista Hospital of Turin, Turin, Italy
,
F. Bar
1   Laboratory of Clinical Informatics, San Giovanni Battista Hospital of Turin, Turin, Italy
,
M. Torchio
1   Laboratory of Clinical Informatics, San Giovanni Battista Hospital of Turin, Turin, Italy
,
S. M. Lavelle
2   Department of Experimental Medicine, Galway University, Ireland
,
G. Corless
2   Department of Experimental Medicine, Galway University, Ireland
,
N. Cappello
3   Department of Genetics, Biology and Biochemistry, University of Turin, Turin, Italy
› Author Affiliations
Further Information

Publication History

Publication Date:
08 February 2018 (online)

    Permissions and Reprints

Abstract:

The study aimed to describe an example of the assessment and validation of knowledge-based clinical expert systems. The paper focuses on ICTERUS, an expert system for jaundice diagnosis. It describes system design, the methodology applied for upgrading and validating the program, and the most important outcomes of the validation procedure. The clinical validation of the system on a very large European database (Euricterus Project) shows that diagnostic conclusions are reliable in about 70% of eligible cases. This figure appears acceptable for a system which provides decision support only on the basis of clinical data, assuming that the final decision is achieved under user responsibility. Expected biases, limitations and inconsistencies in the practical application of the system are discussed.

Keywords

Jaundice - Liver Diseases - Knowledge-based Expert System - System Validation - Diagnostic Accuracy
 

  • REFERENCES

  • 1 Chandrasekeran B. On evaluating AI systems for medical diagnosis. AI Magazine 1983; 4: 34-7.
    PubMedGoogle Scholar
  • 2 Foster D, Buchanan B. Broadening our approach to evaluating medical information systems. Proc 15th SCAMC, Washington DC: 1991: 8-12.
    Google Scholar
  • 3 Heathfield HA, Wyatt J. Philosophies for the design and development of clinical decision-support systems. Method Inform Med 1993; 32: 1-8.
    PubMedGoogle Scholar
  • 4 Hilden J, Habbema DF. Evaluation of clinical decision-aids: more to think about. Med Inform 1990; 15: 275-84.
    PubMedGoogle Scholar
  • 5 Miller PL, Sitting DF. The evaluation of clinical decision support systems: what is necessary versus what is interesting. Med Inform 1990; 15: 185-90.
    PubMedGoogle Scholar
  • 6 Nguyen T, Perkins W, Laffey T. et al. Knowledge base verification. AI Magazine Summer 1987; 8: 69-75.
    PubMedGoogle Scholar
  • 7 O’Neil M, Glowinski A. Evaluating and validating very large knowledge-based systems. Med Inform 1990; 15: 237-52.
    PubMedGoogle Scholar
  • 8 Perrault L, Wiederhold G. System design and evaluation. In: Shortliffe et al (Eds). Med Inform. 1990: 151-77.
    Google Scholar
  • 9 Swets JA. Measuring the accuracy of diagnostic systems. Science 1988; 240: 1285-93.
    CrossrefPubMedGoogle Scholar
  • 10 Friedman C, Wyatt J. Evaluation methods in medical informatics. New York: Springer Verlag; 1997
    Google Scholar
  • 11 Van der Loo RP. Overview of published assessment and evaluation studies. In: Assessment and Evaluation of Information Technologies in Medicine. van Gennip EMSJ, and Talmon JL. Eds Amsterdam: IOS Press; 1995: 261-81.
    Google Scholar
  • 12 Talmon JL, and Van der Loo RP. Literature on assessment of information technology and medical KBS evaluation: studies and methodologies. In: Assessment and Evaluation of Information Technologies in Medicine. van Gennip EMSJ, and Talmon JL. Eds. Amsterdam: IOS Press; 1995: 283-327.
    Google Scholar
  • 13 Molino G, Molino F. Assessment and evaluation of knowledge-based expert systems for medical diagnosis. In: Assessment and Evaluation of Information. van Gennip EMSJ, and Talmon JL. Eds. Amsterdam: IOS Press; 1994: 151-66.
    Google Scholar
  • 14 Lindberg G. Studies on diagnostic decision making in jaundice. Stockholm: Karolinska Institutes (Thesis); 1982: 1-60.
    Google Scholar
  • 15 Matzen P, Malchow-Møller A, Hilden J. et al. Differential diagnosis of jaundice: a pocket diagnostic chart. Liver 1984; 4: 360-71.
    PubMedGoogle Scholar
  • 16 Boom R, Gonzalez C, Fridman L. et al. Looking for “indicants” in the differential diagnosis of jaundice. Med Decis Making 1986; 6: 36-41.
    CrossrefPubMedGoogle Scholar
  • 17 Malchow-Møller A, Thomsen C, Matzen P. et al. Computer diagnosis in jaundice. Bayes’ rules founded on 1002 consecutive cases. J Hepatol 1986; 3: 156-63.
    PubMedGoogle Scholar
  • 18 O’Beirn SF, Given HF, Lavelle SM. Experiences with compiling a diagnostic data base on jaundice presenting to a general hospital. Irish J Med Sci 1987; 156: 93-7.
    CrossrefPubMedGoogle Scholar
  • 19 Malchow-Møller A, Thomsen C, Hilden J. et al. A decision tree for early differentiation between obstructive jaundice and non-obstructive jaundice. Scand J Gastroenterol 1988; 23: 391-401.
    CrossrefPubMedGoogle Scholar
  • 20 Segaar RW, Wilson JHP, Habbema JDF. et al. Transferring a diagnostic decision aid for jaundice. Neth J Med 1988; 33: 5-15.
    PubMedGoogle Scholar
  • 21 Miller RA. Medical diagnostic decision support systems - past, present, and future: a threaded bibliography and brief commentary. J Am Med Inform Assoc 1994; 1: 8-27.
    CrossrefPubMedGoogle Scholar
  • 22 Pauker SG, Gorry GA, Kassirer JP, Schwartz WB. Towards the simulation of clinical cognition. Taking a present illness by computer. Am J Med, 1976; 60: 981-96.
    CrossrefPubMedGoogle Scholar
  • 23 Masarie FE, Miller RA, Myers JD. INTER-NIST-1 properties: representing common sense and good medical practice in a computerized medical knowledge base. Comput Biomed Res, 1985; 18: 458-79.
    CrossrefPubMedGoogle Scholar
  • 24 Barnett GO, Cimino JJ, Hupp JA, Hoffer EP. DXplain. An evolving diagnostic decision-support system. J Am Med Inform Assoc 1987; 258: 67-74.
    PubMedGoogle Scholar
  • 25 Miller RA, Masarie Jr FE. Use of the Quick Medical Reference (QMR) program as a tool for medical education. Method Inform Med 1989; 28: 340-5.
    PubMedGoogle Scholar
  • 26 Warner Jr HR. Iliad: moving medical decision-making into new frontiers. Method Inform Med 1989; 28: 370-2.
    PubMedGoogle Scholar
  • 27 Molino G, Ballarè M, Aurucci PE. et al. Application of artificial intelligence techniques to a well defined clinical problem: jaundice diagnosis. Int J Biomed Comp 1990; 12: 189-202.
    PubMedGoogle Scholar
  • 28 Euricterus Project Management Group – Objective medical decision-making: clinical database for diagnosis of jaundice (EURICTERUS). In: Advances in Biomedical Engineering. Beneken JEW, and Thévenin V. Eds. Amsterdam: IOS Press; 1993: 35-64.
    Google Scholar
  • 29 Molino G, Molino F, Furia D. et al. Computer-aided diagnosis in jaundice: comparison of the knowledge-based and probabilistic approaches. Method Inform Med 1996; 35: 41-51.
    PubMedGoogle Scholar
  • 30 Quaglini S, Stefanelli M, Barosi G. et al. A performance evaluation of the expert system ANEMIA. Comp Biomed Res 1988; 21: 307-23.
    CrossrefPubMedGoogle Scholar
  • 31 Hilden J, Habbema JDF, Bjerregaard B. The measurement of performance in probabilistic diagnosis. II. Trustworthiness of the exact values of the diagnostic probabilities. Method Inform Med 1978; 17: 227-37.
    PubMedGoogle Scholar
  • 32 Hilden J, Habbema JDF, Bjerregaard B. The measurement of performance in probabilistic diagnosis. III. Methods based on continuous functions of the diagnostic probabilities. Method Inform Med 1978; 17: 238-46.
    PubMedGoogle Scholar
  • 33 Hanley JA, McNeil BJ. The meaning and use of the area under a Receiver Operating Characteristic (ROC) curve. Radiology 1982; 143: 26-36.
    PubMedGoogle Scholar