An Approach to Knowledge Base Construction Based on Expert Opinions

Summary

Objectives: To describe, validate and demonstrate an approach for knowledge base construction based on expert opinions.

Methods: A knowledge base containing the frequency of occurrence of manifestations in epileptic seizures is constructed based on information provided by neurologists/epileptologists. The reliability of the responses is determined with the inter-rater intraclass correlation coefficient (ICC). If the ICC is not large enough the Spearman-Brown prophecy formula can be used to predict the number of additional experts. We propose a method to assess whether an additional expert provides information consistent with the already acquired data as well as a method to detect experts with deviating opinions. The power of the first method was determined.

Results: Data were collected for five seizure types. The ICCs determined from the responses for the various seizure types after inclusion of the additional experts was in all cases almost equal to 0.9, the target value. Yet one expert with diverging opinions concerning the frequency of occurrence of manifestations for different seizure types could be identified. Excluding this participant improved the reliability of the data. The power of the methods was good (≥ 0.75).

Conclusions: It is shown that human experts can provide reliable information about the frequency of occurrence of manifestations in epileptic seizures. In addition, the described approach correctly identified neurologists/epileptologists with both consistent and diverging opinions about the frequency of occurrence of manifestations in a number of seizure types.

Keywords

Knowledge base construction - inter-rater intraclass correlation coefficient - reliability - epilepsy - seizures

• References

• 1 Tversky A, Kahneman D. Judgment under uncertainty: heuristics and biases. Science 1974; 185 (Sept) 1124-31.
• 2 Giuse NB. et al. Evaluating consensus among physicians in medical knowledge base construction. Methods Inf Med 1993; 32 (02) 137-45.
• 3 van Ast JF. et al. Development of diagnostic reference frames for seizures. Part 1: interparticipant agreement in the selection of symptoms. Int J of Med Inf 2003; 70 2-3 285-92.
• 4 Hripcsak G. et al. A reliability study for evaluating information extraction from radiology reports. J Am Med Inform Assoc 1999; 6 (02) 143-50.
• 5 Friedman CP, Wyatt JC. Evaluation Methods in Medical Informatics. New York: Springer-Verlag; 1997
• 6 van Ast JF. et al. Diagnostic reference frames for seizures: knowledge elicitation and results. 2004. submitted
• 7 van Ast JF. et al. Development of diagnostic reference frames for seizures. Part 2: Are seizure descriptions discriminative?. Int J of Med Inf 2003; 70 2-3 293-300.
• 8 Shavelson RJ, Webb NM, Rowley GL. Generalizability theory. Am Psychol 1989; 44 (06) 922-32.
• 9 McGraw KO, Wong SP. From inferences about some intraclass correlation coefficients. Psychol Methods 1996; 1 (01) 30-46.
• 10 Cicchetti DV, Feinstein AR. High agreement but low kappa: II. Resolving the paradoxes. J Clin Epidemiol 1990; 43 (06) 551-8.