Mutual Information as an Index of Diagnostic Test Performance

Summary

Objectives: This paper demonstrates that diagnostic test performance can be quantified as the average amount of information the test result (R) provides about the disease state (D).

Methods: A fundamental concept of information theory, mutual information, is directly applicable to this problem. This statistic quantifies the amount of information that one random variable contains about another random variable. Prior to performing a diagnostic test, R and D are random variables. Hence, their mutual information, I(D;R), is the amount of information that R provides about D.

Results: I(D;R) is a function of both 1) the pretest probabilities of the disease state and 2) the set of conditional probabilities relating each possible test result to each possible disease state. The area under the receiver operating characteristic curve (AUC) is a popular measure of diagnostic test performance which, in contrast to I(D;R), is independent of the pretest probabilities; it is a function of only the set of conditional probabilities. The AUC is not a measure of diagnostic information.

Conclusions: Because I(D;R) is dependent upon pretest probabilities, knowledge of the setting in which a diagnostic test is employed is a necessary condition for quantifying the amount of information it provides. Advantages of I(D;R) over the AUC are that it can be calculated without invoking an arbitrary curve fitting routine, it is applicable to situations in which multiple diagnoses are under consideration, and it quantifies test performance in meaningful units (bits of information).

Keywords

Diagnostic tests - entropy - information theory - ROC curve

• References

• 1 Peterson WW, Birdsall TG, Fox WC. The theory of signal detectability. Institute of Radio Engineers Transactions 1954; 4: 171-212.
• 2 Van Meter D, Middleton D. Modern statistical approaches to reception in communication theory. Institute of Radio Engineers Transactions 1954; 4: 119-45.
• 3 Lusted LB. Signal detectability and medical decision-making. Science 1971; 171: 1217-9.
• 4 Simpson AJ, Fitter MJ. What is the best index of detectability?. Psychol Bull 1973; 80: 481-8.
• 5 Lee WC, Chuhsing KH. Alternative summary indices for the receiver operating characteristic curve. Epidemiology 1996; 7: 605-11.
• 6 Bamber D. The area above the ordinal dominance graph and the area below the receiver operating characteristic graph. J Math Psych 1975; 12: 387-415.
• 7 Hilden J. The area under the ROC curve and its competitors. Med Decis Making 1991; 11: 95-101.
• 8 Centor RM. Signal detectability: the use of ROC curves and their analyses. Med Decis Making 1991; 11: 102-6.
• 9 Sox Jr HC, Blatt MA, Higgins MC, Marton KI. Medical Decision Making. Stoneham, MA: Butterworth Publishers; 1988
• 10 Shannon CE, Weaver W. The Mathematical Theory of Communication. Urbana, IL: University of Illinois Press; 1949
• 11 Cover TM, Thomas JA. Elements of Information Theory. New York: John Wiley & Sons; 1991
• 12 Arimoto S. An algorithm for computing the capacity of arbitrary discrete memoryless channels. IEEE Trans Inform Theory 1972; 18: 14-20.
• 13 Blahut RE. Computation of channel capacity and rate-distortion functions. IEEE Trans Inform Theory 1972; 18: 460-73.
• 14 Buchsbaum DG, Buchanan RG, Centor RM. et al. Screening for alcohol abuse using CAGE scores and likelihood ratios. Ann Int Med 1991; 115: 774-7.
• 15 Benish WA. Relative entropy as a measure of diagnostic information. Med Decis Making 1999; 19: 202-6.
• 16 Mossman D. Three-way ROCs. Med Decis Making 1999; 19: 78-89.
• 17 Benish WA. The use of information graphs to evaluate and compare diagnostic tests. Methods Inf Med 2002; 41: 114-8.
• 18 Metz CE, Goodenough DJ, Rossmann K. Evaluation of receiver operating characteristic curve data in terms of information theory, with applications in radiography. Radiology 1973; 109: 297-303.