Comparison of Classifiers Applied to Confocal Scanning Laser Ophthalmoscopy Data

W. Adler; A. Peters; B. Lausen

doi:10.3414/ME0348

Methods of Information in Medicine, Table of Contents

Methods Inf Med 2008; 47(01): 38-46
DOI: 10.3414/ME0348

For Discussion

Schattauer GmbH

Comparison of Classifiers Applied to Confocal Scanning Laser Ophthalmoscopy Data

W. Adler

¹Department of Medical Informatics, Biometry and Epidemiology, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany

,

A. Peters

¹Department of Medical Informatics, Biometry and Epidemiology, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany

,

B. Lausen

¹Department of Medical Informatics, Biometry and Epidemiology, Friedrich-Alexander-University Erlangen-Nuremberg, Erlangen, Germany

› Author Affiliations

Abstract

Summary

Objectives: Comparison of classification methods using data of one clinical study. The tuning of hyperparameters is assessed as part of the methods by nested-loop cross-validation.

Methods: We assess the ability of 18 statistical and machine learning classifiers to detect glaucoma. The training data set is one case-control study consisting of confocal scanning laser ophthalmoscopy measurement values from 98 glaucoma patients and 98 healthy controls. We compare bootstrap estimates of the classification error by the Wilcoxon signed rank test and box-plots of a bootstrap distribution of the estimate.

Results: The comparison of out-of-bag bootstrap estimators of classification errors is assessed by Spearman’s rank correlation, Wilcoxon signed rank tests and box-plots of a bootstrap distribution of the estimate. The classification methods random forests 15.4%, support vector machines 15.9%, bundling 16.3% to 17.8%, and penalized discriminant analysis 16.8% show the best results.

Conclusions: Using nested-loop cross-validation we account for the tuning of hyperparameters and demonstrate the assessment of different classifiers. We recommend a block design of the bootstrap simulation to allow a statistical assessment of the bootstrap estimates of the misclassification error. The results depend on the data of the clinical study and the given size of the bootstrap sample.

Keywords

Classification - confocal scanning laser ophthalmoscopy - Glaucoma - R - machine learning by small data sets - parameter tuning

Full Text

References

References
1 Kononenko I. Machine learning for medical diagnosis: history, state of the art and perspective. Artificial Intelligence in Medicine 2001; 23: 89-109.
2 Mardin CY, Peters A, Horn FK, Jünemann AG, Lausen B. Improving glaucoma diagnosis by the combination of perimetry and HRT measurements. Journal of Glaucoma 2006; 15 (04) 299-305.
3 Quigley HA. Number of people with glaucoma worldwide. British Journal of Ophthalmology 1996; 80: 389-393.
4 Heidelberg-Engineering. Heidelberg Retina Tomograph: Bedienungsanleitung Software Version 2.01. Heidelberg Engineering GmbH; Heidelberg: 1997
5 Mardin CY, Hothorn T, Peters A, Jünemann AG, Nguyen NX, Lausen B. New glaucoma classification method based on standard HRT parameters by bagging classification trees. Journal of Glaucoma 2003; 12: 340-346.
6 Chan K, Lee T, Sample PA, Goldbaum MH, Weinreb RN. Comparison of machine learning and traditional classifiers in glaucoma diagnosis. IEEE Transactions on Biomedical Engineering 2002; 49 (09) 963-974.
7 Adler W, Hothorn T, Lausen B. Simulation based analysis of classification of medical images. Methods Inf Med 2004; 43 (02) 150-155.
8 Bowd C, Chan K, Zangwill LM, Goldbaum MH, Lee T, Sejnowski TJ. et al. Comparing neural networks and linear discriminant functions for glaucoma detection using confocal scanning laser ophthalmoscopy of the optic disc. Investigative Ophthalmology and Visual Science 2002; 43 (11) 3444-3454.
9 Peters A, Lausen B, Michelson G, Gefeller O. Diagnosis of glaucoma by indirect classifiers. Methods Inf Med 2003; 42 (01) 99-103.
10 Schwarzer G, Nagata T, D.M Schmelzeisen D, Schumacher M. Comparison of fuzzy inference, logistic regression, and classification trees (CART). Methods Inf Med 2003; 42 (05) 572-527.
11 Duin RPW. A note on comparing classifiers. Pattern Recognition Letters 1996; 17: 529-536.
12 Van-Gestel T, Suykens JA, Baesens B, Viaene S, Vanthienen J, Dedene G. et al. Benchmarking least squares support vector machine classifiers. Machine Learning 2004; 54 (01) 5-32.
13 Markowetz F, Spang R. Molecular diagnosis – classification, model selection and performance evaluation. Methods Inf Med 2005; 44: 438-443.
14 R Development Core Team. R: A Language and Environment for Statistical Computing: R Foundation for Statistical Computing; 2005
15 Goldbaum MH, Sample PA, Chan K, Williams J, Lee T-W, Blumenthal E. et al. Comparing machine learning classifiers for diagnosing glaucoma from standard automated perimetry. Investigative Ophthalmology and Visual Science 2002; 43 (01) 162-169.
16 Sample PA, Goldbaum MH, Chan K, Boden C, Lee T, Vasile C. et al. Using machine learning classifiers to identify glaucomatous change earlier in standard visual fields. Investigative Ophthalmology and Visual Science 2002; 43 (08) 2660-2665.
17 Jonas J. Biomorphometrie des Nervus Opticus. Stuttgart: Enke; 1989
18 Ripley BD. Pattern Recognition and Neural Networks. Cambridge, UK: Cambridge University Press; 1996
19 Venables WN, Ripley BD. Modern Applied Statistics with S. 4th ed. New York: Springer; 2002
20 Läuter J. Stabile multivariate Verfahren: Diskriminanzanalyse – Regressionsanalyse – Faktoranalyse. Berlin: Akademie Verlag; 1992
21 Läuter J, Glimm E, Kropf S. Multivariate tests based on left-spherically distributed linear scores. TheAnnals of Statistics 1998; 26 (05) 1972-1988.
22 Peters A, Hothorn T, Lausen B. ipred: Improved predictors. R News 2002; 2 (02) 33-36.
23 Hastie T, Tibshirani R, Buja A. Flexible discriminant analysis by optimal scoring. Journal of the American Statistical Association 1994; 1: 1255-1270.
24 Friedman JH. Multivariate adaptive regression splines (with discussion). The Annals of Statistics 1991; 19: 1-141.
25 Hastie T, Buja A, Tibshirani R. Penalized discriminant analysis. The Annals of Statistics 1995; 23: 73-102.
26 Hastie T, Tibshirani R. Discriminant analysis by gaussian mixtures. Journal of the American Statistical Association 1996; 1: 1255-1270.
27 Webb A. Statistical Pattern Recognition. 2nd ed. Chichester, England: Wiley; 2002
28 Heckerling PS, Gerber BS, Tape TG, Wigton RS. Entering the black box of neural networks. Methods Inf Med 2003; 42 (03) 287-296.
29 Seuchter SA, Eisenacher M, Riesbeck M, Gaebel W, Köpcke W. other members of the A.N.I. Study Group Methods for predictor analysis of repeated measurements: application to psychiatric data. Methods Inf Med 2004; 43 (02) 184-191.
30 Rumelhart DE, Hinton GE, Williams RJ. Learning internal representations by error propagation. In: Rumelhart DE, McClelland JL. editors. Parallel distributed processing (Vol 1). Cambridge, MA: MIT Press; 1986. pp 318-362.
31 Vapnik V. Statistical Learning Theory. New York: Wiley; 1998
32 Vapnik V. The Nature of Statistical Learning Theory. 2nd ed. New York: Springer; 2000
33 Meyer D. Support vector machines. R News 2001; 1 (03) 23-26.
34 Kohonen T, Hynninen J, Kangas J, Laaksonen J, Torkkola K. LVQ_PAK: The learning vector quantization program package: Laboratory of Computer and Information Science. Helsinki University of Technology; 1996
35 Breiman L. Bagging predictors. Machine Learning 1996; 24 (02) 123-140.
36 Breiman L. Random forests. Machine Learning 2001; 45 (01) 5-32.
37 Liaw A, Wiener M. Classification and regression by randomForest. R News 2002; 2 (03) 18-22.
38 Hothorn T, Lausen B. Bundling classifiers by bagging trees. Computational Statistics & Data Analysis 2005; 49: 1068-1078.
39 Dimitriadou E, Hornik K, Leisch F, Meyer D, Weingessel A. e1071: Misc functions of the department of statistics (e1071). TU Wien. 2004
40 Efron B, Tibshirani R. Improvements on crossvalidation: the .632+ bootstrap method. Journal of the American Statistical Association 1997; 92: 548-560.
41 Breiman L. Out-of-bag estimation. Technical Report. Berkeley (CA 94708): Statistics Department, University of California; 1996
42 Envgeniou T, Pontil M, Eliseeff A. Leave one out error, stability, and generalization of voting combinations of classifiers. Machine Learning 2004; 55 (01) 71-97.
43 Mikelberg F, Parfitt C, Swindale N, Graham S, Drance S, Gosine R. Ability of the Heidelberg Retina Tomograph to detect early glaucomatous visual field loss. Journal of Glaucoma 1995; 4: 242-247.
44 Mardin CY, Horn FK, Jonas J, Budde WM. Preperimetric glaucoma diagnosis by confocal scanning laser tomography of the optic disc. British Journal of Ophthalmology 1999; 83 (03) 299-304.
45 Bathija R, Zangwill LM, Berry CC, Sample PA, Weinreb RN. Detection of early glaucomatous structural damage with confocal scanning laser tomography. Journal of Glaucoma 1998; 127: 7 (2): 121.
46 Uchida H, Brigatti L, Capriolo J. Detection of structural damage from glaucoma with confocal laser image analysis. Investigative Ophthalmology and Visual Science 1996; 37 (12) 2393-2401.