Severity Identification of Graves Orbitopathy via Random Forest Algorithm

Abstract

This study aims to establish a random forest model for detecting the severity of Graves Orbitopathy (GO) and identify significant classification factors. This is a hospital-based study of 199 patients with GO that were collected between December 2019 and February 2022. Clinical information was collected from medical records. The severity of GO can be categorized as mild, moderate-to-severe, and sight-threatening GO based on guidelines of the European Group on Graves’ orbitopathy. A random forest model was constructed according to the risk factors of GO and the main ocular symptoms of patients to differentiate mild GO from severe GO and finally was compared with logistic regression analysis, Support Vector Machine (SVM), and Naive Bayes. A random forest model with 15 variables was constructed. Blurred vision, disease course, thyroid-stimulating hormone receptor antibodies, and age ranked high both in mini-decreased gini and mini decrease accuracy. The accuracy, positive predictive value, negative predictive value, and the F1 Score of the random forest model are 0.83, 0.82, 0.86, and 0.82, respectively. Compared to the three other models, our random forest model showed a more reliable performance based on AUC (0.85 vs. 0.83 vs. 0.80 vs. 0.76) and accuracy (0.83 vs. 0.78 vs. 0.77 vs. 0.70). In conclusion, this study shows the potential for applying a random forest model as a complementary tool to differentiate GO severity.

Keywords

Graves orbitopathy - severity - random forest algorithm

Introduction

Graves Disease (GD) is an autoimmune thyroid condition and Graves Orbitopathy (GO) is a potential severe consequence of GD. GO is the most common orbital disease in adults and can lead to severe damage to the eyes. The severity of GO can be categorized as mild, moderate-to-severe, or sight-threatening GO according to the European Group on Graves’ orbitopathy (EUGOGO) criteria [1]. Treatment varies according to severity. Thus, it is important to detect the severity of GO [2]. However, previous classification methods required complex eye examinations, which may delay choosing optimal treatment, especially when coming to the primary care physician (PCP) for the first time.

The prevailing view is that gender, age, family history and course of Graves’ disease, smoking history, radioactive ¹³¹I, gut microbiome, thyroid-stimulating hormone receptor antibodies (TRAb), and thyroid hormones are risk factors for the disease, which are strongly associated with the progression of GO [3] [4]. Additionally, most patients seek medical attention because of eye symptoms, such as grittiness, pain, photophobia, tearing, or blurred vision. However, no tools that aid in diagnosing the severity combining different risk factors and clinical features for GO have been previously reported. The random forest algorithm is a machine-learning method. Compared to traditional methods, it could combine all variables and finally figure out their importance weights respectively, which reflects their value in clinical practice [5] [6]. Nowadays, random forest algorithms have been applied in many interdisciplinary. Biscarini et al. used Random Forest to predict GO severity from gut microbiome data [4]. Li et al. used the random forest model to predict poor functional outcomes of patients with anti-LGI1 encephalitis [7]. Moradian et al. applied the random forest algorithms to detect suicidal ideation of digital mental health application users [8].

In our research, we intended to detect the severity of GO by building a random forest model, combining these risk factors and the ocular symptoms, and to identify important indicators related to GO severity. We aimed to find patients who may need special treatments but cannot meet professional ophthalmologists in time at the first visit and subsequently arrange ophthalmology consultation for them. Thus, we hope we can reduce the desire for expert consultation and alleviate the overburden of the health system in some countries.

Patients and Methods

Patients

From March 2020 to July 2022, 199 patients who visited Beijing Tongren eye center were consecutively included in this study. All enrolled patients had complete clinical data and met the diagnostic criteria of the 2016 European Group on Graves’ Orbitopathy (EUGOGO) guidelines. We excluded those patients with other autoimmune diseases, ocular inflammatory diseases, minors, and pregnancies. This study adhered to the Declaration of Helsinki and was approved by the ethics committee of Beijing Tongren Hospital (ID: No.TRECKY2018-056).

Data collection and clinical evaluation

We collected their clinical data by EUGOGO first-diagnosis form. The detailed information was obtained as follows: (1) basic information: gender, age, smoking history, family history, course of Graves’ disease, ¹³¹I treatment history; (2) laboratory test: free triiodothyronine (FT3), free thyroxine (FT4), TRAb; and (3) symptoms: pain on attempted upward or downward gaze, spontaneous retrobulbar pain, tearing, photophobia, grittiness, and blurred vision.

Senior ophthalmologists performed ocular examinations for GO patients. Based on the EUGOGO guidelines, those who had two or more of the following were diagnosed as moderate-to-severe GO: lid retraction+≥+2 mm, moderate or severe soft-tissue involvement, or exophthalmos+≥+3 mm above normal, inconstant or constant diplopia. Patients with dysthyroid optic neuropathy (DON) or corneal breakdown were regarded as vision-threatening GO [1]. Both the moderate-to-severe GO and vision-threatening GO (very severe GO) were regarded as the severe group, while the rest of the patients were in the mild group. We enrolled the eye with a more severe status if two eyes were differently divided for any individual.

Statistical analysis

We conducted statistical analysis through the IBM SPSS Statistics software (Version 20.0 for Mac; IBM Corporation, NY, USA). Categorical variables include gender, smoking, radioactive ¹³¹I treatment history, family history, pain, movement pain, tearing, photophobia, grittiness, and blurred vision. Continuous variables contain age, FT3, FT4, TRAb, and course. Mann–Whitney U-test and Fisher’s exact test were used for descriptive statistical analysis of continuous and categorical variables, respectively. A p-value of<0.05 was considered a statistically significant difference between the variables in the mild group and the severe group.

Model construction

Scikit-learn package (Python3.9 for Mac) was used to build our random forest model, which was constructed according to the risk factors of GO and the main symptoms of patients. The variables included in the model are age, gender, smoking history, family history, course of Graves’ disease, radioactive ¹³¹I treatment history, FT3, FT4, TRAb, spontaneous retrobulbar pain, pain with upward or downward gazing, tearing, photophobia, grittiness, blurred vision, and severity. The severity of GO is regarded as the output variable, while the other 15 variables as input variables. Any subject with outlier and missing data would be excluded. Specially, the binary variables were recorded as “0” or “1” according to their different categories before being enrolled in the model. We divided the patients into a training set and a validation set with a ratio of 7:3. This means that 70% of the sample will be randomly selected to train each decision tree and replaced by bootstrapping. The other 30% will be used to evaluate the performance of the model. As for the maximum features of every decision tree, we chose mtry=4, which was highly recommended as the square root of the sum of all variables. The out-of-bag (OOB) error was chosen to evaluate the generalization error [9]. We kept training the model by varying the number of trees until the OOB error stabilized; in this way, we could take the number of trees at that point as the optimal value. Synthetic Minority Oversampling Technique (SMOTE) was used to enrich the minority in each training set, which could improve the imbalanced dataset. We used the default settings in the Python 3.9 sklearn package for all parameters not mentioned. Based on this model, we ranked the relative importance of the input variables via the mean decrease accuracy (MDA) and mean decrease gini (MDG). We evaluated the validation set based on the sensitivity, specificity, accuracy, F1 score, positive predictive value, and negative predictive value.

Finally, we integrated the same 15 variables into the logistics regression model, Naïve Bayes, and Support vector machine (SVM) algorithms. Regarding the logistic regression model, we did not make any parameter adjustments from the default setting. Besides, we utilized Bernoulli Naive Bayes for model construction. Prior to this, we transformed all data into binary variables (0 or 1). In the algorithm, we also applied Laplace smoothing to avoid scenarios where the probability of a certain variable is zero, setting the smoothing parameter alpha as 1. In addition, we performed normalization for the continuous variables before constructing SVM model because of its requirement for similar data distribution. GridSearchCV function was used to find and consider Gaussian Kernel and penalty coefficient of 0.5 as the optimal parameters, while other parameters were set up using the default value. Their accuracy and area under the receiver operating characteristic curve were calculated respectively.

Results

Baseline characteristics

A total of 199 patients were enrolled in this model, 88 (44.2%) of whom were moderate-to-severe or vision-threatening GO, and the remaining 111(55.8%) patients were mild GO. In the severe group, 37 patients (42.0% of the severe group) were vision-threatening GO. Forty-eight (43.2%) patients were in the active phase in the mild group, while 60 (68.2%) patients were in the active phase in the severe group. Severe and mild groups had significant differences in ocular symptoms, including spontaneous retrobulbar pain (p<0.001), pain on attempted upward or downward gaze (p<0.001), tearing (p=0.001), photophobia (p=0.015), grittiness(p<0.001), and blurred vision(p<0.001). Female (p=0.015), radioactive ¹³¹I treatment history (p=0.032), long course of Graves’ disease (p=0.010), and TRAb (p<0.001) also differed significantly between the two groups ([Table 1]).

Table 1 Demographical and clinical features that are included in the model.

Variables	Mild group (n=111)	Severe group (n=88)	p-Value
Female, n (%)*	48 (43.2)	54 (61.4)	0.015
Smoke, n (%)*	60 (54.1)	60 (68.2)	0.058
131I, n (%)*	16 (14.4)	24 (29.5)	0.032
Family history, n (%)*	8 (7.2)	13 (14.8)	0.105
Pain, n (%)*‡	19 (17.1)	52 (59.1)	<0.001
Movement pain, n (%)*§	11 (9.9)	30 (34.1)	<0.001
Tearing, n (%)*	64 (57.7)	71 (80.7)	0.001
Photophobia, n (%)*	66 (59.5)	67 (76.1)	0.015
Grittiness, n (%)*	48 (43.2)	62 (70.5)	<0.001
Blurred vision, n (%)*	31 (27.9)	69 (78.4)	<0.001
Age, median (IQR)†	46 (37–57)	52 (38–58)	0.223
FT3, median (IQR)†	5.23 (4.63–6.03)	5.08 (4.57–5.73)	0.281
FT4, median (IQR)†	14.91 (11.76–17.69)	14.88 (11.85–16.94)	0.764
TRAb, median (IQR)†	2.97 (1.41–6.54)	6.52 (1.99–16.10)	<0.001
Course, median (IQR), months†	10 (6–20)	14 (7–51)	0.010

^*Categorical variables were analyzed using Fisher’s exact test. ^† Continuous variables were analyzed with Mann–Whitney U-test. ^‡ Pain: spontaneous retrobulbar pain. ^§ Movement pain: pain on attempted upward or downward gaze. ^¶ TRAb: Thyroid hormone receptor antibody.

Performance of the random forest model

Combined with the risk factors of GO and the main symptoms of patients, a random forest model was successfully constructed to detect the severity of GO. Participants consisted of 139 patients in the training set and 60 patients in the validation set. When mtry=4 and ntree=450, our model showed a low OOB error rate of 0.15 ([Fig. 1]). It is apparent that the blurred vision (MDG=0.10, MDA=0.11), TRAb (MDG=0.14, MDA=0.09), course (MDG=0.12, MDA=0.07), and age (MDG=0.10, MDA=0.06) ranked high, which show great importance to the model ([Fig. 2]). The accuracy, positive predictive value, negative predictive value, and the F1 Score random of the forest model are 0.83, 0.82, 0.86, and 0.82 respectively ([Table 2]). Compared to the three other models, the random forest algorithm showed a more convincing performance (AUC, 0.85 vs. 0.83 vs. 0.80 vs. 0.76; accuracy, 0.83 vs. 0.78 vs. 0.77 vs. 0.70) ([Fig. 3]).

Fig. 1 The Out-of-bag (OOB) error. The OOB error was chosen to evaluate the generalization error, which was used to select the optional number of decision trees generated during machine learning. The middle line depicts the OOB error of all data. The top line depicts the OOB error of the severe group, and the bottom line shows the OOB error of the mild group. To be noted, curves begins at 21 of the x-axis (ntree). Finally, we chose 450 as optional number of decision trees because these curves became stable at that point.

Fig. 2 Variables are listed from most important to least important based on the mean decrease in Gini and mean decrease in the accuracy coefficient. (a) Importance of variables based on mean decrease accuracy. (b) Importance of variables based on mean decrease gini. Pain: Spontaneous retrobulbar pain; Movement pain: Pain on attempted upward or downward gaze; TRAb: Thyroid hormone receptor antibody.

Fig. 3 Comparison between different methods based on ROC curves and accuracy. (a) ROC curves of random forest model, logistic regression, Naive Bayes and SVM with different color. Areas under curves (AUC) were labeled on the right side. (b) The bar chart showed the accuracy of random forest model, logistic regression, Naive Bayes, and SVM. SVM: Support vector machine.

Table 2 Prediction of Severity based on a random forest model.

Prediction	Severe group	Mild group	Total
Severe group	31	3	34
Mild group	7	19	26
Total	38	22
Accuracy	0.83
Sensitivity	0.93
Specificity	0.73
PPV*	0.82
NPV†	0.86
F1 Score‡	0.82

^*PPV: Positive predictive value. ^† NPV: Negative predictive value. ^‡ The calculation of F1 Score: 2^*Specificity. ^*Sensitivity/(Specificity+++Sensitivity). It is valuable to evaluate the performance of the random forest model.

Discussion

In this study, we found that the random forest algorithm involving 15 variables could aid in diagnosing the severity of GO. Its performance was more reliable than other models, including logistic regression, SVM, and Naive Bayes metrics. Besides, our model indicated the importance of blurred vision, TRAb, course of disease, and age in the occurrence and development of GO.

In the total of 199 patients, the female-to-male ratio was calculated as 1.1:1, which was similar to previous Asian cohort studies [10] [11]. EUGOGO in its first multicentric study found that mild, moderate-to-severe, and sight-threatening disease accounted for 40%, 33%, and 28% of all the GO patients, respectively [12]. The proportion of mild GO was 55.8% (111 of 199 cases) in our study, with a median age of 46 years old in the mild group and 52 years old in the severe group, which was consistent with a study reporting that the average age of mild GO and moderate-to-severe GO was 43.9 and 53.7 [13]. The overall severity of GO increased with age, which was identical to the findings reported in Denmark, the United States, and Italy [14] [15] [16]. The median of TRAb in the severe group was 6.52, higher than in the mild group, which had been confirmed in previous studies [17]. Moreover, all six clinical features showed remarkable differences based on the incidence in the two groups. As for the thyroid hormones, there was no significant difference between the two groups, which may be due to the fact that our patients developed hyperthyroidism before GO and some patients had received antithyroid therapy in the endocrinology department.

The reason why we chose the random forest algorithm was that it could reduce the probability of overfitting bias by inputting a large number of disease-related clinical variables [5]. By building a random forest model, we could also obtain the relative importance rankings of the predictive characteristic variables. As an autoimmune disease, GO usually has a variety of manifestations. Random forest algorithm can include any possible clinical variables, which is particularly suitable for our study. In addition, since OOB error could provide us with an unbiased estimation of the model’s error, our model does not require cross-validation, which has been validated in previous models [18].

The ranking in our model emphasized that blurred vision, TRAb, course of disease, and age were relatively important factors for severity. These findings were consistent with findings in clinical practice [19]. TRAb ranked high both based on the MDA and MDG as same as the consensus of EUGOGO generally holds that TRAb is the core change of GO [20]. In the severe group, many patients including individuals with DON suffered from blurred vision, which could explain why it was the most important indicator in MDA and it is necessary to consider blurred vision as an early warning factor of GO progression. According to clinical experience and previous literature, severity tends to be positively correlated with disease course within a time frame of approximately 18 months (one and a half years), which is also consistent with our model [21]. Older patients had the potential to suffer from more severe status, which may be attributed to the vulnerability of the older. Besides, long disease course also attached great importance to the severity. It correspondingly inferred that early treatment and recovery may contribute to better outcomes [22]. It is generally held that smoking is closely related to the occurrence and progression of the disease [23] [24] [25] [26]. In our model, there was no significant difference in smoking history between the two groups. The reason may be that compared with other studies, the smoking rates of our two groups of patients were relatively low. As we continue to expand the sample size in the future, smoking may become an important categorical variable in our model.

As we all know, the current management of GO ranges from local supportive measures to intravenous glucocorticoids and surgery [27]. For patients with moderate-to-severe GO or sight-threatening GO, local supportive measures are no longer the first choice; immunosuppressive therapy is mainly used for active patients, while rehabilitative surgery is selected for inactive patients [1] [28]. As a result, accurate identification of GO severity will help doctors to intervene and treat the disease earlier, which is beneficial for the prognosis of the disease.

In this study, we successfully built a random forest model to distinguish patients with different severity, which can be used as an auxiliary method in managing GO patients. However, our study also has several limitations. First, it is well known that race is one of the risk factors of GO, and all patients involved in the research are Chinese, for which our model will be constrained by race. Second, in a cross-sectional study, we could not confirm this model based on the severity status of different courses. Third, the number of samples was not sufficient for establishing a random model. In future studies, we plan to collect more information on patients and expand our dataset.

Conclusion

This study shows the potential for applying a random forest model as a complementary tool to differentiate GO severity, which is more reliable than three other algorithms. In the future, we do hope we can incorporate more samples and clinical factors to make up for the deficiencies of our model.

Conflict of Interest

The authors declare that they have no conflict of interest.

Acknowledgement

This work was supported by the National Natural Science Foundation of China and the Special Fund of the Pediatric Medical Coordinated Development Center of Beijing Hospitals Authority.

• References

• 1 Bartalena L, Kahaly GJ, Baldeschi L. et al. The 2021 European Group on Graves’ orbitopathy (EUGOGO) clinical practice guidelines for the medical management of Graves’ orbitopathy. Eur J Endocrinol 2021; 185: G43-G67
• 2 Drui D, Du Pasquier Fediaevski L, Vignal Clermont C. et al. Graves’ orbitopathy: diagnosis and treatment. Ann Endocrinol (Paris) 2018; 79: 656-664
• 3 Bartalena L. Prevention of Graves’ ophthalmopathy. Best Pract Res Clin Endodrinol Metab 2012; 26: 371-379
• 4 Biscarini F, Masetti G, Muller I. et al. Gut microbiome associated with Graves disease and Graves orbitopathy: the INDIGO multicenter European study. J Clin Endocrinol Metab 2023; 108: 2065-2077
• 5 Breiman L. Random forests. Mach Learn 2001; 45: 5-32
• 6 Kim SY. Effects of sample size on robustness and prediction accuracy of a prognostic gene signature. BMC Bioinformatics 2009; 10: 147
• 7 Li G, Liu X, Wang M. et al. Predicting the functional outcomes of anti-LGI1 encephalitis using a random forest model. Acta Neurol Scand 2022; 146: 137-143
• 8 Moradian H, Lau MA, Miki A. et al. Identifying suicide ideation in mental health application posts: a random forest algorithm. Death Stud 2023; 47: 1044-1052
• 9 Couronné R, Probst P, Boulesteix A-L. Random Forest versus logistic regression: a large-scale benchmark experiment. BMC Bioinformatics 2018; 19: 270
• 10 Lim SL, Lim AK, Mumtaz M. et al. Prevalence, risk factors, and clinical features of thyroid‑associated ophthalmopathy in multiethnic malaysian patients with Graves’ disease. Thyroid 2008; 18: 1297-1301
• 11 Muralidhar A, Das S, Tiple S. Clinical profile of thyroid eye disease and factors predictive of disease severity. Indian J Ophthalmol 2020; 68: 1629-1634
• 12 Prummel MF, Bakker A, Wiersinga WM. et al. Multi‑center study on the characteristics and treatment strategies of patients with Graves’ orbitopathy: the first European group on Graves’ orbitopathy experience. Eur J Endocrinol 2005; 148: 491-495
• 13 Tanda ML, Piantanida E, Liparulo L. et al. Prevalence and natural history of Graves’ orbitopathy in a large series of patients with newly diagnosed Graves’ hyperthyroidism seen at a single center. J Clin Endocrinol Metab 2013; 98: 1443-1449
• 14 Laurberg P, Berman DC, Bülow Pedersen I. et al. Incidence and clinical presentation of moderate to severe graves’ orbitopathy in a Danish population before and after iodine fortification of salt. J Clin Endocrinol Metab 2012; 97: 2325-2332
• 15 Bartley GB. The epidemiologic characteristics and clinical course of ophthalmopathy associated with autoimmune thyroid disease in Olmsted County, Minnesota. Trans Am Ophthalmol Soc 1994; 92: 477-588
• 16 Tanda ML, Piantanida E, Liparulo L. et al. Prevalence and natural history of Graves’ orbitopathy in a large series of patients with newly diagnosed graves’ hyperthyroidism seen at a single center. J Clin Endocrinol Metab 2013; 98: 1443-1449
• 17 Seo S, Sánchez Robledo M. Usefulness of TSH receptor antibodies as biomarkers for Graves’ ophthalmopathy: a systematic review. J Endocrinol Invest 2018; 41: 1457-1468
• 18 Bylander T. Estimating generalization error on two-class datasets using out-of-bag estimates. Mach Learn 2002; 48: 287-297
• 19 Wiersinga WM. Advances in treatment of active, moderate-to-severe Graves’ ophthalmopathy. Lancet Diabetes Endocrinol 2017; 5: 134-142
• 20 Eckstein AK, Plicht M, Lax H. et al. Thyrotropin receptor autoantibodies are independent risk factors for Graves’ ophthalmopathy and help to predict severity and outcome of the disease. J Clin Endocrinol Metab 2006; 91: 3464-3470
• 21 Dolman PJ. Evaluating Graves’ orbitopathy. Best Pract Res Clin Endocrinol Metab 2012; 26: 229-248
• 22 Bahn RS. Graves’ ophthalmopathy. N Engl J Med 2010; 362: 726-738
• 23 Oeverhaus M, Winkler L, Stähr K. et al. Influence of biological sex, age and smoking on Graves’ orbitopathy – a ten-year tertiary referral center analysis. Front Endocrinol (Lausanne) 2023; 14: 1160172
• 24 Wiersinga WM. Smoking and thyroid. Clin Endocrinol (Oxf) 2013; 79: 145-151
• 25 Bartalena L, Tanda ML. Current concepts regarding Graves’ orbitopathy. J Intern Med 2022; 292: 692-716
• 26 Weiler DL. Thyroid eye disease: a review. Clin Exp Optom 2017; 100: 20-25
• 27 González-García A, Sales-Sanz M. Treatment of Graves’ ophthalmopathy. Tratamiento de la oftalmopatía de Graves. Med Clin (Barc). 2021. 156. 180-186
• 28 Genere N, Stan MN. Current and emerging treatment strategies for Graves’ orbitopathy. drugs 2019; 79: 109-124

Publication History

Received: 30 October 2023

Accepted: 11 March 2024

Article published online:
08 April 2024