Validation of the Expression and Emotion Scale for Children with attention deficit hyperactivity disorder into Brazilian Portuguese

ABSTRACT

Objective

To validate the parent-rated Expression and Emotion Scale for Children (EESC) for patients with attention-deficit/hyperactivity disorder (ADHD).

Methods

The EESC was applied to parents of children with and without ADHD. The children were divided into age groups: Group A, between six and eight years old; Group B, between nine and 11 years old; and Group C, between 12 and 15 years old. The validation was carried out according to the steps proposed by Guillemin et al. For the statistical calculation, Cronbach’s α, Pearson’s correlation, the ICC and ROC curve were used.

Results

The statistical tests showed satisfactory coefficients: Cronbach’s α = 0.76; Pearson’s correlation r = 0.91 with CI 95%; replicability ICC = 0.66; sensitivity 0.75; specificity 0.67; accuracy 71%.

Conclusion

According to psychometric data on internal and external consistency (reliability, reproducibility), sensitivity, and specificity, the parent-rated EESC for ADHD is useful in assessing emotional expression.

RESUMO

Objetivo

Validar o instrumento Expression and Emotion Scale for Children (EESC) versão pais para crianças com transtorno do déficit de atenção e hiperatividade (TDAH).

Métodos

O instrumento EESC foi aplicado aos pais de crianças e adolescentes com e sem TDAH. Estes foram divididas em grupos etários: Grupo A entre 6 -8 anos; Grupo B, entre 9-11; Grupo C, entre 12-15 anos de idade. A validação da EESC seguiu as seguintes etapas propostas por Guillemin et al. Para cálculo estatístico: α de Cronbach, Pearson, CIC e Curva ROC.

Resultados

Os testes estatísticos apresentaram coeficientes satisfatórios: Cronbach α = 0,76; correlação de Pearson r = 0,91 com IC 95%; replicabilidade CIC = 0.66; sensibilidade 0,75; especificidade 0,67; acurácia 71%.

Conclusão

Conforme dados psicométricos de consistência interna, externa (confiabilidade, reprodutibilidade), sensibilidade e especificidade, a EESC versão pais é válida para avaliar a Expressão Emocional de crianças e adolescentes com TDAH.

attention deficit disorder with hyperactivity, child - adolescents

transtorno do déficit de atenção com hiperatividade, criança - adolescente

Attention-deficit/hyperactivity disorder (ADHD) is a neurobehavioral disorder characterized by the combination of attention deficit, hyperactivity and impulsivity[1].

The diagnosis is fundamentally clinical and is based on criteria from classification systems such as the Diagnostic and Statistical Manual of Mental Health Disorder, Fourth Edition, 2000 (DSM-IV), effective at the beginning of this research[2]. Attention-deficit/hyperactivity disorder is classified into three types: (a) predominance of hyperactivity-impulsivity; (b) predominance of attention-deficit; and (c) a combination of both, which can be associated with comorbidities[1].

The hyperactive-impulsive individual presents with greater impairment in family/social life, the inattentive child has greater impairment in school performance, and the child affected by the combination has impairment both in family/social life and education[2]. The prevalence of ADHD is 5% to 10% in children and is more common in boys than in girls, at a ratio of up to 3:1[3].

The mechanisms by which ADHD occurs are still not fully understood, but genetics have a very important contribution[1],[4] in the dysfunction of brain dopaminergic and noradrenergic activities that lead to inadequate control of the ventral prefrontal cortex areas over the lower structures, related to automated responses, damaging cortical control of responses to certain stimulae[5],[6]. Therefore, uncontrolled emotions may influence the decision-making process, interfering in cognitive processes[7].

The decrease in dopamine and noradrenaline, and the late maturation of brain circuits associated with the prefrontal cortex lead to a decrease in information processing speed and to a lower self-control capability[8],[9]. The impairment in the functioning of the ventral anterior cingulate cortex is directly associated with the control of primitive responses and impulsive emotional expressions[10]. In a normal state, the action of the prefrontal cortex over the subcortical areas leads to an inhibitory influence on the amygdala and prevents undesirable automatic reactions[11], and also preserves and restores attention, memory and emotional control (executive functions)[12]. Therefore, a broad range of brain control processes connect, prioritize and integrate functions that are needed for self-control.

When some emotions are overtly expressed, they can be of a negative nature, such as: irritability, emotional lability, sadness, dysphoria and crying; or of a positive nature: friendship, joy, happiness and spontaneity[13]. There needs to be monitoring of emotional expression during medical monitoring or drug treatment that will act on the affected brain areas improving brain circuits[13].

Emotional expression refers to the ability to moderate negative emotions and potentiate positive emotions in a balanced way without repressing or exaggerating the information they convey[14].

It has been noted that “dysphoric” emotional effects arise during medication treatment and, therefore, a parent-reported scale was developed to measure the “negative emotional expression,” including affective blunting, mood lability, and the “zombie effect”. The Expression and Emotion Scale for Children (EESC) aims to quantify the impact of medication on emotional expression in children[15].

Kratochvil et al.[15] developed the EESC, taking into consideration the symptoms presented by children with ADHD. This scale assesses the intensity and character of the emotions expressed by the child or adolescent with ADHD. It considers the child’s mood and emotional state (irritability, poor concentration, isolation, crying, liveliness, spontaneity, sympathy, maturity, etc.) over the previous two weeks.

This scale has 29 questions divided into three domains: positive, with 13 questions; negative, with 10 questions; and lability, with 5 questions; question 19 was not considered for the statistical calculation. The items are quantified according the Likert scale: 1 – none; 2 – a little; 3 – reasonable; 4 – very much; 5 – totally. A high level of emotional control is indicated by a lower score. The maximum and minimum values in the positive, negative and lability domains are, respectively: 65-13, 50-10, 25-5; noting that the positive domain is inverted for the sum of the overall score[15].

Considering the impairments in the emotional expression of individuals with ADHD caused by neuroanatomical, chemical, and physiological conditions, the purpose of this study was to validate, for the Portuguese language, the EESC for the assessment of emotional expression control by patients with ADHD between six and 15 years old, who used stimulants or were simply being monitored medically (see [Appendix]).

METHODS

This cross-sectional study was approved by the Ethics Committee for Analysis of Research Projects (CAPPesq) of the Hospital Clinical Board with the number 0613/11. For the validation of the EESC patient version, the study children were all recruited from the Learning Disabilities Clinic of the Children’s Institute of the University of São Paulo Medical School.

Patients

A total of 126 patients with ADHD were assessed and diagnosed according to the Swanson, Nolan and Pelham Teacher and Parent Rating Scale[16] following the DSM-IV criteria (DSM V had not yet been edited when the research began); of these patients, 91% were being treated with a stimulant and 9% were medication-free. The average age of the ADHD group was 11.24 years old and that of the control group (n = 126) was 10.51 years old. Of the 126 patients with ADHD, 48 had mild comorbidities (40%) and 78 had no comorbidities (60%). Data on comorbidities had been previously collected and registered in the medical records. The assessment included the Child Behavior Checklist[17]. The comorbidities identified in the ADHD group were: learning disorder 20 (15%); depression 12 (10%); oppositional defiant disorder 7 (6%); conduct disorder 6 (5%); and bipolar disorder 5 (4%). The characteristics of the patients’ education, socioeconomic profile and hobbies are shown in [Table 1].

Table 1 Clinical characterization of the patients with and without ADHD.

Variable	ADHD	Control
Male gender	83%	52%
Female gender	17%	48%
Slight comorbidity	40%	-
Public school	74%	100%
Electronic games	47%	56%
Years of parent’s education	9.4	11.7

ADHD: attention-deficit/hyperactivity disorder.

Of the 126 patients, it was selected age groups for stablish the cutoff point, at a total 85 subjects, because they were paired according to number, age and gender: Group I – six to eight years old, n = 13 (17.46%); Group II – 9 to 11 years old, n = 40 (46.82%); and Group III – 12 to 15 years old, n = 32 (35.71%).

The study excluded patients under six and over 15 years old; those with a cognitive deficit; with an intelligence quotient lower than 80 (assessed by a neuropsychologist); and those who did not have diligent medical care, or who had interrupted clinical supervision. The control group also had exclusions, which included individuals under six and over 15 years of age; those with a cognitive deficit; those with an intelligence quotient lower than 80 (assessed by a neuropsychologist); and those diagnosed with any psychopathology (assessed by a neuropsychologist).

The tests used to assess the control group were: the School Achievement Test[18]; the Swanson, Nolan and Pelham-IV Teacher and Parent Rating Scale; and the Child Behavior Checklist.

Protocol

Between February and December of 2012, a validation of the parent-rated EESC was carried out. The version was applied to 33% of the individuals with intra-observer and inter-observer variations. The average duration of the application was 12 minutes. It was applied directly to ADHD group parents and indirectly to control group parents. The members of the latter group were parents of students from the Escola Municipal de Ensino Fundamental Presidente Professor João Pinheiro (Vila Matilde, São Paulo, SP) with previous authorization from the school principal.

Initially, a certified translation of the tool was carried out. Then, after the choosing a native speaker for the translation, the document was culturally and semantically adapted. The validation followed the steps proposed by Guillemin et al.[19]: 1) authorization from the authors; 2) initial translation into Portuguese; 3) cultural, conceptual, experimental and idiomatic adaptation to the target population; 4) retroversion; 5) assessment by a revision committee; and 6) pre-test in two stages.

Description of the validation steps

The process of validation of the EESC questionnaire translated into Portuguese had previous authorization (1^st step) by Kratochvil et al[15]. It was translated by an English language native professional translator and by a certified translator (2^nd step). After the comparison of both translations, it was observed that the translation made by the English language native translator was non-literal and, therefore, culturally more appropriate, closer to colloquial language and more effective for the process of linguistic adaptation than the certified translation.

Between the steps of translation and retroversion, the Probe technique[20] was included to assess only the comprehension of the items of the EESC by the patients’ parents. The understanding scale of 1 to 5 was: (1) no understanding, (2) little understanding, (3) average comprehension, (4) understanding almost total, and (5) total understanding. The comprehension level reached was above 85%.

In order to reach satisfactory comprehension, the Probe technique was applied to three groups of 20 patients’ parents each in three phases. During these applications for the improvement of the instrument, a cultural adaptation was made using the following criteria: semantic, idiomatic, conceptual and experimental equivalences (3^rd step).

When the ideal comprehension level was reached, the retroversion step began (4^th step). The Portuguese version, resulting from the Probe technique, was retranslated into English by an English language native translator and compared to the original. The analysis of this retranslation enabled the verification of linguistic equivalence. After this step, the version resulting from the Probe technique was assessed by the revision committee (comprising two neuropediatricians, one pedagogue and one phonoaudiologist), which suggested some changes (5^th step).

The 6^th step was divided into two periods: the reapplication of the Probe technique in order to achieve the satisfactory comprehension level, applied to 41 individuals after the changes suggested by the revision committee and, after this, the questionnaire was applied to one parent of 126 patients for the assessment of the child’s emotional expression, and the results were submitted to statistical calculation for psychometric evaluation.

The evaluation of the psychometric quality of the EESC was carried out according to the following distribution: internal consistency (reliability of questions), n = 126; external consistency (inter-observer reliability), n = 35; external consistency (test-retest reproducibility – twice in the period of three months), n = 24; sensitivity, specificity, accuracy and over all cutoff point, n = 126.

Statistics

The calculation of internal consistency, item reliability, was made using the Cronbach’s Alpha Coefficient test and was classified as: very low (alpha ≤ 0.30), low (alpha between 0.30 and 0.60), moderate (alpha between 0.60 and 0.75), high (alpha from 0.75 to 0.90) and very high (alpha above 0.90), with alpha values above 0.70 being considered satisfactory[21]. This same classification was applied to the other statistical tests used, as shown below. Statistical significance was determined at p < 0.05.

For the calculation of external consistency, the reliability analysis was made through Pearson’s Linear Correlation[22] and reproducibility was analyzed using the Intraclass Correlation Coefficient test (ICC)[23],[24].

The Receiver Operating Characteristic Curve (ROC) is used for the assessment of sensitivity and specificity of the instrument and to indicate optimal cutoff and accuracy. The most elevated point of a curve, corresponding to the upper left angle of the graph, represents 100% of sensitivity and 0% of false positives; in this case, the ideal value of the diagnostic test is the gold standard (d = 0). Line ‘d’ in the curve indicates proximity to the axis of abscissas, which represents the gold standard. When d = 0.20, the distance to the gold standard is low, the distance of d = 0.50 is considered moderate, the distance of d = 0.80 is considered high[25].

The ROC is a method for assessment, organization and selection of diagnostic and/or prediction systems. It is commonly used in medicine to analyze the quality of a given clinical test[26],[27]. The tools used were the SPSS Statistics version 2.2 for Cronbach’s Alpha and the BioStat version 5.3 for the Pearson’s, ICC, and ROC tests. A series of tests was adopted as the gold standard to guarantee the validity of the tool[28].

RESULTS

The level of comprehension reached through the Probe technique was 96%, which is above the required minimum of 85%. The average of the overall score for patients and for the controls was of 60.21 ± 14.02 for patients and 46.96 ± 10.61 for controls; group I, ADHD patients 48.69 ± 9.3 and controls 47.69 ± 10.97; group II, ADHD patients 57.15 ± 11.67 and controls 46.52 ± 13.42; group III, ADHD patients 61. 37 ± 14.13 and controls 51.19 ± 12.97. The averages and standard deviations of the domains can be seen in [Table 2].

Table 2 Mean and standard deviation (SD) of patients with and without ADHD.

Variable	Positive domain	Negative domain	Lability domain	Overall score
ADHD vs control
ADHD mean - SD	24.69 ± 6.91	21.59 ± 7.41	14.38 ± 4.06	60.21 ± 14.02
Control mean - SD	21.51 ± 7.16	14.73 ± 4.84	14.381 ± 4.63	46.96 ± 10.61
Group I vs control
ADHD mean - SD	21.92 ± 8.1	16.38 ± 4.42	12.92 ± 3.82	48.69 ± 9.3
Control mean - SD	20 ± 4.45	14.54 ± 3.81	13 ± 4.06	47.69 ± 10.97
Group II vs control
ADHD mean - SD	22.57 ± 4.35	20.02 ± 6.76	15.1 ± 4.22	57.15 ± 11.67
Control mean - SD	20.57 ± 6.92	13.47 ± 3.96	11.7 ± 4.38	46.52 ± 13.42
Group III vs control
ADHD mean - SD	25.34 ± 6.40	22.31 ± 7.56	13.75 ± 3.8	61. 37 ± 14.13
Control mean - SD	22.84 ± 7.34	16.06 ± 5.78	12.62 ± 5.29	51.19 ± 12.97

ADHD: attention-deficit/hyperactivity disorder.

The internal consistency through Cronbach’s Alpha Test in the reliability assessment of the items in the instrument showed an overall score of α = 0.76 and p < 0.001. This score, and that of other domains, where the values guarantee psychometric reliability of the items, are shown in [Table 3].

Table 3 Values of Cronbach’s alfa, Pearson’s linear correlation coefficient, intraclass correlation (ICC) of the domains of the EESC questionnaire.

Variable	n	Overall score	Positive	Negative	Labile	p-significance
Cronbach’s alpha	13	0.76	0.76	0.75	0.74	p < 0.001
Pearson’s r	35	0.91	0.95	0.97	0.91	p < 0.001
ICC	24	0.66	0.78	0.70	0.76	p < 0.001

EESC: expression and emotion scale for children.

External consistency – reliability – through Pearson’s linear correlation test was r = 0.91 and p < 0.001 in the overall score. The confidence interval of 95% between the observer and the inter-observer in all domains and p < 0.0001 are shown in [Table 3]. For external consistency – reproducibility – the patients were assessed in twice, with an interval of six months, through the ICC test, reaching the value of 0.66; values of the domains are in [Table 3].

The results of the ROC Curve of the EESC for the overall score in all age groups were: sensitivity = 0.75, specificity = 0.67, accuracy = 71% and cutoff = 51. The remaining results of the different domains by age group are shown in [Table 4] and the [Figure].

Table 4 Values of sensibility, specificity, accuracy and cutoff of the domains of the EESC questionnaire.

Variable	Group A	Group B	Group C	All groups
Specificity
Positive domain	0.39	0.53	0.44	0.69
Negative domain	0.62	0.8	0.6	0.69
Labile domain	0.54	0.65	0.5	0.69
Overall score	0.69	0.7	0.53	0.67
Sensibility
Positive domain	0.85	0.78	0.72	0.60
Negative domain	0.69	0.88	0.81	0.79
Labile domain	0.46	0.68	0.69	0.53
Overall score	0.54	0.7	0.84	0.75
Accuracy (%)
Positive domain	62	65	58	65
Negative domain	65	84	70	74
Labile domain	50	66	59	57
Overall score	62	70	69	71
Cut off
Positive domain	17	20	22	23
Negative domain	15	15	17	16
Labile domain	14	13	12	14
Overall score	52	51	52	51

EESC: expression and emotion scale for children

Figure The ROC Curve of the Overall Score of Children with and without ADHD.

DISCUSSION

The EESC is capable of being self-administered and is easy to fill in. It requires 12 minutes to be completed. It is a recommended instrument for clinical supervision in the quantification of the emotional expression of the patient with ADHD in drug treatment to assess the oscillation of the emotions before the impact of medication[13],[15],[29].

A less literal translation was used as a strategy to improve comprehension[30]. The Probe technique was used to measure the level of comprehension; this enabled a better cultural, idiomatic and semantic adaptation. This was necessary due to the fact that words and expressions carry values and meanings inherent to a culture and they must be incorporated into the semantics of the instrument, with the purpose of expressing what the questionnaire intends to show while remaining true to the original version[30]. After the application of the Probe technique, a satisfactory level of comprehension was achieved to enable the collection of data and submission to statistical analysis for the assessment of the psychometric quality of the instrument.

Internal consistency in each domain was moderate to high (α = 0.76). External consistency indicated a strong correlation, guaranteeing reliability (r = 0.95). Test-retest reproducibility was considered satisfactory (ICC = 0.76), qualified as reproducible according to the criteria established by Shrout and Fleiss[23]. The results obtained in this validation are in accordance with those of the original parent-rated EESC[29]. The present validation also measured sensitivity (0.75), specificity (0.67), accuracy (71%) and cutoff (51) through the ROC Curve test. The sensitivity and the specificity were significant, since they remained above the reference line[25]. The level of accuracy confirmed that the instrument adequately assessed that which it aimed to. The validity and efficacy follow the measures of the gold standard, since they are above the diagonal line ([Figure])[25],[29].

By means of the application of the instrument to the age groups, it was possible to establish a cutoff value related to age and domain. Depending on the age group the individual is part of, there is variation in the way they perceive their surroundings, which points to a higher or lower control over emotional expression and over quality of life[28],[31]. Through the standardization of the cutoff regarding the age group, it was possible to assess, with greater precision, the patient’s emotional expression, considering that, in this study, values lower than 51 (overall score) indicated a better emotional expression and, consequently, increased quality of life[28],[31].

The difficulty faced by children and adolescents with ADHD regarding emotional management is notorious[12],[31]. The measurement of the level and quality (positive, negative or labile domains) of emotional expression, which is part of their symptomatic state, must be part of daily therapeutic practice to establish a parameter for the results of medication on the patient’s life[12]. There are studies that confirm the need for the application of an instrument to measure emotional expression[12],[13],[15],[30]. The efficacy of the application of the EESC has also been confirmed in the literature[15],[24].

There is little literature regarding this issue, which was restrictive for this study. It also had other limitations: the EESC was validated in a sample with individuals with comorbidities and was based only on the responses of the parents.

According to the psychometric data on internal and external consistency (reliability and reproducibility), sensitivity, specificity and accuracy, the parent-rated EESC was useful in assessing the intensity and quality of the emotions expressed by the child or the adolescent between six and 15 years old, diagnosed with ADHD and under clinical supervision, verifying the treatment results on the patient’s life.

Conflict of interest:

There is no conflict of interest to declare.

Support:

Higher Education Personnel Improvement Coordination (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior - CAPES). It was carried out in the Department of Learning Disorders of the Children’s Institute, Hospital Clinics, School of Medicine of the University of São Paulo and in the João Pinheiro School.

• References

• 1 Biederman J, Faraone SV. Current concepts on the neurobiology of Attention-Deficit/Hyperactivity Disorder. J Atten Disord. 2002;6(Suppl 1):S7-16.
• 2 American Psychiatric Association. Diagnostic and statistical manual of mental disorders: DSM-IV-TR. Washington, DC: American Psychiatric Association; 2000.
• 3 Spencer TJ, Biederman J, Mick E. Attention-deficit/hyperactivity disorder: diagnosis, lifespan, comorbidities, and neurobiology. Ambul Pediatr. 2007;71 Suppl):73-81. https://doi.org/10.1016/j.ambp.2006.07.006
• 4 Singh AL, Waldman ID. The etiology of associations between negative emotionality and childhood externalizing disorders. J Abnorm Psychol. 2010;119(2):376-88. https://doi.org/10.1037/a0019342
• 5 Comings DE, Blum K. Reward deficiency syndrome: genetic aspects of behavioral disorders. Prog Brain Res. 2000;126:325-41. https://doi.org/10.1016/S0079-6123(00)26022-6
• 6 Viviani, R. Emotion regulation, attention to emotion, and the ventral attentional network. Front Hum Neurosci. 2013;7:746. https://doi.org/10.3389/fnhum.2013.00746
• 7 Elliott R, Sahakian BJ, McKay AP, Herrod JJ, Robbins TW, Paykel ES. Neuropsychological impairments in unipolar depression: the influence of perceived failure on subsequent performance. Psychol Med. 1996;26(5):975-89. https://doi.org/10.1017/S0033291700035303
• 8 Shaw P, Lalonde F, Lepage C, Rabin C, Eckstrand K, Sharp W et al. Development of cortical asymmetry in typically developing children and its disruption in attention-deficit/hyperactivity disorder. Arch Gen Psychiatry. 2009;66(8):888-96. https://doi.org/10.1001/archgenpsychiatry.2009.103
• 9 Sheres A, Tontsch C, Lee Thoeny A. Steep temporal reward discounting in ADHD-Combined type: acting upon feelings. Psychiatry Res. 2013;209(2):207-13. 10.1016/j.psychres.2012.12.007
• 10 Hulvershorn L A, Mennes M, Castellanos FX, Di Martino A, Milham MP, Hummer TA et al. Abnormal amygdala functional connectivity associated with emotional lability in children with attention-deficit/hyperactivity disorder. J Am Acad Child Adolesc Psychiatry. 2014;53(3):351-61. https://doi:10.1016/j.jaac.2013.11.012
• 11 Ochsner KN, Gross JJ. The cognitive control of emotion. Trends Cogn Sci. 2005;9(5):242-8. https://doi.org/10.1016/j.tics.2005.03.010
• 12 Brown TE. Executive functions and attention deficit hyperactivity disorder: Implications of two conflicting views. Interl J Disabil Dev Educ. 2006;53(1):35-46. https://doi.org/10.1080/10349120500510024
• 13 Findling RL, Brams M, Childress AC, López FA, Manos MJ, Jensen PS. Changes in emotions related to medication used to treat ADHD. Part II: clinical approaches. J Atten Disord. 2011;15(2):113-21. https://doi.org/10.1177/1087054710381232
• 14 Mayer J, Salovey P. What is emotional intelligence? In: Salovey P, Sluyter D, editors. Emotional development and emotional intelligence: educational implications. New York: Basic; 1997. p. 3-31.
• 15 Kratochvil CJ, Faries D, Vaughan B, Perwien A, Busner J, Saylor K et al. Emotional expression during attention-deficit/ hyperactivity disorders treatment: initial assessment of treatment effects. J Child Adolesc Psychopharmacol. 2007;17(1):51-62. https://doi.org/10.1089/cap.2006.0018
• 16 Bussing R, Fernandez M, Harwood M, Wei Hou, Garvan CW, Eyberg SM et al. Parent and teacher SNAP-IV ratings of attention deficit/hyperactivity disorder symptoms: psychometric properties and normative ratings from a school district sample. Assessment. 2008;15(3):317-28. https://doi.org/10.1177/1073191107313888
• 17 Naar-King S, Ellis DA, Frey MA, Ondersma ML. Assessing children’s well-being: a handbook of measures. LondoN: Routledge; 2003. p. 61-72.
• 18 Stein LM. TDE - Teste de Desempenho Escolar: manual para aplicação e interpretação. São Paulo: Casa do Psicólogo; 1994.
• 19 Guillemin F, Bombardier C, Beaton D. Cross-cultural adaptation of health-related quality of life measures: literature review and proposed guidelines. J Clin Epidemiol. 1993;46(12):1417-32. https://doi.org/10.1016/0895-4356(93)90142-N
• 20 Clark P, Lavielle P, Martínez H. Learning from pain scales: patient perspective. J Rheumatol. 2003;30(7):1584-8.
• 21 Cronbach LJ. Coefficient alpha and the internal structure of tests. Psychometrika. 1951;16(3):297-334. https://doi.org/10.1007/BF02310555
• 22 Pearson K, Fisher R, Inman HF. Karl Pearson and R. A. Fisher on Statistical Tests: A 1935 Exchange from Nature. American Statistician. 1994;48(1):2-11. https://doi.org/10.2307/2685077
• 23 Shrout PE, Fleiss JL. Intraclass correlations: uses in assessing rater reliability. Psychol Bull. 1979;86(2):420-8. https://doi.org/10.1037/0033-2909.86.2.420
• 24 Fayers PM, Machin D: Quality of life: assessment, analysis and interpretation. New York: Wiley; 2000.
• 25 Sabattini RME. Um programa para o cálculo da acurácia, especificidade e sensibilidade de testes médicos. Rev Informédica. 1995;2:19-21.
• 26 Zhou H, McClish DK, Obuchowski NA. Statistical methods in diagnostic medicine. New York: Wiley Interscience; 2002.
• 27 Silva AC, Gattass M, Carvalho PCP. Diagnosis of solitary lung nodule using texture and geometry in computerized tomography images: preliminary results. IEEE Latin America Transactions. 2004;2(2):75-80. https://doi.org/10.1109/TLA.2004.1468623
• 28 Henegham C, Badenoch D. Ferramentas para medicina baseada em evidências. Porto Alegre: Artmed; 2007.
• 29 Perwien AR, Kratochvil CJ, Faries D, Vaughan B, Busner J, Saylor KE et al. Emotional expression in children treated with ADHD medication development of a new measure. J Atten Disord. 2008;11(5):568-79. https://doi.org/10.1177/1087054707306117
• 30 Barbosa TS, Tureli MC, Gavião MB. Validity and reliability of the Child Perceptions Questionnaires applied in Brazilian children. BMC Oral Health. 2009:9(1):13. https://doi.org/10.1186/1472-6831-9-13.
• 31 Barkley RA, Fischer M. The unique contribution of emotional impulsiveness to impairment in major life activities in hyperactive children as adults. J Am Acad Child Adolesc Psychiatry. 2010;49(5):503-13. https://doi.org/10.1016/j.jaac.2010.01.019

Publication History

Received: 01 December 2016

Accepted: 04 May 2017

Article published online:
05 September 2023