Normative values of the Brief Repeatable Battery of Neuropsychological Tests in a Brazilian population sample: discrete and regression–based norms

Keywords

cognition - multiple sclerosis - neuropsychology

Palavras-chave

cognição - esclerose múltipla - neuropsicologia

Cognitive function abnormalities are increasingly recognized as a major complaint among multiple sclerosis (MS) patients. Neuropsychological studies have shown cognitive dysfunction in up to two thirds of MS patients[1], and it has been described in the earliest stages of a clinically, or even radiologically, isolated syndrome[2]. Cognitive dysfunction generally affects information processing speed and episodic memory and thus has a substantial contribution to disability, impairing daily living and work capacity. Nevertheless, assessing such an important and complex domain requires a systematic approach. Given the low sensitivity of the widely–used Mini Mental State Examination (MMSE) for detecting cognitive dysfunction in MS[3], Rao and co–workers of the American National MS Society developed the Brief Repeatable Battery of Neuropsychological Tests (BRB–N), a battery of neuropsychological measures covering functions most commonly impaired in MS, with 71% sensitivity and 94% specificity[4]. This battery includes tests for the assessment of verbal and visuospatial memory, sustained attention and information processing speed, working memory and verbal fluency. More recently, an international panel introduced a shorter cognitive battery for use when time constraints are strict, comprising tests for information processing speed, verbal and visuospatial memory (the Brief International Cognitive Assessment for Multiple Sclerosis)[5]. Although the BRB–N has been employed extensively in MS research worldwide and is also validated in the Portuguese language[6], normative values for this test battery have been published only in few European countries and the USA[7],[8], limiting a more widespread application in clinical practice, especially in South America[9]. The Brief International Cognitive Assessment for Multiple Sclerosis battery has also recently been validated in Brazil, but normative data is not yet available[10]. An accurate classification of neuropsychological impairment depends on the normative comparison. If normative data are derived from individual samples that do not match the individual under assessment, misclassification may occur. Therefore, development of appropriate normative standards is critical for any neuropsychological measure[11]. Neuropsychological normative data have usually been presented in terms of discrete norms, where means and standard deviations for each age group are provided. Nevertheless, discrete norms have been subject to criticism in the last few years and the use of regression–based demographically–adjusted scores has increasingly been recommended[11]. In this setting, our aim was to provide both discrete and continuous normative values for the BRB–N in a sample of the Brazilian population, while assessing the effect of demographic factors on cognitive performance.

METHODS

Participants

We enrolled a total of 285 healthy individuals recruited from the community at 10 sites across Brazil (Belo Horizonte, Campinas, Rio de Janeiro, Niteroi, Goiania, Porto Alegre, Santos, São Paulo, and Ribeirão Preto).

The sample size was estimated for differences (or correlations) between two independent means by adopting an effect size of 0.5, α error probability of 0.05, and power (1–β error probability) of 0.80 for two groups (version A and B subgroups; two–tailed). In this way, the minimal sample size would be 128 individuals, divided into 64 for each subgroup. However, since most BRB–N normative studies in the USA and European countries employed at least 150 individuals for version A (the most used)[7],[8], we enrolled 237 participants for this version, and 48 participants for version B of the battery.

Inclusion criteria were ages from 18 through 65 years, educational level of at least primary school (four years), visual acuity of at least 0.5 (or 20/40) in both eyes (with or without lens correction), normal neurological examination, and having slept enough the night prior to the testing. Individuals were excluded if they had neurological disease or major psychiatric illness, history of alcohol or drug abuse, serious head trauma, learning disability and a recent major medical illness. The study was approved by the ethics committee of the faculty of medical sciences of the University of Campinas and all participants provided written informed consent.

Neuropsychological test procedures

The BRB–N version A was applied in 237 participants and version B was applied in 48 participants. Different versions are important, to minimize practice effects with longitudinal administration. These groups were not different regarding gender distribution, age and educational level ([Table 1]).

Table 1 Demographic data (expressed as mean values and standard deviation).

Variable	Total (n = 285)	BRB–N Version A (n = 237)	BRB–N Version B (n = 48)	Comparisons*
Male/Female: %	42.6/57.4	44.5/55.5	33.3/66.7	p = 0.199
Age: years (range)	38.14 ± 13.37 (18–66)	37.72 ± 13.40 (18–66)	38.23 ± 11.79 (18–64)	p = 0.599
Education: years (range)	12.34 ± 3.68 (4–22)	12.24 ± 3.59 (3–22)	11.98 ± 4.06 (4–21)	p = 0.693
MMSE: total score (range)	28.51 ± 1.54 (23–30)	28.51 ± 1.57 (23–30)	28.52 ± 1.46 (25–30)	p = 0.948
SRT–LTS	50.17 ± 11.98 (6–71)	49.41 ± 12.33 (6–71)	53.58 ± 9.64 (22–69)	p = 0.037
SRT–CLTR	39.18 ± 14.28 (2–71)	38.18 ± 14.49 (2–71)	43.71 ± 12.46 (17–69)	p = 0.011
SRT–DR	9.27 ± 2.02 (2–12)	9.18 ± 2.07 (2–12)	9.69 ± 1.73 (6–12)	p = 0.167
SpRT	21.35 ± 5.34 (8–30)	21.09 ± 5.22 (8–30)	22.35 ± 5.76 (10–30)	p = 0.104
SpRT–DR	7.54 ± 2.05 (1–10)	7.43 ± 2.00 (1–10)	8.02 ± 2.18 (3–10)	p = 0.036
SDMT	55.89 ± 18.75 (12–110)	55.26 ± 19.54 (12–110)	58.69 ± 14.67 (16–90)	p = 0.054
PASAT	39.28 ± 13.01 (8–60)	39.57 ± 12.88 (8–60)	38.00 ± 13.66 (13–58)	p = 0.452
WLG	24.25 ± 7.04 (5–42)	23.88 ± 7.00 (5–42)	25.85 ± 7.07 (12–42)	p = 0.093

BRB–N: brief repeatable battery of neuropsychological tests; CLTR: consistent long term retrieval; DR: delayed recall; LTS: long term storage; MMSE: mini mental state examination; PASAT: paced auditory serial addition test; SDMT: symbol digit modalities test; SpRT: spatial recall test; SRT: selective reminding test; WLG: word list generation. *Group comparisons were performed with Mann–Whitney U tests, except for percentage distribution, where Fisher's exact test was used.

All neurologists and neuropsychologists who administered the battery had participated in an initial common training session in order to standardize the criteria of administration, data recording and scoring procedures. All individuals underwent testing during daytime and in a quiet room. The administration time of the BRB–N was around 30 minutes.

BRB–N subtests and scores

Verbal memory was tested with the Selective Reminding Test (SRT). The test consists of presenting orally to the individual a list of 12 unrelated words for up to six trials, at the rate of one word per two seconds. After the list has been presented, the participant is instructed to recall as many words as possible in any sequence. Words that are not recalled are repeated (reminded) by the examiner on the next trial, and the individual is requested to recall and say again the whole list of 12 words. The scoring system consists of a Long Term Storage (LTS) and a Consistent Long Term Retrieval (CLTR). If a word is recalled on two successive trials without a reminder, it is assumed to have entered LTS on the first of these two trials. With or without retrieval of this word on the subsequent trials, it is scored as LTS on all following trials. The CLTR score refers to consistent recall of this same word on all succeeding trials until the last one. After about 20 minutes, the subject tries to recall and replicate the list again (delayed recall)[12],[13]. Versions A and B have a different list of 12 words. The scores employed are: LTS (total number of words in LTS in all six trials), CLTR (total number of words in CLTR in all six trials) and the number of correct words after delayed recall.

Visuospatial memory was tested with the 10/36 Spatial Recall Test (SpRT), which consists of a test where participants are shown, for a period of 10 seconds, a 6 x 6 checkerboard with 10 pieces placed in specific locations. Shortly afterwards, the participant is given 10 pieces and asked to replicate the pattern on a blank checkerboard. The test is repeated three times. After about 20 minutes, the participant tries to recall and replicate the pattern again (delayed recall)[13]. Versions A and B differ in the spatial configuration of the 10 pieces. There is an immediate recall score, equal to the total number of correct responses for the three trials, and a delayed recall score.

Sustained attention and information processing speed were tested with the Symbol Digit Modalities Test (SDMT). Only the SDMT oral version was employed in this study. The SDMT consists of a key with two rows, with nine stimulus symbols in the upper row and matched numbers (1–9) in the row below it. The task sequence consists of a series of symbols in random order, each with a blank space underneath, and the participants are asked to respond with the number that matches each symbol as rapidly as possible in 90 seconds, after completing a 10–item practice trial. The score of the test is the number of correct responses completed within the time limit, with the maximum score of 110[13],[14]. Each alternate form (A and B) features a new arrangement of the symbol–digit pairings in the key (the same symbols matched in a different order).

Working memory and information processing speed were tested with the Paced Auditory Serial Addition Test. In this test, participants listen to an audio recording of 61 digits where single digits are presented every three seconds and the participant must add each new digit to the one immediately prior to it (and not to the result of the previous sum), providing each sum orally as quickly as possible. The test score is the number of correct sums given (out of 60 possible) in each trial. Versions A and B differ only in the sequence of digit presentation[13],[15].

Verbal fluency was tested with the Word List Generation. This is a semantic verbal fluency test evaluating the spontaneous production of names of a given category (i.e. vegetables and fruits in version A; animals in version B) within 90 seconds. The score is the number of correct names in either category[13]. Inflections of the same word (cow, cows) or its perseverations are counted as one response, and words not belonging to the category are not counted.

Translation of the BRB–N stimuli and instructions was performed by three translators, fluent in both Portuguese and English. At first, a native English speaker translated from English to Portuguese. Then stimuli and instructions were back–translated into English and then into Portuguese, independently by two neurologists, both experts in neuropsychology and fluent in English, and members of the Neuropsychology and Neurolinguistics Unit of the University of Campinas. The final version was piloted in 10 healthy participants and 10 MS patients for final adaptation of sociocultural and linguistic aspects.

Statistical analysis

Statistical analysis was performed with the Statistical Package for the Social Sciences (v21.0, SPSS Inc, Chicago). The level of statistical significance was p < 0.05. Normality tests for each variable were performed with the Shapiro–Wilk test and q–q plots. Group comparisons on clinical and cognitive tests were performed with Mann–Whitney U tests, except for percentage distribution, where Fisher's exact test was used. Continuous norms were calculated with multiple–regression analysis as previously described[11],[16],[17]. Briefly, healthy participants' raw scores on each neuropsychological measure were first converted to scaled scores based on cumulative distribution (M = 10, SD = 3). Scaled scores were then regressed on age, education and sex (male = 2; female = 1). We tested for multicollinearity among predictor variables. In addition, we inspected the distributions of residuals for normality by analyzing q–q plots. Plots of regression–standardized residual predicted values showed that variance of the residuals was constant (homoscedasticity). Given the small sample size of version B, norms (discrete and continuous) were calculated for version A only.

RESULTS

There were no significant differences of demographic variables between the participants in whom versions A or B were administered ([Table 1]). Mean scores were higher in version B for the SRT (LTS and CLTR) and SpRT (delayed recall only) ([Table 1]).

Mean raw scores and the 5^th percentile for each neuropsychological measure in version A are shown in [Table 2], stratified by age and educational level.

Table 2 BRB–N scores stratified by age and educational level (expressed as mean values and standard deviation).

Age	BRB-N subtests	Educational level
		6–11 years		≥ 12 years
		Mean ± SD	5th percentile	Mean ± SD	5th percentile
18–30	SRT- LTS	50.60 ± 8.97	34.2	54.74 ± 10.54	36.5
	SRT–CLTR	36.65 ± 11.68	18.2	45.68 ± 14.42	23.9
	SpRT	21.58 ± 5.55	10.0	23.62 ± 4.10	15.0
	SDMT	59.88 ± 25.04	38.0	64.82 ± 14.45	44.65
	PASAT	35.11 ± 15.01	10.0	41.63 ± 11.30	23.8
	SRT–DR	9.52 ± 1.54	6.0	10.15 ± 1.54	6.95
	SpRT–DR	7.76 ± 1.98	3.0	8.50 ± 1.47	5.95
	WLG	17.41 ± 6.04	5.0	27.00 ± 6.58	16.75
31–45	SRT- LTS	48.04 ± 12.50	20.45	50.06 ± 11.01	31.5
	SRT–CLTR	34.77 ± 12.80	13.9	41.15 ± 14.39	19.25
	SpRT	20.00 ± 5.39	9.2	22.25 ± 3.92	15.2
	SDMT	46.57 ± 18.27	26.1	60.51 ± 16.30	38.6
	PASAT	33.38 ± 11.78	14.1	47.51 ± 8.61	29.0
	SRT–DR	7.95 ± 2.76	2.3	9.46 ± 1.80	6.0
	SpRT–DR	7.14 ± 2.61	1.2	7.53 ± 2.02	4.0
	WLG	23.23 ± 5.61	9.8	27.53 ± 6.13	20.0
46–66	SRT- LTS	41.63 ± 15.02	15.6	46.83 ± 10.67	28.75
	SRT–CLTR	29.53 ± 14.21	5.1	32.95 ± 10.64	18.25
	SpRT	17.58 ± 3.87	9.4	18.78 ± 5.96	8.2
	SDMT	36.63 ± 12.89	14.2	53.08 ± 14.60	34.6
	PASAT	29.43 ± 11.70	11.1	42.60 ± 10.74	15.2
	SRT–DR	8.19 ± 2.43	3.1	8.86 ± 1.68	6.2
	SpRT–DR	6.12 ± 1.55	4.0	6.86 ± 2.05	2.4
	WLG	19.43 ± 5.82	10.1	23.47 ± 6.14	16.0

BRB–N: brief repeatable battery of neuropsychological tests; CLTR: consistent long term retrieval; DR: delayed recall; LTS: long term storage; PASAT: paced auditory serial addition test; SDMT: symbol digit modalities test; SpRT: spatial recall test; SRT: selective reminding test; WLG: word list generation.

Healthy participants' raw scores in version A were converted to scaled scores ([Table 3]). These raw–to–scale–score conversions can be applied to MS patients. Scaled scores were regressed on age, sex and education yielding equations that can be used to calculate predicted scores ([Table 4]). Age was a significant predictor for all neuropsychological measures, except verbal fluency, while sex was the opposite.

Table 3 BRB–N raw scores to scaled score conversions.

Scaled score	Raw score
	SRT			SpRT		SDMT	PASAT	WLG
	LTS	CLTR	DR	Total	DR
1	< 6	1	1			< 12	< 8	< 5
2	06-09	2–3	2	< 8	1	12–13	8	5–6
3	10–20	4–6	3	8	2	14–16	9–11	7–9
4	21–22	7–13	4	9–10	3	17–22	12–14	10–11
5	23–26	14–18	5	11–12		23–27	15–18	12–13
6	27–33	19–21		13–14	4	28–35	19–23	14–15
7	34–39	22–25	6	15–16	5	36–39	24–26	16–17
8	40–43	26–29	7	17–18		40–45	27–31	18–19
9	44–48	30–33	8	19	6	46–50	32–37	20–21
10	49–52	34–38	9	20–22	7	51–55	38–43	22–23
11	53–56	39–44		23	8	56–62	44–46	24–26
12	57–60	45–50	10	24–25	9	63–68	47–51	27–30
13	61–62	51–57	11	26		69–75	52–55	31–32
14	63–64	58–60		27	10	76–88	56	33–34
15	65–66	61–64	12	28		89–101	57–58	35–36
16	67	65–67				102–103	59–60	37–38
17	68–69	68–69		29		104–110		39–40
18	70–72	70–72		30				41–42
19								> 42

BRB–N: brief repeatable battery of neuropsychological tests; CLTR: consistent long term retrieval; DR: delayed recall; LTS: long term storage; PASAT: paced auditory serial addition test; SDMT: symbol digit modalities test; SpRT: spatial recall test; SRT: selective reminding test; WLG: word list generation.

Table 4 Regression models for BRB–N measures and raw residuals standard deviation.

Measure	Predictors	B	t	Significance	CI95%		R square	SD residual
					Lower bound	Upper bound
SRT – LTS	(Constant)	10.355	10.064	< 0.001	8.32	12.38	0.146	2.885
	Sex	0.042	0.103	0.917	-0.76	-0.76
	Age	-0.064	-4.269	< 0.001	-0.09	-0.03
	Education	0.179	3.129	0.002	0.06	0.29
SRT – CLTR	(Constant)	9.953	10.138	< 0.001	8.01	11.88	0.194	2.753
	Sex	-0.410	-1.056	0.291	-1.17	0.35
	Age	-0.064	-4.470	< 0.001	-0.09	-0.03
	Education	0.227	4.153	< 0.001	0.11	0.33
SpRT	(Constant)	12.553	12.385	< 0.001	10.55	14.55	0.221	2.774
	Sex	0.665	1.681	0.094	-0.11	1.44
	Age	-0.098	-6.670	< 0.001	-0.12	-0.06
	Education	0.095	1.711	0.088	-0.01	0.20
SDMT	(Constant)	9.669	11.292	< 0.001	7.98	11.35	0.372	2.343
	Sex	0.092	0.275	0.783	-0.56	0.75
	Age	-0.090	-7.203	< 0.001	-0.11	-0.06
	Education	0.297	6.294	< 0.001	0.20	0.39
PASAT	(Constant)	7.047	7.187	< 0.001	5.11	8.98	0.197	2.684
	Sex	0.667	1.742	0.083	-0.08	1.42
	Age	-0.031	-2.201	0.028	-0.05	-0.003
	Education	0.314	5.811	< 0.001	0.20	0.42
SRT – DR	(Constant)	10.642	11.285	< 0.001	8.78	12.50	0.171	2.581
	Sex	-0.725	-1.967	0.050	-1.45	0.001
	Age	-0.051	-3.718	< 0.001	-0.07	-0.02
	Education	0.193	3.714	< 0.001	0.09	0.29
SpRT – DR	(Constant)	11.300	12.948	< 0.001	9.57	13.02	0.213	2.388
	Sex	0.128	0.376	0.706	-0.54	0.80
	Age	-0.073	-5.776	< 0.001	-0.09	-0.04
	Education	0.149	3.108	0.002	0.05	0.24
WLG	(Constant)	6.499	6.912	< 0.001	4.64	8.35	0.264	2.573
	Sex	-0.786	-2.141	0.033	-1.51	-0.06
	Age	-0.018	-2.141	0.178	-0.04	0.008
	Education	0.389	7.506	< 0.001	0.28	0.49

BRB–N: brief repeatable battery of neuropsychological tests; CLTR: consistent long term retrieval; DR: delayed recall; LTS: long term storage; PASAT: paced auditory serial addition test; SDMT: symbol digit modalities test; SpRT: spatial recall test; SRT: selective reminding test; WLG: word list generation.

DISCUSSION

Although a remarkable increase in cognitive–related MS research has been observed in the last years, there is still a considerable gap in applying this knowledge into clinical practice. This may be due to several reasons including absence of normative data, time restriction for the use of extensive neuropsychological batteries and lack of established therapeutic options to overcome cognitive deficits. The widely–used MMSE has a low sensitivity to detect MS cognitive dysfunction. Therefore, Rao et al. developed the BRB–N, a neuropsychological battery with better sensitivity and specificity[3],[4]. Accordingly, we aimed to include a fairly significant number of individuals in the present study, representative of areas in Brazil where MS is more prevalent and where most MS centers are located, and we were able to provide normative data for the version A of the battery.

The BRB–N has been widely used in MS research worldwide, including South America[7],[8],[9]. In Brazil, it was initially employed in the context of the CogniCIS, a multinational study evaluating cognitive performance in clinically isolated syndrome[18]. A further study by Brooks et al., has shown that the BRB–N is feasible for assessing cognition of MS patients in the daily clinic[19]. Moreover, this battery showed strong discriminating power between patients and controls in a Brazilian sample, insofar as large effect sizes were observed[20]. However, even though this battery had already been validated in the Portuguese language, Brazilian normative values were not yet available for these studies and analysis had to rely on adequate control groups[6]. Nevertheless, small control groups have potential limitations and one cannot adequately match all possible demographic variables.

We aimed to provide both discrete and continuous normative data. Discrete norms have some limitations and have been subject to criticism over the last years. For example, Morgan et al. have shown that the use of demographically–adjusted T–scores significantly improved sensitivity for discriminating impaired versus neurocognitively normal individuals in comparison to the raw cut score[21]. Similarly, Parmenter et al. demonstrated that using a continuous norm approach, higher rates of impairment were identified compared to standard norms for many of the minimal assessment of cognitive function in multiple sclerosis measures[17], a neuropsychological battery developed with strong psychometrics properties for use in MS research, but with a considerably longer administration time compared to the BRB–N[1],[22]. Recently, the shorter Brief International Cognitive Assessment for Multiple Sclerosis battery was validated in Brazil, but neither continuous nor discrete norms were available in our population[10]. Several authors have advocated the use of continuous norms based on multiple regression analysis accounting for demographic factors[11],[16],[17]. However, whether continuous norms are the most appropriate for answering all research and clinical questions is still a matter of discussion. For example, Silverberg and Millis showed that discrete norms may have worked better in determining whether the patients' cognitive abilities were sufficient for the demands of universal functional tasks, such as activities of daily living[23].

Although, in our sample, raw scores obtained in all BRB–N measures were not normally distributed, we opted to also provide discrete norms stratified by age and education for ease of clinical use on a daily basis and to address selected questions. However, use of these norms should be made with the awareness of potential limitations, as discussed above and illustrated in the following case.

For example, consider a 32–year–old female MS patient with 11 years of education who scores 30 in the SDMT. According to discrete norms in [Table 2], her score is within her age group mean–1 SD, and thus could be considered normal. However, her raw score corresponds to a scaled score of 6, according to [Table 3] and her predicted scaled score on the SDMT is 10.15, based on the regression equation expressed in [Table 4] [9.669 + 1(0.092) + 32(−0.090) + 11(0.297)]. We then divide the difference between her actual and predicted scaled score (6−10.15 =−4.15) by the standard deviation of the residual (2.343), obtaining a z score of–1.77. This value equals a T score of 32 and is considered impaired using an operational definition of “impairment” of either 35 or 40, as suggested by some authors[11],[17].

Regarding the BRB–N version B scores, we found them slightly higher only for the verbal (total learning) and visuospatial memory (delayed recall) tests when compared to version A and these findings were not attributable to gender, education or age differences. Although our sample size for the BRB–N version B test was relatively small, it is interesting that previous studies have found similar results for the SpRT (delayed recall) but not for the verbal memory test[7],[8]. For example, Amato et al. have found higher scores in all BRB–N version B subtests except in the SRT total learning (CLTR and LTS)[8]. Nevertheless, in order to be used longitudinally with reliable results, both versions should be equivalent. Thus, even though the higher scores of the SRT version B may have been influenced by its smaller sample size, we need further studies comparing SRT versions A and B with equivalent words regarding length, frequency of use, and category membership. Subsequently, further research with more individuals performing version B will be able to address whether this version is suitable for longitudinal usage of the battery.

Also in line with previous results, we found that demographic characteristics, and especially age and education, were significantly related to all neuropsychological measures, except the Paced Auditory Serial Addition Test, where there was a trend for only sex and age[7],[8]. Sex was only associated with the Word List Generation score. These findings reinforce the need for adjusting raw scores for demographic variables when interpreting neuropsychological test scores.

In conclusion, our normative data allow a more widespread use of the BRB–N in clinical practice and research, providing both discrete and continuous norms. However, discrete norms should be used with caution, and demographically–adjusted scores are generally preferable when interpreting neuropsychological test scores.

• References

• 1 Benedict RH, Zivadinov R. Risk factors for and management of cognitive dysfunction in multiple sclerosis. Nat Rev Neurol. 2011 May;7(6):332-42. https://doi.org/10.1038/nrneurol.2011.61
• 2 D'Anna L, Lorenzut S, Perelli A, Zanchettin B, Valente M, Gigli GL. The contribution of assessing cognitive impairment in radiologicallyisolated syndrome (RIS): a single case report follow-up study. Mult Scler. 2014 Dec;20(14):1912-5. https://doi.org/10.1177/1352458514523693
• 3 Beatty WW, Goodkin DE. Screening for cognitive impairment in multiple sclerosis. An evaluation of the Mini-Mental State Examination. Arch Neurol. 1990 Mar;47(3):297-301. https://doi.org/10.1001/archneur.1990.00530030069018
• 4 Rao SM, Leo GJ, Bernardin L, Unverzagt F. Cognitive dysfunction in multiple sclerosis. I. Frequency, patterns, and prediction. Neurology. 1991 May;41(5):685-91. https://doi.org/10.1212/WNL.41.5.685
• 5 Langdon DW, Amato MP, Boringa J, Brochet B, Foley F, Fredrikson S et al. Recommendations for a Brief International Cognitive Assessment for Multiple Sclerosis (BICAMS). Mult Scler. 2012 Jun;18(6):891-8. https://doi.org/10.1177/1352458511431076
• 6 Neves MR, Passos AM, Ferreira A, Sousa C, Sá A, Sá MJ. BRBN-T validation: adaptation of the Selective Reminding Test and Word List Generation. Arq Neuropsiquiatr. 2015 0ct;73(10):867-72. https://doi.org/10.1590/0004-282X20150134
• 7 Boringa JB, Lazeron RH, Reuling IE, Adèr HJ, Pfennings L, Lindeboom J et al. The brief repeatable battery of neuropsychological tests: normative values allow application in multiple sclerosis clinical practice. Mult Scler. 2001 Aug;7(4):263-7. https://doi.org/10.1191/135245801680209385
• 8 Amato MP, Portaccio E, Goretti B, Zipoli V, Ricchiuti L, De Caro MF et al. The Rao's Brief Repeatable Battery and Stroop Test: normative values with age, education and gender corrections in an Italian population. Mult Scler. 2006 Dec;12(6):787-93. https://doi.org/10.1177/1352458506070933
• 9 Cáceres F, Vanotti S, Rao S, Workgroup RE. Epidemiological characteristics of cognitive impairment of multiple sclerosis patients in a Latin American country. J Clin Exp Neuropsychol. 2011 Dec;33(10):1094-8. https://doi.org/10.1080/13803395.2011.603690
• 10 Spedo CT, Frndak SE, Marques VD, Foss MP, Pereira DA, Carvalho LF et al. Cross-cultural Adaptation, Reliability, and Validity of the BICAMS in Brazil. Clin Neuropsychol. 2015;29(6):836-46. https://doi.org/10.1080/13854046.2015.1093173
• 11 Casaletto KB, Umlauf A, Beaumont J, Gershon R, Slotkin J, Akshoomoff N et al. Demographically Corrected Normative Standards for the English Version of the NIH Toolbox Cognition Battery. J Int Neuropsychol Soc. 2015 May;21(5):378-91. https://doi.org/10.1017/S1355617715000351
• 12 Buschke H, Fuld PA. Evaluating storage, retention, and retrieval in disordered memory and learning. Neurology. 1974 Nov;24(11):1019-25. https://doi.org/10.1212/WNL.24.11.1019
• 13 Rao SM. A manual for the brief repeatable battery of neuropsychological tests in multiple sclerosis. New York: National Multiple Sclerosis Society; 1990.
• 14 Smith A. Symbol Digit Modalities Test (SDMT): manual (revised). Los Angeles: Western Psychological Services; 1982.
• 15 Gronwall DM. Paced auditory serial-addition task: a measure of recovery from concussion. Percept Mot Skills. 1977 Apr;44(2):367-73. https://doi.org/10.2466/pms.1977.44.2.367PMID:866038
• 16 Testa SM, Winicki JM, Pearlson GD, Gordon B, Schretlen DJ. Accounting for estimated IQ in neuropsychological test performance with regression-based techniques. J Int Neuropsychol Soc. 2009 Nov;15(6):1012-22. https://doi.org/10.1017/S1355617709990713
• 17 Parmenter BA, Testa SM, Schretlen DJ, Weinstock-Guttman B, Benedict RH. The utility of regression-based norms in interpreting the minimal assessment of cognitive function in multiple sclerosis (MACFIMS). J Int Neuropsychol Soc. 2010 Jan;16(1):6-16. https://doi.org/10.1017/S1355617709990750
• 18 Fredrikson S, Wicklein EM, Prayoonwiwat N, Beckmann K, Scherer P, Langdon D. Two-year data from CogniCIS, a multinational longitudinal study on cognition and patient-reported outcomes parameters in clinically isolated syndrome suggestive of multiple sclerosis: demographic and neurological characteristics of the study cohort. Mult Scler. 2010;16 suppl 10:S208.
• 19 Brooks JB, Borela MC, Fragoso YD. Assessment of cognition using the Rao's Brief Repeatable Battery of Neuropsychological Tests on a group of Brazilian patients with multiple sclerosis. Arq Neuropsiquiatr. 2011 Dec;69(6):887-91. https://doi.org/10.1590/S0004-282X2011000700007
• 20 Damasceno A, Damasceno BP, Cendes F. Subclinical MRI disease activity influences cognitive performance in MS patients. Mult Scler Relat Disord. 2015 Mar;4(2):137-43. https://doi.org/10.1016/j.msard.2015.01.006
• 21 Morgan EE, Woods SP, Scott JC, Childers M, Beck JM, Ellis RJ et al. Predictive validity of demographically adjusted normative standards for the HIV Dementia Scale. J Clin Exp Neuropsychol. 2008 Jan;30(1):83-90. https://doi.org/10.1080/13803390701233865
• 22 Strober L, Englert J, Munschauer F, Weinstock-Guttman B, Rao S, Benedict RH. Sensitivity of conventional memory tests in multiple sclerosis: comparing the Rao Brief Repeatable Neuropsychological Battery and the Minimal Assessment of Cognitive Function in MS. Mult Scler. 2009 Sep;15(9):1077-84. https://doi.org/10.1177/1352458509106615
• 23 Silverberg ND, Millis SR. Impairment versus deficiency in neuropsychological assessment: implications for ecological validity. J Int Neuropsychol Soc. 2009 Jan;15(1):94-102. https://doi.org/10.1017/S1355617708090139