Critical reflections about the three 'r', or the barred Brazilian papers

CLINICS has recently proposed to debate the idea that QUALIS 2010 was likely to be perfected by the three-r concept (Remove review papers, Recognize other methods of evaluation, Reevaluate Brazilian papers).[1] Submitted to debate in pairs, on the occasion of the II Satellite Seminar for Full Editors (ABEC, November 2010), it has been concluded that it would rather be focused on only one 'r', namely have SCImago (cites/document) and SciELO (Impact Factor) evaluation methods recognized by CAPES for 2013.

We have previously demonstrated that ISI THOMSON Impact Factor, the only method recognized by CAPES, is virtually identical to SCImago cites/documents. The general correlation by sampling between the two indexes is bigger than 0.9 and the angular coefficient cannot be differentiated into the unit. SCImago Journal & Country 2009 Rank's[2] 2009 collection's recent divulgation now joins those already divulged JCR-ISI and SciELO Impact Factors for that year and allows the three indexes to be collected in real time. We emphasize that such a collate is not applied to any QUALIS table, since 2010 table is history and 2013 Table shall be decided upon in the future. In short, the collate hereinafter related to the year 2009 is offered on an argumentative basis proper to demand course correction regarding the exclusion of SCImago and SciELO indexes.

Let us move to the 2009 scenario as a simulation of a hypothetical Qualis table: In the Journal of Citations Report – ISI's 2009 version, the Brazilian representation jumped from 31 to 71 papers; however, in SCImago Journal Ranking, this representation remained invariable in 235 papers.

Copyright 2011 CLINICS - This is an Open Access article distributed under the terms of the Creative Commons Attribution Non-Commercial License (http://creativecommons.org/licenses/by-nc/3.0/) which permits unrestricted non commercial use, distribution, and reproduction in any medium, provided the original work is properly cited.

There are, accordingly, 164 Brazilian papers (out of which 138 have an impact above zero) not present in the JCR-ISI table. Had Qualis rule been maintained in 2010, these 138 papers would fall into 'no impact factor' categories. [Figure 1] shows the identity between ISI and SCImago for the 64 magazines included in both indexes: the angular unit coefficient and the high coefficient of correlation mean that, if you know one of the indexes, you can estimate the other one with a 95% probability against a chance of error under 5%.

Figure 1. Correlation between impact factors – J. Citation Reports (151 — Thomson) and SCImago (2009) for 64 Brazilian papers represented in both indexes with values above zero. The identity between measures is nearly absolute. The angular coefficient indicates a 45o angle (what means identity) and the correlation (R² = 0,932; p <0.001) is almost perfect.

[Table 1] shows the relation of theses 138 Brazilian papers with Cites/Documents SCImago > zero, but without a JCR-ISI impact factor. It should be emphasized that they are not papers with an impact clearly lower than those of JCR-ISI collection. The first four ones show an impact > 1.00, what would rank them as the best 15 in Brazil. Other 10 show an impact above 0.50, above the average of the Brazilian papers in JCR-ISI. All 138 titles would unquestionably deserve the 'impact factor' classification.

Table 1 One hundred and thirty-eight Brazilian papers excluded from QUALIS with SCIMAGO Impact Factor (simulation 2009).

Rank	Title	cites/doc scimago
1	Acta Scientiarum – Agronomy	1,19
2	International Braz J Urol	1,09
3	Revista do Instituto de Medicina Tropical de Sao Paulo	1,08
4	Brazilian Journal of Plant Physiology	1,05
5	Annual Review of Biomedical Sciences	0,85
6	Jornal Brasileiro de Pneumologia	0,85
7	Revista Brasileira de Epidemiologia	0,73
8	Brazilian Journal of Cardiovascular Surgery	0,72
9	Acta Ortopedica Brasileira	0,71
10	Brazilian Journal of Biology	0,69
11	Brazilian Dental Journal	0,67
12	Ciencia e Saude Coletiva	0,58
13	Materials Research	0,55
14	Brazilian Oral Research	0,53
15	Arquivos de Gastroenterologia	0,49
16	Revista Brasileira de Saude Materno Infantil	0,49
17	Acta Botanica Brasilica	0,48
18	Phyllomedusa	0,48
19	Acta Scientiarum - Biological Sciences	0,46
20	Revista Brasileira de Medicina do Esporte	0,46
21	Revista Brasileira de Botanica	0,45
22	PRO-FONO: Revista de Actualização Cientifica	0,44
23	Revista Brasileira de Engenharia Agricola e Ambiental	0,41
24	Engenharia Agricola	0,40
25	Arquivos Brasileiros de Oftalmologia	0,38
26	Biota Neotropica	0,38
27	Radiologia Brasileira	0,38
28	Revista Brasileira de Otorrinolaringologia (English ed.)	0,38
29	Revista Brasileira de Sementes	0,38
30	Acta Amazonica	0,37
31	Ciencia Rural	0,36
32	Arquivo Brasileiro de Medicina Veterinaria e Zootecnia	0,35
33	Pan-American Journal of Aquatic Sciences	0,35
34	Revista Brasileira de Ginecologia e Obstetricia	0,35
35	Acta Paulista de Enfermagem	0,34
36	Ecletica Quimica	0,34
37	Papeis Avulsos de Zoologia	0,33
38	Revista Brasileira de Plantas Medicinais	0,31
39	Bragantia	0,29
40	Revista Arvore	0,29
41	Revista Brasileira de Anestesiologia	0,29
42	Revista de Psiquiatria Clinica	0,29
43	Jornal Brasileiro de Psiquiatria	0,28
44	Journal of Microwaves and Optoelectronics	0,27
45	Pesquisa Operacional	0,27
46	Scientia Forestalis/Forest Sciences	0,27
47	Revista Brasileira de Ciencia Avicola	0,26
48	Revista Brasileira de Hematologia e Hemoterapia	0,26
49	Revista de Economia Politica	0,26
50	Ceramica	0,25
51	Estudos Avancados	0,25
52	Interface: Comunicação, Saude, Educação	0,25
53	Online Brazilian Journal of Nursing	0,24
54	Boletim de Geociencias - Petrobras	0,23
55	HAHR - Hispanic American Historical Review	0,23
56	Opiniao Publica	0,23
57	Revista Brasileira de Ciencias Farmaceuticas	0,23
58	Cadernos de Pesquisa	0,22
59	Anais Brasileiros de Dermatologia	0,21
60	Journal of Public Child Welfare	0,21
61	Lundiana	0,21
62	Revista de Ciencias Farmaceuticas Basica e Aplicada	0,21
63	Tropical Plant Pathology	0,21
64	Jornal Vascular Brasileiro	0,20
65	Revista Brasileira de Enfermagem	0,20
66	Jornal Brasileiro de Patologia e Medicina Laboratorial	0,19
67	Historia, Ciencias, Saude - Manguinhos	0,18
68	Revista de Psiquiatria do Rio Grande do Sul	0,18
69	Geociencias	0,17
70	Revista Brasileira de Reumatologia	0,17
71	Ararajuba	0,16
72	Revista Brasileira de Cardiologia Invasiva	0,16
73	Revista Brasileira de Economia	0,16
74	Revista Brasileira de Geofisica	0,16
75	Boletim de Ciencias Geodesicas	0,15
76	Controle E Automação	0,15
77	Geologia USP - Serie Cientifica	0,15
78	Economia Aplicada	0,14
79	Engenharia Sanitaria e Ambiental	0,14
80	IRRIGA	0,14
81	Psicologia e Sociedade	0,14
82	Revista do Colegio Brasileiro de Cirurgioes	0,14
83	Revista Gaucha de Enfermagem / EENFUFRGS	0,14
84	Educação e Pesquisa	0,13
85	Lua Nova - Revista de Cultura e Politica	0,13
86	Produção	0,13
87	Psicologia: Teoria e Pesquisa	0,13
88	Revista de Economia e Sociologia Rural	0,13
89	Educação e Sociedade	0,12
90	Physis: Revista de Saude Coletiva	0,12
91	Revista Brasileira de Ciencias Sociais	0,12
92	Sociologias	0,12
93	Ambiente & Sociedade	0,11
94	Brazilian Journal of Oral Sciences	0,11
95	Gestão e Produção	0,11
96	Journal of Epilepsy and Clinical Neurophysiology	0,11
97	Psicologia em Estudo	0,11
98	Soils and Rocks	0,11
99	Archives of Veterinary Science	0,10
100	Estudos Feministas	0,10
101	Jornal Brasileiro de Reprodução Assistida	0,10
102	Psiquiatria Biologica	0,10
103	Anuario do Instituto de Geociencias	0,09
104	Acta Scientiarum - Health Sciences	0,08
105	Coluna/ Columna	0,08
106	GED - Gastrenterologia Endoscopia Digestiva	0,08
107	Revista de Administração Publica	0,08
108	Revista Dental Press de Ortodontia e Ortopedia Facial	0,08
109	Saude e Sociedade	0,08
110	Summa Phytopathologica	0,08
111	Estudos de Psicologia (Campinas)	0,07
112	Cadernos CEDES	0,06
113	Revista Brasileira de Coloproctologia	0,06
114	Revista Brasileira de Educação	0,06
115	Revista Brasileira de Estudos de População	0,06
116	Revista Brasileira de Oftalmologia	0,06
117	Revista de Sociologia e Politica	0,06
118	Ensaio	0,05
119	Estudos Ibero-Americanos	0,05
120	Novos Estudos CEBRAP	0,05
121	Perspectivas em Ciencia da Informação	0,05
122	Revista Brasileira de Educação Especial	0,05
123	Revista Brasileira de Medicina	0,05
124	Revista de Economia Contemporanea	0,05
125	Alea	0,04
126	Horizontes Antropologicos	0,04
127	International Journal of Atherosclerosis	0,04
128	Revista Brasileira de Historia	0,04
129	Trans/Form/Ação	0,04
130	Cadernos Pagu	0,03
131	Historia	0,03
132	Medicina	0,03
133	Psicologia Clinica	0,03
134	Ciencia da Informação	0,02
135	Revista Latinoamericana de Psicopatologia Fundamental	0,02
136	Revista Neurociencias	0,02
137	Sociedade e Estado	0,02
138	Revista Brasileira de Gestao e Desenvolvimento Regional	0,01

Another interesting correlation occurs between SCImago and SciELO. Firstly, it should be noted that there is a broad compliance: SCImago collection contains 235 Brazilian titles and SciElo collection has 223. Inclusions are not 100% compliant: SCImago contains 69 papers not included in SciElo collection; SciElo collection, in turn, contains 56 papers not included in the SCImago collection. This convergence is enough to reveal the quality uniformity of the papers included in SciELO collection.

It is easy to assume that, for any Brazilian paper included in both collections, it must be expected that SCImago impact is higher than SCIELO because SCImago collection has 18,732 papers and there are only 759 papers in SciELO collection. Surprisingly, however, the effect of this huge disproportion between databases is smaller than expected, as you can see from the [figure 2]: out of the 142 Brazilian papers included in the two collections, only 88 (62%) show SciELO > SCImago and nine are equal. This inconsistency between what is expected and what is observed deserves further bibliometric study, but a reasonable hypothesis would be that the Brazilian articles cite other Brazilian articles more intensely by reasons of a deeper concern about a specific location. The correlation between the impacts ([Figure 3]) is similarly instructive: The angular coefficient (0.54) suggests that the average SciELO impact is only 40% lower than SCImago impact. The high ratio of correlation (r = 0.62; p <0.01), however, shows that the two measures evaluate the same parameter in very different databases.

Figure 2. SCImago and SciELO (2009) impact factors for 142 Brazilian papers represented in both indexes with values above zero. It is observed that when SCImago > SciELO, the difference is often big (as indicated by the vertical arrow related to Oswaldo Cruz Memorial Institute), but it is invariably reduced when SciELO > SCImago.

Figure 3. Correlation between SciELO and SCImago (2009) impact factors for 142 Brazilian papers represented in both indexes with values above zero. The angular coefficient indicates a 28° angle and a medium SciELO impact 40% lower than SCImago. The significant correlation indicates that both parameters have the same measure in different databases.

In case CAPES recognizes Cites/Document SCImago, the majority of the Brazilian papers with an impact > 0 shall be rescued. [Table 2], nevertheless, shows that, were the decision made at this moment, there would be 35 Brazilian papers left with a SciELO impact above zero but not included in JCR-ISI and SCImago. Here we also found minor impacts: four papers showed impacts above the average ISI collection. Not to mention the strong possibility that these 35 SciELO impacts shall overestimate what ISI or SCImago impacts would be.

Table 2 – Thirty-five Brazilian papers excluded from QUALIS with SciELO Impact Factor (simulation 2009).

Rank	Title	cites/doc scimago
1	Revista da Sociedade Brasileira de Fonoaudiologia	0,81
2	Texto e Contexto Enfermagem	0,65
3	Revista CEFAC	0,64
4	Revista Brasileira de Educação Mé dica	0,58
5	Ciê ncia e Agrotecnologia	0,42
6	Avaliação: Revista da Avaliação da Educação Superior	0,28
7	Revista Paulista de Pediatria	0,28
8	Paidé ia (Ribeirão Preto)	0,27
9	RAE Electronica	0,21
10	Revista Brasileira de Terapia Intensiva	0,21
11	Psicologia & Sociedade	0,15
12	Revista Estudos Feministas	0,15
13	Economia e Sociedade	0,14
14	Revista Brasileira de Meteorologia	0,14
15	Ciê ncia da Informação	0,12
16	Psicologia Escolar e Educacional (Impresso)	0,12
17	Educação em Revista	0,11
18	Revista de Administração Contemporânea	0,11
19	Revista Brasileira de Ortopedia	0,10
20	Varia Historia	0,09
21	Maté ria (Rio de Janeiro)	0,08
22	Psicologia USP	0,08
23	Caderno CRH	0,07
24	DELTA Documentacao de Estudos em Linguistica Teorica e Aplicada	0,07
25	Ciê ncia & Educação (Bauru)	0,06
26	Nova Economia	0,06
27	Religião & Sociedade	0,06
28	Escola Anna Nery	0,05
29	Estudos Economicos	0,05
30	Revista Contabilidade & Finanças	0,05
31	Sba: Controle & Automação Sociedade Brasileira de Automatica	0,05
32	Contexto Internacional	0,04
33	Educar em Revista	0,04
34	Estudos de Psicologia (Natal)	0,03
35	Interaçõ es (Campo Grande)	0,03

From this simulation, we can conclude that we would have 173 Brazilian papers with IMPACT > ZERO regarded as 'WITH NO IMPACT' by QUALIS, in case the evaluation was made now and the 2010 criteria were repeated. We know that this 'simulation of exclusion' is not static: when Qualis table 'closes' for the next evaluation, a plenty of things shall have changed: some of them would surely be ISI-JCR, SCImago and SciELO, tables as well as (hopefully!) CAPES criteria of evaluation. But we stand that recognizing only ISIJCR Impact factor would be illogical. Hence, we understand that this simulation is the rational basis of an in-time warning to adopt new criteria.

NOTE: this editorial is published by CLINICS and it is exempt from copyright restrictions. We offer them to the Brazilian scientific papers for full or partial reproduction. We alternatively suggest that they support this idea in original editorials. Such actions shall express our politic will of exerting the republican right of petitioning before the public entity to defend what is deemed to be a legitimate and generalized demand of the Brazilian scientific editorial community.

• References

• 1 Rocha e Silva M. Qualis 2011-2013: os três erres. Clinics 2010; 65: 935-6
• 2 SCImago. (2007). SJR - SCImago Journal & Country Rank. Retrieved January 26, 2011, from http://www.scimagojr.com