Methodological Quality Analysis of Systematic Review for the Treatment of Rotator Cuff Disease^[∗]

Abstract

Objective To evaluate the methodological quality of systematic reviews for the surgical and nonsurgical treatment of individuals with rotator cuff syndrome; to compare, through the Assessing the methodological quality of systematic reviews(AMSTAR) instrument, the quality of studies found in the Cochrane Library, PubMed (Publisher Medline), EMBASE andQinsightdatabases.

Methods This is a descriptive and comparative cross-sectional study, in which two independent authors analyzed, through the AMSTAR instrument, the methodological quality of Cochrane and non-Cochrane systematic reviews on the treatment of individuals diagnosed with rotator cuff syndrome.

Results A total of 76 systematic reviews were evaluated by the AMSTAR instrument. The overall mean score was 6.1 (±2.1) and the mean per database was 9.1 (±0.9) for the Cochrane reviews and 5.7 (±1.8) for the non-Cochrane reviews. The lowest-scoring item of AMSTAR was 11, related to the display of the conflict of interests of the publication. In a comparative analysis of the final variable score, there was a statistical difference between the Cochrane and non-Cochrane studies.

Conclusion According to the present study, systematic reviews using the Cochrane methodology have a better methodological quality compared to non-Cochrane studies on the treatment of rotator cuff dysfunctions.

Keywords

rotator cuff - methodology - systematic review

Introduction

Systematic reviews of randomized clinical trials present the highest level of scientific evidence for clinical decision-making.[1] [2] The aim of these studies is to clarify the divergences found in the literature, so that it is possible to answer a specific question and synthesize the findings of primary studies. A good methodological design of these studies is essential so that more assertive interventions become possible.[1] [2] Currently, ∼ 24 instruments are validated and used to determine the level of reliability of scientific studies, such as Preferred reporting items for systematic reviews and meta-analyses (PRISMA) and Assessing the Methodological Quality of Systematic Reviews (AMSTAR).[3] The AMSTAR is a validated measurement tool that assesses the methodological quality of systematic reviews.[3] It has 11 domains that evaluate the methods of construction of systematic reviews.[4] [5]

The scientific literature indicates numerous forms of treatment for the various types of injuries that affect the musculoskeletal system, such as rotator cuff dysfunctions. Thus, it is important to select works with good foundation, high methodological rigor and reliable sources of information for greater efficacy at the time of the therapeutic approach.[6] [7] Evidence indicates that systematic reviews using the methodology proposed by the Cochrane collaboration have a greater methodological rigor when compared to studies that do not adopt this methodology.[8] [9] Thus, the aim of the present research was to evaluate the methodological quality of Cochrane and non-Cochrane systematic reviews on the treatment of individuals with rotator cuff dysfunctions and to compare, through AMSTAR, the quality of studies found in the Cochrane, PubMed (Publisher: Medline), EMBASE and Qinsight databases

Methods

This is a descriptive and comparative cross-sectional investigative study.

Only systematic reviews were included, with Cochrane and non-Cochrane methodology, involving any type of primary studies. The term "PICOS" (population or problem/intervention/control/outcome/study design) was used as a basis. There was no restriction on the language and date of publication of the studies.

Systematic reviews were included in which the research subjects were individuals> 16 years old, diagnosed with rotator cuff syndrome (impact syndrome, subacromial bursitis, tendinosis or tendinopathy of cuff structures rotator, partial or total rupture of rotator cuff and tendinosis structures or calcifying tendinopathy of rotator cuff structures) regardless of the time of disease evolution, treated non surgically or surgically. Only studies that did not have diagnostic confirmation of the disease were excluded.

The search was performed by two authors using the official Medical Subject Heading (MESH) terms– (rotatorcuff, shoulder impingement syndrome, shoulder joint) in the following databases: Cochrane Library, PubMed, EMBASE and Qinsight. There were no restrictions on the language and date of publications during the search process. Search strategies were translated for eachdatabase. The studies were selected by a single evaluator respecting the inclusion criteria described. Initially, after searching all databases, duplicates were removed, then the articles were analyzed based on their titles and abstracts; when the information contained in these two items was not sufficient, it was left for the analysis of the text in full. After these steps, with the final number of studies selected, a download was performed for follow-up with the evaluations with AMSTAR.

Two independent reviewers (Kriebel C. F. and Estevam J. A.) evaluated the methodological quality of the systematic reviews selected using the AMSTAR as an instrument.[3] [5] This tool has 11 domains that investigate the following aspects:

• 1) Presence of a previous protocol with ethical approval and predetermined research objectives

• 2) Extraction and evaluation of the material performed by two independent evaluators

• 3) Comprehensive bibliographic search in at least two databases

• 4) Inclusion of gray literature as a criterion

• 5) Presence of list or references of included and excluded studies

• 6) Presence of list or references of the studies included in the research

• 7) Methodological evaluation of each included study and documentation of the results obtained

• 8) Critical formulation of conclusions based on methodological analysis

• 9) Evaluation of the heterogeneity of each study

• 10) Evaluation of publication bias with the presence of available graphs or tests

• 11) Mention of the conflicts of interests of the publication and the studies included.

Each question has four answer options, which are: 1) Yes; 2) No; 3) Cannot answer and 4) Does not apply. For the calculation of the final score, only the positive answers (Yes) counted, assigning 1 point for each positive answer of the questionnaire.[3] [5]

The information regarding the systematic reviews included in the present study was recorded in Excel software (Microsoft Corporation, Redmond, WA, USA) for the creation of the database. The collected material was analyzed in software R version 3.4.0 (RFoundation for Statistical Computing, Vienna, Austria) Variance analysis with a fixed factor and tukey fixed comparison method were used. A significance number was used for the present study where p< 0.05. Excel software tools[6] we reapplied to measure some variables such as mean, standard deviation and percentage of grades obtained by studies after evaluation with AMSTAR.

The present study was submitted to the research ethics committee of the Universidade Federal de São Paulo, with approval on 07/16/2017 - CEUAxnumber5960140717

Results

A total of 76systematic reviews were analyzed in different databases, namely: 9 Cochrane systematic reviews that used the methodology proposed by the Cochrane organization and published in the Cochrane Library, 26 systematic reviews not indexed in the Cochrane database but which described that their methods follow the Cochrane methodology, 6 reviews indexed in Embase, 26 reviews indexed in Pubmed and 10 reviews indexed in Qinsite.

After statistical analysis, it was observed that the mean score for the 76 studies was 6.1 (±2.1), with 9.1 (±0.9) being the mean of the studies with Cochrane methodology and indexed in the Cochrane database, and 5.7 (±1.8) for the other studies.

[Figure 1] presents the analysis of the methodological quality of these systematic reviews, the percentage of scores obtained by the studies from the analysis made with AMSTAR.

Fig. 1 Percentage of studies that received scores from the Amstar instrument evaluation.

Data on AMSTAR domains with the lowest scores among all papers are shown in [Figure 2].

Fig. 2 Items with the lowest overall score according to the Amstar instrument.

[Figure 3] shows the average of the scores obtained by the studies after applying the AMSTAR instrument.

Fig. 3 Average profiles of the studies, according to the final score obtained by the Amstar instrument.

To compare the quality of systematic reviews in relation to the final scores, the variance analysis model with a fixed factor and tukey multiple comparison method were used. These data are described in [Table 1].

Table 1

Comparisons		Descriptive level
RNI	Cochrane	0.001
Embase	Cochrane	0.014
Pubmed	Cochrane	0.001
Qinsite	Cochrane	0.001
Embase	RNI	0.999
Pubmed	RNI	0.494
Qinsite	RNI	0.672
Pubmed	Embase	0.943
Qinsite	Embase	0.943
Qinsite	Pubmed	0.999

Discussion

Systematic reviews evaluating randomized clinical trials have the best level of scientific evidence. They are research models that offer individuals a synthesis of data, in order to base, theoretically, clinical practices, and guide the construction of new projects; for this, it is important that this material is clear, explicit and reproducible; in addition, to add value to the instrument, primary studies must have high methodological rigor. Only in this way will clinical decision-making be possible based on scientific evidence.[10] [11]

The present study evaluated 76 systematic reviews on the treatment for rotator cuff dysfunctions, selected in different databases. There are consistent data that demonstrate a higher methodological quality of studies using the methodology proposed by the Cochrane collaboration for systematic reviews in comparison with studies that do not follow this methodology, a finding that corroborates a study that compared, using this same instrument, the quality of Cochrane and non-Cochrane systematic reviews related to health interventions.[12]

Some challenges were encountered when evaluating systematic reviews. It is noted that non-Cochrane methodology reviews have limited information, insufficient or missing data, which makes it difficult for AMSTAR to determine points, such as quality analysis of all included primary studies, design of the writing based on an initial protocol, publication bias, list of papers that were excluded from the systematic review, a comprehensive search in the literature and analysis of conflicts of interests of publications.

In the scientific literature there are studies that stimulate discussions about the domains of AMSTAR.[10] [11] The description of each item is thorough and allows the reader to understand the importance of analyzing each topic.

Item 11 of the AMSTAR checklist refers to the declaration of conflict of interests; the authors must clearly and objectively expose any source of funding or support for the research; for scoring, reports on all possible sources of funding should be required because they minimize a potential influence and judgments. In this regard, all the studies evaluated in the present study brought inconsistent information, thus it is suggested a better clarification of this item to allow deeper evaluations. The second item with the lowest score refers to the requirement of an initial research protocol, which would help to delimit the studies that were included and excluded and the quality of these. This domain is important because it identifies possible deviations in the protocol, reducing the risks of publication bias. All studies with Cochrane methodology for systematic reviews use an a priori protocol, but the non-Cochrane studies did not specify the use of this in their reviews, compromising the result of the evaluation. Increasing the search for unpublished literature, published or not, enriches the work. Many authors did not include in their reviews the gray literature (type of unconventional publication); some publications emphasize that more comprehensive studies allow a general analysis on a given theme.[10]

In general, the reviews with Cochrane methodology positively meet the criteria established in AMSTAR, so the quality of these reviews is significantly better compared with works extracted from Pubmed, Qinsite and Embase databases. Similar conclusions were published when evaluating the methodological rigor of Cochrane and non-Cochrane systematic reviews on oral health treatments.[13]

Evidence-based clinical decision-making necessarily depends on the quality of reviews and a high methodological rigor of these publications.[14] In thepresentstudy, it is noted that many systematic reviews do not meet the established methodological standards, thus producing unreliable evidence. The eligibility criteria should be clearly defined, and the methodology should include data on the study population, details of interventions, evaluation methods and the challenges encountered in implementation, so that reproduction and clinical applicability of the conducts are possible.

Study Limitations

The present study used the first version of the Amstar instrument (2007), because the selection and analysis of the studies were published before the publication of Amstar 2.

Conclusion

Based on the results of the present study, it is possible to conclude that systematic reviews with Cochrane methodology have a better scientific quality than the ones with non-Cochrane methodologies. It is necessary to increase the rigor in scientific publications, so that evidence-based clinical practices are better conducted. Further studies are needed to stimulate discussion about the quality of studies published in various databases.

Conflito de Interesses

Os autores declaram não haver conflito de interesses.

Financial Support

There was no financial support from public, commercial, or non-profit sources.

^∗ Work developed at the Universidade Federal de São Paulo, Santos, SP, Brazil.

• Referências

• 1 Pereira MG, Galvão TF. Revisões sistemáticas da literatura: Passos para sua elaboração. Epidemiol Serv Saude 2014; 23 (01) 183-184
• 2 Rother ET. Revisão sistemática X revisão narrativa. Acta Paul Enferm 2007; 20 (02) 1-2
• 3 Shea BJ, Hamel C, Wells GA. et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009; 62 (10) 1013-1020
• 4 Kelly SE, Moher D, Clifford TJ. Quality of conduct and reporting in rapid reviews: an exploration of compliance with PRISMA and AMSTAR guidelines. Syst Rev 2016; 5: 79
• 5 Shea BJ, Bouter LM, Peterson J. et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One 2007; 2 (12) e1350
• 6 Macedo CR, Riera R, Torloni MR. Methodological quality of systematic reviews and clinical trials on women's health published in a Brazilian evidence-based health journal. Clinics (São Paulo) 2013; 68 (04) 563-567
• 7 Pieper D, Buechter RB, Li L, Prediger B, Eikermann M. Systematic review found AMSTAR, but not R(evised)-AMSTAR, to have good measurement properties. J Clin Epidemiol 2015; 68 (05) 574-583
• 8 Atallah AN, Castro AA. Revisão sistemática da literatura e metanálise: Medicina baseada em evidência: Fundamentos da pesquisa clínica. São Paulo: Lemos Editorial; 1998
• 9 Sampaio RF, Mancini MC. Estudos de revisão sistemática: um guia para síntese criteriosa da evidência científica. Rev Bras Fisioter 2007; 11 (01) 83-89
• 10 Burda BU, Holmer HK, Norris SL. Limitations of A Measurement Tool to Assess Systematic Reviews (AMSTAR) and suggestions for improvement. Syst Rev 2016; 5: 58
• 11 Wegewitz U, Weikert B, Fishta A, Jacobs A, Pieper D. Resuming the discussion of AMSTAR: What can (should) be made better?. BMC Med Res Methodol 2016; 16 (01) 111
• 12 Shea BJ, Grimshaw JM, Wells GA. et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007; 7: 10
• 13 Fleming PS, Seehra J, Polychronopoulou A, Fedorowicz Z, Pandis N. Cochrane and non-Cochrane systematic reviews in leading orthodontic journals: a quality paradigm?. Eur J Orthod 2013; 35 (02) 244-248
• 14 Li L, Tian J, Tian H, Sun R, Liu Y, Yang K. Quality and transparency of overviews of systematic reviews. J Evid Based Med 2012; 5 (03) 166-173

Publication History

Received: 29 October 2019

Accepted: 20 February 2020

Article published online:
29 May 2020