Unveiling Bias: Examining the Influence of Positive Results on Ergogenic Aids in Published Sports Science Studies

Abstract

We investigated the potential for publication bias in the field of sports science regarding studies on ergogenic aids and their effects on exercise performance. We found evidence to suggest that journals tend to prioritize studies with positive results (76%) while neglecting those with negative outcomes (2.7%). Worryingly, this could lead to a discrepancy between reported conclusions and actual study outcomes. We also identified inconsistencies between reported outcomes and actual performance variable outcomes. Taken together, these data highlight the need for future research to reduce bias and encourage the publication of studies with both positive and negative results to improve the reliability of scientific evidence in this field.

Key words

bias - exercise - scientific evidence - human performance - physiological data

Introduction

Ergogenic aids, defined as substances, devices, or procedures that have the potential to enhance muscle size, neuromuscular function, or performance recovery [1] are frequently investigated by sports scientists to improve athletes’ performance. Despite the increased number of publications in this field, the quality of these studies may be compromised by methodological limitations and the potential for publication bias, which tends to favor positive results. These issues raise concerns about the reliability of reported effects of ergogenic aids on athletic performance and emphasize the importance of rigorous and transparent research practices in sports science. Researchers often reference positive studies to generate new hypotheses, but it is crucial to acknowledge the potential influence of publication bias on this approach.

Scientific research aims to provide accurate and reliable measures of observed outcomes. To achieve this, it is essential to conduct research with a good methodology that is honest and free of bias [2]. Bias can be defined as the lack of internal validity in data collection and data analysis, and a tendency to interpret outcomes in a target population, causing equivocated conclusions [2] [3]. In fact, different types of bias exist in the literature, and they are typically classified based on the direction of change they produce in a given parameter. For example, bias in data collection occurs when the sample of studies is not randomized and the individual inclusion is manipulated into the study groups.

Data analysis bias occurs when researchers create data that never existed, eliminate data that contradict the study hypothesis, use inappropriate statistical analysis, or perform multiple analyses until they obtain a result with a statistically significant difference [2] [4]. Interpretation bias is another type of bias, which occurs when researchers disregard the original outcomes of the study and interpret them based on their preconceived beliefs, hypotheses, and conflicts of interest [2] [5].

There is also the publication bias, which deals with journals more likely to publish studies that report positive outcomes than studies that report negative outcomes [2] [4]. The dissemination of publications with this type of bias can create the misconception that negative or statistically insignificant results are due to negligent conduction, methodological failures, or inadequate experimental design [6] [7]. Publication bias poses a significant threat to the reliability of scientific studies and can result in various consequences for society. These consequences include the utilization of ineffective tools [8] and failure in replicating experiments [2].

Therefore, we aimed to evaluate whether publications on ergogenic aids are affected by such bias, with a tendency for journals to publish studies presenting positive results. We also aimed to assess the consistency between the actual results and conclusions of publications on ergogenic aids in the field of sports science. Our hypothesis was that editors and journals have a tendency to accept studies demonstrating ergogenic benefits, resulting in a higher impact for the journal due to multiple citations.

Materials and Methods

Two reviewers selected 87 journals between January and December 2020. Of the selected journals, 85 were from the journal of citation reports (JCR) sports sciences list and two were included due to their high number of publications related to human performance. After this selection, the scope of each journal was observed, with the remaining journals having met the following inclusion criteria: 1) have within their scope the terms “sports medicine”, “sports science”, “sports medicine and exercise”, “sports and exercise medicine”, “sports and physiology”, “exercise and physiology”, “exercise science and performance”; and 2) journals not belonging to the sports sciences area if they met the inclusion criteria and contributed to the area of sports sciences. The following non-inclusion criteria were adopted: 1) publication of review articles exclusively; 2) no direct relation to sport and performance; and 3) journals dedicated to medical treatment or rehabilitation.

After selecting the journals, we read the titles and abstracts of all published articles between January and December 2021 to identify the studies that investigated the effect of ergogenic aids. For the inclusion of the studies, the following criteria were adopted: 1) to have been published in 2020; 2) to investigate the effects of at least one ergogenic resource on physical performance; 3) only studies with humans; and 4) having one or two variables directly or indirectly related to physical performance. For the selection of studies, the following non-inclusion criteria were adopted: 1) combination of ergogenic aids with any type of training; 2) evaluation of cognitive performance exclusively; and 3) no full access to the article.

The reported outcomes of the articles were classified as positive, negative, or neutral based on their identification in the conclusion and or title. The outcomes of the variables used in the articles were identified in the actual results and classified as positive, neutral, or negative based on whether a significant increase, no change, or decrease was found between the experimental group and either the placebo or control group.

Statistical analysis

Statistical analysis was performed using the software IBM SPSS (Version 23; IBM Corp., Armonk, NY, USA). The chi-square test of independence was conducted to verify association between variables outcomes and reported outcomes of the articles identified in the conclusion and/or title. When a significant association was found, the z-test adjusted by Bonferroni’s method was used [9].

In addition, when necessary, the adjusted residual was observed to verify a significant difference between count and expected count. Values below –1.96 or above 1.96 were considered significant [10]. Lastly, Fisher’s exact test, conducted when a cell has an expected count less than five, was used to verify the association between reported outcomes of the articles and impact factor of the journals, and between reported outcomes of the articles and publication fees.

To perform the impact factor analysis, the normality of the data was checked and we decided to use the median (=3.0 scores) to determine two groups of journals: scores above 3.0 and below 3.0. The level significance adopted was p<0.05.

Results

Of the 87 journals initially selected, only 20 met the inclusion and non-inclusion criteria, resulting in a total of 219 studies. However, after a thorough screening process, only 150 studies from 18 journals ([Table 1]) were deemed eligible for analysis ([Fig. 1]).

Fig. 1 Flow diagram of search process.

Table 1 Characteristics of selected journals.

Journals	Impact factor	Publication fees*	Number of issues/year	Total articles
A	12.68	No	24	333
B	5.20	No	6	90
C	5.07	No	24	53
D	4.03	No	6	333
E	3.61	No	1	193
F	3.53	Yes	12	220
G	3.37	Yes	1	55
H	3.25	No	12	245
I	3.04	Yes	12	330
J	2.97	No	12	430
K	2.85	Yes	24	209
L	2.78	No	10	149
M	2.74	Yes	12	202
N	2.60	No	24	303
O	2.58	No	12	249
P	2.56	No	14	112
Q	2.52	No	15	197
R	2.00	Yes	4	57

*or submission fees.

In our analysis of 150 studies from 18 journals, we identified 449 variables related to physical performance. Of these variables, 181 (40.3%) showed statistically significant improvements in performance, 18 (4%) showed impairment, and 250 (56.7%) did not present significant changes ([Fig. 2]).

Fig. 2 Total number of articles and variables with positive, negative, and neutral outcomes. The articles were classified based on their conclusion and or title, while the variables used by the articles themselves were classified based on their statistical result. a=significant difference compared to articles.

In each study, the reported outcomes were identified in the conclusion and in the title. Of the outcomes observed, 114 studies (76%) reported positive outcomes, 4 studies (2.7%) reported negative outcomes, and 32 studies (21.3%) reported neutral outcomes ([Fig. 2]). The chi-square test of independence showed a significant association between reported outcomes of the articles and reported outcomes of the variables (X² ₍₂₎=57.803; p<0.001), detecting a significant difference between them in the positive and neutral outcomes but not in the negative outcomes ([Fig. 2]).

For the positive outcomes, the expected count of the reported outcomes of the articles was significantly lower than that found (observed), while the expected count of the variables outcomes was significantly higher than that found (observed). On the other hand, for the neutral outcomes, the expected count of the reported outcomes of the articles was significantly higher than that found (observed), while for the variables outcomes it was significantly lower than that found (observed). For the negative outcomes, no significant difference was found ([Table 2]).

Table 2 Count, expected count, and adjusted residual of the reported outcomes of the variables and scientific articles.

Outcomes		Articles	Variables	Total
Positive	Count	114	181	295
	Expected Count	73.9	221.1	295.0
	Adjusted Residual	7.6	–7.6	–
Negative	Count	4	18	22
	Expected Count	5.5	16.5	22.0
	Adjusted Residual	–0.8	0.8	–
Neutral	Count	32	250	282
	Expected Count	70.6	211.4	282.0
	Adjusted Residual	–7.3	7.3	–

Adjusted residual higher than 1.96 or less than –1.96 means a significant difference between count and expected count.

Fisher’s exact test showed no significant association between reported outcomes of the articles and either the impact factor (X² ₍₂₎=3.365; p=0.159) or publication fees (X² ₍₂₎=2.718; p=0.271) ([Table 3]).

Table 3 Association between reported outcomes of the articles and impact factor and publication fees.

	Reported outcomes of the articles
Impact factor	Positive	Negative	Neutral	Total
Above 3.0	39	3	14	56
Below 3.0	75	1	18	94
Total	114	4	32	150
	Reported outcomes of the articles
Publication fees*	Positive	Negative	Neutral	Total
Yes	40	3	13	56
No	74	1	19	94
Total	114	4	32	150

^*or submission fees.

Discussion

This study evaluated whether scientific publications on ergogenic aids and exercise performance are affected by bias and also assessed the consistency between the actual results and titles/conclusions presented. The main results confirm our hypothesis, indicating a tendency for journals to publish studies with positive results while neglecting those with negative results. Additionally, our analysis revealed inconsistencies between the reported outcomes in the titles and conclusions and the actual performance variable outcomes. Specifically, we observed a predominance of non-significant outcomes in the performance variables, suggesting a potential discrepancy between the reported conclusions and the actual study outcomes.

We observed an inconsistency between reported positive outcomes (76%) and statistically significant variable outcomes (40.3%). Furthermore, a very small rate of studies reported a negative outcome (4%). This inconsistency can hinder the formation of knowledge about the effects of ergogenic aids, confusing the interpretation of the reported outcomes. This confusion can lead to errors in the application of ergogenic aids in the field of physical performance, such as athletes using aids that may not be effective or safe, or coaches and trainers making training and performance decisions based on incomplete or biased information.

It is important to highlight that such errors could have serious consequences, not only for the athletes’ performance but also for their health and well-being [11]. Therefore, it is essential to address these issues and ensure that the scientific evidence on ergogenic aids is reliable and unbiased to guide the safe and effective use of these aids in sports science.

A similar study to ours analyzed the effects of antidepressant medications by comparing the results of published research in scientific journals with the results of work in the United States Food and Drug Administration (FDA) database [12]. According to their analysis, the published literature showed that 94%of trials investigating antidepressants reported significant positive effects. This percentage differs from the FDA database, which found significant positive effects in only 51%of the trials.

Several factors can contribute to an increase in results that do not show statistical significance. These include variations in individual and temporal physiological responses, a lack of clear protocols that elicit responses, and inherent measurement errors in the testing process [13] [14]. Non-positive results, on the other hand, can contribute to the progress of research by providing evidence for alternative hypotheses and directing attention towards key variables for improved performance.

Our analysis of [Table 3] suggests that there is no significant relationship between the reported outcomes of the articles and either the impact factor or publication fees. This finding is notable, as previous research has suggested that journals with high impact factors may be more likely to publish articles reporting positive results [15]. However, our study found no statistically significant difference in the proportion of positive, negative, or neutral outcomes between articles published in high-impact and low-impact journals, or between those that required payment of publication fees and those that did not.

While there may be other factors at play, such as sample size or study design, that can influence the outcomes reported in scientific articles, our analysis indicates that impact factor and publication fees alone are not reliable indicators of the reported outcomes of research studies involving ergogenic aids and exercise performance.

The findings of this study show that journals tend to prioritize studies with positive results, often neglecting those with negative outcomes. Furthermore, our analysis indicates inconsistencies between reported outcomes in titles and conclusions and actual performance variable outcomes, with a higher prevalence of non-significant results in performance variables. These observations suggest a potential discrepancy between the reported conclusions and actual study outcomes. Future research should aim to reduce bias and encourage the publication of studies with both positive and negative results to improve the reliability of scientific evidence on ergogenic aids and exercise performance.

Conflict of Interest

The authors declare that they have no conflict of interest.

Acknowledgement

Moacir Marocolo is supported by National Council for Scientific and Technological Development – CNPq (process no. 308138/2022–8) and Rhaí Arriel by Fundação de Amparo à Pesquisa de Minas Gerais – FAPEMIG (process no. BPD-00905–22).

• References

• 1 Kim J, Kim E-K. Nutritional strategies to optimize performance and recovery in rowing athletes. Nutrients 2020; 12: 1685 DOI: 10.3390/nu12061685.
• 2 Šimundić A-M. Bias in research. Biochem Med (Zagreb) 2013; 23: 12-15 DOI: 10.11613/BM.2013.003.
• 3 Yarborough M. Moving towards less biased research. BMJ Open Sci 2021; 5: e100116 DOI: 10.1136/bmjos-2020-100116.
• 4 Vassar M, Roberts W, Cooper CM. et al. Evaluation of selective outcome reporting and trial registration practices among addiction clinical trials. Addiction 2020; 115: 1172-1179 DOI: 10.1111/add.14902.
• 5 Souza NV, Nicolini AC, Dos Reis INR. et al. Selective outcome reporting bias is highly prevalent in randomized clinical trials of nonsurgical periodontal therapy. J Periodontal Res 2023; 58: 1-11 DOI: 10.1111/jre.13066.
• 6 Dwan K, Gamble C, Williamson PR. et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias - an updated review. PLoS One 2013; 8: e66844 DOI: 10.1371/journal.pone.0066844.
• 7 Teixeira Da Silva JA. Negative results: negative perceptions limit their potential for increasing reproducibility. J Negat Results Biomed 2015; 14: 12 DOI: 10.1186/s12952-015-0033-9.
• 8 Chan A-W, Altman DG. Identifying outcome reporting bias in randomised trials on PubMed: review of publications and survey of authors. BMJ 2005; 330: 753 DOI: 10.1136/bmj.38356.424606.8f.
• 9 MacDonald PL, Gardner RC. Type I error rate comparisons of post hoc procedures for I j chi-square tables. Educ Psychol Meas 2000; 60: 735-754 DOI: 10.1177/00131640021970871.
• 10 Sharpe D. Chi-square test is statistically significant: now what?. Pract Assess Res Evaluation 2015; 20 DOI: 10.7275/tbfa-x148.
• 11 Adami PE, Koutlianos N, Baggish A. et al. Cardiovascular effects of doping substances, commonly prescribed medications and ergogenic aids in relation to sports: a position statement of the sport cardiology and exercise nucleus of the European Association of Preventive Cardiology. Eur J Prev Cardiol 2022; 29: 559-575 DOI: 10.1093/eurjpc/zwab198.
• 12 Turner EH, Matthews AM, Linardatos E. et al. Selective publication of antidepressant trials and its influence on apparent efficacy. N Engl J Med 2008; 358: 252-260 DOI: 10.1056/nejmsa065779.
• 13 Grgic J. Are there non-responders to the ergogenic effects of caffeine ingestion on exercise performance?. Nutrients 2018; 10: 1736 DOI: 10.3390/nu10111736.
• 14 Marocolo M, Marocolo IC, Da Mota GR. et al. Beneficial effects of ischemic preconditioning in resistance exercise fade over time. Int J Sports Med 2016; 37: 819-824 DOI: 10.1055/s-0042-109066.
• 15 Duyx B, Urlings MJE, Swaen GMH. et al. Scientific citations favor positive results: a systematic review and meta-analysis. J Clin Epidemiol 2017; 88: 92-101 DOI: 10.1016/j.jclinepi.2017.06.002.

Publication History

Received: 22 May 2023

Accepted: 15 September 2023

Accepted Manuscript online:
25 October 2023

Article published online:
08 January 2024