Problematic statistical analyses in Zacharias et al.

In Homeopathy July 2008, Zacharias et al. evaluate the effects of homeopathic treatment on sheep carrying the parasite Haemonchus contortus.[ ¹ ] Their paper's [Table 1] comprises 3 correlation matrices, each showing 37 correlation coefficients that quantify the correlations between assorted physiological variables of the sheep. They use some of these correlations to try to attribute positive effects to the homeopathic treatment.

Unfortunately, their attempt is stymied by several problems: (1) transcription errors; (2) dubious significance testing of correlations; (3) failure to account for doing multiple significance tests on correlations derived from the same dataset; and (4) comparison of the treatment group's correlations with zero and not the corresponding correlations in the other groups.

For two correlation coefficients, Zacharias et al. give two different values, one in their main text and the other in its [Table 1]:

• The homeopathy group's FEC (fecal egg count)–Hb correlation is −0.893 in the main text, but in [Table 1] it's −0.834. (A neighbouring cell in the table displays an Hct–Hb correlation of 0.893; Zacharias et al. appear to have quoted this latter number instead, mistakenly prefixing it with a minus sign).

• The anthelminthic group's FEC–Hb correlation is −0.039 in the main text and −0.399 in [Table 1].

The inconsistencies render the correlation coefficients' true values uncertain. For the purposes of this reanalysis, I took those presented in Zacharias et al.'s [Table 1] as correct, and ignore those in the paper's main text.

Several times, the Zacharias et al. paper explicitly claims that a treatment group's correlation coefficient is statistically significant, or treats a correlation coefficient as significant by saying it ‘suggest[s]’ a causal effect.

Where correlations are taken as evidence of real treatment effects, it should be on the basis of rigorous significance testing. Consequently, to verify Zacharias et al.'s correlation-based claims of the homeopathy's efficacy, I attempted to replicate their significance tests.

I consulted the statistical analysis subsection of their paper's materials and methods section, which mentions that “[c]orrelation between values from different parameters was calculated for each group using Pearson's correlation coefficient”. It gives no other details regarding the correlations, so it's unclear which significance test they used. (I have emailed Dr Mendonça-Lima, the corresponding author, to ask how significance was tested, but have not received a reply).

The format of Zacharias et al.'s [Table 1] suggests that they tested each correlation individually for whether it differed significantly from zero. There are two standard procedures for carrying out this test on a Pearson correlation coefficient r, given the number of sample pairs N. One can transform r to a t-value and, taking N − 2 as the number of degrees of freedom, test that t-value for significance.[ ² ] Alternatively, one can use the Fisher transformation to obtain a z-value (z = arctanh r), take (N − 3)^−1/2 as its standard error, and perform a z-test.[ ³ ] As I didn't know which (if either) procedure Zacharias et al. used, I calculated both t- and z-values for the correlations that Zacharias et al. interpret as it significant ([Table 1]).

Table 1 r-values reported in Zacharias et al.'s Table 1, their equivalent t- and z-values, and the p-values that result from performing two-tailed t- and z-tests on those values

Group	Variables	r	t	z	pt	pz
Homeopathy	FEC, haematocrit	−0.656	−1.943	−0.786	0.110	0.116
Homeopathy	FEC, Hb	−0.834	−3.380	−1.201	0.020	0.016
Control	FEC, haematocrit	−0.425	−0.939	−0.454	0.401	0.432
Control	FEC, Hb	−0.624	−1.597	−0.732	0.185	0.205
Anthelmintic	FEC, haematocrit	−0.412	−1.011	−0.438	0.358	0.381
Anthelmintic	FEC, Hb	−0.399	−0.973	−0.422	0.375	0.398
Homeopathy	EOS, FEC	−0.344	−0.819	−0.359	0.450	0.473
Homeopathy	Globulin, IgG	0.610	1.721	0.709	0.146	0.156
Control	Globulin, IgG	0.239	0.492	0.244	0.648	0.673
Anthelmintic	Globulin, IgG	0.255	0.590	0.261	0.581	0.602

As my [Table 1] shows, Zacharias et al.'s reported p-values for the homeopathy and control groups' FEC–haematocrit and FEC–Hb correlations seem not to have come from a standard test. For the homeopathy group, the correct p-values are not less than 0.01, but are all at least 0.016. For the control group, they are unambiguously insignificant, with p-values of 0.401 or 0.432 (Zacharias et al. claim ‘p < 0.05’) and 0.185 or 0.205 (Zacharias et al. claim ‘p < 0.01’). The correlation cited as ‘suggesting’ that the homeopathy “stimulated the production of circulating eosinophils (EOS)” is insignificant (p > 0.4), as is the correlation cited as ‘suggesting’ a superior immune response in the homeopathy group (p > 0.1).

Using one-tailed or two-tailed tests makes little difference. Switching to a one-tailed test simply halves all of the p-values, which brings the FEC–Hb correlation in the homeopathy group into line with Zacharias et al.'s statement that it's less than 0.01. All of the other p-values, however, remain greater than 0.05.

There is another oddity. For the homeopathy group, the FEC–whole proteins correlation is −0.475, which is marked insignificant. For the anthelminthic group, the whole proteins–Hb correlation is 0.457, which is marked significant. But both groups are the same size, and so if a correlation in one group is significant, a correlation in the other group with a higher absolute value should also be significant. This isn't the case for the two correlations quoted: if the anthelmintic group's 0.457 correlation is significant, the homeopathy group's −0.475 correlation is also, but the paper says otherwise.

No standard test of individual Pearson correlations would have significance levels that shift in such a way or the deflated p-values that Zacharias et al. reported. At best, they have used an unusual significance test with a high type I error rate.

So far I, like Zacharias et al., have used a significance level of 0.05 when testing correlations. This significance level is in fact too high, as both Zacharias et al. and I are performing repeated significance tests on the same dataset, which inflates the risk of type I error far beyond the nominal 0.05 level. With 111 significance tests (one for each of their reported correlations), a nominal significance level of 0.05 means a more than 99% chance of at least one test giving a p-value below 0.05, even with all 111 correlations being zero in the population. Under the null hypotheses, to maintain a real significance level of 0.05 over all 111 significance tests, a nominal significance level of 0.00046 is necessary. None of the correlations come close to reaching this.

Being less stringent hardly helps. If I consider only the five correlations that Zacharias et al. allege are significant (the FEC–haematocrit, FEC–Hb, and globulin–IgG correlations in the homeopathy group and the FEC–haematocrit and FEC–Hb correlations in the control group) and put aside the other 106, the nominal significance level need only be reduced to 0.01 instead of 0.00046. None of the five correlations are significant at the 0.01 level.

Often, when a correlation is calculated for a study, the question of interest is whether the correlation is statistically distinguishable from zero or not. However, Zacharias et al. try to use correlations to determine whether homeopathic treatment is more effective than no treatment and an anthelmintic, and what the homeopathy's unique physiological effects are. This implies that they wish to compare the homeopathy group to the other two groups, not to imaginary benchmarks of zero correlations. This means they tested the wrong null hypotheses: the pertinent null hypotheses are whether the homeopathy group's correlations differ from those of the other two groups, not whether they differ from zero.

Testing the correct null hypotheses reveals that the homeopathy group correlations that Zacharias et al. treat as significant are less significant than [Table 1]'s p-values suggest. [Table 2] shows that none of them differ at all significantly from either of the other groups' correlations.

Table 2 Comparison of sample correlations from Zacharias et al.: r _h, r _c, and r _a are the correlations in the homeopathy group, control group, and anthelmintic group respectively, and z _h, z _c, and z _a their Fisher transformations. p _hc and p _ha are p-values for two-tailed tests of whether the homeopathy group's correlation differs significantly from the corresponding correlation in, respectively, the control group or the anthelmintic group

Variables	r h	r c	r a	|z h − z c|	|z h − z a|	p hc	p ha
FEC, haematocrit	−0.656	−0.425	−0.412	0.332	0.348	0.66	0.62
FEC, Hb	−0.834	−0.624	−0.399	0.470	0.779	0.54	0.27
EOS, FEC	−0.344	−0.132	−0.142	0.226	0.216	0.77	0.76
Globulin, IgG	0.610	0.239	0.255	0.465	0.448	0.54	0.53

Appropriate analyses demonstrate that all of Zacharias et al.'s relevant correlations are insignificant, leaving the inferences they draw from them without basis. It is not true that the correlations “indicat[e] that the homeopathic medicine improved vital functions” [p. 145], “demonstrate the different responses to the homeopathic and antihelminthic treatments” [p. 149], or “[suggest] that the immune response was better in the homeopathy group” [p. 150]. Their speculations that “the homeopathic medicine acts by stimulating the entire body … This led to a better recovery from haematopoiesis” [p. 149] and that “the homeopathic medicine stimulated the production of circulating eosinophils” [p. 150] remain unfounded.

It's worth briefly noting that Zacharias et al.'s statistical peccadilloes aren't confined to their handling of correlation coefficients. For instance, in the space of one page (p. 150), they observe that the sheep treated with homeopathy gained more weight than the other sheep, go on to conclude that “these results would have a considerable impact in sheep flocks bred on a commercial scale”, and that “cost benefit analysis, showed a favourable trend to homeopathy compared to other groups”. Also on page 150 is their Figure 2 and its caption, which states that the “weight gain comparison did not show statistic significance”! It is a bad idea to make sweeping veterinary recommendations on the basis of a result that could well be a fluke.

Ultimately, Zacharias et al.'s overenthusiastic interpretation of insignificant results, flawed significance testing, and numeric inconsistencies lead me to decide that most of their conclusions still await experimental confirmation.

The authors have not responded.

• References

• 1 Zacharias F., Guimaraes J.E., Araújo R.R. et al. Effect of homeopathic medicines on helminth parasitism and resistance of Haemonchus contortus infected sheep. Homeopathy 2008; 97: 145-151.
• 2 Samiuddin M. On a test for an assigned value of correlation in a bivariate normal distribution. Biometrika 1970; 57: 461-464.
• 3 Fisher R.A. On the “probable error” of a coefficient of correlation deduced from a small sample. Metron 1921; 1: 3-32.