Systematic Review and Meta-Analysis of Randomised, Other-than-Placebo Controlled, Trials of Individualised Homeopathic Treatment

Abstract

Background This study focuses on randomised controlled trials (RCTs) of individualised homeopathic treatment (IHT) in which the control (comparator) group was other than placebo (OTP).

Aims To determine the comparative effectiveness of IHT on health-related outcomes in adults and children for any clinical condition that has been the subject of at least one OTP-controlled trial. For each study, to assess the risk of bias and to determine whether its study attitude was predominantly ‘pragmatic’ or ‘explanatory’.

Methods Systematic review. For each eligible trial, published in the peer-reviewed literature up to the end of 2015, we assessed its risk of bias (internal validity) using the seven-domain Cochrane tool, and its relative pragmatic or explanatory attitude (external validity) using the 10-domain PRECIS tool. We grouped RCTs by whether they examined IHT as an alternative treatment (study design Ia), adjunctively with another intervention (design Ib), or compared with a no-intervention group (design II). For each RCT, we identified a ‘main outcome measure’ to use in meta-analysis: ‘relative effect size’ was reported as odds ratio (OR; values >1 favouring homeopathy) or standardised mean difference (SMD; values < 0 favouring homeopathy).

Results Eleven RCTs, representing 11 different medical conditions, were eligible for study. Five of the RCTs (four of which in design Ib) were judged to have pragmatic study attitude, two were explanatory, and four were equally pragmatic and explanatory. Ten trials were rated ‘high risk of bias’ overall: one of these, a pragmatic study with design Ib, had high risk of bias solely regarding participant blinding (a bias that is intrinsic to such trials); the other trial was rated ‘uncertain risk of bias’ overall. Eight trials had data that were extractable for analysis: for four heterogeneous trials with design Ia, the pooled OR was statistically non-significant; collectively for three clinically heterogeneous trials with design Ib, there was a statistically significant SMD favouring adjunctive IHT; in the remaining trial of design 1a, IHT was non-inferior to fluoxetine in the treatment of depression.

Conclusions Due to the low quality, the small number and the heterogeneity of studies, the current data preclude a decisive conclusion about the comparative effectiveness of IHT. Generalisability of findings is limited by the variable external validity identified overall; the most pragmatic study attitude was associated with RCTs of adjunctive IHT. Future OTP-controlled trials in homeopathy should aim, as far as possible, to promote both internal validity and external validity.

Keywords

comparative effectiveness - explanatory trial - individualised homeopathic treatment - meta-analysis - pragmatic trial - randomised controlled trial - risk of bias - systematic review

Introduction

There are several distinct forms of homeopathy, the main types being ‘individualised’ (or ‘classical’) homeopathy, ‘clinical’ homeopathy, ‘complex’ homeopathy and isopathy. In individualised homeopathy—as originally defined by its founder, Samuel Hahnemann—typically a single homeopathic medicine is selected on the basis of the ‘symptom picture’ of a patient, including his/her mental, general and constitutional type. In clinical homeopathy, one or more homeopathic medicines are administered for standard clinical situations or conventional diagnoses. In complex homeopathy, several homeopathic medicines are combined in a fixed (‘complex’) formulation. Isopathy is the use of homeopathic medicines prepared from the causative agent of the disease itself, or from a product of the disease process, to treat the condition.[1]

Our previous reviews focused on individualised or non-individualised homeopathy in the context of placebo-controlled randomised controlled trials (RCTs).[2] [3] The current study focuses again on the ‘whole-person’ therapeutic approach of individualised homeopathic treatment (IHT): individualised consultation together with the consequently prescribed homeopathic medicine/s (and sometimes including dietary and/or lifestyle recommendations). The context here is an RCT in which the control (comparator) group is something other than placebo (OTP), and which can be regarded as a ‘comparative effectiveness’ study. Whereas for placebo-controlled trials of IHT, we investigated whether individually prescribed homeopathic medicines can have effects above those of placebos,[2] our study of OTP-controlled trials aims to ascertain the comparative effectiveness of IHT as a whole entity: that is, the package of homeopathic care that includes the prescribed medicines.

Two essentially different comparator options exist for OTP study design of RCTs: (I) other therapeutic intervention (e.g. a conventional medicine or a physical therapy), which can be sub-divided into (a) trials in which IHT is given as an alternative to the comparator intervention, and (b) trials in which IHT combined with the other intervention is compared with the other intervention alone (the ‘[A + B] versus B’ approach); (II) no therapeutic intervention (usually waiting-list controls).

Some systematic reviews (SRs) of homeopathy have focused on specific clinical conditions, typically examining placebo-controlled trials, with conclusions about efficacy of homeopathic medicines that are variously positive,[4] [5] [6] negative[7] [8] [9] or non-conclusive.[10] [11] [12] Other SRs have adopted a ‘global’ or ‘comprehensive’ approach, examining the RCT research literature on homeopathy in general, including the broad spectrum of clinical conditions that have been researched, and by all forms of homeopathy. Most of these comprehensive SRs reached the cautious conclusion that, overall, the effect of a homeopathic medicine differs from that of placebo.[2] [13] [14] [15] [16] [17] One comprehensive SR concluded that there was ‘weak evidence for a specific effect of homoeopathic remedies…compatible with the notion that the clinical effects of homoeopathy are placebo effects’.[18] There has been insufficient quantity or quality of evidence in these comprehensive SRs—including a recent SR of placebo-controlled RCTs in non-individualised homeopathic treatment[3]—to enable clear conclusions about the efficacy of homeopathic medicines for any specific clinical condition.

Only one general SR has considered solely RCTs of IHT that were controlled by an OTP intervention.[19] Published 19 years ago, this SR identified just six eligible trials, each of which was judged to contain serious methodological flaws. Findings for these trials were mixed: two favoured IHT, two favoured conventional drugs and two were non-conclusive. Each trial investigated a different clinical condition, and none had been replicated by an independent research group. It was concluded that the ‘value of individualised homoeopathy relative to allopathic treatments’ was not known.

An RCT's lack of bias (its internal validity), the generalisability of its findings (external validity) and the quality of its clinical intervention (model validity) are key to its overall worth. Though, in general, it is seldom formally addressed by trialists or in SRs,[20] [21] external validity can be appraised using a tool (PRECIS), developed by Thorpe et al in 2009, that considers a trial design's positioning on ‘the pragmatic–explanatory continuum’.[22] For the current SR of OTP-controlled trials, we assess each study's internal validity as well as its external validity, with a view to appraising both its intrinsic quality and the extent of its pragmatic/explanatory attitude.

Aim of the Study

Our objective was to examine comparative effectiveness of IHT in OTP-controlled trials of any clinical condition, in adults or children, for which there was at least one eligible RCT. Using meta-analysis methods, we aimed to evaluate RCTs that have investigated IHT: (study design Ia) in comparison to another therapeutic intervention; (study design Ib) adjunctively with another treatment intervention in comparison to the other intervention alone (‘[A + B] versus B’); or (study design II) compared with no other intervention. An additional aim was, where possible, to evaluate by meta-analysis the comparative effectiveness of IHT for any given clinical condition or category of conditions. In all cases, we reflected matters of internal validity (risk of bias) and external validity (pragmatic/explanatory study attitude).

Methods

Methods comply fully with the PRISMA 2009 Checklist ([Supplementary File 1], available in online version only) and with our published study protocol[23] (not registered with PROSPERO).

Search Strategy, Data Sources and Trial Eligibility

We conducted a systematic literature search to identify RCTs that compared IHT with something other than placebo, for any clinical condition. Each of the following electronic databases had previously been searched from its inception up to the end of 2011: AMED; CAM-Quest; CINAHL; Cochrane Central Register of Controlled Trials; Embase; Hom-Inform; LILACS; PubMed; Science Citation Index; Scopus.[24] For the present SR, the same database searches were updated—each with the same strategy as for 2011—to the end of 2015. For the update, CORE-Hom was also searched, using the term ‘randomized’ or ‘unknown’ in the Sequence Generation field.

The full electronic search strategy for PubMed (Cochrane Highly Sensitive Search Strategy) is given in our previous article:[24] ‘(homeopath* or homoeopath*) and (randomised controlled trial[pt]) or (controlled clinical trial[pt]) or (randomised [tiab]) or (placebo [tiab]) or (clinical trials as topic[mesh:noexp]) or (randomly [tiab]) or (trial[ti]) not (animals[mh] not humans[mh])’.

Specific pre-defined exclusion criteria were then applied:

• Trials of homeopathic prophylaxis

• Trials with crossover design

• Research using radionically prepared ‘homeopathic’ medicines[25]

• The tested intervention is IHT in conjunction with other (complementary or conventional) medicine or therapy, and where the nature of the combined comparator intervention makes it impossible to distinguish any effects due to IHT.[a]

• Other specified reason.

Whereas a placebo-controlled trial of individualised homeopathy can be fully blinded, it is more difficult—and sometimes impossible—to achieve such blinding in a corresponding OTP-controlled trial. Unlike the case for our SR of placebo-controlled trials, therefore, patient- and practitioner-unblinded trials are eligible for the current SR of OTP trials.

Only published data were eligible for analysis. Because it is recognised that contacting the original authors of RCTs may lead to limited or overly positive answers,[26] the authors of eligible articles were not approached for clarification on unclear or missing facts; however, original authors' cross-reference to their previously published study methods were followed up and taken into account as necessary. For trials with more than two study groups, and where such trials had not previously been catalogued under ‘placebo-controlled’, only the data concerning comparisons between IHT and OTP were extracted from the articles; in relevant cases of more than one OTP control, a study group comprising actual treatment was favoured for analysis over one comprising ‘no treatment’.

Outcome Definitions

For each trial, and for the purposes of risk-of-bias assessment, we identified a ‘main outcome measure’ using a refinement of the approaches adopted by Linde et al and by Shang et al.[15] [18] As for our previous reviews,[2] [3] each trial's ‘main outcome measure’ was identified based on the following hierarchical ranking order (consistent with the WHO ICF Classification System for Levels of Functioning Linked to Health Condition):[b]

• Mortality

• Morbidity

  • ○ Treatment failure

  • ○ Pathology; symptoms of disease

• Health impairment (loss/abnormality of function, including presence of pain)

• Limitation of activity (disability, including days off work/school because of ill health)

• Restriction of participation (quality of life)

• Surrogate outcome (e.g. blood test data, bone mineral density).

We followed the WHO ICF system without reference to any ‘primary outcome measure’ that may have been identified by the original investigators. Unless otherwise indicated, the single end-point (measured from the start of the intervention) associated with our designated ‘main outcome measure’ was taken as the last follow-up at which data were reported for that outcome.

Data Extraction

Two reviewers (RTM and SU-Z) independently extracted relevant data using a standard data recording approach, in spreadsheet format (Microsoft Excel). The data extracted per trial included, as appropriate: demographics of participants (gender, age range, clinical condition); study setting; potency or potencies of homeopathic medicines; dosage frequency; whether a pilot trial; ‘main outcome measure’ and measured end-point; other outcome measures reported; funding source/s. The statistical items noted were sample size and missing data for each intervention group; whether power calculation performed; whether intention-to-treat (ITT), per-protocol, complier-average-causal-effect[27] or other type of primary analysis.

Assessment of Risk of Bias (Internal Validity)

Using the standard criteria defined by Cochrane,[26] the extraction of information enabled appraisal of ‘low risk’, ‘uncertain risk’ or ‘high risk’ of bias with respect to: (domain I) the methods used to generate the random sequence; (domain II) the method of allocation concealment used to implement the random sequence; (domain IIIA) the blinding of participants and/or study personnel; (domain IIIB) the blinding of outcome assessors; (domain IV) completeness of the outcome data included in the article's analysis; (domain V) evidence of selective outcome reporting; (domain VI) evidence of other bias, including data imbalance between the groups at baseline.

Two assessors (RTM and SU-Z) performed their assessments independently, with discrepancies between them resolved by consensus discussion and, if necessary, the input of a third assessor (LAL or JRTD). For domain IV, a trial was normally regarded as no better than ‘unclear’ if there was greater than 20% participant attrition rate, irrespective of whether ITT analysis had been performed. For domain V, judgment on selectiveness of Results reporting in a given article was based on a comparison not with an original trial protocol (for few exist in the public domain for RCTs of homeopathy), but rather with details for outcomes provided in that same article's Methods section. Also, domain V was automatically designated ‘high risk of bias’ if its main outcome measure could not be extracted to enable calculation of ‘relative effect size’ (see below). The source of any research sponsorship was taken into account for sub-group analysis (see below), not in risk-of-bias assessment per se.

Rating of Trials for Risk of Bias

As per the standard Cochrane approach, each trial was designated: low risk of bias for all key domains; or uncertain risk of bias for one or more key domains; or high risk of bias for one or more key domains.[26] We used our novel method of nomenclature, based on the Cochrane approach, for rating risk-of-bias characteristics across all domains per trial:[2] [3]

• A = Low risk of bias in all seven domains.

• Bx = Uncertain risk of bias in x domains; low risk of bias in all other domains.

• Cy.x = High risk of bias in y domains; uncertain risk of bias in x domains; low risk of bias in all other domains.

An ‘A’-rated trial is designated reliable evidence. We also designate a ‘B1’-rated trial reliable evidence (and listed as ‘B1* [minimal risk of bias]’) if the uncertainty in its risk of bias is for one of domains IV, V or VI only (i.e. it is required to be judged free of bias for each of domains I, II, IIIA and IIIB).[23]

It is expected that an OTP-controlled trial will be rated ‘high risk of bias’ in assessment domain IIIA. We recognise that this is a normal feature of an OTP-controlled trial, and which thus inevitably limits its internal validity—see also the section Sensitivity analyses, below.

Assessment of Pragmatic/Explanatory Attitude (External Validity)

Equating external validity to study attitude, we adopted the PRECIS approach[22] to assess each trial's positioning on the pragmatic–explanatory continuum, taking account of the 10 domains:

1. Participant eligibility criteria;

2. Experimental intervention flexibility;

3. Experimental intervention practitioner expertise;

4. Comparison intervention;

5. Comparison intervention practitioner expertise;

6. Follow-up intensity;

7. Primary trial outcome;

8. Participant compliance with ‘prescribed’ intervention;

9. Practitioner adherence to study protocol;

10. Analysis of primary (‘main’) outcome.

Against a set of standard judgmental criteria,[22] two of us (RTM and PV) independently assessed each of the 10 attributes as ‘much more explanatory than pragmatic’, ‘more explanatory than pragmatic’, ‘equally pragmatic and explanatory’, ‘more pragmatic than explanatory’ or ‘much more pragmatic than explanatory’. Consistent with the PRECIS authors' receptiveness to methodological innovation,[22] we devised our own notation, giving a score of 1, 2, 3, 4 or 5 in each case respectively. Where an article contained no relevant information to enable assessment of a given domain, a score of 3 was given. The range of possible total scores per trial was 10 to 50. The full PRECIS-based approach that we used to assess each domain is given in [Supplementary File 3] (available in online version only).

We characterised each trial overall using the following empirical consideration of its total score:

• 10–17: Much more explanatory than pragmatic;

• 18–25: More explanatory than pragmatic;

• 26–34: Equally pragmatic and explanatory;

• 35–42: More pragmatic than explanatory;

• 43–50: Much more pragmatic than explanatory.

Discrepancies between assessments (per attribute and by overall characterisation per trial) were resolved by consensus discussion. Initial attention focused on each domain for which the two independent scores were a value of 2 or more apart, and with the aim of narrowing the discrepancy to no more than a value of 1. For any studies whose two sets of total scores lay on either side of a threshold for a given pragmatic/explanatory attitude, we agreed a final designation by overall re-appraisal and discussion; we did not calculate the mean value of the two independent total PRECIS scores.

Study Selection for Meta-Analysis

All RCTs that were included in the SR were potentially eligible for meta-analysis. If the original article did not provide or inform adequate data on the selected ‘main outcome measure’ to enable extraction or calculation of the SMD or the OR, we described the selected main outcome as ‘not estimable’: an alternative, estimable, outcome was not sought.

Consistent with the above, the following studies were excluded from meta-analysis:

• Those that presented non-parametric data only, and where there was no information that enabled the data distribution to be assessed;

• Those from which the necessary data could not be extracted (not provided or uninterpretable).

Summary Measures for ‘Main Outcome’

For the remaining relevant records of IHT, we aimed to examine: (1) overall relative effect sizes; (2) disease-specific relative effect sizes; (3) disease category-specific relative effect sizes. In each of these three cases, ‘relative effect size’ was taken as the difference (if relevant—see below) between the homeopathy and the control groups at the identified end-point of the trial, and using per-protocol data:

• For dichotomous measures: odds ratio (OR), with 95% confidence interval (CI);[c]

• For continuous measures: standardised mean difference (SMD), with 95% CI.[d]

In trials where the main outcome measure was a continuous variable, and where there were insufficient data presented to identify the mean and/or the standard deviation (SD) per group at the defined end-point, the necessary data were estimated, if possible, by imputing relevant other data from the same study.[28]

Statistical Interpretation Per Study Design

Interpretation of statistical finding: p ≤ 0.05 (direction of effect towards homeopathy):

• (Ia) Other-intervention control: IHT is more effective than the other intervention;

• (Ib) ‘[A + B] versus B’: IHT+ other intervention is more effective than the other intervention alone;

• (II) No-treatment control: IHT is more effective than no intervention.

Interpretation of statistical finding: p ≤ 0.05 (direction of effect towards control):

• (Ia) Other-intervention control: IHT is less effective than the other intervention;

• (Ib) ‘[A + B] versus B’: IHT+ other intervention is less effective than the other intervention alone;

• (II) No-treatment control: IHT is ineffective.

Interpretation of statistical finding: p > 0.05 (direction of effect towards either homeopathy or control):

• (Ia) Other-intervention control: Inconclusive whether IHT and the other intervention differ in effectiveness;

• (Ib) ‘[A + B] versus B’: Inconclusive whether the effectiveness of IHT+ other intervention differs from that of the other intervention alone;

• (II) No-treatment control: IHT is probably ineffective.

Detailed interpretation of the above is described in the study protocol[23] and reflects our approach to superiority trials. We separately interpreted studies explicitly designed as ‘equivalence’ or ‘non-inferiority’ trials,[29] [30] reflecting the original authors' margin of equivalence or non-inferiority as appropriate.

For any RCT or group of RCTs on a given clinical condition/category, the interpretation of IHT as ‘effective’, ‘ineffective’ or ‘inconclusive’ applied solely to the particular clinical condition/category examined.

Synthesis of Quantitative Results

1) Overall ‘relative effect size’ of IHT

For groups of eligible RCTs that have compared IHT (Ia) with another intervention, or (Ib) adjunctively with another intervention, or (II) with no treatment, we aimed to pool for meta-analysis the ‘main outcome’ data in two separate sets of studies as appropriate, using (1) the OR or (2) the SMD of each relevant trial.[31] For each study design (Ia, Ib, II), it was then planned to combine data from the two sets of studies (OR and SMD) into a single forest plot, re-expressing SMDs by transformation to OR, using an approximation method proposed by Chinn[32] and recommended by the Cochrane Statistical Methods Group.[33]

Based on the assumption of at least moderate clinical heterogeneity, the ‘random effects’ statistical model for meta-analysis was selected rather than the ‘fixed effect’ model.[31]

2) Disease- and category-specific ‘relative effect size’ of IHT

For each specific clinical condition or category of conditions, for each of study designs (Ia), (Ib) and (II), and for which there was >1 RCT of given type and with extractable main outcome, we planned to pool the data using meta-analysis.

3) Heterogeneity and asymmetry

The I ² statistic was used to assess the variability between studies in a given meta-analysis: it gives the percentage of the total variability in the estimated effect size (which is composed of between-study heterogeneity plus sampling variability) that is attributable to heterogeneity. The I ² statistic can take values between 0 and 100%: I ² = 0% means that all of the heterogeneity is due to sampling error; I ² = 100% means that all variability is due to true heterogeneity between studies. Where feasible, it was intended to use funnel plots to assess the impact of publication bias.

Additional Quantitative Analyses on Overall ‘Relative Effect Size’ of IHT

Sensitivity Analyses

We planned sensitivity analyses based separately on (1) our risk-of-bias ratings and on (2) our assessments of external validity. We aimed to reflect in sensitivity analysis any trial that was categorised as ‘(much) more pragmatic than explanatory’ and whose internal validity was limited by high risk of bias in domain IIIA only (blinding of participants and/or study personnel)—see also Rating of trials for risk of bias, above.

Sub-group Analyses

Comparative forest plots were planned—for each of study designs (Ia), (Ib) and (II)—on the following sub-groups of trial attributes:

• Whether or not a pilot (or ‘preliminary’ or ‘feasibility’) study, as defined by the original authors;

• Whether or not sample size > median for all trials with extractable data;

• Whether or not potency/potencies of homeopathic medicines ≥ 12C;

• Whether or not the research sponsor is an organisation (e.g. homeopathic pharmacy) with potential vested interest in the trial findings.

Results

Included Studies

The PRISMA flowchart from the original comprehensive literature search (up to and including 2011) was published previously.[24] An updated PRISMA flowchart is given in [Fig. 1], identifying a total of 565 records;[e] 466 remained after removal of duplicates. After excluding 97 due to type of record (book chapter, thesis, abstract and other minor article), 369 full-text records were then assessed for eligibility. Three hundred and fifty-eight records were excluded for the general reasons summarised in [Fig. 1]; seven of these same 358 were excluded from the present SR for the additionally specified reasons shown in [Supplementary File 4] (available in online version only).[f] The finally remaining 11 records (11 RCTs) were thus included in this SR; data were not extractable from three of those, leaving eight records potentially available for meta-analysis—see [Supplementary File 4] for details of the three records excluded from quantitative analysis.

Fig. 1 Updated PRISMA flowchart for all records published up to and including 2015.

Characteristics of Included Studies

The 11 RCTs (publication details given in [Supplementary File 4]) represent 11 clinical conditions across eight categories ([Table 1]). Homeopathic potency was >12C in all five trials for which relevant information was available. Seven trials were free of vested interest; one trial was not free of vested interest; three trials did not enable certainty in this assessment.

Table 1 Demographic data for 11 OTP-controlled RCTs of individualised homeopathic treatment: three studies excluded from meta-analysis are shown in italics

No.	First author	Year	Study design	Category	Condition	Participants' demographics	Study setting	Homeopathic potency; frequency	Comparator	Funding source	Free of vested interest [a]
A138	Adler	2009	Ia	Mental disorder	Depression	Referred patients who met DSM-IV criteria for depression	Outpatient Clinic of Homeopathy and Depression, Medical School, São Paulo, Brazil	Q; 3 times per week	[b]Fluoxetine; 20 mg once per day	HN-Pharmacy donated the homeopathic potencies	N
A140	Gmünder	2002	Ia	Musculoskeletal	Low back pain	Patients with chronic back pain lasting longer than 7 weeks: male and female patients, aged 18–70 years	Department for Physical Medicine and Rheumatology, Orthopaedic University Clinic, Switzerland	Not stated	[b]'Standard physiotherapy'; 8 sessions with stabilising, strengthening measures, supported by heat applications and massage	None stated	U
A141	Harrison	1999	Ia	Ear, nose and throat	Otitis media (chronic)	Patients aged 18 months to 8 years with positive diagnosis of OME; hearing loss >20dB; abnormal tympanogram	Two paediatric/ENT specialist centres in England	Not stated	Standard care ('watch and wait'); course of low-dose antibiotics for 4–6 weeks in some cases	Research Council for Complementary Medicine	Y
A143	Paterson	2003	Ib	Gastroenterology	Dyspepsia	Patients with dyspepsia presenting in one general practice	Regional Medical Center, England	Not stated	Normal GP care	NHS South & West Region R&D project grant. Medical Centre funded as research general practice	Y
A144	Relton	2009	Ib	Rheumatology	Fibromyalgia	Adults with formal diagnosis of primary fibromyalgia	Regional Rheumatology Department, England	Not stated	[b]Usual care: one or more of physiotherapy, aerobic exercise, analgesics, NSAIDs, anti-depressants	NHS Hospital Foundation Trust; Homeopathy Action Trust Charity; Department of Health funded Centre for Research Capacity Development	Y
A148	van Erp	1996	Ia	Tropical disease	Malaria	Patients with standardised clinical signs of malaria, aged > 10 years	Clinic in Rural Ghana	200C; one granule	[b]Chloroquine; 25mg/kg body weight	None stated	U
A149	Witt	2009	Ia	Obstetrics and gynaecology	Vulvovaginal candidiasis	Women ≥18 years, with ≥ four episodes of Candida vaginitis in previous year and with current acute symptoms	Medical School Outpatient Department, Austria	30–1,000C; frequency not stated	[b]Itraconazole; 200 mg b.i.d., 1 day per month, with and without local lactobacilli	‘None’	Y
A282	Relton	2012	Ia	Obstetrics and gynaecology	Menopausal syndrome	Women aged 45–64 years, experiencing 14 or more hot flushes or night sweats per week	Menopause Clinic in Sheffield, England	Not stated	Standard care: not specified (not HRT)	Department of Health National Co-ordinating Centre for Research Capacity Development	Y
A283	Sinha	2012	Ia	Ear, nose and throat	Otitis media (acute)	Children of either gender, aged 2–6 years; earache of not more than 36 hours duration; bulging tympanic membrane	Regional Research Institute of Homeopathy, Jaipur, India	LM; 2–6 hourly, depending on need	[b]Symptomatic relief: analgesic, anti-inflammatory and antipyretic (antibiotics prescribed if less than 50% improved at day 3)	None stated	U
A296	Peckham	2014	Ib	Gastroenterology	Irritable bowel syndrome	IBS diagnosis using Rome III criteria; aged ≥ 18	Regional Hospital Clinic, England	Not stated	Usual care: not specified	Barnsley Hospital Small Grants Fund; Friends of Barnsley Hospital; Homeopathy Research Institute	Y
A304	Frass	2015	Ib	Miscellaneous	Cancer	All cancer patients aged > 18 years, prior to first conventional treatment (chemotherapy, radiation therapy, surgical therapy)	Medical School Oncology Department, Vienna, Austria	Q, C, LM; daily	Conventional cancer therapy: chemotherapy, radiation, or surgery	‘None’	Y

Abbreviations: DSM-IV, Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition; HRT, hormone replacement therapy; NSAIDs, non-steroidal anti-inflammatory drugs; OME, otitis media with effusion; OTP, other than placebo; RCT, randomised controlled trial; N, no; U, unclear; Y, yes.

^a Vested interest: Support (direct, through research sponsorship; indirect, via gifted medicines) from company that provided homeopathic medicines for the trial.

^b Best effective standard care.

Summary of Findings

For each trial, [Table 2] includes details of the sample size, the identified main outcome measure (and whether dichotomous or continuous), the end-point, and several other study attributes. Six trials were described in the original article as a ‘pilot’ (or ‘preliminary’ or ‘feasibility’) study. A power calculation was performed for four of the trials. ITT was the basis for original analysis in six trials. Mean attrition rate per trial was 20.1%. The main outcome variable was dichotomous in four studies and continuous in the other seven. The total sample size for the eight meta-analysable trials was 675; the median sample size was 59 (inter-quartile range, 45.5–75.5). The eight analysable studies included eight different main outcome measures and for an end-point that ranged from 7 days to 12 months.

Table 2 Summary of findings table: three studies excluded from meta-analysis shown in italics

Study design Ia: IHT versus other intervention
No.	First author	Year	Pilot	Power calc.	ITT sample	PP sample	PP sample > median (59)	Attrition rate %	Original primary analysis	‘Main’ outcome identified	Nature of 'main' outcome	End-point
A138	[a]Adler	2009	N	N	91	55	N	39.6	PP	MADRS score	Continuous	8 weeks
A140	Gmünder	2002	Y	N	43	37		14.0	PP	Oswestry Questionnaire Score	Continuous	8 weeks
A141	Harrison	1999	Y	N	35	33	N	5.7	PP	Treatment success: hearing loss of less than 20 dB	Dichotomous	12 months
A148	van Erp	1996	N	N	74	55	N	25.7	PP	Improvement in at least three of listed symptoms of malaria	Dichotomous	1 week (for daily attacks); 3 weeks (for weekly relapses)
A149	Witt	2009	N	Y	150	71	Y	52.7	PP	Candida-free status	Dichotomous	12 months
A282	Relton	2012	Y	N	41	40		2.4	ITT (included pts who refused Rx)	Hot flush frequency severity score	Continuous	36 weeks
A283	Sinha	2012	Y	N	81	80	Y	1.2	ITT (LOCF; but N/A at 7d end-point)	‘Cure rate’	Dichotomous	7 days duration
Study design Ib: IHT + other intervention versus other intervention alone
No.	First author	Year	Pilot	Power calc.	ITT sample	PP sample	PP sample > median (62.5)	Attrition rate %	Original primary analysis	‘Main’ outcome identified	Nature of 'main' outcome	End-point
A143	Paterson	2003	Y	N	40	32		20.0	ITT	MYMOP	Continuous	6 months
A144	Relton	2009	Y	Y	47	36	N	23.4	ITT/PP	FIQ total score	Continuous	22 weeks
A296	Peckham	2014	N	Y	72	63	Y	12.5	ITT/PP	Number who achieved clinically relevant change of 50% in IBS SSS	Continuous	26 weeks
A304	Frass	2015	N	Y	373	282	Y	24.4	ITT	Global health status	Continuous	4 months

Abbreviations: FIQ, fibromyalgia impact questionnaire; IBS, Irritable Bowel Syndrome; IHT, individualised homeopathic treatment; ITT, intention to treat; MADRS, Montgomery & Åsberg Depression Rating Scale; MYMOP, Measure Yourself Medical Outcome Profile; PP, per protocol; SSS, Symptom Severity Score; N, no; Y, yes.

^a Non-inferiority trial.

Risk of Bias

[Table 3] provides the risk-of-bias details for each of the 11 trials, and sub-divided by study design: (Ia) the seven with other-intervention control; (Ib) the four that were ‘[A + B] versus B’. Domain IIIA (blinding of participants and/or study personnel), domain IIIB (blinding of outcome assessors) and domain IV (completeness of outcome data) presented the greatest methodological concerns. The three trials with high risk of bias for domain V were so because they did not contain data that were extractable for meta-analysis. Domain I (sequence generation) and domain II (allocation concealment) presented the most uncertain methodological judgments.

Table 3 Risk-of-bias and external validity assessments for RCTs: three studies excluded from meta-analysis are shown in italics

Study design Ia: IHT versus other intervention
			Risk-of-bias domain
No.	First author	Year	I	II	IIIA	IIIB	IV	[a] V	VI	Risk of bias	Risk-of-bias rating	PRECIS assessment (total score per assessor)
A138	Adler	2009	Y	Y	Y	Y	N	Y	Y	High	C1.0	More explanatory than pragmatic (24; 24)
A140	Gmünder	2002	U	U	N	N	Y	[b]N	U	High	C3.3	Equally pragmatic and explanatory (34; 31)
A141	Harrison	1999	U	N	N	U	Y	Y	N	High	C3.2	Equally pragmatic and explanatory (33; 28)
A148	van Erp	1996	U	U	Y	U	N	Y	U	High	C1.4	Equally pragmatic and explanatory (33; 33)
A149	Witt	2009	Y	U	N	N	N	U	U	High	C3.3	More explanatory than pragmatic (21; 19)
A282	Relton	2012	U	Y	[c]N	N	Y	[b]N	N	High	C4.1	Much more pragmatic than explanatory (44; 43)
A283	Sinha	2012	Y	U	U	Y	Y	Y	Y	Uncertain	B2	Equally pragmatic and explanatory (31; 32)
Study design Ib: IHT + other intervention versus other intervention alone
A143	Paterson	2003	U	Y	[c]N	N	U	[b]N	Y	High	C3.1	More pragmatic than explanatory (38; 38)
A144	Relton	2009	Y	Y	[c]N	N	N	Y	Y	High	C3.0	More pragmatic than explanatory (36; 36)
A296	Peckham	2014	Y	Y	[c]N	Y	U	Y	Y	High[d]	C1.1	Much more pragmatic than explanatory (43; 45)
A304	Frass	2015	Y	Y	[c]N	N	N	Y	U	High	C3.1	More pragmatic than explanatory (37; 38)

Abbreviations: IHT, individualised homeopathic treatment; RCT, randomised controlled trial; N, no (not free of bias); U, unclear; Y, yes (free of bias).

^a Unless a published study protocol was available, completeness of reporting was judged solely on correspondence of Results section with details in Methods section of article.

^b Data not extractable for meta-analysis.

^c Expected high risk of bias in domain IIIA—trial assessed as ‘(much) more pragmatic than explanatory’.

^d High risk of bias in domain IIIA only: trial is otherwise ‘Uncertain risk of bias’ overall.

There were no A- or B1*-rated trials. One trial (A283, Sinha) was rated uncertain risk of bias (‘B2’-rated). Ten trials were assessed as high risk of bias (‘C’-rated); one of those (A296, Peckham) was judged high risk of bias for domain IIIA only. Seven ‘C’-rated trials were deficient in three or more domains of assessment (‘C3.0’-rated or worse). A summary risk-of-bias bar graph is shown in [Supplementary File 5] (available in online version only).

Pragmatic/Explanatory Attitude

Independent assessment identified a total of 26 domains (one to six domains per trial) for which the two scores were ≥ 2 apart. In all cases, consensus discussion narrowed the score discrepancy per domain to no more than 1, which was our target. There was no trial for which the consequent two total scores lay on either side of a threshold for overall characterisation of pragmatic/explanatory attitude: a final designation per trial was thus readily apparent ([Table 3]). Five of the 11 trials were more pragmatic than explanatory, two were more explanatory than pragmatic and four were equally pragmatic and explanatory. All four of the trials with study design Ib were more (or much more) pragmatic than explanatory in attitude.

Across all trials, each of 19 domains was given a score of 3 due to the absence of sufficient information in the original article: this lack of information pertained especially to domain 8 (participant compliance) and domain 9 (protocol adherence).

Meta-Analysis

The selected main outcome was not estimable for three studies (see [Supplementary File 4]), for which group SDs for the designated end-point could not be derived. For the remaining eight studies, five were in study-design category Ia (Other-intervention control) and three were in category Ib (‘[A + B] versus B’); there were no trials in category II (No-treatment control). One category Ia study (A138, Adler) was a non-inferiority trial and was examined separately.

Overall Relative Effect Sizes

For the superiority trials, it transpired that the distribution of dichotomous and continuous main outcome measures was the same as that of trials with study design Ia and Ib; the non-inferiority trial's main outcome measure was a continuous variable. For a given study design, therefore, there was no need to transform/combine data from effect estimates expressed as OR or SMD.

Study design Ia: Other-intervention control: The pooled effect estimate for all four superiority trials was OR = 0.87 (95% CI: 0.30 to 2.48; p = 0.79)—see [Fig. 2]. I ² = 70%, indicating significant statistical heterogeneity between studies. There was also very substantive clinical heterogeneity amongst the four studies, with conditions of investigation comprising otitis media, vulvovaginal candidiasis and malaria. Three of the trials had a non-significant effect favouring homeopathy; the fourth trial (A149, Witt) had a statistically significant effect favouring the comparator.

Fig. 2 Forest plot for four analysable randomised controlled trials of individualised homeopathic treatment (IHT): study design Ia (IHT versus other intervention). Abbreviations: CI, confidence interval; df, degrees of freedom; M-H, Mantel-Haenszel method.

As shown in [Table 3] and summarised in [Fig. 2], three of the four trials displayed high risk of bias; the fourth trial (A283, Sinha) had uncertain risk of bias. Three trials were equally pragmatic and explanatory in attitude; the fourth trial (A149, Witt) was more explanatory than pragmatic.

With such similar attributes and small numbers of studies, we did not pursue sensitivity analyses based on risk of bias or on external validity. Small numbers also made redundant our planned sub-group analyses.

Non-inferiority trial (A138, Adler): The effect estimate for this single trial was SMD = –0.41 (95% CI: –0.95 to 0.12; p = 0.13)—see [Fig. 3]. This finding, reflecting also the original authors' designated margin of non-inferiority, is consistent with a conclusion that IHT was non-inferior to fluoxetine in the treatment of depression for these subjects. The trial displayed high risk of bias due to its 39.6% patient attrition rate. Its study attitude was more explanatory than pragmatic.

Fig. 3 Forest plot for non-inferiority randomised controlled trial of individualised homeopathic treatment (IHT): study design Ia (IHT versus other intervention). Abbreviations: CI, confidence interval; IV, inverse-variance method; SD, standard deviation; Std., Standardised.

Study design Ib: ‘[A + B] versus B’: The pooled effect estimate for all three trials was SMD = –0.26 (95% CI: –0.47 to –0.05; p = 0.02)—see [Fig. 4]. I ² = 0%, indicating that heterogeneity was due to sampling error alone. Again, however, there was very substantive clinical heterogeneity amongst the studies, with conditions of investigation comprising menopausal syndrome, irritable bowel syndrome and cancer. Each of two trials had a non-significant effect favouring adjunctive IHT; the third trial (A304, Frass) had a marginally statistically significant effect favouring adjunctive IHT.

Fig. 4 Forest plot for three analysable RCTs of IHT: study design Ib (IHT + other intervention versus other intervention alone). Abbreviations: CI, confidence interval; IV, inverse-variance method; SD, standard deviation; Std., Standardised.

As shown in [Table 3] and summarised in [Fig. 4], each of the three trials displayed high risk of bias. The study attitude of each was judged to be more (or much more) pragmatic than explanatory. One of the RCTs (A296, Peckham) had high risk of bias in domain IIIa only: a sensitivity analysis would have taken that factor into account.

As for the Ia trials above, we did not undertake sensitivity or sub-group analyses because of the overall similarity of study quality and the small number of studies.

Disease- and Category-Specific Relative Effect Sizes

With 11 unique clinical conditions, it was not possible to carry out disease-specific analyses. Ear Nose & Throat was the only category of condition for which data were available from at least two studies: chronic otitis media (A141, Harrison) and acute otitis media (A283, Sinha). The pooled effect estimate was OR = 1.28 (95% CI: 0.61 to 2.70; p = 0.52)—[Fig. 5]. It is thus inconclusive whether IHT and the comparator interventions differed in effectiveness. These two trials had high/unclear risk of bias; in both, the study attitude was equally pragmatic and explanatory.

Fig. 5 Forest plot for two RCTs of IHT: clinical category, Ear Nose and Throat. Abbreviations: CI, confidence interval; df, degrees of freedom; M-H, Mantel-Haenszel method.

Discussion

None of the 11 studies was judged to comprise reliable evidence (i.e. there were no A- or B1*-rated trials), 10 being assessed as high risk of bias. Three of the 11 articles failed to yield data suitable for meta-analysis, leaving eight studies in three different categories of study design: collectively for four heterogeneous trials with study design Ia, the effects of IHT were inconclusive; for three clinically heterogeneous studies with design Ib, there was a statistically significant effect favouring adjunctive IHT; in the remaining trial IHT was found non-inferior to fluoxetine in the treatment of depression. Lack of replication of subject matter prevented meta-analysis of IHT by clinical condition, and our analysis by category of condition was limited to pooling data from two trials of otitis media (chronic and acute disease), which found a non-significant effect of IHT (pooled OR = 1.28; p = 0.52). The low intrinsic quality and number of studies preclude any clear conclusions about comparative effectiveness of IHT. The same limitations prevented analysis of any sub-group data. An additional literature search has revealed only two further OTP-controlled trials that might have been eligible for inclusion in the current article.[34] [35]

Within the pragmatic–explanatory continuum, the internal validity for pragmatic study design would typically be lower than that for explanatory design.[36] Logically, therefore, an intrinsically low-quality OTP-controlled study might be expected to have more emphasis placed on external validity; however, it is notable that, despite their OTP-controlled design, only five of the studies possessed a truly pragmatic study attitude. The external validity of six trials was therefore no better than that of an overtly explanatory trial, with limited generalisability of findings to a wider population of patients. The non-inferiority of IHT to fluoxetine in the treatment of depression, observed in one trial (A138, Adler), was found to lack the external validity that the original authors presumably intended. By contrast, the studies that examined adjunctive homeopathy were all positioned within the pragmatic band of the pragmatic–explanatory continuum, affirming generalisability of their findings.

Although the PRECIS tool was originally developed to help at the design stage of clinical trials,[22] we found it fairly straightforward to apply in assessing the pragmatic/explanatory attitude of a completed trial. We recognise also that the empirical scoring notation we devised diverges from the specific methods suggested in the original PRECIS article;[22] however, our approach has allowed us to approximate a given trial's positioning on the pragmatic–explanatory continuum, and is in keeping with the original PRECIS authors' receptiveness to further methodological innovation.[22] The 10-domain judgmental criteria were operationalised successfully, though there was often a lack of information in the published articles to form a clear opinion on participant compliance (domain 8) and protocol adherence (domain 9). Our scoring method may be viewed—like the original PRECIS authors' approach—as a ‘work in progress’.

The homeopathy community expresses reasonable concern about the sensitivity of the placebo-controlled trial design to detect a treatment effect of its medicines.[37] Indeed, the wider Complementary/Alternative Medicine (CAM) community also recognises that research on a ‘whole system of care’, such as IHT, should avoid focus solely on ‘active ingredients’ (i.e. the efficacy of the medicine).[38] With its particular focus and emphasis, each type of study design in IHT is thus important: placebo-controlled design to examine proof of concept that homeopathic medicines may have a specific treatment effect; OTP-controlled design to evaluate that concept within a package of care that is everyday homeopathy practice. Our meta-analysis of placebo-controlled RCTs of IHT suggested evidence for a distinct effect of homeopathic medicines taken as individualised prescriptions;[2] our current analysis of OTP-controlled RCTs, however, has been less successful in evaluating that concept due to the low quality and quantity of relevant studies.

Unless the ‘double-dummy’ approach is adopted, OTP-controlled trials have expected high risk of bias regarding blinding of participants. In fact, our SR has highlighted that low risk of bias in OTP research may be feasible only in double-dummy trials (A138, Adler; A148, van Erp); however, such trials also have limited pragmatic attitude and a focus that is on the efficacy of the homeopathic medicine rather than on the whole package of homeopathic care. By contrast, OTP-controlled design may be optimal in studies of adjunctive IHT (‘[A + B] versus B’ design), where we found a prevailing pragmatic attitude and, in one case (A296, Peckham), a high risk of bias in domain IIIa only: our aim to accommodate that sole bias within sensitivity analysis was thwarted by the low quality and number of relevant trials. Some observers might argue, nevertheless, that the Cochrane-type system for rating internal validity is inappropriate for more pragmatically orientated trials of ‘whole-system’ interventional approaches such as homeopathy.

We emphasise that the extreme clinical heterogeneity of the current studies seriously limits the interpretation of the corresponding meta-analyses. The pooled effect estimates presented in [Fig. 2] and [Fig. 4] must be viewed with appropriate caution: indeed, these results are not represented in Conclusions. It should also be noted that ‘[A + B] versus B’ trial design in individualised CAM therapy has been criticised as being prone to ‘false positive’ results due to ‘non-specific effects’ of the intervention.[39] Any issues concerning model validity of these current trials—a key attribute, as previously emphasised[38] [40]—are unknown at present.

Conclusions

Due to the low quality, the small number and the extreme heterogeneity of studies, no decisive conclusion is currently possible regarding comparative effectiveness of IHT on health-related outcomes, assessed within the context of OTP-controlled trials. Generalisability of findings is limited by the variable extent of external validity observed overall. To date, the highest quality and the most pragmatic study attitude for RCTs of IHT is associated with the ‘[A + B] versus A’ approach. Future OTP-controlled trials in homeopathy should promote both internal validity and external validity, as informed by the domain-based assessments in the present article.

Conflicts of Interest

Authors RTM, SU-Z, PV, ERR and ESB are (or were) associated with a homeopathy charity whose principal aim is to clarify and extend an evidence base in homeopathy. RTM holds an independent research consultancy contract with the Deutsche Homöopathie-Union, Karlsruhe, Germany. RTM, SU-Z, PV, ERR and ESB have no other relationships or activities that could appear to have influenced the submitted work. Author LAL has no relationships or activities that could appear to have influenced the submitted work. Author JRTD had no support from any organisation for the submitted work; in the past 3 years, and for activities outside the submitted study, he received personal fees, royalties or out-of-pocket expenses for advisory work, invitational lectures, use of rating scales, published book chapters or committee membership; he receives royalties from Springer Publishing Company for his book, A Century of Homeopaths: Their Influence on Medicine and Health. J.T.R.D. has no other relationships or activities that could appear to have influenced the submitted study.

Acknowledgments

We thank Daniela Hacke, Karl und Veronica Carstens-Stiftung, Essen, Germany, for assistance with the literature search and retrieval of relevant full-text articles.

Highlights

• This systematic review focuses on randomised controlled trials (RCTs) of individualised homeopathic treatment (IHT) in which the control (comparator) group was other than placebo.

• For each eligible trial, risk of bias was assessed using Cochrane methods, and its relative pragmatic or explanatory attitude was approximated using the PRECIS tool.

• Eleven RCTs, representing 11 different medical conditions, were eligible for inclusion.

• Ten RCTs were rated ‘high risk of bias’; the other trial was rated ‘uncertain risk of bias’.

• Only five RCTs were judged to have pragmatic study attitude.

• Quantitative data extraction did not yield a decisive conclusion about the comparative effectiveness of IHT.

Authors' Contributions

RTM devised and led the study, developed the study protocol and contributed to all facets of the work. SU-Z helped to develop the study protocol, co-assessed trials for risk of bias, contributed to data interpretation and edited the manuscript. PV helped to develop the study protocol, co-assessed trials for pragmatic/explanatory study attitude, contributed to data interpretation and edited the manuscript. ERR helped to develop the study protocol, contributed to data interpretation and edited the manuscript. ESB contributed to statistical analysis and data interpretation and edited the manuscript. LAL helped to develop the study protocol, contributed to data interpretation and edited the manuscript. JRTD helped to develop the study protocol, contributed to data interpretation and edited the manuscript. All authors have applied the normal high standards of scientific method in the conduct of the work and of complete and transparent reporting in the write-up of the manuscript; they have all read and approved the submitted manuscript.

^a This study design is distinct from the eligible ‘A versus [A + B]’ design, and from eligible studies that allow concomitant conventional medication to remain ongoing in the subjects of each study group. Also, this criterion does not, per se, disallow studies where IHT intervention includes lifestyle and/or dietary advice in addition to a homeopathic prescription. See [Supplementary File 2] for tabulation of eligible and ineligible study designs.

^b Towards a Common Language for Functioning, Disability and Health. ICF: The International Classification of Functioning, Disability and Health. Geneva; World Health Organization, 2002.

^c OR > 1 favours homeopathy.

^d SMD < 0 favours homeopathy.

^e Complete details of all 565 records are available from the same website that contains the study protocol.[23]

^f [Supplementary File 4] represents an update of the flowchart that is included in the study protocol.[23]

• References

• 1 Swayne J. International Dictionary of Homeopathy. Edinburgh: Churchill Livingstone; 2000
• 2 Mathie RT, Lloyd SM, Legg LA. , et al. Randomised placebo-controlled trials of individualised homeopathic treatment: systematic review and meta-analysis. Syst Rev 2014; 3: 142
• 3 Mathie RT, Ramparsad N, Legg LA. , et al. Randomised, double-blind, placebo-controlled trials of non-individualised homeopathic treatment: systematic review and meta-analysis. Syst Rev 2017; 6: 63
• 4 Jacobs J, Jonas WB, Jiménez-Pérez M, Crothers D. Homeopathy for childhood diarrhea: combined results and metaanalysis from three randomized, controlled clinical trials. Pediatr Infect Dis J 2003; 22: 229-234
• 5 Taylor MA, Reilly D, Llewellyn-Jones RH, McSharry C, Aitchison TC. Randomised controlled trial of homoeopathy versus placebo in perennial allergic rhinitis with overview of four trial series. BMJ 2000; 321: 471-476
• 6 Schneider B, Klein P, Weiser M. Treatment of vertigo with a homeopathic complex remedy compared with usual treatments: a meta-analysis of clinical trials. Arzneimittelforschung 2005; 55: 23-29
• 7 Ernst E, Barnes J. Are homoeopathic remedies effective for delayed-onset muscle soreness? A systematic review of placebo-controlled trials. Perfusion (Nürnberg) 1998; 11: 4-8
• 8 Ernst E. Homeopathic prophylaxis of headaches and migraine? A systematic review. J Pain Symptom Manage 1999; 18: 353-357
• 9 Ernst E. Homeopathy for insomnia and sleep-related disorders: a systematic review of randomised controlled trials. Focus Altern Complement Ther 2011; 16: 195-199
• 10 Smith CA. Homoeopathy for induction of labour. Cochrane Database Syst Rev 2004; CD003399
• 11 Mathie RT, Frye J, Fisher P. Homeopathic Oscillococcinum® for preventing and treating influenza and influenza-like illness. Cochrane Database Syst Rev 2015; 1: CD001957
• 12 Long L, Ernst E. Homeopathic remedies for the treatment of osteoarthritis: a systematic review. Br Homeopath J 2001; 90: 37-43
• 13 Kleijnen J, Knipschild P, ter Riet G. Clinical trials of homoeopathy. BMJ 1991; 302: 316-323
• 14 Boissel JP, Cucherat M, Haugh M, Gauthier E. Critical literature review on the effectiveness of homoeopathy: overview of data from homoeopathic medicine trials. In: Homoeopathic Medicine Research Group Report. Commission of the European Communities' Directorate-General XII – Science, Research and Development, Directorate E – RTD Actions: Life Sciences and Technologies – Medical Research. Brussels: Commission of the European Communities; 1996
• 15 Linde K, Clausius N, Ramirez G. , et al. Are the clinical effects of homeopathy placebo effects? A meta-analysis of placebo-controlled trials. Lancet 1997; 350: 834-843
• 16 Linde K, Scholz M, Ramirez G, Clausius N, Melchart D, Jonas WB. Impact of study quality on outcome in placebo-controlled trials of homeopathy. J Clin Epidemiol 1999; 52: 631-636
• 17 Cucherat M, Haugh MC, Gooch M, Boissel JP. ; Homeopathic Medicines Research Advisory Group. Evidence of clinical efficacy of homeopathy. A meta-analysis of clinical trials. Eur J Clin Pharmacol 2000; 56: 27-33
• 18 Shang A, Huwiler-Müntener K, Nartey L. , et al. Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy. Lancet 2005; 366: 726-732
• 19 Ernst E. Classical homeopathy versus conventional treatments: a systematic review. Perfusion (Nürnberg) 1999; 12: 13-15
• 20 Treweek S, Zwarenstein M. Making trials matter: pragmatic and explanatory trials and the problem of applicability. Trials 2009; 10: 37
• 21 Bornhöft G, Maxion-Bergemann S, Wolf U. , et al. Checklist for the qualitative evaluation of clinical studies with particular focus on external validity and model validity. BMC Med Res Methodol 2006; 6: 56
• 22 Thorpe KE, Zwarenstein M, Oxman AD. , et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol 2009; 62: 464-475
• 23 Mathie RT, Ulbrich-Zürni S, Viksveen P. , et al. Systematic review and meta-analysis of randomised, other-than-placebo (OTP) controlled, trials of individualised homeopathic treatment (IHT): Study protocol. Version 1.2; 21 April 2017. Available at: https://www.hri-research.org/hri-research/learning-more-from-existing-evidence/systematic-review-programme/
• 24 Mathie RT, Hacke D, Clausen J, Nicolai T, Riley DS, Fisher P. Randomised controlled trials of homeopathy in humans: characterising the research journal literature for systematic review. Homeopathy 2013; 102: 3-24
• 25 Baker DG, Myers SP, Howden I, Brooks L. The effects of homeopathic Argentum nitricum on test anxiety. Complement Ther Med 2003; 11: 65-71
• 26 Higgins JPT, Altman DG. Assessing risk of bias in included studies. In: Higgins JPT, Green S. , eds. Cochrane Handbook for Systematic Reviews of Interventions; Version 5.1.0. The Cochrane Collaboration; 2011. . Chapter 8
• 27 Hewitt CE, Torgerson DJ, Miles JNV. Is there another way to take account of non-compliance in randomized controlled trials?. Can Med Assoc J 2007; 175: 347-348
• 28 Higgins JPT, Deeks JJ, Altman DG. Special topics in statistics. In: Higgins JPT, Green S. , eds. Cochrane Handbook for Systematic Reviews of Interventions; Version 5.1.0. The Cochrane Collaboration; 2011. . Chapter 16
• 29 Sedgwick P. Equivalence trials. BMJ 2013; 346: f184
• 30 Sedgwick P. What is a non-inferiority trial?. BMJ 2013; 347: f6853
• 31 Deeks JJ, Higgins JPT, Altman DG. Analysing data and undertaking meta-analyses. In: Higgins JPT, Green S. , eds. Cochrane Handbook for Systematic Reviews of Interventions; Version 5.1.0. The Cochrane Collaboration; 2011. . Chapter 9
• 32 Chinn S. A simple method for converting an odds ratio to effect size for use in meta-analysis. Stat Med 2000; 19: 3127-3131
• 33 Schünemann HJ, Oxman AD, Vist GE. , et al. Interpreting results and drawing conclusions. In: Higgins JPT, Green S. , eds. Cochrane Handbook for Systematic Reviews of Interventions; Version 5.1.0. The Cochrane Collaboration; 2011. . Chapter 12
• 34 Manchanda RK, Nair KRJ, Varanasi R. , et al. A randomized comparative trial in the management of alcohol dependence: individualized homoeopathy versus standard allopathic treatment. Indian J Res Homeopathy 2016; 10: 172-181
• 35 Viksveen P, Relton C, Nicholl J. Depressed patients treated by homeopaths: a randomised controlled trial using the “cohort multiple randomised controlled trial” (cmRCT) design. Trials 2017; 18: 299
• 36 Sedgwick P. Explanatory trials versus pragmatic trials. BMJ 2014; 349: g6694
• 37 Weatherley-Jones E, Thompson EA, Thomas KJ. The placebo-controlled trial as a test of complementary and alternative medicine: observations from research experience of individualised homeopathic treatment. Homeopathy 2004; 93: 186-189
• 38 Verhoef MJ, Lewith G, Ritenbaugh C, Boon H, Fleishman S, Leis A. Complementary and alternative medicine whole systems research: beyond identification of inadequacies of the RCT. Complement Ther Med 2005; 13: 206-212
• 39 Ernst E, Lee MS. A trial design that generates only “positive” results. J Postgrad Med 2008; 54: 214-216
• 40 Mathie RT, Roniger H, Van Wassenhoven M. , et al. Method for appraising model validity of randomised controlled trials of homeopathic treatment: multi-rater concordance study. BMC Med Res Methodol 2012; 12: 49