Clinical trials, clinical evidence, and selective citation

Dr Lionel Milgrom's reliance on the authority of Sir Michael Rawlins may be misplaced.[ ^1,2 ] To give the quotation its proper context, Sir Michael actually said this about randomised controlled trials (RCTs) in his Harveian Oration of October 2008:

“The technique has three great advantages. It minimizes bias, especially selection bias because every patient has an equal chance of receiving each of the treatments. It minimizes confounding factors because as a result of randomization these are likely to be evenly distributed among the groups. And provided the groups are of an appropriate size it minimizes the play of chance or random error as statisticians like to call it”. [ ³ ]

Shortly after that event, he said:

“As far as homeopathy is concerned it breaks every rule in the evidential base! It is biologically implausible; it is almost always used to treat conditions where the natural history is unpredictable; and the signal to noise ratio is close to one!” [ ⁴ ]

Perhaps more importantly, Milgrom misunderstands the now infamous pie chart published by BMJ Clinical Evidence. What the chart does not tell us is what proportion of patients receive a treatment that is ‘beneficial’. Unfortunately we are not told how many diseases are covered in this survey. If we look up the advice on a given condition (eg. low back pain) there are 18 treatments considered, but probably for other less common diseases there will be fewer treatments considered. Let us be generous and assume that on average there are 10 treatments per disease, then 2500 treatments are being used for 250 diseases. But we know that there are at least 12% of beneficial treatments, so 12% of 2500 is 300, so the data are compatible with 100% of patients having at least one effective treatment, with a spare 50 treatments for a second choice. What we need (and should try to collect) is the evidence for the treatments that patients actually receive.

Indeed that has already been done for several specialisms. Several published studies put the proportion of evidence based clinical practice (not treatments) at between 55% and 97%.[ ^5–15 ] Not good enough, but far better than the Clinical Evidence pie chart suggests. In any case we are unconvinced by the ‘tu quoque’ argument.

Milgrom also misunderstands the role of the RCT. We know of no-one who is advocating the ‘monoculture’ he describes. Sir Michael Rawlins was arguing quite rightly for a more inclusive approach to evidence, in particular because we need to know whether RCT evidence is generalisable to normal clinical practice. He did not say that RCTs should routinely be replaced by less rigorous methodologies. It may be tempting for chiropractors and homeopaths to rely on uncontrolled observational data when RCTs fail to give them the answers they want, but would that be good science?

• References

• 1 Milgrom L.R. When sorry seems to be the hardest word: CAM, free speech, and the British legal system. Homeopathy 2010; 99: 83-84.
• 2 Milgrom L.R. Gold standards, golden calves, and random reproducibility: why homeopaths at last have something to smile about. J Altern Complement Med 2009; 15 (03) 205-207.
• 3 Rawlins M. De Testimonio. Harveian Oration delivered to the Royal College of Physicians, London. 16th October 2008.
• 4 Rawlins M. Personal communication, 21st October 2008. Quoted with permission.
• 5 Myles P.S., Bain D.L., Johnson F., McMahon R. Is anaesthesia evidence-based? A survey of anaesthetic practice. Br J Anaesth 1999 Apr; 82 (04) 591-595.
• 6 Jemec G.B., Thorsteinsdottir H., Wulf H.C. Evidence-based dermatologic out-patient treatment. Int J Dermatol 1998 Nov; 37 (11) 850-854.
• 7 Michaud G., McGowan J.L., van der Jagt R., Wells G., Tugwell P. Are therapeutic decisions supported by evidence from health care research?. Arch Intern Med 1998; 158 (15) 1665-1668.
• 8 Howes N., Chagla L., Thorpe M. Surgical practice is evidence based. Br J Surg 1997; 84 (09) 1220-1223.
• 9 Kenny S.E., Shankar K.R., Rintala R., Lamont G.L., Lloyd D.A. Evidence-based surgery: interventions in a regional paediatric surgical unit. Arch Dis Child 1997; 76: 50-53 doi:10.1136/adc.76.1.50.
• 10 Geddes J.R., Game D., Jenkins N.E., Peterson L.A., Pottinger G.R., Sackett D.L. What proportion of primary psychiatric interventions are based on evidence from randomised controlled trials?. Qual Health Care 1996 Dec; 5 (04) 215-217.
• 11 Gill P., Dowell A.C., Neal R.D., Smith N., Heywood P., Wilson A.E. Evidence based general practice: a retrospective study of interventions in one training practice. BMJ 1996 Mar; 312 (7034): 819-821.
• 12 Ellis J., Mulligan I., Rowe J., Sackett D.L. Inpatient general medicine is evidence based. Lancet 1995 Aug; 12 (8972): 407-410.
• 13 Suárez-Varela M.M., Llopis-González A., Bell J., Tallán-Guerola M., Pérez-Benajas A., Carrión-Carrión C. Evidence based general practice. Eur J Epidemiol 1999 Oct; 15 (09) 815-819.
• 14 Slim K., Lescure G., Voitellier M., Ferrandis P., Le Roux S., Dumas P.J. et al. Is laparoscopic surgical practice “factual” (evidence based)? Results of a prospective regional survey. Presse Med 1998 Nov; 21 (36) 1829-1833.
• 15 Djulbegovic B., Loughran Jr. T.P., Hornung C.A., Kloecker G., Efthimiadis E.N., Hadley T.J. et al. The quality of medical evidence in hematology-oncology. Am J Med 1999; 106 (02) 198-205.