Thursday, September 11, 2008

A great article about Evidence and Bias

NeuroLogica Blog » An Acupuncture Debate
An Acupuncture Debate
Published by Steven Novella under Science and Medicine
Comments: 15

Recently I was invited to write my views on acupuncture for a website called Opposing Views. I pre-published (with permission) my side of the debate on “Does Acupuncture Work” here at NeuroLogica. Taking the pro-acupuncture side is Bill Reddy - his profile states that he is “currently serving on the Executive Committee of the American Association of Acupuncture and Oriental Medicine.”

The format of the website allows for moderated comments, which are intended to allow for a written debate with the two sides. Here are my responses to the first round of arguments.

Bill Reddy wrote:

The National Center for Complementary and Alternative Medicine (NCCAM) of the National Institutes of Health (NIH) worked with the World Health Organization (WHO) in 1983 to develop a consensus on the use of Acupuncture to treat a number of common illnesses.

He then give a long list of medical conditions acupuncture is supposed to treat.

Bill Reddy asserts that the WHO and NCCAM in 1983 have reviewed the evidence and support acupuncture as safe and effective for a number of medical conditions, but this assertion is simply wrong. Bill does not provide a specific reference or any quotes.

One reason to doubt the claim is that the NCCAM did not exist in 1983. It began its existence as the Office of Alternative Medicine in 1993.

The WHO had this to say about acupuncture in 2002:

“Since the methodology of clinical research on acupuncture is still under debate, it is very difficult to evaluate acupuncture practice by any generally accepted measure.”


“Only national health authorities can determine the diseases, symptoms and conditions for which acupuncture treatment can be recommended.”

Finally, reviews of the acupuncture literature performed by scientists and academics, rather than political organizations, have consistently found that there is no compelling evidence that acupuncture is effective for any condition.

Even politically motivated and biased panels, when constrained by the scientific evidence, must admit that the evidence does not support any specific effect from acupuncture. The 1997 NIH consensus conference on acupuncture was packed with acupuncture proponents and did not contain any skeptics or scientists who had found negative results. Despite this obvious bias the panel had to concede that the best controlled trials showed no difference between “sham” acupuncture and “true” acupuncture for any indication, but then tried to spin these results by saying that well-controlled trials must therefore be unreliable.

Bill’s strategy is ultimately deceptive. He cites a 25 year old political assessment of acupuncture, then vaguely refers to later research. In fact later better-designed trials of acupuncture have been essentially negative.

In his next point he writes:

As a former Aerospace Engineer who flight tested helicopters and performed cutting-edge research in the field of turbulent aerodynamics, I can honestly say that acupuncture is a thoroughly proven system of healthcare. Proving acupuncture efficacy is relatively easy - western medicine and modern science are having a difficult time understanding the underlying mechanisms as to how acupuncture works.

If you type in the key word “acupuncture” into the NIH national library of medicine “PubMed” database, it will return over 13,000 peer reviewed journal articles.

This argument shows very poor scientific reasoning. Arguments concerning mechanism of action and efficacy are mixed together in a confusing way. The lack of a plausible mechanism makes acupuncture very suspect, and raises the bar for an appropriate level of evidence before accepting acupuncture as a treatment. But the clinical evidence of acupuncture stands on its own – and is convincingly negative.

Bill Reddy gives us his anecdotal experience (combined with an argument from authority logical fallacy) and cherry-picked studies to support his claims for acupuncture effectiveness in fertility. However, systematic reviews of the evidence are negative. Most recently, this review of acupuncture for fertility rates following IVF was completely negative. (A systematic review and meta-analysis of acupuncture in in vitro fertilisation. BJOG. 2008 Sep;115(10):1203-13. Epub 2008 Jul 23.)

In response to this study, Paul Robin, the chairman of the Acupuncture Society, said:

“I’m really surprised by these findings. I’ve been treating people for 20 years and in my experience treatment does seem to improve their chances of becoming pregnant. This study has shown that there’s no proof that acupuncture can help - so that suggests that there should be lots more studies to examine the question. I’m convinced it can help.”

Robin shows the closed-mindedness of dedicated proponents. He is perplexed that the scientific evidence does not support his personal anecdotal experience, and concludes that therefore we need more evidence. Rather he should conclude that anecdotal experience is unreliable – something the scientific community has learned long ago.

Bill concludes with the argument that because there is a great deal of research going on there must be something to acupuncture, but this is a flawed argument ad populi. The research reflects cultural interest and belief, not necessarily scientific validity. If you actually review that research, as many have, it is clearly negative.

Bill Reddy next launches into the tired argument that scientific medicine is dangerous and acupuncture is at least safe, writing:

In a Japanese survey of 55,291 acupuncture treatments given over five years by 73 acupuncturists, 99.8% of them were performed with no significant minor adverse effects and zero major adverse incidents (Hitoshi Yamashita, Bac, Hiroshi Tsukayama, BA, Yasuo Tanno, MD, PhD. Kazushi Nishijo, PhD, JAMA).

According to wikipedia “A recent study by Healthgrades found that an average of 195,000 hospital deaths in each of the years 2000, 2001 and 2002 in the U.S. were due to potentially preventable medical errors.

This argument is yet another logical fallacy – a non sequitur. The state of modern science-based medicine says nothing about the purported mechanisms and effectiveness of acupuncture. You could use the same line of argument to falsely support any medical claim. The data cited, which is now a standard argument among proponents of dubious health claims, are also highly misleading. They only consider the risk of medical intervention, not the benefit. A proper assessment of any intervention should include risk vs benefit. This is the standard calculation of science-based medicine, and treatments are only considered justified if they can demonstrate benefit in excess of risk. Focusing only on risk is designed to create a misleading impression.Direct risk is generally a function of the invasiveness of any intervention. No one doubts the fact that less invasive interventions are less directly risky. But if only risk were considered then doing nothing would always be the best option – and it clearly isn’t. Acupuncture is certainly closer to doing nothing than many legitimate medical interventions, but even the small risk of acupuncture is not justified until there is compelling evidence of benefit – and there isn’t.

Further, indirect risk must also be considered. While direct harm from acupuncture is rare (but does occur) there is the potential for tremendous indirect harm, including the delay in proper medical assessment and treatment.

Bill concludes with a classic argument ad populi:

If you had a severe case of tennis elbow, and you went to see an oriental medicine practitioner and he stuck needles in you and you didn’t get better, would you ever go back to see him again? Probably not.

1.3 billion Chinese are hard to argue with. The system of healing would not last over 5000 years if it weren’t effective.

This is nothing more than an argument ad populi and appeal to anecdotal evidence. As I have already argued, blood letting survived for over 2,000 years as the standard accepted practice. The placebo effect is well documented, as well as a variety of psychological effects, such as confirmation bias. There are also statistical effects, such as regression to the mean (the tendency for any ailment to get better if one seeks treatment during an exacerbation). All of these combine to create the illusion that ineffective treatments have some effect.

The scientific method as applied to medicine evolved to address this potential for self-deception. Appealing to anecdotes is a decidedly unscientific approach to any medical question, and defies the well-documented history of medicine – which gives countless examples of treatments that were believed to work but failed to show any effect when properly studied.

Bill concludes with a discussion of the limitations of acupuncture:

Musculoskeletal pain, internal conditions, dermatological and neurological conditions all respond favorably to acupuncture. Where the modality falls short is in structural problems such as spinal stenosis (gradual narrowing of the spinal canal leading to nerve compression). All the needles in the world won’t remove the built-up calcifications to relieve the pressure on a nerve root.

I certainly agree with this point - acupuncture does not treat any anatomical or mechanical medical problem. However, this does not imply that it does treat non-mechanical problems, nor does it render any of the other arguments put forth for acupuncture more reasonable.

I would also point out that this is a common feature of dubious medical interventions - they do not treat anything which is objectively demonstrable. They thrive in the realm of subjective symptoms where anecdotes and placebo effects are strong and carefully controlled trials more difficult. They tend to thrive where the data is poor. Eventually, when well-controlled data is obtained, any alleged effects vanish.


I only gave representative quotes from Bill Reddy’s arguments here, not wanting to reprint the entire debate. Go to Oppossing Views to take a look at the full debate and even leave a comment if you wish.


Amelia Hirota said...

As long as we try to speak the foreign language of Western science when looking at acupuncture, we acupuncturists in the West will continue to be misunderstood and criticized. Let's speak our language of TCM and be evaluated on the "true" effects of our medicine. In the clinic, I educate my patients on the basics of TCM and then speak in terms of TCM, not nerve impulses and neurotransmitters. For the most part they get it and more importantly, the results speak to them.

Anonymous said...

Hi there! Would you mind if I share your blog with
my twitter group? There's a lot of people that I think would really enjoy your content. Please let me know. Thank you

Here is my homepage :: home cellulite treatment

Anonymous said...

Larimore laser treatment rosacea

My blog post; El Toro Marine Corps Air laser treatment rosacea