Thursday, September 11, 2008

Criticism of an Acapuncture trial

Respectful Insolence: The largest "randomized" acupuncture study ever done: Why did they even bother?
Believe it or not, there was one area of so-called "alternative" medicine that I used to be a lot less skeptical about than I am now. Homeopathy, I always realized to be a load of pseudoscientific magical thinking. Ditto reiki, therapeutic touch, and other forms of "energy healing." It didn't take an extensive review of the literature to figure that out, although I did ultimately end up doing fairly extensive literature reviews anyway. Then, the more I looked into the hodge-podge of "healing" modalities whose basis is not science but rather prescientific and often mystical thought, the less impressed I was.

Even so, there was always one modality that I gave a bit of a pass to. There was one modality that, or so I thought, might actually have something to it. There was one modality that seemed to have a bit of suggestive evidence that it might do something more than a placebo. I'm referring to acupuncture. No, I never bought all the mystical mumbo-jumbo about how sticking needles into "meridians" somehow alters or "unblocks" the flow of a mysterious "life force" known as qi that is undetectable by science. I did wonder if perhaps it worked as a counterirritant or by releasing endorphins.

Then I actually started paying attention to the scientific literature regarding acupuncture, including literature like this and this. The more I read, the more I realized something. I realized that there was far less to acupuncture than I had previously thought, and, even with my previous openness to it, I hadn't thought all that much about it anyway. What I had thought about it was that it might have a very mild beneficial effect. What I know now is that acupuncture is almost certainly no more than an elaborate placebo. What I know now is that virtually every study of acupuncture claiming to show a positive effect has serious methodological flaws and that the better-designed the study the less likely there is to be an effect. What I now know is that any study without a true "sham" acupuncture arm is worthless, and that well-designed studies show "sham" acupuncture to be no different than "real" acupuncture; i.e., no different than placebo.

And now comes yet another in a long line of studies that is consistent with just that, and, worse, it's billed as (and probably is) the "largest randomized study of acupuncture ever done." Too bad it depends on what you mean by "randomized." Too bad the press coverage misses the point:

For the current study, published in the journal Cephalalgia, German researchers followed more than 15,000 adults with chronic headaches; all had been suffering from either migraine or tension-type headaches at least twice a month for 1 year or more.

Of these patients, nearly 3,200 agreed to be randomly assigned to either have acupuncture added to their regular therapy or to stay with their usual care alone. The rest of the patients began on acupuncture treatment.

All of the acupuncture patients received up to 15 sessions over 3 months, and all study patients were reassessed after 6 months.

In the end, the study found, acupuncture patients reported greater pain improvements than those who stayed with their usual care only. At the outset, they reported an average of 8.4 headache days over 3 months; that dropped to 4.7 by the study's end.

Take a minute here. If you're a regular reader of this blog, I'm betting that you can pick out the huge methodological flaw in this study from just the press report alone. Do you have it yet? Don't worry, I'll get to it very soon. However, I don't like to rely on just the news coverage of such a study. Whenever possible, I always like to go to the original study, and, as I usually do, I did just that. Here's the abstract:

We aimed to investigate the effectiveness of acupuncture in addition to routine care in patients with primary headache (> 12 months, two or more headaches/month) compared with treatment with routine care alone and whether the effects of acupuncture differ in randomized and non-randomized patients. In a randomized controlled trial plus non-randomized cohort, patients with headache were allocated to receive up to 15 acupuncture sessions over 3 months or to a control group receiving no acupuncture during the first 3 months. Patients who did not consent to randomization received acupuncture treatment immediately. All subjects were allowed usual medical care in addition to study treatment. Number of days with headache, intensity of pain and health-related quality of life (SF-36) were assessed at baseline, and after 3 and 6 months using standardized questionnaires. Of 15 056 headache patients (mean age 44.1 ± 12.8 years, 77% female), 1613 were randomized to acupuncture and 1569 to control, and 11 874 included in the non-randomized acupuncture group. At 3 months, the number of days with headache decreased from 8.4 ± 7.2 (estimated mean ±s.e.) to 4.7 ± 5.6 in the acupuncture group and from 8.1 ± 6.8 to 7.5 ± 6.3 in the control group (P < 0.001). Similarly, intensity of pain and quality of life improvements were more pronounced in the acupuncture vs. control group (P < 0.001). Treatment success was maintained through 6 months. The outcome changes in non-randomized patients were similar to those in randomized patients. Acupuncture plus routine care in patients with headache was associated with marked clinical improvements compared with routine care alone.

Got it yet?

I'm sure that most of you do; so I'll just move on. This study has three design flaws so glaring that I almost don't even care what it shows because the flaws are so significant that they scuttled the study before it even started. Here they are:

1. The study is not only not double-blind, it's not even blinded in any way. Both the patients and the health care practitioners know who is receiving what therapy. That alone makes its result entirely explainable by placebo effects.
2. There isn't even an attempt at a sham acupuncture group. Remember my previous posts on the importance of sham acupuncture and how sham acupuncture is indistinguishable from "real" acupuncture.
3. The "randomization" isn't even really a randomization. Of 15,056 patients with a complaint of headache, only 3,404 accepted randomization to control or acupuncture. Normally a clinical investigator, when faced with this situation, studies only the patients who agreed to be randomized. Not these intrepid woo-mavens! They included the remaining 11,652 patients anyway. Actually, they included 11,874 nonrandomized patients.

There are a number of other flaws, but, really, they pale in significance to the three above and are hardly worth mentioning except in passing. For example, there were significant differences between the randomized and nonrandomized groups, including higher pain intensity and a shorter duration of chronic headaches, making them prime candidates to be prone to regression to the mean. The study is also suspect because it lumps together all headaches, rather than separating out the migraine headaches, which have a different physiological mechanism behind them than run-of-the-mill headaches. Another problem was that the authors relied on questionnaires, rather than a pain diary. Because a questionnaire relies on patient memory, rather than the patient writing down an incident as it happens, it's prone to recall bias. There are also numerous other nits to pick, but none of them even come close to the three flaws listed above.

Regarding the failure to use a sham acupuncture group or to blind, it makes me wonder if all the studies coming out showing that sham acupuncture and "real" acupuncture are indistinguishable are starting to get to acupuncture advocates to the point where they really aren't even trying anymore. After all, an unblinded study is almost guaranteed to produce an effect, but an investigator has no way of knowing whether that effect is greater than placebo if there is no valid placebo group. True, the authors did all sorts of fancy statistics and handwaving to try to take the reader's mind off of this fundamental fatal flaw, but none of that changes anything.

Reading the discussion is very instructive, as it demonstrates very well the torturing of language and logic that is used by advocates of "alternative" medicine. While they admit flat out that this was an unblinded study and that , the excuse used was that this was a "pragmatic" study designed "chosen to reflect general medical practice." Yes, that's a great excuse not to do the necessary placebo/sham acupuncture control that would make the results of this study interpretable. It's also a lovely excuse to allow patients in essence to self-select for acupuncture by refusing randomization, thus making the likelihood of a placebo effect even greater--except that there's no control that allow us to know if it's just a placebo effect or not. Here's part of what they argue:

Although differences with respect to both baseline characteristics and treatment outcomes between randomized and non-randomized patients were small in absolute numbers, our findings indicate that randomization was associated with some selection effects. Therefore, the use of study designs that also include non-randomized patients appears to be desirable.

It is of note, however, that treatment benefits were similar in the randomized and nonrandomized acupuncture groups after adjusting for baseline differences. This suggests that the results of randomized trials can be representative of routine medical care situations, at least in large pragmatic studies.

No, it suggests that the placebo effect was operative in both the randomized patients and the patients who refused to be randomized. The use of study designs that include non-randomized patients is only "desirable" if you want to maximize the chances of a seemingly positive result. Bravo, though, to the study authors for having the chutzpah to try to change this study's most glaring weakness into a strength. It was a nice try, but it won't fly.

What's truly depressing is the editorial by Dr. H-C Diener of the University Hospital Essen in Germany, where he actually makes this argument:

Despite the fact that I have major design issues with the study, my view is that studies like this have to be published in high ranked journals to promote discussion on trial design in non-drug trials

Dr. Diener actually lists two of the same flaws that I did, namely the unblinded nature of the study and the lack of adequate "sham" acupuncture controls. In spite of this, he still argues that this article should be published in high ranked journals "to promote discussion"? Funny, but I always thought that high ranked journals are high ranked because they post the most scientifically sound and medically interesting articles. Think New England Journal of Medicine. I always thought the reason for such journals to publish an article is because it is scientifically sound and studies a clinically important and/or interesting question. As a reason to publish such an article, "to promote discussion" is about as far down on the list of reasons as "because the author has nice hair."

The bottom line is that this study is yet another of a long line of studies of "complementary and alternative" medicine that are entirely consistent with the placebo effect. Worse, it didn't even really try to distinguish between a treatment effect and placebo effect. Maybe that's the point. Whatever the point was, what I do know is that if I were a German citizen, I'd be mightily pissed off that so much money was wasted on this study. I'd also wonder why it was a consortium of insurance companies who funded the study. Maybe I was wrong about insurance companies funding woo. Maybe it is cheaper in the long run for them to pay for CAM than actual scientific medicine that's more than just a highly elaborate placebo.


1. Jena, S., Witt, C., Brinkhaus, B., Wegscheider, K., Willich, S. (2008). Acupuncture in patients with headache. Cephalalgia, 28(9), 969-979. DOI: 10.1111/j.1468-2982.2008.01640.x

2. (2008). Acupuncture for the treatment of headaches: more than sticking needles into humans?. Cephalalgia, 28(9), 911-913. DOI: 10.1111/j.1468-2982.2008.01650.x


Anonymous said...

The Updated funny celebrity gossips mostly consists of on the scandals of famous or popular celebrities like James waltz dating with
David Arquette. These make great funny pictures, and can be as blatant as the example I just gave, or as simple as a spelling mistake.

While almost every cat is a work of aesthetic beauty, they are often animals
that combine silliness and a fierce disposition.

Feel free to visit my homepage :: lol pictures
Also see my page: lol pictures

Anonymous said...

These fitness tips in above some basic ideas for keeping fit and healthy body.
The second option is to use headphones as you
probably did at the gym in the past. Just like with the above diet tips,
you want to make sure that each fitness tip you incorporate
into your routine is doable.

Review my blog post ... Http://Homeopatias.Org/

Anonymous said...

Your means of telling all in this piece of writing is really pleasant, every one be
capable of easily know it, Thanks a lot.

Visit my website; simply click the up coming document

Anonymous said...

With the New Year quickly approaching, many people are setting their sights towards total physical
fitness. Don't jump cold, warm up first until you break a light sweat. Yet, before you can get to this point, you first have to make a start.

Also visit my weblog ... fitness tips abs

Anonymous said...

Outstanding quest there. What occurred after?

Feel free to surf to my blog post ...|

Anonymous said...

Just go through the internet and discover the best source to
get all these humorous images. Some pictures will remind you
of your ambitions about your life while other humorous ones will relieve your stress
and make your life more comfortable and easier.
The moment you think of funny video the first name that probably comes to your mind is You - Tube.

My blog post ::

Anonymous said...

Based on my own experiences I am going to make five recommendations
that I wish someone had made to me to help take the fear out of fitness.
Men can follow these health and fitness tips to lose weight and get healthy and
fit. The decisive factor is that men would want to show themselves
as hot and rock solid before the fashionable ladies as ever.

Feel free to surf to my web site

Anonymous said...

And as we all know, too much stress can be harmful to health.
One of my favorite ones is where the thanksgiving turkey is hiding from
a guy, who apparently wants to get him in the oven, by placing a lampshade on his head and standing on a dresser so he
looks like a lamp. The videos relax, they make people
smile and as they say, laughter is the best

Also visit my web page :: just click the next site

Anonymous said...

I've listened to this a dozen or more times and still find it delightful and sooooo original in concept. A person needs to have an active internet connection to be able to access all the headlines and information on internet. Due to the recent horrifying terror attacks and other such couple of events people want to be aware and alerted of the whereabouts of the nation.

Visit my blog post: Latest Daily News

Anonymous said...

Your payments will not only get to you faster but it will provide you with a
history of your earnings for your records. Even if your small business doesn't have some sort of refrigerator (a rarity today) you can choose vegetable and fruit that do not even require refrigeration. Remember; there is a difference between a fattening diet and a healthy diet.

Here is my website ::

Anonymous said...

White biotechnology is for the business field and the
marine biotechnology for all sea life. Russian Prime Minister Vladimir
Putin emphasized this commitment, "We have allocated substantial resources for the development of such promising areas as nano-and biotechnology, nuclear energy, aerospace and other research. Bioaugmentation is the procedure chosen if the treatability study of groundwater comes back with no signs of degradation, or an extended time before degradation is found.

Also visit my web page :: Biotechnology Today

Anonymous said...

This simple app provides lots of funny facts that you have ever known.
The goal with a funny picture is to have the photograph be
almost a blank canvas for you to put the punch line on.
A few young superstars have made their name just from being spread across the internet in videos
and photos.

my web site - lol pictures

Anonymous said...

So little brother's been a bit of a couch potato lately. The other day, I was talking with an acquaintance who stated that "There are many issues in modern society that bother me, I feel that everybody is a zombie, people seem to be blind to what actually happening around them, newspaper headlines are full of celebrity nonsense, everything seems designed to distract people from the real world, everything seems so detached. If you plan on creating a simple game for the app store, Fruit Ninja can give you a good amount of ideas.

Here is my page ::