It's annoying when the high-and-mighty statisticians get the statistics wrong

I'm having a conversation with my mother about the evidence for acupuncture being an effective treatment. This is not my area of expertise nor my mom's, but through discussions my mom is now starting to scour the primarily literature for evidence. I think this is great! In trying to understand where I think the evidence is, I came across the website Science-based Medicine which has a post about a particular acupuncture study. The post proceeds to point out all the deficiencies in the study. Of course, you can do this with all studies and all statistical analyses, i.e. you can always include more variables, more treatments, more parameters in your model. I did agree that I wasn't sure what the point of the electrical stimulation was and that this appears to unnecessarily complicate the study and analysis. I also agree that the study seemed to be set up as a confirmatory study with the primary outcome being the number of migraines in the 5-8 weeks after randomization. The original study mentions that the acupuncture groups had fewer migraines, but the result was not statistical significant (p>0.05). My main problem with the post from Science-based Medicine is this statement:

we can conclude that acupuncture does not work for migraines.

This is making the statistical error of accepting the null hypothesis when, in fact, all the pvalue tells you is that your sample size was insufficient to reject the null hypothesis. I realize this post makes me a bit high-and-mighty, but I am willing to give people who aren't so high-and-mighty some leeway. In particular, the researchers who are generating the data are working under time, money, and publication pressure. They probably didn't include a full control group because then people would have complained that the patients weren't blinded. They needed to have found something significant for their manuscript to get published at all. They were honest in their abstract about their primary outcome and its statistical insignificance. For those who are interested in discrediting acupuncture (again I have no idea if there is any real effect), they should just have stopped with `the primary outcome in this blinded study of acupuncture showed no significant difference between sham acupuncture and 3 particular acupuncture practices.'

---

---

blog comments powered by Disqus

Published